Lecture 4 - Data mining introduction

Question 1

Business Intelligence definitions

Answer 1

• Creative process that provides results useful for decision making
• Requires several different skills:
  *

Question 2

Data mining

Answer 2

• Creative process that provides results useful for decision making
• Requires several different skills, e.g.: Statistics, ML, Programming, etc
• Ability to ope with huge amounts of data → used frequently for Big Data

Question 3

Big Data & Challenges

Answer 3

Current data is big (by reference to the past)
Challenges of Big Data are typically characterised by:

• Volume: amount of data
• Velocity: flow rate, i.e. speed at which data is being generated and changed
• Variety: different types of data being generated, i.e., currency, dates numbers, text, etc.
• Veracity: data is being generated by organic distributed processes, ie.e, quality (missing, clean value)
  
  • Value: data have no value for the company unless yo can turn it into something useful

Question 4

Methods

Answer 4

• Classification and Prediction
  
  • Regression
  • K-nearest neighbors
  • Decision trees
• Association
  
  • Association rules
• Clustering
  
  • k-means

Question 5

Existence of various methods

Answer 5

Each method has advantages and disadvantages, depending on several factors:

• Size of the dataset
• Types of patterns that exist in the data
• If the data meet some underlying assumptions on the method
• How noisy is the given data
• The goal of the analysis

Question 6

Data

Answer 6

• A collection of facts usually stained as the result of experiences, web page visits, observations, or experiments, extracted from documents
• Lowest level of abstraction (from which information and knowledge are derived
• May consist of numbers, words, images, etc.

Question 7

Terminology

Answer 7

Question 8

Data & Types of Variables

Answer 8

• Data may consist of numbers, words, images, etc
• Classification:

Question 9

NOIR

Answer 9

• *Nominal data** - values are distinct symbols
• *Ordinal** - order on values, but no distance between values defined
• *Interval** - quantities are not only ordered but measured in fixed and equal units
• *Ratio** - quantities for which the measurement scheme defines a zero point

Question 10

Steps in a typical data mining effort

Answer 10

1. Develop an understanding of the purpose of the data mining project
2. Obtain the dataset to be used in the analysis
3. Explore, clean, and preprocess the data
4. Reduce the data dimension, if necessary
5. Determine the data mining task
6. Partition the data (for supervised tasks)
7. Choose the data mining technique(s)
8. Use algorithms to perform the task
9. Interpret the results of the algorithm
10. Deploy the model

Question 11

SEMMA methodology

Answer 11

• Includes the 10 data mining steps
  
  • Developed by the software company SAS

Sample, Explore, Modify, Model, Assess

Question 12

CRISP-DM

Answer 12

• Similar methodology to SEMMA, by IBM SPSS Modeler
• CRoss Industry Standard Process for Data Mining

1. Business understanding
2. Data understanding
3. Data preparation
4. Model Building
5. Testing and Evaluation
6. Deployment

Question 13

CRISP-DM Steps

Answer 13