Big Data Projects

Question 1

What is volume in big data

Answer 1

This refers to the quantity of data, big data needs a lots of information,

Question 2

What is variety in big data

Answer 2

This refers to the diversity of the ata sources
You need user generated content, images, traditional data sources, ecetra

Question 3

What is velocity of big data
What is another v that oyu need to consider

Answer 3

Velocity is the speed that data is created and collected.
High velocity is examples like social media, this is generated very quickly.
You also have to consider validity. This refers to the reliability of data.

Question 4

What are the 5 steps involved in data analaysis

Answer 4

1
Conceptualisation of the modeling task.
2
Data collection
3
Data preparation and wrangling
4
Data exploration
5
Model training

Question 5

What is involved in detail with conceptualising the modeling task

Answer 5

This si when you define the problem and you work out what you want to get out of the model
You also then identify who will use the model and how it will be incorporated into business processes

Question 6

Explain the concept of data collection in detail
What is this step called when you are using unstructured data

Answer 6

You have to identify and gather from INTERNAL AND EXTERNAL SOURCES
This step is often referred to as curation when referencing unstructured data.
This step is also the stage where web scraping, annotating target variables for what you want the output to be and other data analysis processes happen.

Question 7

Explain data preparation and wrangling

Answer 7

Data preparation and wrangling when you clean and organise the raw data
Cleansing is when oyu adress missing values and identify invalid or out of range values.
Wrangling is when you transform data through extraction and aggregation., this is done to make sure that it is suitable for the model. For unstructured text this means removing punctuation and other things like that

Question 8

What are the three main things in data exploration

Answer 8

The three main things in data exploration are
Explanitory data analysis
Feature selection
Feature engineering

Question 9

What is the meaning of explanatory data analysis

Answer 9

Explanatory data analysis uses statistics and visualisations, to understand data properties nad the relationships between the data,

Question 10

What is feature selection

Answer 10

feature selection is when you choose only the most relevant attributes to reduce the model complexity, this also helps you to reduce noise

Question 11

What is feature engineering

Answer 11

Feature engineering is when you create new variables by transforming or combing existing ones,
This includes combining several valitbales into a single score which gives the probabliyly or indicator that you want.

Question 12

What is model training in big data

Answer 12

Model training invlolves selecting the correct machine learning algorithm based on the task that you want to complete.
The researcher then evaluates the algorithm based on the training data set and tunes the algorithm’s parameters in order to increase the performance.

Question 13

What is the longest and most difficult part of a data analysis project

Answer 13

Usually the data wrangling element, where oyu have to synthesise lots of data into a usable form

Question 14

What are the 3 main steps of data collection .

Answer 14

Data identification - working out what data is worth collecting
Data collection - data gathered through internal databases or vendors. Or though api’s
Data documentation - when you use a README file to say where the data is stored and what the starting point is for identifying errors

Question 15

What is involved int eh cleansing stage of the data wrangling stage
What are the specific errors that you have to adress with cleansing

Answer 15

Data cleansing is when oyu try to reduce the errors in the raw data set.
MISSING VALUES
Invalid values - which fall outside the logical range
Inaccurate values - just wrong
Non uniform values - resulting form inconsistent formatting
Duplicate observations - redundant

Question 16

What is data wrangling

Answer 16

When you process data through transformation and scaling, so that it is ready to use

Question 17

What are the main data wrangling/data transformation types

Answer 17

Extraction - when you create a new usable vaiable from the raw data. Eg creating a years of employment metric that starts at a beginning start data
Aggregation - when you create muiltiple related variables into a single variable using weightings
Filtration- when oyu remove observations that are irrelevant to the task
Selection - when you remove entire features that aren’t needed
Conversion - when you change data types such as converting nominal data to ordinal forms

Question 18

What are some of the common transformations that are used to deal with handling outliers

Answer 18

Trimming
Windsorisation - when you take the outlier values and make them equal to a maximum or minimum allowed value
Harmonisation - when you take the outlier values and make them equal to the value in a certain percentile 95%

Question 19

What is the meaning of scaling as it relates to homogenous data

Answer 19

Normalisation rescales variables to fall between 0-1
Standardisation centres values at a mean of 0 and scales them in standard deviation units. It assumes a normal distribution but it is not sensitive to outliers,

Question 20

What are the three main phases for the conversion of unstructured text based data into a structured format that a computer can ingest.

Answer 20

Preparation
Wrangling
Exploration

Question 21

Explain cleansing process as it relates to text based data.

Answer 21

Remove the HTML tags, you do this using regex which will identify specific charter patterns
Remove the punctuation. Most of the punctuation is deleted however if there are specific symbols that are needed then they are replaced with ANNOTATIONS
Remove the numbers digits are typically removed or replaced with generic annotations. Unless their specific value is needed.

Question 22

What is the process of text wrangling in text based machine learning
What is stemming and what is lemmatisation
What is tokenisation

Answer 22

Text wrangling is normalised into a standard format:
Lowercase
Removing stop words. These include the and is. These have little value to a modle
Stemming is when you break a word down into a common base. Integrate and integrating become intergrat
Lemmatisation is more advanced and resource intensive that transforms words into their lemma (morphological root)
Tokenisation is when you split sentences into their individual words which are referred to as tokens

Question 23

What is the structuring section of dealing with text based machine learning
What are the structuring steps

Answer 23

Bag of words - this si when you just put all the unique tokens in a list without regard for their original sequence
Document term matrix - then you take the final step, this is when the unstructured data becomes the matrix. Each row represents a document, and then each row represents a token. The values within the matrix refer to how often the document contains that token which can be identified by the row.
If the sequence of the words matters then the tokens can be grouped into multi word sentences.
These can be called bi grams if there are two words that always go together. (Market_up)

Question 24

What is text exploration as it relates to machine learning using text.

Answer 24

Text exploration is when you evaluate the data set using methods to see what the best model for training might be
One method is visualisation where you use word clouds, tokens which appear more frequently are displayed in a larger font and helps analaysis.
One method is feature selecting which to reduce the model nouse you can estimate the high frequency and low frequency words.
Another is document frequency. With this the number of documents containing a specific token divided by the documents.

Question 25

What is the meaning of feature engineering

Answer 25

Feature engineering assigns tags to Tolkien’s based on their entity type,

Question 26

What is parts of speech

Answer 26

Parts of speech uses dictionaries to assign strtuctural tags. This mean you can give each word a grammatical label, that then allows a machine to be able to construct sentences becuase the machine knows what elements of the sentence each part relates to.

Question 27

What is data exploration for unstructured data

Answer 27

Unstructured text means you have to initially process the data in order to identify contextually important elements. - this involves tokenisaition - summary statistics - visualisation

Question 28

What is the meaning of feature selecting when it comes to unstructured data What does chi squared do What is mutual information

Answer 28

Feature selection is to create a parsimonious model by selecting a subset of tokens from the bow (Bag of words) Therefore you reduce the feature induced noise an oyu improve the predictive accuracy. You remove words like ‘the’ Chi square ranks the tokens by their usefulness in distinguishing between classes. Tokens with a higher chi squared statistic are highly assosiated with classes and very good for analysis. Mutual information - measures how much a token contributes to specific classes If a token appears in all classes it’s mutual information index is 0 If it only is included in a few,then the mutual information index gets closer to one

Question 29

What is an N gram in feature engineering

Answer 29

This is a multi word pattern where, if they are useful the order is preserved,

Question 30

What is name entity recognition as it relates to feature engineering

Answer 30

Entity name recognition searches for key token values, int he context that it was used against their internal library and assigns a NER tag to the token. This lets you know what catagory the token shoudl go in

Question 31

What is parts of speech in feature engineering

Answer 31

Parts of speech uses language structure dictionaries to asssign tags to text. This lets you know what gramtatical meaning the word has.

Question 32

What can cause model fitting errors

Answer 32

Number of features Size of the data sample

Question 33

What is the model method selection catagories based on

Answer 33

Supervised or unsupervised learning Type of data - for numberical data you might use a classification or a regression tree method for text you might use a GLM Size of data large data sets can be handled with support vector machines

Question 34

What can model fitting errors be caused by

Answer 34

Modle fitting errors can be caused by the size of the small or the number of features.

Question 35

How do you evaluate the performance of a machine learning algorithm Classification models What are the 4 metrics you can use to quantify performance

Answer 35

For classification tasks. The main tool is the confusion matrix. This classifies results into 4 outcomes True positive True negative False positive False negative Prescision Recall Accuracy F1 score

Question 36

What is a type 1 error What is a type 2 error

Answer 36

A type one error is incorrectly predicting a false positive A type two error is when you fail to identify a positive class

Question 37

What is the definition of prescisison and how is it calculated When would oyu want a model that is highly accurate

Answer 37

Prescision is basically how accurate positive predictions are. This is calculated by taking the number of true positives and working out the proportion of true positives there were as a proportion of all the positive outcomes that the model predicted Therefore the answer is TP/FP+TP You use this when the cost of recording a false positive is very high

Question 38

What is the definition of recall and when are you likely to prioritise strong recall

Answer 38

Recall is the models ability to find all positive instances. High recall is prioritised when the total cost of a type two error is high Recall is basically what proportion of all the positive readings that we woudl above wanted you to pick up have you been able to pick up So equal to TP/(TP+FN) Used basically when the cost of you not identifying something is high.

Question 39

What is the definition of accuracy and what is the calculation for it

Answer 39

Accuracy is equal to basically the number of true positives and trie negatives that the model was able to identify in total, so the percentage of correctly identified items out of the total items that it predicted Formula equals TP+TN/FP+FN+TP+TN

Question 40

What is the definition of the F1 score and what is it used for

Answer 40

The F1 score is the harmonic mean of the Prescision and recall metrics 2 x P X R/(P+R)

Question 41

What is the receiver operating statistic and what is it used to measure What is the calculation for the false positives rate Waht does the area under the curve represent

Answer 41

The ROC curve is uesed to plot the tradoff between the true positives call rate (recall) and the False positive rate The calculation for the false positive rate is just FP/(FP+TN) The area under the curve, is a single metric between zero and one that measures the predictive accuracy. If the area under the curve is equal to 0.5 then it means that hte model is just making random guesses. The more that the line is curved the better the model fit and the more predictive accuracy the model has.

Question 42

What is the performance analaysis for continuous data.

Answer 42

For continuous data there are three main metrics. Root mean squared error Mean absolute error Mean absolute percentage error

Question 43

What is the meaning of root mean squared error

Answer 43

root mean squared error is a metric which summarises the avergage prediction error in a sample The formula is the sum squared errors of the predicted value versus the actual value all squared divided by n then all square rooted,

Question 44

What is the mean absolute error and the mean absolute percentage error

Answer 44

The mean absolute error is just what it says on the tin, you take the absolute differences and then average them out The mean absolute percentage error is the same but you just are using the percentages

Question 45

What is the fitting curve

Answer 45

The fitting curve basically says that the model has to be revised until it reaches an acceptable level of performance However if you modify the model too much then you get to the point where you have variance error The Burke is upward sloping and oyu basically want to get to the pint on the curve wich is the lowest that means that oyu have the optimal complexity model The two tails of the curve are points where you have bias error and where yo have variance error,