What’s the difference between structured and unstructured data?
Structured data: data that can be easily organized into data tables (eg. Stock returns)
Unstructured data: data that can’t be easily organized (eg. Social media images, text messages)
What are 4 V’s in big data investment management?
- Volume: The quantity of data being collected
- Variety: The array of available data sources
- Velocity: The speed at which data are created
- Veracity: The credibility and reliability of data sources (eg. 20% of internet is spam & 10-15% of social media is fake)
What are 5 steps in ordinary Machine Learning Model Building Steps (for structured data)? DDDDM
Ordinary ML Model Building Steps
- Step 1: determining the modeling task/output
Step 2: Data collection
Step 3: Data preparation and wrangling (data cleaned, eg. Removing outliers.)
Step 4: Data exploration
Step 5: Model training (model begins working and make necessary adjustments.)
What are 4 steps in text machine learning model? DDTT
Text ML Model Building Steps
- Step 1: Text problem formulation (define objective).
- Step 2: Data (text) collection
- Step 3: Text preparation and wrangling (data needs to be cleansed & preprocessed.)
- Step 4: Text exploration
What is data cleansing/preparation vs data wrangling/preprocessing?
Data Cleansing (Preparation): Improve data quality by removing or correcting errors. (Raw data contain errors like inaccuracies and duplications, must be cleaned up before use in an ML model. Data cleansing involves identifying and dealing with these errors.)
Data Wrangling (Preprocessing): Transform raw data into a usable format for analysis. (Once cleansed, data prepared to be used as inputs in ML model. This activity involves addressing outliers, identifying useful variables, as well as formatting data appropriately so they can be used as inputs in an ML model.)
What are 6 possible errors when preparing data to be read by computers? IIIIND
- Incompleteness: some data is missing
- Invalidity: some data doesn’t make sense (eg. House price -100k)
- Inaccuracy: Some data is wrong but looks reasonable (eg. One row says a house was sold for $500K, but another part of the dataset says the same house was sold for $450K.)
- Inconsistency: same type of data is recorded in different ways. (eg. One row says a house was sold for $500K, but another part of the dataset says the same house was sold for $450K.)
- Non-uniformity: Data are provided in different formats.
- Duplication: Two different entries have the same data.
What are 5 techniques used in data wrangling & processing step? FAFSC
- Feature extraction: creation of new variable from an existing variable for better analysis. (eg. using the date of birth to create a variable for age.)
- Aggregation: combining variables that contain similar information. (eg. price return + roll return + collateral return can be aggregated into total return for valuing commodities.)
- Filtration: removal of rows that are not applicable to the project.
- Selection: involves removing data columns that are not needed.
- Conversion: process of making appropriate adjustments to data so they are relevant for analytical purposes. (Eg. values of properties in various countries are not comparable unless stated in common currency terms.)
What is difference between trimming, winsorization (for structured data)?
trimming: removing all extreme values and outliers. (eg. top 5% and bottom 5% of observations are removed.)
winsorization: replaces extreme values with the maximum or minimum of the data not containing outliers.
What’s the difference between normalization and standardization & formulas for each (for structured data)
- Normalization rescales each observation to a range of 0 to 1 based on where it falls in the range from lowest (minimum) to highest (maximum)
(Xi - Xmin) / (Xmax - Xmin)
- Standardization produces a scaled value based on how many standard deviations an observation differs from the mean
X - u / o
x = variable
u = mean
o = standard deviation
What is text processing?
process of taking unstructured data and turning it into structured data.
What are 4 common steps of Text preparation (cleansing)?
- Removing html tags — Many web pages include html markup tags which do not contribute to the text being analyzed.
- Removing punctuation — In most cases, punctuation does not help in text analysis and should be removed. (Eg. Commas(
- Removing numbers (eg. removing 12 million into /number/) (we do this or else computer may treat each number as a separate word)
- Removing white spaces — Extra white spaces are unnecessary and should be deleted. (remove any extra spaces)
What is text wrangling (preprocessing)?
process of separating text into tokens (tokens aka words)
What are 4 techniques used in the process of normalizing text data: LSSL
- lowercasing (Money into money so they all look the same)
- stop words (removing words like “the & “is” to reduce number of tokens)
- stemming (convert word into base word. Eg. Increasing or increase to increas)
- lemmatization (convert word into its morphological root. eg. Better & best changed to good)
what is a bag of words after the normalization process?
- a technique that represents text data as a collection of words, ignoring word order and context, while focusing on the frequency of each word’s occurrence
What is document term matrix?
- text analysis technique that represents the frequency or how many times a word occurs across a collection of documents, used in areas like natural language processing
The bag of words doesn’t tell us the sequence of words so what can we use instead?
- we can use n grams
What is 1 n gram, 2 n grams, and 3 n grams?
- n gram: number of words combined to make one token
Unigram
Stock prices closed higher today > “stock” “prices” “closed” “higher” “today”
Bigram
Stock prices closed higher today > “stock_prices” “prices_closed” “closed_higher” “higher_today”
Trigram
Stock prices closed higher today > “stock_prices_closed” “prices_closed_higher” “closed_higher_today”
What are the 3 activities conducts as part of the data exploration for structured data?
- exploratory data analysis
- feature selection
- feature engineering.
What is exploratory data analysis (EDA) for structured data?
- Exploratory data analysis (EDA) for structured data involves creating data visualizations to summarize and explore information
eg. One dimensional data can be summarized into mean, median
eg. Two dimension data can be summarized into correlation matrix
What is feature selection for structured data?
- using the most relevant factors (independent variables) for machine learning models.
What is feature engineering for structured data?
- transforming existing features (independent variables) to either create new features that are more descriptive and meaningful or breaking down existing features.
What is exploratory data analysis for unstructured data (aka text analytics)?
- identifies patterns in unstructured data
What are 3 common applications of text analytics?
Text classification — This uses supervised ML methods to place text into different classes.
Topic modeling — This creates clusters of text using unsupervised ML methods.
Sentiment analysis — This is similar to text classification in that the text is grouped (positive, neutral, or negative). Either supervised or unsupervised ML methods can be used in this analysis.
What is a collection of text called?
- corpus (which is a sequence of tokens)
