lecture 4

Question 1

sources of bias

Answer 1

1. selection phase (influences data)
2. annotation (influences data)
3. input representation: how language is encoded and fed to models
4. models
5. research design

Question 2

importance of data

Answer 2

1. datasets form the basis of model training, evaluating, and benchmarking
2. the ways in which we collect/construct/share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development
3. good quality data ensures models perform well, are fair, and can be generalized across various contexts

Question 3

text classification

Answer 3

corpora help us with text classification

goal: assign a label or category to a specific piece of text

Question 4

why use text classification

Answer 4

1. categorize language at word, sentence, and document level
2. predict future outcomes
3. find patterns

Question 5

sentiment analysis

Answer 5

goal: predict the sentiment expressed in a piece of text (+, - , scale rating)

Question 6

why is sentiment analysis hard

Answer 6

1. sentiment is a measure of a speaker’s private state, which is unobservable
2. sometimes words are a good indicator of sentiment, but many times it requires deep world + contextual knowledge

Question 7

other text classification problems

Answer 7

1. language identification: which language the text is in
2. spam classificiation
3. authorship attribution
4. genre classificiation
5. senitment analysis: understanding public opinion

Question 8

questions when building a sentiment classifier

Answer 8

1. what is the input for each prediction (e.g., sentence, text, etc.)
   –> requires substantial data
2. what are the possible outputs (e.g., +, -, scale)
3. how will the model decide (model decision mechanism)
4. how to measure effectiveness (evaluation metrics)
   –> requires substantial data

Question 9

data-driven evaluation

Answer 9

choose a dataset for evaluation before you build a system

Question 10

why is data-driven evaluation important

Answer 10

1. controlled experimentation
2. benchmarks: serve as reference points to evaluate the performance of a system
3. your intuitions about inputs are probably wrong

Question 11

where to get a corpus

Answer 11

1. many corpora are prepared specifically for linguistic/NLP research with text from providers
2. collect a new one by scraping websites

Question 12

gold labels

Answer 12

annotations used to evaluate and compare sentiment analyzers

these can be
1. derived automatically from the original data artifact (metadata such as starratings)
2. added by human annotator who reads the text (but how to address trouble with deciding and agreeing between annotators)

Question 13

sentiment analysis training data

Answer 13

(X,Y) pairs to learn h(X)
–> (input, output)
–> relies heavily on accurately labeled data
–> this is text classification

Question 14

accuracy

Answer 14

• #correct / #total
• simplest measure
• not a good measure when there are class imbalances: when a classifier always predicts the majority class it will seem accurate but is ineffective in reality
• doesnt show the quality of predictions

Question 15

confusion matrix

Answer 15

• gives more detailed insight into classification
• used for precision, recall, F1 score

Question 16

precision

Answer 16

• accuracy of positive predictions (how often is my prediction correct)
• TP/ (TP + FP)
• measure of quality

Question 17

precise model

Answer 17

might not find all positives, but the ones that the model does classify as positive are very likely to be correct

Question 18

not precise model

Answer 18

may find a lot of positives, but its selection method is noisy. it wrongly detects many positives that arent true positives.

Question 19

recall (sensitivity)

Answer 19

• how well are we capturing positive instances (how many of the positive instances do i find)
• TP / (TP + FN)
• measure of quantity

Question 20

model with high recall

Answer 20

succeeds in finding all positive cases, even though it might also wrongly identify some negative cases as positive cases

Question 21

model with low recall

Answer 21

not able to find all or a large part of positive cases

Question 22

when to use precision vs recall

Answer 22

• precision: when we prioritize the quality of positive predictions over finding all positive instances
• recall: when the aim is to capture all positive cases, even if it leads to some false positives

Question 23

tuning for high precision

Answer 23

the system should not make a mistake

Question 24

tuning for high recall

Answer 24

the system should not miss a case

Question 25

F1 measure

Answer 25

- balance between precision and recall (harmonic mean) - offers better insight about model performance based on quality - especially important for class imbalance - 2*(precision*recall)/(precision+recall)

Question 26

F1 score

Answer 26

1: high: both P and R are high 2. low: both P and R are low 3. medium: one of P and R is low and the other is high

Question 27

random baseline

Answer 27

- method to provide a reference point for evaluating classification model performance - labels are assigned to observations at random 1. fix random seed 2. repeat n times 3. average results - serves as benchmark against which better models are evaluated

Question 28

majority baseline

Answer 28

- assign most frequent class label to all instances, calculate results - results in high accuracy when one class significantly outweighs the other, but poor performance in identifying the majority class - **ensures that models not only have high overall accuracy, but can also correctly identify less frequent classes**

Question 29

evaluation for multiple classes

Answer 29

1. calculate precision and recall for every class separately 2. average the results over classes --> macro average: does not take class imbalance into account --> weighted average: weighted by class size

Question 30

sentiment lexicon

Answer 30

1. predefined list of words classified as positive/negative 2. count positive and negative words within the text. predict whichever is greater.

Question 31

problems with sentiment lexicon

Answer 31

1. hard to know if words that seem pos/neg are actually used that way 2. opinion words might describe a characters attitude rather than an evaluation of the film 3. some words are semantic modifiers

Question 32

solutions for sentiment lexicon problems

Answer 32

data-driven method: use frequency counts to ascertain which words in corpora tend to be positive or negative

Question 33

h(x)

Answer 33

- for text classification - a mapping h from input data x to a label y - two components 1. representation of the data 2. formal structure of the learning method

Question 34

representation of data for text classification

Answer 34

1. sentiment analysis: only positive and negative words 2. only words in isolation (BoW) 3. conjunctions of words (sequential, ngrams, other nonlinear combinations) 4. higher order linguistic structure (syntax)

Question 35

bag of words

Answer 35

- simplest representation - text represented as counts of words that it contains - frequency of occurrence of each word is used as a feature for training a classifier

Question 36

BoW process

Answer 36

1. tokenize 2. count 3. vectorize: each dimension represents a unique word in the entire corpus, and the value in each dimension is the word's frequency in the document

Question 37

why is BoW not sufficient for modeling language

Answer 37

1. insensitive to word order or semantics 2. vectors are sparse and high-dimensional 3. 'words' are not always the most meaningful units of information

Question 38

ngrams

Answer 38

- assign probabilities to sentences - looking at more than one word at a time - estimate P(S = w1...wn). this is a joint probability over all the words in S.

Question 39

ngrams: chain rule

Answer 39

P(S = w1...wn) = product of conditional probabilities

Question 40

problem with chain rule + solution

Answer 40

- problem: many conditional probabilities are just as sparse due to the vast number of possible word combinations - solution: independence assumption. the probability of a word only depends on a fixed number of previous words (history) P(mast| i spent three years before the mast) = P(mast|before,the)

Question 41

ngrams usefulness for sentiment analysis

Answer 41

ngrams capture sentiment beyond the word level, since they have more context awareness

Question 42

why do we need text corpora

Answer 42

1. to evaluate our systems - good science requires controlled experimentation - good engineering requires benchmarks 2. to help our systems work well - data-driven methods instead of rule-based - learning

Question 43

learning

Answer 43

collecting statistics or patterns from corpora to govern the system's behavior 1. supervised learning 2. core behavior: training, refining behavior: tuning.