Vector Spaces

Question 1

Distributional hypothesis

Answer 1

If two words have similar contexts, we can assume that they have similar meanings.

Question 2

A distributional approach to lexical semantics

Answer 2

• Record contexts of words across a large collection of texts (corpus)
• Each word is represented by a set of features
• Each feature records some property of the observed context
• Words that are found to have similar contexts are expected to also have similar meaning.

Question 3

Context windows

Answer 3

​”I bake bread for breakfast”

• Context = neighborhood of +-n words left/right of the focus word.
• Features for +-1: {left: bake, right: for}
• Some variants: distance weighting, ngrams.

Question 4

Bag of Words (BoW)

Answer 4

• Context: all co-occuring words, ignoring the linear ordering
• Features: {I, bake, for, breakfast}
• Some variants: Sentence-level, document-level

Question 5

Grammatical context

Answer 5

• Context: The grammatical relations to other words
• Intuition: When words combine in a construction they often impose semantic constraints on each-other.
• Requires deeper linguistic analysis than simple BoW-approaches
• Features: {dir_obj(bake), prep_for(breakfast)}

Question 6

Tokenization

Answer 6

Splitting a word into sentences and words or other units.

Question 7

Stop-list

Answer 7

Filter out closed-class words or function words. The idea is that only content words provivde relevant context

Question 8

Lemmatized string from the raw string “The programmer’s program had been programmed”.

Answer 8

“The programmer ‘s program have be program”

Question 9

“Relatedness” vs. “Sameness”

Answer 9

Similarity in domain:
{car, gas, road, service, traffic, driver}

Similarity in content:
{car, train, bicycle, truck, vehicle, airplane}

Question 10

Vector space model

Answer 10

• A general model for representing data based on a spatial metaphor
• Each object is represented as a vector (or point) positioned in a coordinate system
• Each coordinate (or dimension) of the space corresponds to some descriptive and measurable property (feature) of the objects.
• To measure similarity of two objects, we can measure their geometrical distance / closeness in the model.
• Vector representations are foundational to a wide range of ML methods.

Question 11

Semantic spaces

Answer 11

Semantic spaces AKA distributional semantic models or word space models

A semantic space is a vector space model where:

• Points represent words
• Dimensions represent contexts of use
• Distance in the space represents semantic similarity.

Question 13

One standard metric for spatial proximity

Answer 13

Euclidian distance:

Question 14

Norm of a vector

Answer 14

Norm of a vector

Question 15

Potential problem with Euclidian distance

Answer 15

It is very sensitive to extreme values and the length of the vectors.

Question 16

Overcome length biaz

Answer 16

• Normalization to unit vectors
• Cosine similarity

Question 17

Normalize vector to unit length

Answer 17

Divide vector by the length of the vector

Question 19

Cosine of angle between vectors

Answer 19

Question 20

Cosine of angle between unit vectors

Answer 20

Question 21

The cosine measures ____ rather than distance

Answer 21

The cosine measures proximity rather than distance

Question 22

___ has the same relative rank order as the Euclidean distance for _____

Answer 22

Cosine measure has the same relative rank order as the Euclidean distance for unit vectors!