Vl 2 Document Processing

Question 1

problems parsing documents

Answer 1

• format
• language
• reading direction
• character set

Question 2

token

Answer 2

individual words in a text (syntaktische Einheit)

Question 3

term/type

Answer 3

individual words in a text vocabular

Question 4

morphem

Answer 4

smallest grammatical/meaningful unit in a language

Question 5

inflection

Answer 5

case, conjugation, comparative words

Question 6

basic form

Answer 6

word without any inflection: nomative singular for nouns, infinitive for verbs, adjectives without comparative

Question 7

derivation

Answer 7

forming a new word from an existing word, often by adding prefix or suffix

Question 8

compound

Answer 8

a word that consist of more than one stem

Question 9

noun phrase

Answer 9

a phrase wich has generally a noun as its head word

Question 10

tokenization problems
specific problems

Answer 10

• elimination of stop words
• end of sentences
• contraction
• where to seperate words
• numbers
• apostrophes
• special words
• same word different meaning (lower/ upper case)
• differences in languages
• no whitespaces (compound words)
• accents
• accents are often not used even if grammaticaly needed
• umlauts

Question 11

tokenization

Answer 11

splitting text into smaller semantical units

Question 12

tokenization problems
general problems

Answer 12

• how are users likely to write their queries
• tokenization steps for queries must be identical to steps for documents

Question 13

normalization

Answer 13

reuce text to pre-defined form

Question 14

normalization techniques

Answer 14

grammatical markings
stemming
lemmatization

Question 15

(normalization techniques)
grammatical markings

Answer 15

• used to differentiate different forms of a word

Question 16

(normalization techniques)
stemming

Answer 16

• focus on syntax
• an operation that strips off grammatical markings to leave the stem
• chopping of end

Question 17

(normalization techniques)
lemmatization

Answer 17

• focus on semantics
• an operation that specifies the lemma corresponding to a word form together with corresponding morpho-syntactic and possible morpho-derivational information

Question 18

case folding

Answer 18

• reduces all letters to lower case
• exceptions possible

Question 19

stop words

Answer 19

• extremly common words
• not helpfull usally
• exceptions

Question 20

(equivalence classing)
soundex

Answer 20

• use heuristics to equivalence class or expand terms with phonetic equivalents
• used for foreign/ complex words

Question 21

(equivalence classing)
thesauri

Answer 21

• semantic equivalence

Question 22

(equivalence classing)
context dependent typings

Answer 22

-same word, meaning dependent on context

Question 23

(equivalence classing)
transliteration

Answer 23

• sience that is concerend with 1:1 mapping of symbols in a script
• 1:1 mapping not always possible

Question 24

problems with stemming

Answer 24

-error prone
- language dependent
- often inflectional and derivational

Question 25

stemming algorithms

Answer 25

- table lookup approach - successor variety - n-gram stemmers - affix removal stemmers

Question 26

(stemming algorithms) table lookup approach

Answer 26

-table with stems and fitting terms - searching terms, returning the associated stem

Question 27

(stemming algorithms) table lookup approach problems

Answer 27

- labor intensive - missing exceptional cases - storage overhead - often only with technical terms

Question 28

(stemming algorithms) N-gram Stemmers

Answer 28

-association measures calculated between pairs of terms based on shared unique n-grams - use Dices coefficient to determine similarity

Question 29

(stemming algorithms) affix removal stemmers

Answer 29

-remove suffixes and/or prefixes -Lovins, Dawsons, Porters, Paice

Question 30

(stemming algorithms) Porter Algorithm

Answer 30

-standard for english - set of context-sensetive rewriting rules

Question 31

Part-of-Speech-Tagging

Answer 31

- labeling each word in a sentence with its appropriate part of speech

Question 32

(POS Tagging) Simple Stochastic Tagger

Answer 32

- each word is assigned its most frequent tag - may create unacceptable tag sequence

Question 33

Named Enitity Recognition

Answer 33

- locates and classifies named entities in a unstrucured text into predefined categories

Question 34

Dependency Parsing

Answer 34

- words are connected to each other by direct links - parses a sentences into its grammatical strucure