Tokenisation

Question 1

Describe the task of tokenisation

Answer 1

Break down a piece of text into individual words

Question 2

What is the initial approach to tokenisation

Answer 2

whitespace

Question 3

Where does ambiguity come from in tokenisation

Answer 3

end of sentences “mary.”, abbreviations “dr.”, punctuation “they’re”

Question 4

What are the three classes of token

Answer 4

morphosyntactic word, punctuation or symbol, number

Question 5

What is the challenge of character encoding

Answer 5

Do we choose ascii only or unicode to include emojis.

Question 6

What are the challenges of tokenisation (7)

Answer 6

• character encoding
• transliterations
• poor ocr results
• writing systems
• hyphenation
• telephone numbers
• dates, decimals, money

Question 7

What challenges arise from domain dependence (3)

Answer 7

• organism/species names/authorities, families, orders
• coordinates
• protein sequence, protein names