8b. Text Mining Process

Question 1

What are the 4 main steps in the “text mining” process?

Answer 1

Question 2

What are the 6 steps in data “pre-processing”?

Answer 2

• Data Acquisition
• Tokenisation
• Cleaning
• Remove stop words
• Spelling
• Stemming and Iemmatisation

Question 3

What is the first (most foundational) step in data “pre-processing”?

Answer 3

DATA ACQUISITION
(either downloading or through web scrapers)

Question 4

What is the “tokenisation” step in data “pre-processing”?

Answer 4

Deciding what your unit of analysis is going to be. -

-> eg. the unit of analysis is the individual tweet - the words in the tweet are the TOKENS of the document

Question 5

What is the “cleaning” step in data “pre-processing”?

Answer 5

Removal of all non-meaningful text.

(HTML tags removed…. Emojis are not removed but converted into words..)

Question 6

What is the “removing stop words” step in data “pre-processing”?

Answer 6

Just the removal of common non-meaningful words

-> eg. “a” or “the”

Question 7

What is the “spelling” step in data “pre-processing”?

Answer 7

Correcting misspelt words.

Question 8

What is the “stemming and Immatisation” step in data “pre-processing”?

Answer 8

Standardising words by reducing them to their common stem or Iemma.

-> eg. “car” and “cars” are stemmed to just “car” (stem)
-> eg. “car” and “cars” are turned to the Iemma “automobile” (iemma)

Stem = A stem is the basic form you get by chopping off endings. It is often mechanical and may not be a real dictionary word.
Iemma = True dictionary/base form of a word, based on its meaning and part of speech.

Question 9

Why is the pre-processing vital?

Answer 9

remember: rubbish in = rubbish out

Question 10

What are the 3 types of text analysis “Extraction”?

Answer 10

1. Entity extraction
2. Topic extraction
3. Relation extraction

Question 11

What is “entity extraction”?

Answer 11

Extraction of individual entities (words) from text..
(most basic and common level of text mining)

Question 12

What is “topic extraction”?

Answer 12

Identifies general topics that appear ina text document.
(reduces to a few general topics)

Question 13

What are some applications of “entity extraction”?

Answer 13

• Social media monitoring
• Predictive modelling
• Input to dictionaries

Question 14

What is “relation extraction”?

Answer 14

Focuses on discovering textual relationships among extracted text entities.

Question 15

What are some applications of “topic extraction”?

Answer 15

• Useful to explore non-observable customer preferences

Question 16

What are some applications of “relation extraction”?

Answer 16

• Understand customer sentiment/ preferences in relation to specific product or service features

(very limited use today)

Question 17

What is the most common level of text mining (text analysis “Extraction”)?

Answer 17

Entity extraction

Most common + Most basic level

Question 18

Which level of text mining (“Extraction”) is focused on in this module?

Answer 18

TOPIC extraction

Question 19

What are the 4 text metrics used?

Answer 19

• Count (measure the frequency of each entity’s occurence)
• Similarity (measure the similarity of the text between documents)
• Accuracy (judges the accuracy of text measures relative to human-coded or externally validated documents)
• Readability (judges readability level of text)

Question 20

What is done in step 3 (text metrics)?

Answer 20

Text is converted into quantifiable measures

Question 21

What is the last step (#4) of text mining?

Answer 21

Assessing the validity of extracted text and measures

Question 22

What are the two types of validity that are dealt with in the last step (#4) of text mining?

Answer 22

• Internal validity
• External validity

Question 23

What is “internal validity”?

Answer 23

Internal validity is everything that has to do about whether the text accurately measures the constructs and the relationship between them

Question 24

What is “external validity”?

Answer 24

External validity looks at whether the text-based findings do also apply to phenomena outside the study

Question 25

Overview of the text mining process:

Answer 25