Model Evaluation (1)

Question 1

Model Evaluation

What are you evaluating?

Answer 1

Quality Control

Question 2

Model Evaluation

Examples of tasks in model evaluation?

Answer 2

text summarization, Q&A, text classification

Question 3

Model Evaluation

What do you supply to the model for Model Evalution?

Answer 3

Prompt Data set

Question 4

Model Evaluation

Where can you create them?

Answer 4

Create custom yourself, or use curated prompt datasets

Question 5

Model Evaluation

What’s in a prompt data set?

Answer 5

Questions and what you think a good answer should be

Question 6

Model Evaluation

How do you score the model during Model Evaluation?

Answer 6

You don’t. The scores are calculated automatically.

Question 7

Model Evaluation

How does the model calculate a score?

Answer 7

It doesn’t. A separate judge model scores the actual response vs. your supplied exemplar response.

Question 8

Model Evaluation

What statistical methods are used to score a response?

Answer 8

BERT, F1, others

Question 9

Model Evaluation

What’s an example of something Benchmark tests can help detect?

Answer 9

Bias and discrimination

Question 10

Model Evaluation

Metrics to eval an FM?

Answer 10

ROUGE, BLEU, BERT

Question 11

Model Evaluation

What is ROUGE?

Answer 11

Eval automatic summarization and machine translation

Question 12

Model Evaluation

What is ROUGE-N?

Answer 12

Measure the number of n-grams between reference and generated text

Question 13

Model Evaluation

What is ROUGE-L?

Answer 13

Longest sequence of text between reference and generated text

Question 14

Model Evaluation

What is BLEU?

Answer 14

Bilingual Evaluation Understudy

Question 15

Model Evaluation

What is BLEU used for?

Answer 15

Quality of generated text, especially for translations

Question 16

Model Evaluation

How does BLEU work?

Answer 16

Looks at combination of n-grams

Question 17

Model Evaluation

What is BERT?

Answer 17

Semantic similarity: looks at how close the /meanings/ are to each other

Question 18

Model Evaluation

What’s the special techie term for what BERT does?

Answer 18

Computes the cosine similarity

Question 19

Model Evaluation

Which evaluation metric is the best for nuanced meaning between reference and generated text?

Answer 19

BERT: it uses semantics, not n-grams

Question 20

Model Evaluation

What is Perplexity metric?

Answer 20

How well the model predicts the next token

Question 21

Model Evaluation

For Perplexity, is a higher or lower score better?

Answer 21

Lower is better