CIA - Models

Question 1

Define ‘model’.

Answer 1

It is a practical representation of relationships among entities, using FEMS concepts:

• Financial
• Economical
• Mathematical
• Statistical

Question 2

What are the elements of a model? (3) (hint: SIR)

Answer 2

• model Specification
• model Implementation
• model Run

Question 3

Define ‘model specification’.

Answer 3

It is a description of the parts of a model and their interactions, including:

• data
• assumptions
• methods
• entities
• events

Question 4

Define ‘model implementation’.

Answer 4

It is the systems that perform the calculations:

• computer programs
• spreadsheets
• etc.

Question 5

Define ‘model run’.

Answer 5

It is the inputs and outputs of the implementation.

Question 6

Define ‘model risk’.

Answer 6

It is the risk that the user will draw inappropriate conclusions due to the shortcomings of the model or its use

Question 7

What is the main distinction between a calculation and model?

Answer 7

That a model requires more documentation:

• how it was chosen
• how it is used

Question 8

Why is there always risk in using a model?

Answer 8

It is because a model is a simplification of reality.

Question 9

How can model risk be measured? (2)

Answer 9

• severity of model failure

- likelihood of model failure

Question 10

Describe the considerations in assessing the severity of a model failure. (3)

Answer 10

• financial significance (eg: the severity is higher if estimating a major balance sheet item)
• importance of model (eg: severity is lower if multiple models are being used)
• frequency of use of model (eg: severity is higher if model is used frequently)

Question 11

Describe the considerations in assessing the likelihood of model failure. (4)

Answer 11

• complexity (eg: higher complexity means a higher likelihood of misuse of the model)
• expertise (eg: non-expert users may not understand the model’s limitations)
• documentation (eg: bad documentation means that there is a higher likelihood that there will be a model failure)
• testing (eg: inadequate testing means that there is a higher likelihood of model failure)

Question 12

Does the actuary have more control over the SEVERITY or the LIKELIHOOD of model failure? Justify your answer.

Answer 12

The actuary has more control over the LIKELIHOOD:
it is within the actuary’s control to:
- CHOOSE a more reliable model
- TEST the model more thoroughly

Question 13

Identify the steps an actuary should take before using a new model. (4)

Answer 13

• review specifications
• validate implementation
• deal with the limitations
• keep documentation

Question 14

Describe what an actuary does when reviewing a model’s specifications. (3) (hint: DAMs)

Answer 14

Verify that the :

• DATA fits the model requirements
• METHODS are sound
• ASSUMPTIONS are appropriate

Question 15

Describe what an actuary does when validating a model’s implementations. (5)

Answer 15

• compare with other tested models
• maintain a set of test cases
• backtesting (testing with historical data where you already know the answer)
• run an entire live file through successive version of the model (for models with a higher risk-rating)
• peer review of testing procedure

Question 16

Describe what an actuary does when dealing with a model’s limitations.

Answer 16

The actuary must understand the range of uses for which the model was designed and tested.

Question 17

Describe what an actuary should include when documenting a model. (3)

Answer 17

• how the model was CHOSEN
• how it was TESTED
• what are its LIMITATIONS

Question 18

What is an important tool for validating models?

Answer 18

A model’s risk rating (riskier models need more thorough validation).

Question 19

How should an actuary evaluate an existing model that’s being used in a NEW way? (2)

Answer 19

• check that the initial model was properly validated

- review limitations in the new application that may not have been relevant in the initial application

Question 20

How should an actuary evaluate a model approved by the use BY OTHERS?

Answer 20

The actuary should review & approve the initial validation report.

Question 21

How should an actuary evaluate a model OUTSIDE the ACTUARY’S EXPERTISE? (5)

Answer 21

Make a reasonable attempt at understanding the model’s:

• specifications
• validation (extent to which experts were involved)
• risk-rating
• complexity
• control framework

Question 22

Give an example of a model outside of the actuary’s expertise.

Answer 22

A credit-scoring model.

Question 23

What is the purpose of sensitivity testing regarding models? (3)

Answer 23

• to validate a model
• to understand the relationship between inputs and outputs
• to develop a sense of comfort

Question 24

How can model assumptions be tested in the context of sensitivity-testing? (3)

Answer 24

• test assumptions OUTSIDE the expected range
• test assumptions singly and then IN COMBINATION
• test assumptions with a NON-LINEAR relationship between inputs and outputs

Question 25

What types of validation should be done when USING a model? (3)

Answer 25

Validation of Data, Assumptions and Results: - data should be Reliable and Sufficient - validate non-global assumptions that vary by model run - results should be 'reasonable' relative to the input

Question 26

What does it mean for data to be RELIABLE? (2)

Answer 26

- data RECONCILES to audited sources (eg: Balance Sheet | - data is REASONABLE with respect to prior period data (eg: data from prior quarter shouldn't be hugely different)

Question 27

What does it mean for data to be SUFFICIENT? (2)

Answer 27

- data FITS model specification | - data is available in a CONSISTENT format

Question 28

What checks can be done to validate the results of a model? (2)

Answer 28

- inputs and outputs should be CONSISTENT (input data should match similar fields in output file) - results should be REASONABLE in both magnitude & direction (ie: small change in inputs causes a small change in outputs)

Question 29

Compare a uni-dimensional model risk rating with a two-dimensional model risk rating.

Answer 29

uni-dimensional approach: - rating from 1-20 (20 is high) - based on financial significance, complexity, expertise of users, and documentation two-dimensional approach: - assessed separately for severity & likelihood of failure - final rating is a balance of these