1
Q
According to the PBA/APS guidelines on the use and interpretation of psychological tests what technical information relating to a test should a psychologist take into account when interpreting the information derived from a test?
A
- understand nature of construct(s) underlying test score
- understand basic psychometric principles + procedures
- understand technical properties + limitations
- context + purpose + integrate test results with other important info about the individual
- importance of communicating CIs
- monitor and periodically review continuing effectiveness of tests you use
BASED ON ACCEPTED STANDARDS OF USE
- clear directions for admin/scoring
- info about purpose, SEM, validity + reliability
- test is valid for your purpose and sub-population
- acceptable validity + reliability in well-designed, independently replicated studies
- ensure it’s valid for your context/sub-group
- adequate reliability
- appropriate norm/reference group > need info about norm sample + procedure
2
Q
Apart from technical information relating to any tests used, what other considerations should a psychologist bear in mind when conducting a psychological assessment – according to the PBA/APS guidelines on the use and interpretation of psychological tests?
A
- all relevant info: language proficiency, medications/health problems, cultural background
- age, gender, language, background, social class (important for norms!)
- affect and mental state
- client’s previous experience with same/similar test(s)
- own competence: training, knowledge, expertise
- select words carefully in a report
- understand the limits of the test results
3
Q
According to Strauss and Smith (2009) discussion of test validity has been dominated by two main themes. Describe these themes and give examples of each approach.
A
- all validity can be thought of as contributing to construct validity
CONSTRUCT VALIDITY
- degree to which a test measure what it claims to measures
- theoretical understanding, latent variable or latent construct modelling
- rship to other relevant constructs + latent variables
- simultaneous process of measre and theory validation
- subsumes content and criterion validity
- EG: Personality 5 Factor Model - a statement of theory + (when used to describe the psych meaning in a test) a statement of construct validity of a test that measures 1+ of the five factors
CRITERION-RELATED
- practical evaluation of the usefulness of the test in the relevant population(s)
- eg. correlation b/w test score and criterion variable + t-tests to determine mean diff b/w criterion groups on test scores
- concurrent eg: depression test scores (BDI) vs. DSM consensus of MDD diagnosis
- predictive eg: poor scores on memory test predict future care needs in patient with diagnosis of alzhiemers
- once this is established, calc Se and Sp
4
Q
According to Strauss and Smith (2009) what contemporary research method is best suited to evaluating the multi-trait multi-method approach to construct validity?
A
- CFA
- analyses MTMM matrices
- allows for simultaneous evaluation of convergent and discriminant validity + contribution of method variance to observed rships
- allows direct comparison of alternative models of rships among constructs
- statistical tools are available to do this and are increasingly accessible to researchers
5
Q
In practice two members of the family of standard errors of measurement are most useful. What are they and how are they interpreted?
A
STANDARD ERROR OF PREDICTION
- to observe change in one individual’s score over time
- can produce CIs to predict future scores
STANDARD ERROR OF ESTIMATION
- use to get CIs around PTS
- to determine likely location of individual’s score on test scale + to compare observed score with population parameters
6
Q
Describe the Predicted True Score and explain its use with examples.
A
- PTS = theoretically correct score
- CIs placed around for an individual score
- most important to consider when Rxx (reliability) is relatively low
- provides the center of the distribution of scores that would be observed if an individual were tested many times with the same test (assuming no real change in scores)
- PTS will always be closer to mean than observed score (how much closer depends on reliability of test: more reliability = PTS closer to observed score)
EXAMPLE
- observed score 130
- test: M 100, SD 15
- if reliability 0.7 > PTS is 121
- if reliability 0.9 > PTS is 127 (closer to observed)
7
Q
Describe some reasons why knowledge of the reliability of the score derived from a test aids interpretation of that test.
A
- provides info on proportion of variance in observed score that is due to variance in true score, rather than error variance
- helps determine predicted stability of an individual’s score over time
- helps determine change (worsen/improve) in condition b/w one testing occasion and another
- determines the distro of one person’s score (less reliable = wider distro)
- while reliability does not imply validity, reliability does place a limit on the overall validity of a test
- known reliability of two variables allows us to estimate max validity correlation
8
Q
In applied research, correlations between a pair of tests scores are commonly interpreted in terms of convergent and discriminant validity. How does the reliability of either or both test score affect the interpretation of this kind of correlation?
A
- reliability can have an attenuating effect on validity coefficients
- validity correlations are diluted/weakened/underestimated by measurement error
- disattenuation provides for a more accurate estimate of the correlation between the parameters by accounting for this effect
- when considering convergent and discriminant validity coefficients, they should always be interpreted in their dis-attenuate form
- attenuation formula: corrects observed validity coefficient on basis of reliability
- corrected Rxy = Rxy / sqrt( Rxx x Ryy )
- can manipulate formula to determine max validity correlation b/w 2 variables
- assume corrected Rxy = 1
- known reliability of 2 variables allows us to estimate max validity correlation
9
Q
Describe the use of the Spearman-Brown prophecy formula for test development.
A
- relates psychometric reliability with test length
- SBPF allows a test developer to determine the reliability of a new test, as expanded by a factor of n
- tells you how the reliability of a test will change after expanding the test length
- produces a negatively accelerating growth curve with increasing test length (“diminishing returns” function)
- should use formula to determine optimal trade-offs b/w reliability and time devoted to data collection
10
Q
What do we mean by relying on “clinical judgement” to form professional opinions?
A
- using past clinical experience + intuitive learning experiences
- some say it is just anecdotal evidence
- drawing knowledge on basis of causal inferences, drawn from clinical samples you have experienced
- knowledge accumulated more or less informally over (many) years of clinical practice
- use of intuitive (generalizing from the particular) logic to guide clinical practice and de-emphasizing objectivity based on good test standardization and norms
- repudiating scientific methods in clinical research and clinical decision making
- no formal data analysis
- collected haphazardly
11
Q
Describe some of the common biases which may influence learning from clinical experience.
A
- representativeness heuristic: identify a client’s symptoms as conforming to or understanding of a disorder (pigeonholing) and then assuming that we understand the client and their prognosis
- confirmation bias: bc we are experts in normality, we make decisions in-line with abnormality
- availability heuristic: experiences that are vivid in memory are most likely to be invoked to explain clinical observations
- overconfidence, under-use of base rates etc.
- hindsight bias: tendency to view events as more predictable than they really are
- ontological-epistemological identity fallacy: human behaviour is complex and therefore our study and understanding of it must be complex (can’t be simple)
- Kreapelinian hierarchy: if it’s psychopathology, it can’t be brain disease + if it’s brain disease, it can’t be psychopathology (but overlap actually more common than not, but this comorbidity is dealt with very poorly in our diagnostic classification systems)
12
Q
Outline some of the recommended methods to improve the accuracy of clinical judgement.
A
- actively consider alternative outcomes
- minimise the role of memory
- incorporate base rates and prevalence rates into diagnostic thinking
- do your own diagnostic validity and treatment outcome research
- avoid using assessment/predictive techniques with unsatisfactory/unknown reliability + validity
- don’t assume that you understand a client > always obtain follow-up data
- develop models of diagnostic judgment and decision-making
- adopt an evidence-based approach > read literature for quality of evidence, interpret clinical importance of the evidence, apply high quality evidence to patients
- train psychs to be aware of own biases + limitations of their clinical judgment
- ALWAYS be cautious in clinical opinions
13
Q
Grove and Meehl’s (1996) describe the results of a meta-analysis of the empirical literature comparing clinical with statistical prediction. Briefly summarize the findings of the meta-analysis.
A
- meta-analysis of empirical literature comparing clinical with statistical prediction since 1920
- studies of predicting health-related phenomena (eg. diagnosis) or human behaviour
- 136 studies, 167 distinct comparisons b/w the two methods of prediction
- all had empirical outcomes of 1+ human-based prediction and 1+ statistical prediction
- 64 favoured stat prediction
- 64 showed equivalent accuracy
- 8 favoured clinician (but no reason why, random, not concentrated in any one predictive area or represent any one type of clinician, no obvious characteristics in common)
- according to “total evidence rule: most plausible explanation of these deviant findings is a combo of random sampling errors + clinicians’ informal advantage in being privy to more data than actuarial formulae
- less studies favouring clinicians than would be expected due to chance if clinician and statistical were equivalent
- clinician experience makes little/no diff in predictive accuracy BUT type of stat prediction did make a diff (best = weighted linear prediction)
- OVERALL: stat prediction methods superior to clinical prediction
14
Q
“I Cannot Use Actuarial Prediction Because the Available (Published or Unpublished) Code Books, Tables, and Regression Equations May Not
Apply to My Clinic Population”
Summarize Grove and Meehl’s (1996) responses to this objection.
A
- relies on notion that slightly non-optimal stat parameters due to validity generalisation would liquidate the superiority of the stat over clinical model > no evidence for this + does not make mathematical sense for cases where the actuarial method’s superiority is high
- if stat predicts something with 20% greater accuracy than clinicians in a no. of studies around the world + no reason to think that your patient is extremely unlike all other psych outpatients = improbable that clinicians are so superior that the stat method would reduce efficacy to level of clinicians
- cannot simply assume that if an actuarial formula works in several outpatient psych pops, and each one does better than the local clinicians or better, the formula will not work well in one’s own clinic
- all clinicians have diff training in diff settings with diff supervisors etc. > subject to same validity generalisation concern as stat methods, if not more so
- some think it unethical to apply a stat predictive system to their specific client without having validated it > but this is strange coming from people who rely on anecdotal evidence daily to make big decisions given all the literature says that anecdotal evidence is untrustworthy
- if re-validating predictor equation/table is deemed necessary, this is quick (just have to record hits/misses from someone else’s discriminant function) and then make a new predictor system if it doesn’t work well (quick, doesn’t need fancy maths)
15
Q
“Statistical Predictionists Aggregate, Whereas We Seek to Make Predictions for the Individual, so the Actuarial Figures Are Irrelevant in Dealing With the Unique Person”
Summarize Grove and Meehl’s (1996) responses to this objection.
A
- they say: actuarial figures give probs that are relevant for making predictions with an individual
SCENERIO
- medical illness > radical surgery
- you would ask if it work, how risk, % works, how many die etc
- doctor: why are you asking me about stats? you are a unique individual, nobody is exactly like you, do you want to be a mere statistic? what diff does % make anyway?
- physician cannot tell you beforehand into which group (success/failure) you will surely fall, but probs are still important for decisions
- claim cannot be made: stats give mere probabilities (average results/aggregates), whereas in dealing with the unique person one will know exactly what will befall this person
- could assign client to patient categories and get the probability of the event > resulting proportions would differ depending on which reference class was used
- there are as many probabilities as there are reference classes > choose the narrowest reference class with no. of cases big enough to provide stable relative frequencies available
- hardly any clinical decisions (using formal or informal procedures) that people claim to be absolutely certain
- clinicians focus on cases they could have saved an actuarial mistake + ignore the obvious point that any decision policy (unless infallible) will involve making some mistakes
- research should be done to show that clinicians to realise that, in general, they do not do as well as the actuarial method + then realise how they can improve upon the equation once in a while by clear-cut “broken-leg” countervailings > note there should be a high threshold for countervailings though
16
Q
According to Grove and Meehl (1996) what are some of the reasons practitioners continue to resist actuarial, or data-based clinical predictions of diagnosis or treatment outcomes?
A
- no. of sociopsych factors at play
POOR EDUCATION
- training: don’t place value on cultivation of skeptical, scientific habits or thoughts
- role models in psych: don’t place high value on scientific thinking; don’t do research themselves
- many clin psychs not aware of controversy b/w clinical and stat prediction methods
FEAR
- increased technology > unemployment
- resent/dislike that computers can compete with human minds
- self-concept: threat to personal security > societal value places on scholarly + technical functions
FONDNESS TO THEORY
- for certain theories/concepts
- idea that theory-mediated prediction do not contribute any more than a stat table (or could be worse) = cog dissonance
DEHUMANISING
- continued misperception that actuarial method is dehumanising to clients
17
Q
Define Sensitivity and Specificity. Feel free to use a diagram, or figure.
A
SENSITIVITY:
- proportion of cases who have target condition, in whom the test result is +ve
- how good a test is at correctly identifying people with the condition
- a/(a+c)
- 0-100 (>50% useful)
- interpret with CIs
- SpPin
SPECIFICITY:
- proportion of cases who do not have the target condition, in whom the test result is -ve
- how good a test is at correctly identifying people without the condition
- d/(d+b)
- 0-100 (>50% useful)
- interpret with CIs
- SnNout
18
Q
Define base-rates (or clinical prevalence or pre-test probability) and explain why this quantity varies across professional settings.
A
- rate/freq of target condition in a particular population
- varied over profess settings bc. typically higher in profess setting than general pop
- eg. rate of ALZ in primary care setting higher than general population, but even higher still in dementia/cog impairment setting
- failure to know the base rate in your setting > likely to wrongly interpret results
- diagnostic validity is a joint function of criterion-related validity and base rate
19
Q
Explain how base rates impact on the interpretation of Sensitivity and Specificity.
A
- Se and Sp are properties of the test and do not vary as base rates vary (assuming they come from a representative sample, we assume they can generalise to other samples)
- BR affects interpretation of Se and Sp by allowing +ve and -ve predictive powers of a test (PPP and NPP) to be calculated
- even with high sensitivity + specificity > point at which base rate is so low that test ceases to be clinically useful/valid
- all tests, point at which BR so low that PPP <0.5 and point where BR so high that NPP <0.5
- ^ at these point, interpreting the test results at face value as indicating presence/absence of disorder is more likely to be incorrect than correct
- Se or Sp may be low, or 50%, but the test may still have high PPP and NPP > thus the test still works
- dangerous to blindly follow base rates, but they should always be carefully considered
20
Q
Define Positive and Negative Predictive Power (feel free to use a diagram, or figure) and explain the common sources of misinterpretation of these values.
A
- PPP: prob that a +ve test result is correct at a specific base rate
- prob that someone with a +ve test result has the target condition
- NPP: prob that a -ve test result is correct at a specific base rate (1 - NPP = prob that the person has the disorder, given a -ve test result)
- prob that someone with a -ve test result does not have the target condition
- both range 0-1, >0.5 useful
- they are properties of the test as well as the local base rate
- PPP and NPP always need to be interpreted in terms of local base rates
MISINTERPRETATIONS
- assume fixed values across diff clinical settings (don’t consider unique base rates)
- assume that NPP and PP are same in all samples as reported in a test validation study (don’t consider base rates)
21
Q
Define Likelihood ratios by formula (no need to mention odds) and explain some of the advantages of likelihood ratios compared to PPP and NPP.
A
POSITIVE: sensitivity/(1-specificitiy)
- rules in diagnosis (want high)
NEGATIVE: (1-sensitivity)/specificity
- rules out diagnosis (want low)
- LR are independent of prev, they are re-worked column ratios and thus properties of the test
- how much more likely someone is to get a +ve test result is they have the disorder, compared to someone without the disorder
- highly valid test: LR+ much larger than 1 (+ve result increases prob of target condition), LR- much smaller than 1 (-ve test result lowers prob of target condition)
- LR(+) = increase in pre-test odds, assuming a +ve test result
- LR(-) = decrease in pre-test odds, assuming a -ve test result
- change in prob from base rate of target condition after obtaining +ve/-ve test result
- used to calculate post-test odds + post-test probability
22
Q
Define the construct of Fluid Intelligence (Gf) according to the CHC model and give examples of tests that evaluate this construct from a contemporary adult ability test
A
- ability to solve novel problems
- ability to process or manipulate abstractions, rules, generalisations and logical rships
- form/recognise concepts, perceive rships among patterns, draw inferences, problem solving etc.
- broad ability to reason, form concepts + solve problems using unamility info/novel procedures
WAIS
- matrix reasoning (narrow: general sequential reasoning)
- arithmetic (narrow: quantitative reasoning)
23
Q
Define the construct of Crystallized Intelligence (Gc) according to the CHC model and give examples of tests that evaluate this construct from a contemporary adult ability test
A
- breadth + depth of acquired knowledge acquired from one’s culture
- dependent on education and experience
- ability to communicate one’s knowledge
- ability to reason using previously learned experiences/procedures
WAIS
- vocab (narrow: general info + lexical knowledge)
- information (narrow: general info)
24
Q
Define the construct of Processing Speed (Gs) according to the CHC model and give examples of tests that evaluate this construct from a contemporary adult ability test
A
- ability to perform automatic cog tasks quickly and fluently (esp. when under pressure to maintain attention)
- ability to process information rapidly
- perception, number, reading/writing speed, test speed etc.
WAIS
- symbol search
- coding
25
Define the construct of Short-term Memory (Gsm) or Working Memory according to the CHC model and give examples of tests that evaluate this construct from a contemporary adult ability test
- ability to apprehend + hold info in immediate awareness and use it within a few seconds
- encode, maintain, manipulate info in one's immediate awareness
- WM is a narrow Gsm > ability to temporarily store and perform a set of cog operations on info that requires divided attn and management of limited STM capacity
WAIS
- digit span backward (narrow: WM)
- digit span sequencing (narrow: WM)
26
Define the construct of Long-term Retrieval (Glr) according to the CHC model and give examples of tests that evaluate this construct from a contemporary adult ability test.
- ability to store info and fluently retrieve it later in the process of thinking
- store, consolidate, retrieve over time (anything longer than seconds)
WMS:
- verbal paired associates
- logical memory
27
What CHC constructs does the WAIS-IV measure? Which subtest scores correspond to which constructs?
5-factor CHC:
- crystallised intelligence (vocab + language) > verbal comprehension
- short-term memory (digit span backward and sequencing) > working memory
- fluid intelligence (matrix reasoning + arithmetic) > perceptual reasoning
- visual processing (block design + visual puzzles)
- processing speed (symbol search + coding) > processing speed
28
Outline some of the reasons Reynolds and Milam (2012) caution against intelligence test subtest profile analysis.
- issue with subtest interpretation = subtest scores are too unreliable to be a useful source of info about individual differences
- subtest score diffs too common in normal individuals to be used to identify learning problems etc.
- subtest profiles lack reliability (rarely >.5) >> not interpretable
- run the risk of mistaking common ability patterns as being rare/noteworthy > issues with diagnostic decision-making process
29
Briefly outline the key components of Kaufman’s (Lichtenberger and Kaufman, 2009) “intelligent testing” approach.
- subtests measure what the individual has learned (within a culture)
- subtests are samples of behaviour and are not exhaustive (caution with generalisation)
- standardised, individually administered tests assess mental functioning under fixed experimental conditions (be aware of the artificiality, don't over-interpret results, consider test data in broader context)
- test batteries are optimally useful when interpreted from a theoretical model (organise data in meaningful way, understand strengths and weaknesses and the theoretical meaning of them)
- hypotheses generated from the test profile should be supported with data from multiple sources (prevent test misuse, integrate with background info
+ observations)
- remember, the focus of ax is the person being ax NOT the test
30
Describe the approach to Wechsler test score analysis described by Lichtenberger and Kaufman, 2009) down to the level of evaluating factor-composite scores as unitary constructs. Be sure to identify the criteria by which you decide to move from one level of analysis to another.
STEP 1:
- is FSIQ a reasonable summary of client's ability profile?
- seek evidence to reject hypothesis that 4 indicies can be summarised as a single ability (diff <23 in 4 indexes = use FSIQ)
STEP 2:
- if you reject a 1-factor solution (FSIQ), consider interpreting the test profile in terms of five-factor CHC model (scaled score diff <5 = use indexed)
- after FSIQ, 4 indexes (VC, PR, WM + PS) are most reliable scores and may be trait-homogenous
STEP 3:
- examine subtest scores contributing to each index
- is there any evidence to reject assumption that each index is trait homogenous?
- YES > consider reformulating ability profile in terms of more discrete abilities (beware of Type 1 errors)
- ONLY use discrete approach is rejecting a 4/5 factor solution
NOTE: a 'homogenoenous' score is only re-interpreted as heterogenous is there is tangible (e.g base-rate) evidence to reject the nullhypothesis of homogeneity and the alt. interpretation has relevant theoretical/clinical meaning
31
Assuming your analysis of a Wechsler test profile supports retention of Full Scale IQ (FSIQ) as a good summary of a patient/client’s current cognitive ability. What are the advantages of relying on FSIQ as an indicator of current ability?
- most reliable score
- narrowest CI
- most sensitive measure of cognitive abilities in terms of to change over time + impairment in cross-section assessment
- extensive criterion-related validity evidence
- the psychologically sophisticated alternative a cog mental status score (eg. MMSE)
Psych Assessment
flashcards
Decks in class (5)
# Cards
