applied scale construction

Question 1

what does construct validity mean?

Answer 1

Does a test measure the construct that it claims to measure
Unscientific to presume validity - face validity can deceive
Cannot rely on authority - must prove case using standard rigorous steps

Question 2

what is face validity?

Answer 2

when you assume someone is valid because it looks valid
may hide underlying invalid constructs
desirable, not enough, not even necessary

Question 3

what are the preconditions that must be met before you statistically measure construct validity?

Answer 3

Necessary steps for validity must be present - involves discrimination, reliability, structure
Validity must be present - involves patterns of links to other things

Question 4

what are the steps taken to statistically measure construct validity?

Answer 4

• Measure a construct statistically by giving it a score called the observed score
• If you measured it perfectly you get a hypothetical score called the true score - can never measure any construct perfectly - always some error/ invalidity
• If the link between the observed score and the true score were perfectly matched, you would have a perfect measurement (validity)
• If the link is mismatched/ not perfectly linked, you would have an imperfect measurement (error/invalidity)
• Find the correlation between the observed score and the true score on a scale on 0–>1

Question 5

what are the two sources of invalidity?

Answer 5

Systematic error - bias in a particular direction - caused by particular thing you can identify
Random error - bias in no particular direction - caused by different things you cannot identify

Question 6

what are the 5 standard steps to achieve construct validity in questionnaires?

Answer 6

1. Item design - coming up with the questions itself - can be open-ended or close-ended
2. Item analysis - to achieve discrimination
3. Reliability analysis - to achieve reliability
4. Factor analysis - to achieve structure
5. Scale validation - involves convergent and discriminant validation

Question 7

item design: what type of items are included in a questionnaire?

Answer 7

Close-ended - quantitative information, more common, more convenient, efficient, more top-down
Open-ended - qualitatively rich, generate own thoughts in response, must be coded to turn into numbers, labour-intensive, more bottom-up

Question 8

item design: how to write good items?

Answer 8

• Avoid vague, complex, obscure expressions
• Avoid items that pull for biased responses
• Bear in mind what people can and can’t know
• Try to cover fully the construct of interest

Question 9

item design: bad examples of items

Answer 9

Oxford Capacity Analysis by Scientologist

Question 10

item design: what is scaling?

Answer 10

applying a particular type of number to a response on a questionnaire
used for close-ended items

Question 11

item design: how is an item scaled?

Answer 11

• Convert psychological content to a number
• Various methods of varying sophistication
• Interval-level measurement assumed

Question 12

item design: what are the response options?

Answer 12

Decide on how much or few options there are
More options get more information - studies show that validity approaches its maximum between 5-7 items

Question 13

item design: what is the impact of labelling on responses to questionnaires?

Answer 13

Label every item to reduce ambiguity of what items mean - by standardising you are reducing variation

Question 14

item design: what is the impact of neural/uncertain responses on responses to questionnaires?

Answer 14

can increase information capture and therefore validity and accuracy
OR
can increase laziness and therefore decrease information capture and accuracy and validity SO
no overall benefit

Question 15

item design: what is the impact of forward-scored and reverse-scored items on responses to questionnaires?

Answer 15

should be roughly equal
reduces acquiescence bias
permits test of completing scale seriously by screening out non-standard respondents

Question 16

item analysis: discrimination in questionnaires and items

Answer 16

• An item should discrimination between different types of people otherwise no individual differences are assessed
• Items eliciting the same response are useless

Question 17

item analysis: how is variation measured statistically?

Answer 17

Desirable statistical features:
○ More dispersion –> higher SD - SD of scores on a scale is a direct measurement of dispersion of variability or variation in scores
○ Central average –> middling M - items scores then to clump into a bundle
○ Symmetric distribution –> lower SKEW - skew is a direct measure of the asymmetry or imbalance in a distribution of scores

Question 18

item analysis: what are the levels of discrimination?

Answer 18

○ Good distribution - broader spread of scores, mean is close to the scale midpoint, distribution is balanced
○ Bad distribution - narrower spread of scores, mean is lower than scale midpoint, distribution is positively skewed

Question 19

item analysis: what is the ideal criteria when using a 5-point scale?

Answer 19

M because 2-4
SD less than 1

Question 20

what is internal consistency?

Answer 20

consistency between items

Question 21

what is test-retest reliability?

Answer 21

consistency over time

Question 22

what is inter-rater reliability?

Answer 22

consistency between scores

Question 23

what is scale reliability?

Answer 23

how consistently a scale produces similar results when measuring the same construct multiple times

Question 24

reliability analysis: why does scale reliability matter?

Answer 24

Scale reliability matters because it is a precondition for validity
- A scale must be reliable and consistent to be valid but a reliable scale can be invalid due to random error

Question 25

reliability analysis: how should items relate to one another?

Answer 25

- Items should be related (on the same team) but not redundant - The more scores on items are related to one another, the higher the internal consistency of a scale - Frame in terms of correlations - should intercorrelate well but not perfectly - content validity matters

Question 26

reliability analysis: how is internal consistency assessed?

Answer 26

overall form of reliability Key item-level indices: - average correlation with other items (> r=0.2) - correlation with scale total (> r=0.3)

Question 27

reliability analysis: what is a scale-level index?

Answer 27

- Overall internal consistency: alpha - Average of all possible split-half correlations - A lower bound estimate - it could be higher - Increases with number of items - Lower alpha reduces possible correlations

Question 28

factor analysis: what is a factor and why does a factor analysis need to be conducted?

Answer 28

A factor is a single underlying dimension - Suggested but not guaranteed by high scale reliability (internal consistency) - Possible to have two or more factors underlying a highly reliable scale - Hence a factor analysis must be conducted

Question 29

factor analysis: how does factor analysis work?

Answer 29

- Asks: where do the correlations clump? - Reduces variances across many variables to fewer distinct clusters of shared variance

Question 30

Factor analysis: what are the three stages of factor analysis?

Answer 30

factor extraction factor rotation factor interpretation

Question 31

factor analysis: what is factor extraction?

Answer 31

Process of determining the number of the underlying factors Use scree plot gap to infer number of factors Eigenvalue must exceed 1 to be more important Use Principal Axes Factoring (PAF) or Maximum Likelihood (ML) - analyses shared variances - suited to finding underlying factors Do not use Principal Components Analysis (PCA) - analyses all variance - suited to collapsing variables

Question 32

factor analysis: what is factor rotation?

Answer 32

Aids interpretation and helps to move data towards a simple structure Two types of factor rotation: Orthogonal rotation - assumes factors independent (unlikely) - solution is more interpretable Oblique rotation - allows factors to be correlated (likely) - solution is less interpretable

Question 33

factor analysis: what is factor interpretation?

Answer 33

Items load on different factors Items don't load on the same factors Outcome can be messy sometimes - want it to be neat Confirmatory Factor Analysis (CFA) -hypothesise links between variables - check how well it fits the data - less interpretation Item Response Theory (IRT) - models responding as function of person and item - more informative but more complex

Question 34

scale validation: what does validation mean?

Answer 34

Involves showing that a scale, which has met all the preconditions so far, actually measures what its supposed to measure

Question 35

scale validation: how is validation assessed?

Answer 35

- Should correlate significantly, in a positive or negative direction with scores on other measures of related constructs - associative validity - Should not correlate significantly with scores on other measures of unrelated constructs - dissociative validity

Question 36

what is associative validation with example?

Answer 36

shows the external links it should e.g. convergent validity - where different measures of the same construct converge

Question 37

what is dissociative validation with example?

Answer 37

doesn't show those it shouldn't e.g. divergent validity - where measures of the same type but of different constructs diverge

Question 38

scale validation: what is the criteria for validation?

Answer 38

- Can be concurrent, predictive, retrodictive - All forms of criterion validity - Via questionnaires, behaviour, groups

Question 39

what is predictive validity?

Answer 39

if your measure of construct predicts something in the future e.g. IQ predicting future exam performance

Question 40

what is concurrent validity?

Answer 40

if your measure of construct predicts something at the same time e.g. questionnaires administered at the same time

Question 41

what is retrospective validity?

Answer 41

if your measure of construct predicts something in the past e.g. IQ predicting previous exam performance

Question 42

other forms of validity

Answer 42

External (generalisability) Internal (IV causes DV) Statistical (assumptions, procedures, etc)

Question 43

what is blirtatiousness?

Answer 43

- Tendency to respond quickly and effusively to others - Tends to amplify impressions of personality - Like extraversion/ dominance but not the same - "BLIRT" scale

Question 44

findings from research on blirtatiousness

Answer 44

- Higher in salespeople than librarians - Higher in Euro-Yanks than Asian-Yanks - Predicts amount of and speed of speech - Predicts shared self-observer impressions - Predicts engagement with chatty stooge - Predicts cheerful responding to annoying stooge - Predicts lower blood pressure in such situations