Reliability & Validity

Question 1

What is reliability?

Use an example

Answer 1

Reliability is a measure of consistency

Example: Does an individual get the same result on an IQ test when they take it again

If a particular measurement is repeated and the same result is obtained, then that measurement is reliable.

Reliable = Consistency

Question 2

What are the ways to assess reliability?

Answer 2

Test-retest

Inter-observer

Note: For both methods, if a significant positive correlation is found between the two sets of results, they can be said to be reliable.

Question 3

What is test-retest?

Use an example

Answer 3

Test the same person twice or repeat the study using the same procedure, design and measurements.

E.g., The same test or questionnaire is given to the same person two or more different occasions. If the test or questionnaire is reliable the results should be the same (or very similar) each time it is administered.

Question 4

What is inter-observer?

Use an example

Answer 4

Compares observations from different (two or more) observers.

E.g., Two or more observers compare their data by conducting a pilot study of the observation to check that observers are applying behavioural categories in the same way. Observers should watch the same event, sequence of events, but record their data independently.

Question 5

How to improve reliability of questionnaires and interviews?

Answer 5

Use structured interviews

Use more closed questions

Make sure questions are clear and unambiguous

Remove questions that are causing inconsistency

Question 6

How to improve reliability of lab experiments?

Answer 6

Control extraneous variables

Use standardised procedures

Question 7

How to improve reliability of observations?

Answer 7

Operationalise behavioural categories

Train observers on the behavioural categories

Do a pilot study to trial the behavioural categories

Question 8

What is validity?

Provide example

Answer 8

Validity is the extent to which a test measures what it claims to measure.

Data can be reliable but not valid.

e.g., an IQ test may produce the same result every time when the same people are tested but not measure what is it is designed to.

Question 9

What are the types of validity?

Answer 9

Internal

External

Ecological

Temporal

Population

Question 10

What is internal validity?

Answer 10

Controls within a study e.g., demand characteristics

Question 11

What is external validity?

Answer 11

The extent to which results can be generalised to other settings, population or eras.

Question 12

What is ecological validity?

Answer 12

refers to whether findings can be generalised from one setting to another, most particularly generalised to everyday life. This may not relate to the setting (e.g., a lab) but more to the realism of the participant’s task.

Question 13

What is temporal validity?

Answer 13

the extent to which the findings and conclusions of study are valid when we consider the differences and progressions that come with time. E.g., Asch’s study may lack temporal validity now because it was conducted during a conformist era in US history.

Question 14

What is population validity?

Answer 14

generalising results to the target population.

Question 15

What are the ways of assessing validity?

Answer 15

Face validity

Concurrent validity

Question 16

What is face validity?

Answer 16

how valid your results seem based on what they look like. This is the least scientific method of validity, as it is not quantified using statistical methods.

Question 17

What is concurrent validity?

Answer 17

Concurrent validity refers to the extent to which the results and conclusions concur with other studies and evidence. If a significant positive correlation is found between the two sets of results, the measure can be said to be valid.

Question 18

How to improve validity for questionnaires?

Answer 18

Make them anonymous

Incorporate lie scales (questions that no one would honestly answer yes to, e.g. I always tell the truth)

Question 19

How to improve validity for lab experiments?

Answer 19

Control extraneous variables

Use standardised procedures

Use single- and double-blind procedures

Question 20

How to improve validity for observations?

Answer 20

Operationalise behavioural categories

Use covert observation

Question 21

Summary of reliability and validity

Answer 21

Reliability and validity are both about how well a method measures something:

Reliability refers to the consistency of a measure (whether the results can be reproduced under the same conditions).

Validity refers to the accuracy of a measure (whether the results really do represent what they are supposed to measure).