reliability

Question 1

what does reliability refer to?

Answer 1

how consistent a measuring device is

Question 2

how could a particular measurement be described as being reliable?

Answer 2

if it is made twice and produces the same result

Question 3

what are 4 ways of assessing reliability?

Answer 3

• test-retest
• inter-observer
• inter-rater
• inter-interviewer

Question 4

what does test-retest reliability involve?

Answer 4

• give the same test / questionnaire to the same person (or people) on different occassions
• if the test / questionnaire is reliable, the results should be the same or similar each time it is given
• can also be applied to interviews

Question 5

why must there be sufficient time between test and retest?

Answer 5

• to ensure pt cannot recall their answers to the questions
• not so long that their attitudes / abilites have changed

Question 6

what happens to the data after the retest?

Answer 6

• two sets of scores are correlated to make sure they are similar
• if correlation is significant (and positive) then measuring instrument has good reliability

Question 7

what is an issue of observational research?

Answer 7

• one observer’s interpretation of events may differ widely from someone else’s
• this introduces subjectivity, bias and unreliability into the data collection process

Question 8

how many observers should be present in an investigation?

Answer 8

teams of at least two

Question 9

how can inter-rater reliability be established?

Answer 9

• small-scale trial run (pilot study) of observation to check that observers are applying behavioural categories in the same way
• comparison may be reported at end of study

Question 10

how should data be colected during an observation?

Answer 10

• observers should watch the same event or sequence of events
• record data independently
• data collected by the two observers should be correlated to assess its reliability

Question 11

what is inter-rater reliability used for?

Answer 11

content analysis

Question 12

what is inter-interview reliability used for?

Answer 12

interviews

Question 13

how is reliability measured?

Answer 13

• using correlational analysis
• in test-retest and inter-observer reliability, the two sets of scores are correlated
• correlation coefficient should exceed +0.80 for reliability

Question 14

how could a questionnaire produce low test-retest reliability?

Answer 14

questions could be:

• complex
• ambiguous
• interpreted differently be the same person on different occasions

Question 15

how can the reliability of questionnaires be improved?

Answer 15

• remove or rewrite some questions
• replace open questions (with more room for misinterpretation) with closed alternatives which may be less ambiguous

Question 16

how can the reliability of interviews be improved?

Answer 16

• use same interviewer each time
• if this is not possible or practical, all interviewers must be properly trained so one particular interview is not asking questions that are too leading or ambiguous
• use a structured interview, as interviewer’s behaviour is more controlled by the fixed questions

Question 17

how can the reliability of observations be improved?

Answer 17

ensure behavioural categories:

• are properly operationalised
• are measurable
• are less open to interpretation
• do not overlap
• cover all possible behaviours

Question 18

what happens if behavioural categories are not operationalised well, are overlapping or absent?

Answer 18

different observers have to make their own judgements of what to record, ending up with differing and inconsistent records

Question 19

what needs to be done if reliability for observations is low?

Answer 19

• observers may need further training in using the behavioural categories
• observers may wish to discuss their decision with each other so they can apply their categories more consistently

Question 20

what is the focus of reliability in experiments?

Answer 20

• standardised procedures
• procedures must be the same every time to compare the performance of different pts and to compare results from different studies