WEEK 2 - Measuring variables, sampling, validity and reliability

Question 1

Why is generalisability in sampling important?

Answer 1

Generalisable results are results that reflect the true state of affairs in the population of interest - to claim this your sample needs to be as representative of the population as you can make it

Question 2

What is ‘population’

Answer 2

The totality to whom/which you wish to generalise your study findings

Question 3

What is ‘sample’

Answer 3

the participants in your study

Question 4

What are the two types of sampling procedures?

Answer 4

Probability sampling - simple random, systematic random, stratified, multi-stage cluster

Non-probability sampling - convenience, snowball, purposive

Question 5

What is probability sampling?

Answer 5

• A way to ensure that your sample is representative of the population (on the characteristics deemed important to the study)
• Basic principle: A sample will be representative of the population if all members of the population have an equal chance of being selected for the sample
• Allows the researcher to calculate the relationship between the sample and the population

Question 6

What are the types of probability sample?

Answer 6

• Simple Random sample
• Systematic random sample
• Stratified random sampling
• Multi-stage cluster sampling

Question 7

What is a simple random sample?

Answer 7

• each member has an equal and independent chance of being selected
• define the population, list all members, assign numbers then randomly select number

Question 8

What is systematic random sampling?

Answer 8

• Every kth person

-Randomly select the first person then divide the size of the population by the size of the desired sample and then use this to determine the interval at which the sample is selected

**size of population/size of desired sample **

Example: to select a sample of 1000 people from a list of 10 000 randomly select the first person and then select every 10th person from the list

Question 9

What is stratified sampling?

Answer 9

• If you want to make sure the profile of the sample matches the profile of the population on some important characteristics (for example, age or ethnicity)
• Researcher divides population into sub-populations (strata) and then randomly samples from strata

Question 10

Why do we use stratified sampling?

Answer 10

• Can reduce sampling error by ensuring ratios reflect actual populations (example -ratio of different ethnic groups)
• To ensure that small sub-populations are included in the sample

Question 11

What is multi-stage cluster sampling

Answer 11

• Begins with a sample of groupings then samples individuals

example: Rural sample
- Define rural sample as those with populations <X
- Get a listing of all relevant towns
- Take a random sample of towns
- Randomly sample people from within the randomly sampled town

Question 12

What is the difference between multi-stage cluster sampling and stratified sampling?

Answer 12

Multi-stage cluster is not the same as stratified sampling as each cluster does not need to be sampled.

Question 13

What is multi-stage / multi-phase sampling?

Answer 13

• Larger sample obtained first to identify members of a sub-sample
• Sub-sample randomly chosen from study

Example: Large community survey in Australia which asks if they had diagnosis X disease –> X disease sufferers followed up again for sampling

Question 14

What is non-probability sampling?

Answer 14

• Not every member of the population has equal chance of being part of the sample

Question 15

Why use non-probability sampling?

Answer 15

There are no lists for some populations under study, for example:
- The homeless
- Certain occupations (e.g. farmers)
- Hidden or specific populations (e.g. farmers with mental health issues
- convenience/resource restriction

Question 16

Types of non-probability samples

Answer 16

• convenience sample
  -snowball sample
• Purposive sample

Question 17

What is a convenience sample

Answer 17

A sample of available participants
Example: students enrolled in a particular course or people passing a particular location

Question 18

What are advantages and disadvantages of convenience sampling

Answer 18

Advantages:
Easy
Inexpensive

Disadvantages:
No control over representativeness
Bias

Question 19

What is snowball sampling?

Answer 19

• Involves collecting data with members of the population that can be located and then asking those members to provide information/contacts for other members of the population
• Used mainly for hard-to-study populations
  for example: homeless young people, people with not commonly listed characteristics

Question 20

What is Quota sample?

Answer 20

• Non-probability sampling equivalent of stratified random sample
• Want to reflect relative proportion of population but you don’t/can’t sample randomly from each strata as you do in stratified random sampling

Question 21

What is purposive/judgment sampling?

Answer 21

• Selecting a sample based on knowledge of the population, its elements and the purpose of the study
• Clear purpose to sampling strategy. Select key informants, atypical cases, deviant cases or a diversity of cases

Example: If a study aimed to find problems experienced by new immigrants it may sample key people involved in agencies that help immigrants such as ethnic welfare groups, community immigration legal aid groups

Question 22

Why is purposive sampling often used?

Answer 22

• To select cases that may be especially informative
• Select cases in a difficult to reach population
• Select cases for in depth investigation

Question 23

Which method of sampling do I use?

Answer 23

• best method is normally a probability sampling one (as the aim of research is to generalise findings to the population)
• Sometimes different sampling methods aren’t feasible given resources, time etc.

-

Question 24

How do you determine sample size?

Answer 24

• Largely determined by the analysis you plan to conduct with the data derived
• Generally: the more complex the analysis the larger the sample

Question 25

When are larger sample sizes needed?

Answer 25

- When the sample is heterogeneous (when the sample is composed of widely different people) - When you want to break down the sample into subcategories (e.g. look at gender separately) - If you want to obtain a more narrow or precise confidence interval - when you expect a small effect or weak relationship - for some statistical techniques

Question 26

What are the five simple rules for determining sample size?

Answer 26

- If less than 100, use entire population - Larger sample sizes make it easier to detect an effect or relationship in the population - Compare to other research studies in area by doing a literature review - Use a power table for a rough estimate - Use a sample size calculator (e.g. G power)

Question 27

What is a metric? (levels of measurement)

Answer 27

When we want to measure something (e.g religion, self-esteem, tennis ability) we need to choose a metric with which we can measure it The metric will determine the statistical analysis we will perform

Question 28

What are the levels of measurement?

Answer 28

Nominal Ordinal Ratio Interval (NORI)

Question 29

What is nominal?

Answer 29

Something which is purely categorical information (quality or kind of something) Example: religion, gender

Question 30

What is ordinal?

Answer 30

A rank order Ordinal variables do indicate an underlying quantity but they do not obey mathematical laws (you cannot meaningful subtract or divide)

Question 31

What is interval?

Answer 31

A true number in the sense that there are equal intervals implied but no true zero point - example: temperature in degrees

Question 32

Ratio

Answer 32

A true number. The distinguishing feature of a ratio scale variable is that it has a meaningful zero point, which participants could use to indicate the quantity is completely absent

Question 33

What is an issue with reliability and validity

Answer 33

The issue is you can't assess these until after you have developed your questionnaire and use them . Therefore, a pilot test can be beneficial Many people chose to use established measures instead of their own

Question 34

What are the two broad types of validity

Answer 34

Internal validity External validity

Question 35

Why is validity an issue?

Answer 35

- Many (if not most) variables in social research cannot be directly observed example: motivation, satisfaction, helplessness Therefore, the challenge is to make a judgment call on weather we are measuring what we think we are measuring

Question 36

What are the types of validity

Answer 36

Face validity content validity criterion-related validity - (concurrent validity +predictive validity) construct validity - (convergent + divergent)

Question 37

What is face validity

Answer 37

- Asks the question: on the face of it, does my measure seem to relate to the construct? - Measures that lack face validity have the potential to alienate research participants (what are they really trying to measure) - a weak subjective method for assessing validity but a first step

Question 38

What is content validity

Answer 38

- The extent to which the measure represents a balanced adequate sampling of relevant dimensions - Considers what should go into a measure and what should stay out - considers boundaries - How much does the measure cover the content of the definition? Example: which of the following would be a more valid test of mathematical ability; - a 20-question test containing addition problems - a 20 question test containing addition, subtraction, multiplication problems

Question 39

What is criterion-related validity

Answer 39

- involves checking the performance of your measure against some external criterion

Question 40

What are the two types of criterion-related validity:

Answer 40

Concurrent - Establish the validity of your measure by comparing it to a “gold standard” (i.e., Existing validated measure of the same construct) Predictive - does the measure predict/relate to some criterion that you would expect it to predict

Question 41

What is predictive validity (in criterion-related validity)

Answer 41

- Does the measure predict something that it is theoretically supposed to predict - Does the measure differentiate between people in a way that you would expect * What should a measure of the following constructs predict? * Iq -> perhaps some cognitive-based performance task * Workplace depression scale -> number of mental health sick days

Question 42

What is construct validity

Answer 42

Demonstrating that the measure relates to the theoretical construct of interest

Question 43

What are the two types of construct validity

Answer 43

convergent divergent

Question 44

What is convergent validity (in construct validity)

Answer 44

Demonstrates that the measure relates to other similar measures

Question 45

What is divergent validity (in construct validity)

Answer 45

Demonstrates that the measure does not relate to unrelated constructs

Question 46

Summary of validity types

Answer 46

Face - in the judgment of others items appear to relate to construct content - Captures the entire meaning (all elements of definition) of a construct criterion - agrees with external source concurrent - agrees with existing gold standard measure Predictive- agrees with future behaviour construct - how well multiple indicators relate to each other (consistent with theory) convergent: similar measures (or measures of theoretically related constructs) are related divergent - different measures are unrelated

Question 47

What is reliability?

Answer 47

- The consistency or repeatability of your measurement

Question 48

What are the types of reliability?

Answer 48

1. Stability of the measure (test-retest) 2. Internal consistency of the measure (split-half, Cronbach's alpha) 3. Agreement or consistency across raters (inter-rater)

Question 49

What is test-retest reliability?

Answer 49

- Addresses the stability of your measure - you administer the measure at one point in time and then you give the same measure to the same participant at a later point in time - You correlate the scores on the two measures

Question 50

What is the problem with test-retest?

Answer 50

- memory effect - practice effect (performance improves because person has had practice taking test before) - Another thing to consider is time between intervals. If too short there is a greater risk of memory effect. If too long there is a risk of other variables (e.g. additional learning) influencing results

Question 51

What is split half reliability

Answer 51

- administer a battery of questions - split the measure into two halves - correlate the score on the two halves of the measure higher correlation means greater reliability strengths: eliminates memory and practice effects limitations: are the two halves equivalent

Question 52

What is inter-item reliability

Answer 52

- Assesses the internal consistency of the measure ie. tells you how well items or questions in your measure appear to reflect the same underlying construct. - You will get good internal consistency if individuals respond in approximately the same way on your survey Cronbach’s alpha can range from 0 (when the items are not correlated with one another) to 1.00 (when all items are perfectly correlated to each other). The closer the alpha is to 1.00, the better the reliability of the measure

Question 53

Inter-rater or inter-observer reliability

Answer 53

- Checking the match between two or more raters or judges in your study

Question 54

Calculations of inter-rater reliability

Answer 54

- Nominal or Ordinal scale (the percentage of times different raters agree) - Interval or ratio scale - correlation coefficient

Question 55

What kind of reliabilities co-efficients should I be aiming for?

Answer 55

- Test-retest coefficients > .70 - Internal consistency >.70 (but ideally much higher) - Rating consistency >.90

Question 56

Reliability and measurement error

Answer 56

*measurement error serves to weaken our statistical tests * All other things being equal, more error in measurement means lower power * Choosing a measure which is highly reliable decreases measurement error and increases the power of your design

Question 57

Can a measure be reliable but not valid?

Answer 57

Yes! You could have a consistent measure that does not actually measure the construct

Question 58

Can a measure be valid but not reliable?

Answer 58

Yes. Example of valid tool but is unreliable – something that is difficult to implement (e.g., Skin fold tests – require technical skill) – may be unreliable across multiple administrators.

Question 59

Summary of validity types

Answer 59

Test-Retest - Same question given on different occasions and data correlated Split half - Split questions in half and correlate data from two halves Inter-item reliability - Overall correlation between items in scale Inter-rater - Checking for agreements between multiple raters or judges