1
Q
What is secondary data analysis?
A
A type of research that analyzes data collected by others
2
Q
What is secondary data?
A
Data collected and recorded by someone else before and for a purpose other than the current project
3
Q
What is a systematic review?
A
A review of the evidence on a research question that uses systematic and explicit methods to identify, select, and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review
4
Q
What is meta-analysis?
A
- Quantitative techniques to summarize or integrate findings from a body of literature
- Statistical analysis that combines the results of multiple quantitative studies
- Uses the results of individual research projects on the same topic
5
Q
What does PRISMA stand for?
A
Preferred Reporting Items of Systematic Reviews and Meta-Analysis
6
Q
What is PRISMA?
A
- useful research tool that helps to declare or report on how you choose the articles for your review
- It consists of a 27 item checklist and a 4 phase flow diagram
- Identification, screening, eligibility and included
7
Q
What are the advantages of secondary data analysis?
A
- saves time and effort in research instrument development
- does not require a pilot study
- may include a standardized research tool with proven validity
- may focus on the topic that you would not ask about
8
Q
What are the disadvantages of secondary data analysis?
A
- requires review of the original instrument
- may not have all the questions you need
- questions may be in a format that is inappropriate for your study
- contextual info (protocols, showcards, etc.) may be missing
9
Q
What is content analysis (general usage)?
A
- Refers to research techniques for analyzing the content of written, spoken, or pictorial communication
- The analysis may relate the occurrence of coded content to other factors (e.g characteristics of the producer or effects on the receiver)
10
Q
What is content analysis (specific usage)?
A
- refers to quantitative technique that attempts to quantify the meaning of a communication content
- critical to answering the classic question: who is saying what, to whom, why, how, and with what effect?
11
Q
What is the process in developing the coding frame?
A
- read through all the responses
- establish categories from the responses
- can every response be assigned to a category? If not then refine the categories by merging similar ones and deleting redundant ones. If yes, then the final categories emerge
- cross-category analysis - concentrate upon the relationship between the categories
- the big picture - explain the contents of each category in relation to the others
- present your (time and context-bound) theory
12
Q
How long does it take to develop the coding frame?
A
Depends on the amount of knowledge you retrieve from the original data analysis
13
Q
What are the types of observation?
A
- participant vs non-participant
- standardized vs non-standardized
- covert vs overt
- direct vs indirect
- natural vs contrived
- personal vs mechanical
14
Q
Explain participant vs non-participant observational research
A
Participant
- as observers you are part of the observation, actively manipulating the situation you are observing
Non-participant
- you stay at a distance and don’t manipulate anything
15
Q
Explain standardized vs non-standardized observational research
A
Standardized
- very detailed, actual behavioural pattern described
Non-standardized
- you don’t have coding schemes, observational schedule, or don’t know what you are going to observe and try to notice all aspects that you can and believe might be relevant with the aim of your research question
16
Q
Explain covert vs overt observational research
A
Covert
- when you as a researcher behave like regular customers who just watch and observe
Overt
- you don’t hide yourself and declare yourself as a researcher
17
Q
Explain direct vs indirect observational research
A
Direct
- e.g. observing people at bus stop
Indirect
- analyse the content of the direct process and based on that you might discover something new - you look at the outcomes of the processes and not at the process as such
18
Q
Explain natural vs contrived observational research
A
Natural
- natural environment
Contrived
- typically done in a lab
19
Q
Explain personal vs mechanical observational research
A
Personal
- e.g. your own limits
Mechanical
- e.g. audio recordings, video recordings
- in case of mechanical you don’t use an instrument because it depends on what the machine can pick up
20
Q
Can you observe attitudes?
A
No, because it’s something that happens in our heads, but you can observe the behavioural patterns
21
Q
What is an observation plan/schedule?
A
- can be a simple list of things to look for in a particular situation
- can be far more complex; a minute-by-minute count of events such as verbal interactions between subjects
22
Q
What goes into qualitative research designs?
A
In-depth interviews
- one to one communication
Focus groups
- researcher and group
- shows the group dynamic and arguments and counterarguments
Semi-structured interviews
- some questions must be asked in a standardized way
- intention is not to describe the sample or group but to detect issue on individual level or make a social diagnosis
- one to one communication
23
Q
What are projective techniques?
A
indirect interview methods that allow respondents to project their ideas, beliefs, and feelings onto a third party or into a task situation
24
Q
Describe projective techniques
A
- the researcher sets up a situation for the respondents and ask them to express their own views, or to complete/interpret some ambiguous situations presented to them
- involves situations in which participants are placed in simulated activities in the hope that they will reveal things about themselves that they might not reveal under direct questioning
25
When are projective techniques generally used?
when language barriers, respondents illiteracy, social barrier or psychological barriers create difficulties
26
Give the types of projective techniques and examples of each
Association techniques
- word association
Completion techniques
- sentence completion
- story completion
Construction techniques
- cartoon test
- collage
Expressive techniques
- role playing
-third person technique
- brand party
- obituary
27
What is a word association task?
- it records the first thought that comes to a respondent in response to a stimulus
- respondents are presented with a list of words, one at a time and asked to respond to each with the first word that comes to mind
28
What is the sentence completion test?
Respondents are asked to complete a series of incomplete sentence, often related or neutral to the topic of interest
29
What is the cartoon test?
Researcher shows an ambiguous picture and the respondent tells about the story
30
What is a collage?
- a pattern (larger picture, story etc.) created by sticking pictures or materials onto a surface
- respondents assemble images that represent their thoughts and feelings
31
What are the expressive techniques?
Role playing
- it's important to make them stop afterwards because participants might enjoy their role
Third person technique
Brand party
- they need to explain if a brand had a party: who would be there, what kind of music, who is not invited etc.
Obituary
- give the task that a certain brand died and talk about its life and its failures and successes
32
What are the guidelines for using projective techniques?
- should be used because the required info cannot be obtained accurately by direct methods
- should be used for exploratory research to gain initial insights and understanding
- because of the complexity, they should not be used naively
33
What are the 9 ways to evaluate questions?
- desk based evals
- expert panels
- respondent debriefing
- analysis of existing data
- split-ballot experiment
- behavioural coding
- interviewer rating
- standard field pilots
- cognitive testing
34
What are desk based evaluations and why might they be helpful?
The use of textbook etc. as reference to check if your questions follow the rules etc
Actively try to find and solve problems with your questions
May be helpful to spot mistakes that you didn't see before
35
What are expert panels?
Bringing experts in to review your work/questionnaire and to get their inputW
36
What is the risk with expert panels?
It may make the researcher inclined to prepare the survey for the experts, as opposed to for the general public
More likely to use jargon etc. rather than layman's terms
37
What is respondent debriefing?
Asking respondents to provide feedback on how it was to take the survey
38
What is analysis of existing data and what are the benefits?
- similar to desk based evals
- may use parts of surveys from previous questionnaires
- you will have the data on the performance of the questions which is a big advantage (may for example see that a q didn't provide variability which is problematic)
39
What is a split-ballot experiment and name a disadvantage?
When you take a pool of respondents and then divide them (randomly but evenly)
Then distribute different instrument to each sub-group and see which performs best
Example: see if it is better to use a 5 point or a 10 point scale
Disadvantage: expensive
40
What is behavioural coding?
when you observe the reactions of respondents to the questions (how long it takes, facial reaction etc.)
41
What is interviewer rating?
when you get feedback from interviewer on the performance of questions
42
What are standard field pilots?
when you test the questionnaire on a test group
43
What is cognitive testing and why is it used?
- Process used to study the manner in which target audiences understand, mentally process and respond to the survey
questions
- Used to reveal mental processes used when answering
- Used to understand how respondents interpret the questions
- We want to make qs as easy to understand and answer as possible
- Helps to perfect the question
44
What are the four stages in question response process?
- understand the question
- retrieval from memory
- judgement
- responseH
45
How can cognitive interviewing be used to prevent problems in step 1 of question response process?
- when trying to understand the question, the interviewee may translate the question to their own terms
- this is not desirable as we want everyone to understand our phrasing to prevent people interpreting the question differently
- we can use cognitive interviewing to check for this
- this is generally not a very problematic step
46
How can cognitive interviewing be used to prevent problems in step 2 of question response process?
- can be a problematic step
- to address this, ask additional qs such as "how did you decide your answer"
47
How can cognitive interviewing be used to prevent problems in step 3 of question response process?
- the respondent may not want to answer the question (possibly due to social desirability factor)
- to address the problem, ask "is this distressing for you" etc. during cognitive interviewing
48
How can cognitive interviewing be used to prevent problems in step 4 of question response process?
- for example, in scale question, respondent may be deciding between 7 & 8
- in cog. int. ask why they chose their answer, ask for other examples, ask if the question design provides them with adequate response options
49
What are the cognitive interviewing methods?
- Think aloud interviews
- Probe interviews
50
Describe think aloud interviews.
Requires that respondents verbalize their thought processes as they answer the survey
51
Where are think aloud interviews more or less effective?
More:
- verbal info (not spatial or non-verbal)
- problem oriented questions
- conscious processing of info
Less:
- understanding of terms
It may rely on short term memory recall
52
What are the types of probes used in probe interviewing?
Types of probes:
- scripted and unscripted
- concurrent (between qs) and retrospective (prov at end of survey)
- general and specific
53
Give the pros and cons of scripted vs unscripted probes
Scripted:
- pros: interview is focused around objectives
- cons: might be too rigid
Unscripted/spontaneous:
- pros: allows more flexibility
- cons: no coordination of probing across interviewers
54
Give the pros and cons of concurrent vs retrospective probes
Concurrent:
- pros: question is fresh on the mind
- cons: potential bias (switching of tasks can be distracting)
Restrospective:
- pros: avoids bias and task switching
- cons: long gap between question and probe
55
Name some other types of probes
- frame of reference (what were you thinking about while answering?)
- encourage narrative (to learn what the short response to the question means)
- redundancy (how is the phrase "give advice about X" different from the phrase "talk about X")
- acceptability (asking if respondent was offended by sensitive qs)
56
What are the advantages of think aloud interviews?
- less interviewer bias
- less interviewer training
- open-ended format (unanticipated answers)
- interviewer more free to listen and less potential for bias
57
What are the disadvantages of think aloud interviews?
- need to subject training
- less control
- more difficult to tell if respondent can answer
- potential bias in info processing
58
What are the advantages of verbal probing?
- more control (topics; depth)
- little training of subject
59
What are the disadvantages of verbal probing?
- greater risk of reactivity
- potential for bias
- need for interviewer training
60
What are the possible visual effects in scale development?
Horizontal x vertical
- horizontal usually best, never use diagonal writing
One column x two or more columns
- avoid more columns as can be confusing as they may read the info in rows and mix it up
Stand alone x matrix
- matrix better on paper but not on screen
- matrix hard if not one word items
- matrices can be bigger - can frustrate respondents
Status bar, sections, return option
- status bar can be inaccurate if there's skip questions or questions that require more info -demotivating for respondents
- keep whole survey in one section for self monitored status
Reminders
- 2 reminders can make response rate double
- >2 reminders unhelpful
61
What are ways in which you can improve scale questions?
- provide a meaningful scale
- use a balanced scale (but you can rescale it)
- mind the meaning of the mid category (don't use not sure)
- consider a "don't know" response (if you don't, people may choose mid option and skew your data)
62
What should you take into account when trying to provide a meaningful scale?
Low vs high number of categories
- usually short is best for respondents, long best for researchers (if they need to do factor analysis)
- sometimes long is better if people don't want to fall into the extreme categories
Odd vs even number of categories
- odd provides mid category which many may choose as an "out" or safe option
All categories vs top boxes (outer categories) labeled
- don't label all if its a long scale
- labels like "somewhat likely" might have different interpretations based on culture etc
63
What is gamification?
The process of taking game design features and incorporating them into learning activities or research
64
What are the effects of gamification?
Higher attractiveness
- more cognitively demanding as you have to interpret the pictures; not recommended
Colours
- different perceptions of colours can cause problems
Context and images
- may affect construal of category
- powerful contextual stimuli
65
How does the order in which the alternatives are listed affect the distribution of replies?
- if you start with positive options, people will answer more positively
- e.g. poor to excellent or the other way around
66
What effect can fatigue have on responses?
- surveys that took more than 17.5 minutes led to predicted completion rates of less than 70%
- surveys with more than 30 screens/questions are predicted to exceed acceptable dropout rates
- other reports state 55 clicks as a threshold level
67
What are the types of constructs?
- single item (single pointer) (can be risky to rely on just one question)
- composite indicator/measure
- observed characteristics
68
How do you operationalize?
- conceptualise the concepts being worked with
- specify the method of measurement
- define explanatory models
69
What are the possible negative outcomes if operationalisation is done incorrectly?
Completely wrong operational definition
- we are measuring something else than what we expect
Imprecise operational definition
- real characteristics and indicators used are different but there's some overlap
Reductionist operational definition
- all the items you have are part of the construct but they don't cover the construct in its whole
Extensive operational definition
- cover too much; what you aim to cover is contained in it but it also includes unrelated info
70
What are the steps in the scale development process?
- determine clearly what it is you want to measure
- generate an item pool (through research or focus group)
- determine the format for measurement
- have the initial item pool reviewed by experts (face validity etc)
- consider inclusion of validation items
- apply items to a development scale (must also filter out items)
- evaluate the items: the heart of scale development (can run an exploratory factor analysis)
- optimize scale length
71
What are the three stages of the scale development process?
- item generation and selection phase
- scale refinement phase
- scale validation phase
72
Explain the item generation and selection phase
- contains item generation and selection
- qualitative inquiry stage
- content and face validity assessment
73
Explain the scale refinement phase
Pilot test sample/stage
- item analysis
- exploratory factor analysis
- consistency and reliability assessment
Purification stage/callibration sample
- confirmatory factor analysis
- unidimensionality and reliability assessment
- convergent and discriminant validity assessment
74
Explain the scale validation phase
Validation sample/stage
- replication of confirmatory factor analysis
- unidimensionality and reliability assessment
- convergent and discriminant validity assessment
- nomological validity assessment
75
What are some questions you may ask during the planning of scale development?
- how many items are necessary
- which response scale is appropriate
- how to score the test
- which psychometric model is appropriate
- what item evaluation process is suitable
- how to administer the test
76
What are the 3 measurement models?
- unidimensional (all items in matrix building one single construct)
- oblique 3-factor model (3 dimensions that are interrelated; confirmatory factor analysis will help to see if this is the right model, and if the items are measuring what they are meant to measure)
- hierarchical model (see if there's other roots that are deeper behind this model)
77
What are the 3 critical interactions in operationalization?
- client/consultant - researcher
- research objective - items in the instrument
- instrument - respondent (about understanding of questions and whether they are appropriate for the target group)
78
What is reliability?
- the degree to which a variable has nearly the same value when measured several times
- precision: absence of random error
- error: discrepancy between the observed result and the true value
79
What is validity?
- the degree to which a variable actually represents what it is supposed to represent
- accuracy: getting the correct result
- bias: systematic error; estimates are moved in one direction
80
What is the best way to assess reliability?
Comparison among repeated measures
81
What is the best way to assess validity?
Comparison with a reference standard
82
What is reliability's value to a study?
Increases power to detect random effects
83
What is validity's value to a study?
Increases validity of conclusions
84
What is reliability threatened by?
Random error (variance)
85
What is validity threatened by?
Systematic error (bias)
86
Describe reliability vs validity on a dartboard diagram
- reliability is when all the points are close to together, regardless of their positioning on the board
- validity is when the points are when the points are at the bullseye
87
What are some areas across which there must be consistency in order to be reliable
- items/subscales/total scales (internal consistency)
- data collectors (inter-rater reliability or inter-observed agreement)
- time (test-retest reliability)
88
Define internal consistency
Extent to which the items on an instrument adequately and randomly sample one construct
89
How do you assess internal consistency?
If the instrument adequately and randomly samples one construct, and if it were divided into two equal parts, both parts should correlate strongly
90
What is the metric for internal consistency?
Cronbachs' coefficient alpha
- the average split-half correlation based on all possible divisions of an instrument into two parts
91
How should one interpret Cronbach's alpha?
a ≥ 0.7 - adequate for measures under development
a ≥ 0.8 - adequate for basic research
a ≥ 0.9 - adequate for measures on which decisions are based
92
Define inter-rater reliability/inter-observer agreement
Extent to which the instrument measures the same construct regardless of who collects the data
93
How do you assess inter-rater reliability/inter-observer agreement?
If the same construct were observed by two data collectors, their ratings should be almost identical
94
What is the metric for inter-rater reliability?
Cohen's kappa
- percentage of agreement between two data collectors
95
How do you interpret Cohen's kappa?
≥ 90% good
≥ 80% acceptable
< 80% problematic
96
Define test-retest reliability
Extent to which the instrument yields consistent results at two points in time
97
How do you assess test-retest reliability?
- administer the measure at two points in time
- the time interval is set so that no improvement is expected to occur between the first and second administrations
98
What is the metric for test-retest reliability?
Pearson correlation coefficient
Expressed as a correlation between pairs of scores from the same schools obtained at the measurement administrations
99
How do you interpret the metric for test-retest reliability?
r ≥ 0.7 acceptable
100
What are the two types of validity and their subgroups?
Theory-related validity
- face validity
- content validity
- construct validity
Criterion-related validity
- concurrent validity
- predictive validity
101
What is face validity?
- looks to measure what it is supposed to measure
- looks at items for appropriateness (client, sample respondents)
- least scientific validity measure
102
Define content validity
Extent to which the items on an instrument relate to the construct of interest e.g. student behavious
103
How do you assess content validity?
Expert judgement if items measure content theoretically or empirically linked to the construct
104
What is the metric for content validity and how do you interpret it?
Expressed as a percentage of expert agreement
≥ 80% agreement desirable
105
Define construct validity
Extent to which the instrument measures what it is supposed to measure (e.g. the theorized construct "student behaviour")
106
How do you assess construct validity?
- factor analyses yielding information about the instrument's dimensions (e.g. aspects of "student behaviour)
- correlations between constructs hypothesized to impact each other (e.g "student behaviour" and "student reading achievement")
107
What is the metric for construct validity and how do you interpret it?
Statistical model fit indices (e.g. chi square, PCA, CFA)
Interpret it through statistical significance
108
Define criterion-related validity
Extent to which the instrument correlates with another instrument measuring a similar aspect of the construct
109
How do you assess criterion-related validity?
- concurrent validity: compare data from concurrently administered measures for agreement
- predictive validity: compare data from subsequently administered measures for predictive accuracy
110
What is the metric used when assessing criterion-related validity and how is it interpreted?
Expressed as a correlation between two measurements
Moderate to high correlations are desirable
Concurrent validity: very high correlations might indicated redundancy of measures
111
How might you modify a scales?
- wording
- length
- dimensions
- multiple modifications
Research Tools
flashcards
Decks in class (2)
# Cards
