1
Q
The majority of survey research is conducted to
A
answer descriptive or predictive
2
Q
when might a survey answer a causal question
A
This researcher might conduct an experiment to compare the therapies* effect on scores from a depression questionnaire, such as the Beck Depression Inventory
Another example of experiments that involve surveys is in testing the effect of new products on consumer behaviors and attitudes.
3
Q
Descriptive Research Question:
A
a search question that asks about the presence of behavior, how frequently it is exhibited, or whether there is a relationship Between different behaviors
4
Q
Predictive Research Question:
A
a research question that asks if one behavior an be predicted from another behavior to allow predictions of future behavior
5
Q
Causal Research Question:
A
a research question that asks what causes specific behaviors to occur
6
Q
Psychometrics:
A
area of psychological research that involves the development, validation, and refinement of surveys and tests for measuring psychological constructs
7
Q
There are two good sources for standardized questionnaires that may contain the type of survey you are looking for
A
- One is called the Health and Psychosocial Instruments (HAPI) database.
- The other source is called the Mental Measurements Yearbook (also available through EBSCO if you have access to that database).
8
Q
why is it important to write questions carefully, and what are the steps through which you do so?
A
- Hence, it is important to write questions carefully so as to reduce bias in responses.
- Consider whether participants are able to recall the behaviors you are asking about
- Consider how your participants perceive the question
- Avoid double-barreled questions that are actually two questions in one
- In addition, avoid loaded questions that assume too much
9
Q
types of survey response scales
A
open ended and close ended
10
Q
writing close ended question responses
A
- Consider how your participants view the response choices Vou provide
- Will the response categories be confusing to your participants or bias the way they respond?
- Consider the number of response categories provided to the participants.
- more response options allow for more variable responses, which can increase the validity of the scores
- Also, consider that the use of ordinal response scales, such as those described above, may not provide choices that match the participants* exact behavior
- Thus, an open-ended ratio scale that allows participants to report the exact number of hours of TV they watch per day may be accurate.
- Include equal numbers of positive and negative response categories.
11
Q
construct validity
A
- Construct Validity: indicates that a survey measures the behavior it is designed to measure
- It is very important that they write survey items carefully to ensure that they address the behaviors that they intend to measure
12
Q
nonresponse as a type of coverage error
A
- nonresponse Error: a sampling error that occurs when individuals chosen for the sample do not respond to the survey, biasing the sample
- Nonresponse errors are a particular type of coverage error that occurs when the sample completing the survey does not represent the entire population that the researchers wish to use to generalize the scores.
- Coverage errors reduce the external validity of your survey
13
Q
criterion related validity
A
- determining the validity of the scores of a survey by examining the relationship between the survey scores and other established measures of the behavior of interest
- Often, a researcher tests the criterion-related validity of a survey to determine if the survey can predict other behaviors.
14
Q
social desirability bias
A
bias created in survey responses from respondents’ desi re to be viewed more favorably by others. Typically resulting in overreporting of positive” behaviors and underreporting of negative” behaviors
15
Q
how do researchers deal with social desirability bias
A
- include a measure of this bias such as the Marlowe-Crowne social desirability scale.
- These measures include items designed to test a respondent’s level of social desirability in order to allow a measurement of the bias within the sample
16
Q
how do Unreliable surveys reduce the validity of the scores of a study
A
- If participants respond to the questions in a different way at different times or respond in different ways :o different sets of similar questions in the survey* the researcher is unable to draw accurate conclusions from the survey responses.
17
Q
length of survey and reliability
A
- One way to increase the reliability of scores on a survey is to use a longer survey
- Shorter surveys tend to be less reliable, because an unusual response on a single item can skew the results quite a bit
18
Q
test retest reliability
A
- Test-Retest Reliability; indicates that the scores on a survey wilI be similar when participants complete the survey more than once; means that the scores on a measure are consistent over time
*
19
Q
why is test-retest problematic
A
Getting the same participants to take a questionnaire twice can be problematic for two reasons:
participants may not come back the second time, and the sample size is then reduced-an issue known as attrition/mortality
having participants take the same questionnaire more than once can change their results through testing effects-these occur when taking a test or questionnaire once affects future scores on the scale
In addition, there may be occasions when researchers expect to find changes in scores over time based on changes in personality or attitudes as individuals develop or events that individuals experience (e.g., starting a new job, getting married) between testings.
20
Q
internal consistency
A
Internal consistency of scores indicates how similar scores on the different items of a survey are compared to one another. This is another means of evaluating the reliability of the scores on a survey
21
Q
split half reliability
A
- The items on the questionnaire are split into two halves or sections and the Relationship between the scores on the two sets of items is tested.
- This is called split-half reliability- If a strong, positive relationship exists between the scores on the two sets of items, then the questionnaire has good split-half reliability
- The advantage of split-half is that you can test the reliability with a single testing of a group of participants
22
Q
cronbach’s alpha
A
- Another method of testing the internal consistency of scores is to examine the overall correlation between pairs of items.
- In other words, the relationship between the scores for each pair of terns on a survey is calculated, and then a statistical averaging of these correlations is determined for the whole survey.
- This method is called Cronbach’s alpha (a), which is also the name of the statistical test that is used to calculate the overall correlation
23
Q
correlational studies
A
- Correlational Study: a type of research design that examines the relationships between multiple dependent variables without manipulating any of the variables
- Correlational studies can also suggest the incidence or likelihood of something occurring in the presence or absence of something else.
24
Q
Correlational studies are designed to address
A
descriptive and predictive research questions.
25
predictor and outcome variables
For correlational studies with predictive research questions, the variable that is used for the prediction is called the predictor variable, and the variable being predicted is called the outcome variable
26
The goal of any correlational study is to
examine relationships between two or more measures of nehavior
27
why does correlation not prove causation
This is due to the lack of manipulation of an independent variable in correlational studies and subsequent lack of control of other extraneous variables.
28
Third-Variable problem:
the presence of extraneous factors in a study that affect the dependent variable can decrease the internal validity of the study
29
manipulation of an independent variable in a study increases
the internal validity of the study
30
The manipulation of the independent variable can occur in two ways:
the variable can be manipulated between subjects or within subjects.
31
between subjects
* In between-subjects manipulations, each participant receives only one level of the independent variable.
* participants are typically randomly assigned to the different levels of between-subjects variables'
32
random assignment
* Random assignment of participants to levels allows random distribution of participant differences across the levels, making it less likely that the participants' differences cause (and more likely that the independent variable causes) a difference across groups for the dependent variable being measured
* Random assignment is a means of controlling for participant differences across groups and increases the internal validity of the experiment
33
within subjects
* In within-subjects manipulations, each participant receives all levels of the independent variable.
* Random assignment is important for within-subjects variables in terms of the order in which participants receive the levels of the variable to control for possible order effects of the different levels of the variable.
34
order effects
* Order Effects: occur when the order in which the participants experience conditions in an experiment affects the resu Its of the study
* Ex. If participants receive the hard copy version or the online version of the text whether the participants receive the hard copy version or the online version of the text first.
* Order effects are a particular form of testing effects, with bias occurring because of multiple testings of a participant in a study
35
what is the greatest concern for between subjects experiments
The presence of individual differences across groups is the greatest concern for between-subjects experiments.
36
when might random assignment not be sufficient in between subjects designs?
In I experiments where individual differences are likely to be present (such as the example about background music) and a small sample size is use d, Random assignment may not be sufficient to control for these differences.
37
what shoudl ther esearcher do when random assignment is not sufficient for a between subjects experiment
* the researcher has two additional means of controlling for individual differences
* The first is to manipulate the independent variable as a within subjects variable\* This allows participants to serve as their own controls and individual differences across groups are no longer of concern
* , the second means of control is to use a matched design for between-subjects experiments. In a matched design, participants are matched on a characteristic that may contribute to group differences. Then each participant within a matched set is randomly assigned to different groups.
38
adjusting for between group differences using comparison
* Another way to ensure that participants in the groups are matched on some characteristic is to measure that characteristic (e.g.. Years of education, socioeconomic status, language abilities) during the experiment and then compare the groups in an additional analysis
* This comparison indicates if the groups are similar or different on that characteristic.
* However, if the groups are different, you are then left with the possibility that this difference affected your results. Thus, it you are concerned that a group difference is likely to affect your results, it is best to either use a within-subjects manipulation or match your participants before you assign them to the groups.
39
primary concern for within subjects designs
* The primary concern of within-subjects variables is the order in which the participants receive the different levels of the independent variable.
* Because the participants receive all levels of the independent variable, it is possible that the order of the levels can affect the dependent variable measure
40
To solve the problem of order effects for within-subjects manipulations
* TO solve the problem of order effects for within-subjects manipulations, a researcher typically counterbalances the order of the levels within the study
* This means that different groups of participants receive different orders of the levels of the independent variable
* The counterbalancing for a simple factorial experiment with four conditions is very complicated. To simplify the counterbalancing in such a case, a partial counterbalancing technique, called a Latin square, can be used instead.
* In a Latin square, the number of orders used in the experiment is equal to the number of conditions in the design, and each condition is in each ordinal position (i.e., first, second, third, fourth) exactly once. Latin squares can be useful for within-subjects designs where independent variables have the same number of levels.
41
factorial designs
many experiments conducted by researchers are factorial designs, meaning they contain more than one independent variable
42
advantage of factorial
The primary advantages of a factorial design over simpler experiments is that a researcher can be more efficient in testing the effects of multiple independent variables in one experiment and can also examine the effects of the interactions of those independent variables on the dependent variable
43
main effect (factorials)
* In factorial designs, the comparison of means for the levels of one independent variable is the test of the main effect of that independent variable
* In other words, the main effects indicate the independent causal effect of each independent variable in a factorial design.
* The main effect is one type of effect tested in an analysis of variance (ANOVA), Which is the type of statistical analysis used most often for a factorial experiment
44
interaction effects, ANOVA
The other type of effect tested in an ANOVA is an interaction effect he interaction effect tests the effect of one independent variable for each level of another independent variable to determine how the independent variables interact to affect the dependent variable
45
simple effects tests
Simple Effects Tests: statistical tests conducted to characterize an interaction effect when one is found in an ANOVA
46
What is an interaction between independent variables?
An interaction can occur between independent variables such that the effect of one independent variable depends on which level of the other independent variable one is looking at. For example, an independent variable can show a difference between levels for Level 1 of another independent variable but show no difference between levels for Level 2 of the other independent variable.
47
quasi experiments
* a researcher often conducts a quasi-experiment, w here groups of participants are compared on a behavior of interest, as in an experiment, but the quasi-independent variable is not manipulated, which leaves the study open to other possible explanations o£ he results.
* In other words, groups that have not been randomly assigned can be compared in a quasi-experiment, but one cannot gain strong causal information, because of the lack of random assignment
* 3uasi Experiment: a type of research design where a comparison is made, as I n an experiment, but no random assignment of participants to groups occurs
48
when is behavior measured in pretest post test
twice: once before a treatment or condition is implemented (the pretest) and once after it has been implemented (the posttest).
49
why is pretest posttest quasi experimental
* This design is a quasi-experiment because there is no random assignment of participants to the treatment. In this case, all the participants get the treatment. The researcher compares the scores from the pretest and the posttest, looking for a change based on the treatment or condition occurring between the two measurements
* However, if a change occurs in this design, the researcher cannot automatically conclud e lat the treatment or condition caused the change because other factors (besides the treatment or condition) may have occurred as well between the two measurements
50
how can researchers deal with problems of pretest posttest
* Researchers can attempt to deal with some of the alternative explanations in pretest- posttest designs by including a control group\*.
* If participants are randomly assigned to the control group and the treatment group, the study becomes an experiment.
51
pretest posttest with nonequivalent groups
group differences that might account for the results are not controlled by random assignment to groups
In this design, the researcher considers the difference between the pretest and posttest scores across the control and experimental groups
52
regression to the mean
Regression toward the mean-a high score achieved at posttest may be an extreme score in some cases, and with additional testing, these students may score closer to their original mean (which is typically lower than the norm)
53
regression to the mean in pretest posttest
Pretest-posttest designs are susceptible to bias due to regression toward the mean because one extreme score (high or low) can skew the change in scores between the pretests and posttests
One major drawback to the pretest-posttest design is that the participants are tested vice on the same behavior.
This can cause a source of bias known as testing effects. Testing effects occur when participants are tested multiple times and each subsequent test s affected by the previous tests
54
solomon four group design
One method of evaluating testing effects in pretest-posttest studies is the use of a Solomon four-group design
In the Solomon four-group design, he pretest-posttest design with nonequivalent groups is used.
However, two sets of each roup type are included: one set that takes the pretest and posttest as illustrated in Figure 12.2 and one set that takes only the posttest to allow comparison of the two sets of groups
Comparison across the group sets Indicates if testing effects occur For example,
Solomon Four-Croup Design: pretest posttest design with two sets of nonequivalent groups, one set that takes the pretest and posttest and one set that takes only the posttest
55
time series designs
* For some measures of behavior or attitudes, the measures may fluctuate a good deal from week to week or month to month. In these cases, a single week or month to month. In these cases, a single e true nature of the behavior. The pretest can occur at a time when scores happen to be particularly high or particularly low. Which can bias the change in scores from pretest to posttest.
* Time series designs are special pretest-posttest designs that account for these fluctuations t\>y measuring the behavior or attitude multiple times before the treatment and multiple times after the treatment over the same time period.
* the patterns of scores before and after the treatment compared to determine if a difference has occurred over time.
56
difference between regular pre test post test and time series designs
The patterns of scores are compared in a time series design, rather than two scores as in the simple pretest-posttest design.
57
interrupted time series designs
in Interrupted time series design, the treatment is an independent event that the researchers have no control over:
events such as a war, passage of a new law, or other historical events are considered "treatments" in an interrupted time series design in that patterns of scores are compared
Archival data are used as observations in the study
58
noninterrupted time series designs
* designs with a researcher-implemented treatment are called non interrupted time series designs.
* The behavior of interest is measured over time both before and after a treatment has been introduced
59
limitations of time series designs
As with pretest-posttest designs, limitations of time series designs exist due to possible extraneous factors that can affect a score pattern change other than the treatment or event of interest
60
sources of bias in quasi experiments
history effects
maturation
attrition and mortality
61
history effects
events that occur during the course of a study to all participants or to individual participants that can result in bias
62
history effects most likely to occur in
pretest-posttest designs
63
The best way to minimize effects that occur over time in a pretest-posttest design is
to include a control group that takes the pretest and posttest at the same time as the experimental group but does not receive the treatment given to the experimental group
64
maturation and best way to minimize it
Maturation occurs when participants exhibit natural changes over the course of a study
He best way to minimize f fects of maturation is to include a control group that does not receive the treatment to allow a compariso n )f groups that have similar experiences except for the treatment.
65
attrition
Attrition (also called mortality) occurs when participants drop out of a study after completing only part of the study
66
developmental designs
There are three main types of developmental designs ) that treat the factor of age in a different way: longitudinal designs, cross-sectional designs, and cohort-sequential designs
67
longitudinal design
Longitudinal designs treat age as a within-subjects variable
Participants are tested at different ages in their lives
68
how is longitudinal design considered within subjects?
* Participants are tested at different ages in their lives
* 'This is how age is considered a within-subjects variable in the longitudinal design, although age is not an independent variable in this case because age cannot be manipulated.
69
disadvantages of longitudinal designs
* One of the disadvantages of the longitudinal design: It takes time to wait for the participants to age.
* Another disadvantage is that attrition/ mortality (i.e., some participants do not complete the study) may occur over time as the study is conducted
* A third disadvantage is that testing effects can occur with multiple testings of the same participants.
* In other words, being tested on the measures early in the study can affect the later testings, as participants scores can show effects of practice or fatigue
70
cross sectional designs
* Developmental designs that treat age as a between-subjects variable are called cross-sectional designs.
* These designs compare different age groups of participants, where each participant contributes data for only one age group
* The cross-sectional design solves many of the problems that can occur with longitudinal designs.
71
advantages of cross sectional designs
* Each participant is tested only once, which reduces the chance of attrition.
* In addition, because the researcher collects data from all age groups at he same time, the study can be completed more quickly with a cross-sectional design.
72
disadvantages of cross sectional
* However, there are some drawbacks to the cross-sectional design that are more problematic :han in longitudinal designs.
* For example, if the age groups being tested contain participants om different generations, generation effects (also called cohort effects) might affect the results
* These effects occur when the experiences of one generation (e.g., growing up with or without computers) are very different from those of another generation and affect the way the participants complete the task or measure in the study
73
cohort sequential design
* The third type of developmental design combines elements or tne longitudinal and cross sectional designs.
* In a cohort-sequential design, age is treated as both a between-subjects and within-subjects factor.
* Cohort-sequential designs begin with separate samples of different they develop to allow participants to be tested at multiple ages, as in a longitudinal design.
74
small n designs
* These designs are sometimes called single-subject or single-case designs, but they often include more than one participant: thus, small n is a better descriptor of this design type
* The goal of a small-n study is to understand an individual's behavior, either to better describe the behavior as it occurs for many individuals or in order to change that behavior.
75
difference between small n and case study
this design is different from a case study in that the goal of a case study is to explore an individual's behavior when little is known about the behavior.
76
what is typically being tested in small n designs
In a small-n design, a researcher is typically testing a theory about how a behavior works for most individuals or testing a treatment for a problematic behavior of an individual or group of individuals this is accomplished through repeated measurement of behavior )
77
small n baseline designs
baseline designs, where the repeated measurement of the baseline behavior of one or a few participants is compared with their behavior during the implementation of a treatment
Involve experimental comparison of baseline Behavior and behavior with a treatment.
The goal is to determine if a treatment creates a desired change in the behavior of interest (often an undesirable behavior), a technique also called behavior modification
A-B-A design or reversal design
78
small n discrete trials designs
discrete trials designs, where one or a few participants complete a large number of trials of a task to describe how performance on the task operates
Discrete Trials Design: a sma I l-n design that involves a large number of trials completed by one or a few individuals and conducted to describe basic behaviors
With the controls used in an experiment and the large amount of data collected under these conditions, discrete trials designs allow for good tests of causal relationships for behaviors where there are few individual differences.
Thus discrete trials designs tend to achieve stable measures of behavior with high internal validity.
Mathematical description of behavior is often a common goal of discrete trials designs
79
advantages of small n designs
* The primary advantage of small-n designs comes from the large number of observations collected from one or a few participants.
* Using a large number of observations reduces the error in the data (seen in the low variability of the scores) and makes it easier for a researcher to detect an effect of an independent variable.
* Using a small number of participants also makes it easier for a researcher to control for extraneous factors that may bias the data.
* This control increases the internal validity of the study
80
disadvantages of small n designs
* The main disadvantage to small-n designs is that the results cannot always be generalized to people outside the study
* This is why they are typically used to study very basic behaviors (eg., sensory processes, learning processes), where the behaviors being measured should be very similar from person to person, and for studies where the goal is to tailor a treatment to a specific person.
* In addition, small-n designs cannot be used to measure many types of behavior.
* Due to the large number of observations collected, carryover effects can occur for tasks that may affect future performance over time.
* In other words, participants experiences in the treatment condition can affect their later behavior in a second baseline condition that the follows the condition treatment
81
A-B-A design or reversal design
* the baseline measure of behavior (designated as Condition A) is first taken
* Then the treatment (designated as Condition b) is implemented, and behavior is measured again, this time with the treatment.
* Finally\* a baseline measure (A) is made again after the treatment has been stopped to determine if the baseline behavior appears once again after the treatment has been stopped
* % A common variant of this procedure is the A-B-A-B design, where the treatment is then implemented a second time to determine if the behavior still changes with implementation of the treatment.
82
Data Analysis in Small-n Designs
* Because there are no group means to present in small-n designs, data are often presented for the individual participants in the study (with no identifying information to protect their confidentiality)
* Inferential statistics are sometimes reported for discrete trials designs for within-subjects variables that are manipulated in these types of designs.
* Inferential statistics can only be used in baseline designs if a large number of observations are collected for each individual.
PSYC 270
flashcards
Decks in class (3)
# Cards
