3.2

Question 1

Basic steps of conducting experiments

Answer 1

1. Define purpose of the experiment
2. Define concise research question
3. Define dependent variable
4. Define independent variable
5. Manipulate independent variable/design treatment
6. Define participants
7. Theory work (research model & hypotheses)
8. Identify control variables
9. Craft experiment design
10. Craft experiment instructions/define incentive
11. Conduct pretest
12. Refine experiment design
13. Conduct experiment
14. Analyze experiment
15. Write your work down

1-12 = work before experiment
13-15 = work after experiment

Question 2

6. Define participants

Answer 2

• Can be in a physical lab, then it will be mostly students
• Can be through an online micro-task platform (Amazon Mturk, Prolific, etc.)
• If it is a field experiment, then it can be employees, customers, a lot of possible options

Question 3

10. Craft experiment instructions/define incentive

Answer 3

• Write a clear and concise introduction for the participants.
• DO NOT write exactly what your aim is (seeing if people rate differently when they are exposed to chatbots). Just give a high-level intro (we are interested in your evaluation of this video).
• Workers were paid 0.13£ per minute upon completion of the task.
• We also implemented attention checks (asking random questions), to check if participants actually pay attention.

Question 4

11. Conduct pretest

Answer 4

• Conduct pre-tests for all the important variables in your experiment
• Independent variable (survey vs. chatbot):
  Do people understand the survey, are questions clear (more on that in the survey session)
  Does the chatbot work?
• Dependent variable (rating):
  Which scale to use (5-point, 7-point, 10-point)?
• Mechanism (interactivity & social presence):
  What are the accepted scales in the literature to measure interactivity & social presence?
• When you are all set and done, launch the experiment.
• Platforms like Amazon Mturk or Prolific will be able to recruit a couple of hundred participants within a few hours.

Question 5

13. Analyze experiment

Answer 5

1. Randomization check
2. Manipulation check
3. Main results / hypothesis test

Question 6

13.1 Analyze experiment - Randomization check

Answer 6

• Probably the most important task is to check if your randomization has worked properly. If not, everything is in danger.
• You do this by testing if your control variables are significantly different between your treatment and control groups with a t-test, for instance

Question 7

13.2 Analyze experiment - Manipulation Check

Answer 7

• Very important: You want to test whether your treatment actually had the desired effect. In our case, you want to see if chatbots actually had higher interactivity and social presence than conventional review forms.
• T-test for the variable between treatment and control group

Question 8

13.4 Analyze experiment - Tests to use

Data normally distributed

Answer 8

• Independent t-test

- Dependent t-test

Question 9

13.4 Analyze experiment - Tests to use

Data not normally distributed

Answer 9

Family of non parametric tests:

• Mann-Whitney U test (instead of independent t-test)
• Wilcoxon Signed Rank Test (instead of dependent t-test)