what do generalisability, precision and realism mean
generalisabilty - the Results can apply to other people or situations
precision - Control of factors that were not studied
realism - Context of study is like context of use
what is quantitative evaluation?
- require a hypothesis
- Quantitative observations are made using scientific tools and measurements.
- The results can be measured or counted, and any other person trying to quantitatively assess the same situation should end up with the same results.
- numeric data!
- questionnaires using likert scales
what is qualitative evaluation?
- start with an open-ended question
- we study aspects that cannot necessarily be counted
- consider the interplay among factors
- can lead to a hypothesis or research question for further studies
- interviews, focus groups, observations (people related, ethnographical)
- descriptive data
what is the motivation for a qualitative study?
- when the research questions are open ended
- when we want to see the big picture
- when its inappropriate to abstract technology away from its context of use
- want to incorporate people’s perspectives rather than just performance
what are a field studies pros and cons
high realism validity because we’re in the context
low generalizability because the results may vary in other situations
low precision because we can’t measure all factors that may affect the results
what are differences between field trails/experiments and studies?
both real world setting
trials/experiments have recruite participants and semi-defined tasks. gives rich first hand info with focus, but potential bias of participants
studies have spontaneous participants and activities
there may be a moral issue with particpants being unaware that they are being studies. rich observation of spontaneous activity but stronger reliance on interpretation
you can combine them
pros and cons of diary studies?
+ diverse long-term accounts for real life experience
+ feasible with large numbers of participants
- high work load for participant
- risk of bad data quality
how should a questionnaire be designed?
- short as possible
- consider the order of questions, it may bias
- avoid biasing with loaded terms like “fun”
- avoid intrusive questions. age range > age
pros and cons of questionnaires?
+ good for large scale
+ can be conducted online
+ quick to aggregate and analyse response
+ mix between qualit and quanti measures
- potential misinterpretation of questions
- not suitable for in-depth questions
- no follow up questions to clarify
pros and cons of interviews?
participant
- seeks a positive experience with the interviewer, avoids negativity
interviewer
- asks biasing questions
- dominates the interview
- showing agreement of disagreement
both parties have pre-assumptions about the study and the participant’s experience
can elaborate on questions
how should an interview be designed?
- avoid leading questions such as “was it easy?” instead, say “how was it?”
- use the same words as the participant when following up
- reverse triangle, start with general questions then specific follow up questions after
what is the grounded theory approach?
a common method for analysing qualitative or quant data
highly rigorous method that follows a very particular strategy
ITS NOT A THEORY, ITS A METHOD FOR THEORY BUILDING
- iterative approach
- collect data and analyse it right away
- based on the data you begin to formulate a theory
- you continue to collect and analyse more data to validate, refine and expand your theory, test the limit of it and adapt your data collection methods
- use theoretical sampling, which means that the researcher is deliberately choosing where to collect data
- repeat the iterative data collection & analysis until saturation
- saturation: each new data item can be fitted within the existing theory without requiring the theory to be modified
- theory is complete!
what are cons of grounded theory
- its complex, training and practice is required
- time consuming data collection and analysis
- subjective, choice of theoretical sampling, choice of coding, choice of saturation point
- lack of repeatability
- lack of generalisability
what is the data coding in grounded theory?
a data analsis strategy
helps to break down and categorise the data into a standard format
- open coding phase:
identifying concepts and joining them
- axial coding phase:
indentifying interrelationships of categories
- selective coding phase:
indentifying a core cateogry and defining a high level narative around it
cons of qualititative studies
lack of reproducibility
less precision
less generalisable
but highly realistic
potential bias of research
-
Design7
-
Heuristics and Modelling9
-
Fitts' Law7
-
Models + Linear Regression9
-
Experimental Design14
-
Data Visualisation8
-
ANOVA12
-
Quantifying Errors4
-
Non-Parametrics4
-
Qualitative Evaluation15
-
Cognitive Dimensions15
-
Intelligent UIs14
-
Intelligent UIs 28
-
UbiComp8
-
CSCW4
-
Social Networks & Crowdsourcing11
