Connection between Reliability and Validity
Reliability is needed for Validity, but Validity it not needed for reliability
Scenarios:
1. Reliable, Not Valid
2. Low Validity, Low Reliability
3. Not Reliable, Not Valid
4. Both Reliable and Valid
Face Validity
Occurs when a measure appears to be a reasonable measure of some trait or when the test takers and the test user believes the test is accurately measuring what is intended to and they believe the test results will be used in a useful and appropriate manner.
The actual test or measure must reasonably assess the construct that you are interested in understanding
*Easiest to Demonstrate
*Lowest Level of Validity
Content Validity
Accuracy with which an item provides information about the construct being measured and/or evaluates how well the test covers all relevant aspects or parts of the construct it aims to measure.
- Relates to both how the question is phrased as well as how the specific responses are
- Assessed through expert analysis of the questions and determining if the content can actually measure the construct of interest
Criterion Validity
Concurrent or Predicitve Nature
Can either measure what is going on currently or provide a prediction at what will happen in the future.
- It is how well a measure aligns with predeteremined criterion
- It is form of validity in which a psychological measure is able to predict some furture behavior or is meaning fully related to some measure.
- Concurrent Validity
- Predicitve Validity
Concurrent Validity
Occurs when a criterion referenced test is accurately providing information on a person’s current state, trait, or behavior
Ex…Having a measure of self-esteem relate to a measure of depression
Predictive Validity
Occurs when a criterion-referenced test accurately predicts a test takers future state, trait, or behavior
Ex…Does you measure of math GRE predict your ability in future math course during gradaute school
Construct Validity
Related to the attibute being Measured
An examination of how accurately and usefully a test measures the specified construct(s). When the measure being used accurately assesses some hypothetical construct, refers to when the construct itself is valid; refers to whether the operational definition used for independent and dependent variables are valid.
- Discriminant Validity
- Convergent Validity
If you have evidence for both discriminant and convergent validity then you can say that you have evidence for constuct validity
Discriminant Validity
Determines if a test that is supposed to discriminate between two types of people or two types of situations actually do so.
*The degree to which concepts that should be not related theoretically are in fact not interrelated in reality (or correlated with each other)
Convergent Validity
Occurs when there is a theoretical prediction that two measures will be related
- The degree to which concepts that should be related theoretically are interrelated in reality (or correlated with each other)
Multitrait-MultiMethod Matrix (MTMM)
Metric table of correlations arranged to facilitate the interpretation of the association of construct validity by
assessing convergent and and discriminant validity
- To claim your measures have construct validity you have to demonstrate both convergent and discriminant validity.
Assumes that you measure each of several concepts (called traits) by each several methods (e.g. paper-and-pencil test, direct observation, a performace measure)
Reliability Diagonal (Monotrait-Monomethod)
Tells you the correlation of reliability for each of the traits with the methods described.
- Coefficients in this diagonal should be consistently the highest in the matrix*
Validity Diagonal (Monotrait-Monomethod)
Correlations between measure of the same trait measures using different methods. Two measures are of the same trait or concept, we would expect them to be strongly correlated.
Coefficients in this diagonal should be significantly different from zero and high enough to warrant further investigation
Heterotrait-Monomethod Triangles
The correlation among measures that share the same method of measurements.
If they are high, it is due to having a stong methods factor
- Validity coefficient should be higher than all coefficients in the heterotrait-method triangles*
Heterotrait-Heteromethod Traingles
Correlations that differ in both trait and method.
- We expect these to be the lowest in the matrix*
Mononmethod Blocks
Consist of correlations that share the same methods
Heteromethod Blocks
Consists of correlations that do not share the same methods.
There are (K(K-1)/2, so it there are 3 methods then, (3(3-1)/2=3 blocks
- Validity coefficient should be higher than values lying in its column and row in the same heteromethod block.*
Advantages of MTMM
1.Possible in one matrix examine both convergent and discriminant validity
2. Provide rigorous framework for assessing contruct validity
Disadvantages of MTMM
- In terms of praticallity, it is not feasible to make methods an explicit part of research
- Judgemental nature of the MTMM and the inability to have a single measure to access construct validity makes it possible for different researchers to come to different conclusions.
