[0:01]So now that you've learned more about and seen more examples about internal and external validity, you can imagine that the two sometimes are in competition with each other, where focusing on internal validity can sacrifice external validity and vice versa. So if the measurements and research strategies used in a study don't adequately capture the true impact of interventions on outcomes of interest or the relationships between variables, or in other words, they have low internal validity, then any conclusion that you make based on the research is unreliable and unlikely to apply in the real world or the population and have low external validity. Even if results are statistically significant, if you're not measuring or manipulating what you think you are, then your results will not translate into the real world.
[0:50]To gain a high level of internal validity and to be confident that significant results actually show that the independent variable causes the dependent variable, a researcher must eliminate or minimize extraneous variables that could be correlated with the independent variable and the dependent variable, third variables, right? Or confounds that interfere with the ability to conclusively state that changes in the dependent variable were due to the independent variable. So to maximize internal validity, a study must be really tightly controlled, so that no extraneous variables can influence the results. However, controlling a study to this degree may create a research environment that is so artificial and unnatural that the results that you obtain within the study may not occur in the real outside world. Thus, attempts to increase internal validity can reduce external validity. On the other hand, research that attempts to gain a high level of external validity or realism, often creates a research environment that closely resembles the outside world or does field research is performed on site. The risk in this type of research comes from the fact that the real world is often a chaotic jungle of uncontrolled variables, especially in comparison with the highly regulated environment of a controlled study in the lab. Thus, striving for increased external validity can allow extraneous and potentially confounding variables and chaos or noise in your data into a study and thereby threaten the internal validity. So for example, let's say that your hypothesis is that first-year college students who receive an intervention designed to increase their sense of belonging will have better first-year completion rates than students who receive information about study strategies. So we'd have the two conditions, like the belongingness intervention and then the control group who just received information about study strategies and we want to see if there's differences in retention. So, there would be poor internal validity if the intervention did not increase student sense of belonging or the conditions were not randomly assigned. So for example, if the in-person students who were taking classes face-to-face got the belongingness intervention and then the online students got the study strategies. This lack of confidence in the findings within the sample make it really difficult to make generalizations about the effectiveness of the intervention for students outside of the sample or external validity. The study could focus on internal validity by controlling for all the extraneous variables and requiring all the students in the sample to take the same classes, study the same number of hours, eat the same diet, live in the same type of campus housing, and interact with the same number of other students, faculty and staff on campus. While this would make it easier to conclude that the belongingness intervention led to any significant improvements in first-year retention, having such a restricted sample wouldn't really translate very well to all first-year college students. The confines of the study may also have an influence on student behavior that make it difficult to isolate the effects of the intervention. So maybe students drop out because they want to study as much as they want to and they want to take whatever classes they want, right? The study could also focus on external validity by observing the natural relationship between first-year completion rates and belongingness indicators. So, researchers could look at the frequency of social interactions on campus among students, the number of on-campus events attended, the number of memberships to clubs and organizations on campus, and this would provide a more realistic situation that college students experience. But there's no control over extraneous variables that may influence those first-year completion rates, like the quality of social interactions, or the number of classes, or the living arrangements that they have. So in that case, you'd be focusing on external validity, but sacrificing the internal validity and the confidence that what you observed in the study was real. Fortunately, researchers can strike a balance between internal and external validity in research. We don't have to totally sacrifice external validity for internal validity or vice versa. We can have a balanced level of control that's focused on eliminating confounds while maintaining realism. So you can hold potential confounds constant as much as possible without tampering with environmental variables that are essential to realism. So, for instance, in the belongingness study, you could only focus on online students for the belongingness intervention study to control for being online versus face-to-face. And only focus on first-year students who are not transfers to control for the influence of previous college experiences on sense of belonging and first-year completion. And chances are, they would represent your population of interest anyway, and you would not be hurting your external validity. You could also randomly or strategically assign research participants in your sample to conditions, so that those pre-existing individual differences are evenly distributed across conditions of your study. Instead of relying on comparing pre-existing groups in a quasi experiment. So for example, you could randomly assign first-year non-transfer online students to receive a belongingness intervention or a study skills intervention. Not letting them choose, not relying on pre-existing groups, but randomly assigning them. You could also balance internal and external validity by statistically controlling for extraneous variables that you may think could be confounds and those third variables. So you could measure essential environmental variables, measure third variables, measure the perceived novelty of research participation, and record differences in study features, and then look to see with the analysis if those had an effect on your results. So for example, past research suggests that students who have parents who went to college and who had a higher high school GPA tend to feel a stronger sense of belonging in college and are more likely to graduate. As such, you should measure parent and guardian highest level of education to see if they're first gen students and high school GPA in a survey or through institutional data and control for those variables in the data analysis. So this strategy exerts control over potential confounds and third variables to maximize the internal validity while maintaining some realism and giving you that external validity. So now I would love for you to apply your knowledge and take a moment to think about it. So read over this scenario and see if you can answer the questions here on the screen. So take a moment, pause, and then come back to me when you're ready for the answers and my thoughts. All right, so hopefully you took an opportunity to think on your own and think about the answers to these questions. So now I'll share with you what my thoughts are. So, Keith is wanting to see if a yoga program will reduce stress and increase life satisfaction for SciD students. And to do that, he takes a sample of 12 of his SciD students at Marshall University and he measures their stress and life satisfaction with single item measures. Has them do a yoga program for two weeks and then measures their stress and life satisfaction again. So, in this case, the independent variable, the thing that was manipulated, the cause in this relationship, is the yoga program. The dependent variable, the effect, right? The measured outcome of the yoga program. There's two of them, stress and life satisfaction. All right, so now let's look at the type of research strategy being used. So, that would be a quasi experimental pretest posttest. We measure the dependent variable, manipulate the independent variable, and measure the dependent variable again. When we think about the threats to internal validity and how to improve internal validity without sacrificing external validity, I definitely have some thoughts. So one of the first threats that I saw here was poor measurement. We need multi-item empirically validated measures to operationalize complex constructs like stress and life satisfaction. A single item just won't do. If people are responding to their life satisfaction from dissatisfied to very satisfied, you can't be sure that everybody who's responding that question has the same thing in mind. It would be better to have multiple items that tap into what we know theoretically and from past research to be life satisfaction and the same goes for stress. Another issue would be the third variable problem. So we need to measure and control for other variables that are related to the independent variable and dependent variable, in this case, may influence stress and life satisfaction. Like physical health quality, academic social support, and the number of work and study hours per week. So you could measure those and control for them, and you wouldn't be harming the realism of the study, but you would be controlling for some third variables that may influence your results. Another threat to internal validity could be the participants. So, you need to make sure graduate students, who often aim to please, are not taking on the good subject role. They may suspect that Keith, their program director, is expecting the SciD students to be less stressed and more satisfied with life after the yoga program, so they provide overly favorable responses in the posttest survey to kind of give Keith what he wants. Another threat to internal validity could be time. So Keith should make sure that no major program requirements or milestones are due over the course of the study. Those external factors, right? Or events that could manipulate or have an impact on results. Think about it, if comprehensive exams happened during the yoga program and end before the post-test, SciD students are going to be less stressed and happier with life, at least in part due to being done with comps, not necessarily due to the yoga. Keith could also use a Solomon four-groups design to rule out the impact of external events. So you could compare the pretest in one control condition to the posttest, and that would be in controlled condition B, to the posttest in the other control condition who never took the pretest, condition D. And if there's significant differences, it's possible that something happened between the pretest and the posttest to influence your results. Now, let's focus on threats to external validity and how you would improve external validity without sacrificing internal validity. So if you think about the sample, that sample of 12 SciD students at Marshall University is probably not representative of all SciD graduate students. The population of interest based on the first sense of the study's description is the all, you know, SciD students. Keith should contact his colleagues at other institutions and have them run the study with their SciD students too to see if they replicate his results. Another potential threat to external validity would be measurement sensitivity or reactivity. So taking the first survey on stress and life satisfaction may have made them more receptive to the yoga treatment. So again, you could use a Solomon four-groups design, or at least have two conditions, where one gets the pretest, yoga, and a posttest, and the other condition who skips the pretest, and just gets the yoga intervention and the posttest. If the condition who took the pretest and did yoga, condition A, has significantly less stress and more life satisfaction than students in the condition who did the yoga without taking the pretest, or condition C, then it's likely that the pretest influenced the impact of yoga on posttest results. Another issue with external validity or threat could be replication or response latency. The effects of the yoga intervention on stress and life satisfaction may degrade over time. So maybe have a follow-up study where students um stress and life satisfaction are tracked for a year with one condition doing yoga for two weeks and the other doing it the whole year to see if you have to keep doing the yoga for the effects to be retained. So I hope that this video helped you think a lot about practical implications of balancing internal and external validity and kind of the happy medium that we can strive for in research so that we can be confident that what we are measuring and what we are manipulating in the study is actually valid and is actually accurate, and then also have the external validity where we are confident that whatever we observed in the results that we got in our sample would translate beyond the sample. So think about this in terms of your own research and also the research that you read about and make decisions based on.
