S4’s Presentation at UFERN

Last week, S4 Founding Principal Manisha Patel presented a 5-minute “lightning talk” at the 2021 Undergraduate Field Experiences Research Network (UFERN) Meeting. Her presentation, co-authored with S4 Founding Principal Aaron Ellison and Andrew McDevitt, a Bio Education Researcher at the University of Colorado Denver described how we have been thinking about and measuring student impacts from undergraduate research experiences. Continue reading to see the slides and learn more.

Our data and insights are based on our work co-directing the Harvard Forest Summer Research Program in Ecology from 2004-2019. HF-SRPE began in 1985 with one student researcher.  It gradually grew into an REU Site and by 2005 was supporting >15 students/year.

At that point we started asking ourselves “Are we making an impact?”. Unfortunately, since we had not been collecting data, we had no way of evaluating or interpreting our program’s outcomes or evidence to support changes we wanted to make.

So, in 2005, we worked with a professional evaluator (Brad Rose) to develop and implement a Pre/Post survey to understand whether the undergraduate research experience was having a positive impact on our student’s learning about and ability to do science. We found that:

• the absence of prior laboratory research experience and the perception of being a respected member of a research team
• were strongly and positively associated with gains in research skills, confidence in research skills, and abilities to do and present research.

We published the outcome of this work in 2016 (McDevitt et al. 2016). But we couldn’t use anonymous pre/post data collected in a single summer to evaluate long-term effects of the summer research experience.

Around 2010, NSF supported the development and deployment of the Undergraduate Research Student Self-Assessment (URSSA). The results we found using URSSA suggested that:

• there are differences in attitudes and behaviors towards STEM,
• with higher variance among minoritized students traditionally underrepresented in science.

But URSSA is only a post-hoc survey. Although it is very informative for understanding national or network-level effects of UREs, for us working at an individual site it only allowed us to identify short-term changes.

Finally, we wanted to know about the long-term impacts of the REU Site experience for our students. We have been thrilled to learn that our alumnae/i stay in touch and that > 75% of them have obtained advanced degrees and work in ecology and environmental sciences (McDevitt et al. 2020).

All these data suggest that UREs make real short-term and long-term differences in student’s learning about scientific research and their persistence in ecology and environmental science fields. However, these data don’t really help us understand why UREs are impactful.

“Systems thinking” can help us deal with quantitative and qualitative complexities simultaneously. We can use theoretical models to guide our programmatic assessments and generate testable hypotheses about what factors contribute to affective gains in student learning outcomes.

Currently, our data emphasize easy-to-collect quantitative data, including

• demographic information
• Likert-scaled self-reported learning gains

But these data are not collected in a way that allow us to meaningfully test hypotheses and ignore the social, cultural, and historical contexts of our students and our REU site.

To help us understand and analyze the relationships between what people think and feel and what they actually do, we have been using cultural-historical activity theory (CHAT) as a theoretical framework (McDevitt et al. 2020). Our next phase of assessment and evaluation will use CHAT to integrate short- and long term survey data, URSSA, and qualitative information into a systems-based approach that will help us expand our thinking about causal factors that lead to impacts and gains by individual students.

It’s difficult to extrapolate results from the Harvard Forest to other sites, but CHAT also helps frame why it likely worked for us and could allow others to evaluate whether similar approaches might work for them. We believe that using systems thinking, applying models, and adopting conceptual frameworks such as CHAT would help improve discourse across undergraduate experiential research programs.

Although we are no longer directing the HF-SRPE, we are continuing to study impacts on Undergraduate Research Experience. Through S4, we welcome opportunities to support other REU sites and PIs interested in addressing similar issues. Contact us for more information!