Generative AI in assessment
In the presentation below (download slides), I start with some operational assumptions around assessment that inform my thinking. I then talk about how large language models are different to other types of technology that influence assessment practices. I describe the standard assessment paradigm, and the negative impact of working within this paradigm (this is when the recording begins), and then talk about how AI-supported assessment may move us into a different assessment paradigm.
I talk about some of the recent changes to large language models and how these changes move us towards what I’ve previously called ‘universal anything machines’ (see expertise). I go on to discuss principles of assessment design with specific reference to platforms like ChatGPT, and then work through an example of an assessment task making explicit use of ChatGPT.
Finally, I briefly describe a faculty development framework, as well as the implications of AI-supported assessment on the academy, and on learning more broadly.
In Beta podcast: Generative AI and assessment
This was a wide-ranging conversation that explored some of the detail around how language models work, it’s inability to compare responses to valid models of the world, practical uses for AI in teaching, learning, and assessment, and the risks of having AI being trained on data generated by AI. We explore the implications of a higher education system that embraces AI, and ask if integrating AI has inherent value, or if it’s value is simply instrumental. And we discuss an awkward conclusion that at least of us feels is inevitable (Ellis & Rowe, 2023).
When designing assessment tasks, try to frame the activity so that it challenges students’ understanding rather than their memory. For example:
- Create prompts which allow for self-reflection, critical analysis and connection to real-life situations.
- Consider asking for drafts, outlines and notes to be submitted as formative assessments, allowing the students’ development to be documented.
- Ask students to include references from texts used during their studies, but not freely available on the internet.
- Multimodal assessment, including elements such as infographics and video presentations, allow students to show their understanding in other ways.
Supercharged assessments
I think we need to seriously consider the possibility that the only reasonable way forward is to allow students to use AI for their work, but we significantly increase the baseline of what we expect that work to look like. In my opinion, AI gives everyone a massive boost in capability in terms of what they are able to do. And so we should increase our expectations of what they submit for assessments.
For example, instead of asking students to:
- …design a building and create detailed blueprints, ask them to not only design the building but also create a fully immersive virtual reality walkthrough of the structure, complete with realistic textures, lighting, and interactive elements. They must also optimise the building’s energy efficiency, structural integrity, and environmental impact.
- …create a proposal for a public health initiative, ask them to launch an actual public health campaign that identifies at-risk populations, optimises resource allocation, and personalises outreach messages. In addition, the campaign should demonstrate a measurable impact on public health outcomes.
- …write a research paper on an environmental policy issue, ask them to develop an AI-powered decision support system for environmental policymakers. They should also create an interactive interface that allows policymakers to explore different policy options and understand their potential impacts.
If AI enables ‘superhuman’ performance, then our assessments should be modified to evaluate superhuman outputs.
Additional resources
- Ellis, B. & Rowe, M. (2023). Generative AI and assessment. In Beta podcast. Retrieved from http://inbetaphysio.com/2023/07/14/29-generative-ai-and-assessment/.
- Rowe, M. (2023). Generative AI in higher education: Paradigm shifts in assessment. Keynote presentation, CPUT Learning and Teaching webinar. Retrieved from https://www.mrowe.co.za/blog/2023/08/presentation-generative-ai-in-higher-education-paradigm-shifts-in-assessment/