There have been some interesting discussions around the use of Artificial Intelligence (AI) – and the fear of how it will impact assessment in HE. I will reflect on this myself in time, but for now, I want to muse on a possible use case. I’ve been following an email thread on AI in higher education via a publically accessible JiscMail. One participant had acknowledged how sprawling the debate had become – and asked if ChatGPT could lend a hand. I decided to give this a go using OpenAI‘s text-davinci-003 model (and it should be noted that this post’s feature image was drawn by OpenAI’s DALL·E 2)!
First, I downloaded the content text from 30 publically accessible emails on this topic of ChatGPT. Secondly, I removed all personally identifiable information (i.e. names, emails, links, institutions) from the text. This was an ethical issue as while the email text is in the public domain via jiscmail.ac.uk, the tool I used is allowed to retain text to train the algorithm further. Thirdly, I split the text into two separate queries to fit within the 4,000 token limit on text-davinci-003 model. Finally, I took the two text-davinci-003 outputs and asked text-davinci-003 to summarise the discussion from each text sample and re-write it for a knowledgeable audience.
Artificial Intelligence summary of the discussion:
This email thread is discussing a variety of topics related to ChatGPT, a new technology, in higher education assessment. Ideas have been suggested to mitigate any short-term disruption to assessment this semester, such as informing teachers of the technology, revising current assessment regulations, teaching students about academic misconduct, incorporating the model into assessment design, encouraging the greater use of oral-based assessment, group project work, and classroom-based assessment, using browser lockdown capabilities, testing the model against a variety of tasks, and promoting assessment (re)design that requires the creation of multi-media artefacts. There has also been discussion about curating resources on the topic, as well as different types of assessment that could be used, such as handwritten assignments and orals, and the ethics of the technology, with suggestions to use Turnitin to cross-reference essay submissions and to teach students to question the production and consumption of their cultural landscape. Additionally, it has been suggested to have an event which focuses on the implications of GPT for educational development and developers, as well as a SIG on AI and Academic Development. It was also noted that OpenAI (the company behind ChatGPT) is likely to become an organisation within the Microsoft ‘empire’, and that there is software that can detect ChatGPT-generated text.text-davinci-003 summary of academic development discussion
Evaluating text-davinci-003’s output
… and do you know what. It isn’t a bad summary. It has missed some of the nuance and detail – and obviously the weight of some arguments over others. Some of the more powerful arguments (that were buried in the thread) were lost. For me, the summary was much poorer for excluding some of the deeper ethical concerns, for example, OpenAI Used Kenyan Workers on Less Than $2 Per Hour .
Yet, for someone who is ‘time-poor’, such a summary presents an exciting way to engage with significant chunks of information in little time. This summary took about five minutes – including both my time editing the text and that of text-davinci-003. This cost around 20 USD cents ($0.20). The original text was computed to take 18 minutes to read – the refined output was 60 seconds. Paying 20 cents to save 17 minutes of reading… not a bad cost for some.
Caution is, however, needed. When I asked text-davinci-003 to summarise the above for a 3rd grader – it was quite off the mark!
ChatGPT is a new technology that can help teachers assess students. It can help teachers check if students are cheating and can help them create different types of assessments. It can also help teachers create projects and activities that involve creating multi-media artefacts. There are also some ethical considerations to think about when using this technology.text-davinci-003 3rd grader summary of academic development discussion
At this point, it has over-simplified the discussion to the point it is incorrect. At no point was ChatGPT pitched as something to help assess students or help create multi-media artefacts.
It was quite right on the ethical mark, though. I took great care to only use public-domain information with no personal information in this experiment. But even then – I imagine some people would feel uncomfortable about this.
Artificial Intelligence summaries as an automated proccess
This was a manual process – but I wonder how far off an automated future could be. From anyone using Microsoft 365/Outlook for emails, you may have received Microsoft Viva briefings. Microsoft Viva emails to bring together data from across your Microsoft 365 account (including your email) to bring connection and insights. This all uses a form of Artificial Intelligence to provide things like helpful reminders on outstanding actions, advice on how to manage your time in the context of your diary and information it feels might be helpful. Given Microsoft are rumoured to want OpenAI’s technology further integrated into Windows and Microsoft 365 – perhaps email summaries (on this scale) would be a natural extension of Microsoft Viva.