How Irish Colleges have modified their grading in response to AI

By Katherine Donnelly

Fri 15 Sep 2023 at 03:30

Radical changes in how students are assessed are on the way in a move to combat the threat of artificial intelligence (AI) platforms such as ChatGPT.

The initial focus is on third level, but the Department of Education is keeping abreast of developments.

It is early days, but new guidelines are intended to lead to a complete reimagining of how students’ knowledge and skills are measured.

Already, higher education colleges are being told to abandon certain forms of assessment because they are “no longer be considered to be sufficiently robust to award scores which count towards official grades”.

These include do-at-home essays or similar assignments focused on subject knowledge, with a submission by a single deadline and where marks are based on structure, style, and information.

Third-level colleges are also advised to avoid unsupervised online assessments, such as multiple choice question (MCQ) exams, which count toward grades. There will be a greater reliance on oral assessments to check students’ understanding of a topic and to identify if they have cheated by using AI in an assignment without declaring it.

As third-level colleges return, guidelines have issued on how to tackle the threat posed by Generative AI (GenAI) platforms, which can mimic the work of humans and may encourage difficult-to-detect cheating by some students.

The same principles will also apply to the further education sector.

The guidelines have been drawn up under the auspices of the academic standards watchdog, Quality and Qualifications Ireland (QQI), and have recommendations for both educators and students. Crucially, QQI has also been sharing its expertise with Department of Education officials in the recent months.

While the QQI remit does not extend to second level, the agency is leading the way in Ireland in dealing with the AI challenge to education, leading to these discussions.

As post-primary school leaders consider the implications for their pupil assessments, a QQI representative briefed the Association of Community and Comprehensive Schools (ACCS) annual education conference yesterday.

The new GenAI guidelines take on a particular relevance as detailed work gets under way on reform of the Leaving Certificate, including plans to change the way school-leavers are assessed.

The guidelines set out what students and lecturers need to know and do in a world where GenAI can produce plausible written texts, images and graphics, computer code and mathematical solutions, and more on demand.

AI can have a legitimate educational use and students may even be encouraged to use such tools in their work, once they declare that they have done so.

But the way GenAI platforms such as ChatGPT and Bard are trained runs the risk of increasing intentional and unintentional academic misconduct by students.

The guidelines – called Generative Artificial Intelligence: Guidelines for Educators – have been drafted by QQI’s National Academic Integrity Network (NAIN) as a response to the swiftly evolving and developing field of GenAI.

Colleges are advised to act quickly and take short-term measures while also working toward a reimagination of student assessment in the longer term.

NAIN has advised an early review of all assessments at the level of individual programmes, the range of assessment types and the overall volume of assessment, to identify vulnerabilities.

Then, any assessment that may be completed satisfactorily by someone using AI without an appropriate level of understanding of the subject, or which is vulnerable to breaches of integrity, should be replaced or modified, the guidelines state.

However, colleges are told to resist any temptation to switch back to traditional end-of semester formal exams as the easiest way to ensure the integrity of assessment. The guidelines say that this would run counter to the strength of more authentic assessment, which aims to develop skills, knowledge in context and other professional and graduate attributes

Instead, they suggest a short term reweighting of assessments may be necessary to respond quickly, with a longer-term goal of a more holistic approach with a range of assessment types.

Colleges are also urged to be alert to the danger of over-assessment, including overlapping assessments, which can cause undue stress and pressure for students and can lead to cheating.

NAIN suggests lecturers agree a schedule of assessments and deadlines to address workload issues.

Rather than switching one assessment type for another, NAIN is urging some fresh approaches, such as focussing on the “process rather than the product” and giving credit for the stages that a student goes through in completing an assessment.

Other suggestions include in-class writing assignments or problem-solving tasks, or the inclusion of an oral component in which the students are asked to answer questions around a topic.

The guidelines will be fully aired at the Academic and research Integrity Conference Ireland 2023 , jointly hosted by NAIN and the University of Galway next month.