Foto van Muskan Achhpilia & Karen De Meyst

ChatGPT put to the test: how did it do in its accounting exams?

05 July 2023 Research news item

Accounting lecturers Karen de Meyst and Muskan Achhpilia recently contributed to a unique piece of collaborative research, in which ChatGPT was fed thousands of accounting assessment questions to see how it would perform. Although the AI chatbot largely failed its exams, the results give rise to an urgent discussion about the future of accounting education. ‘It’s a force that can’t be ignored.’

Ever since its launch in November 2022, ChatGPT has been the subject of discussion. Amazed by its potential, people in all kinds of occupations have started wondering: how is this going to affect my job? Just look at accounting, for example: managing transactions and balance sheets, creating financial statements, computing tax returns. Aren’t these things that AI technology like ChatGPT could take over?

Well, no, is the simplified answer to that question. At least not for now. That is according to a unique study led by American Accounting Professor David Wood. In a collaborative effort with 327 co-authors across 18 institutions worldwide, he decided to put ChatGPT to the test. The researchers subjected the AI language-learning model to more than 25 thousand accounting assessment questions and compared its results to those of their students.

Nijmegen School of Management contributes

Assistant Professor in Accounting Karen De Meyst, Junior Lecturer in Accounting and Technology Muskan Achhpilia and Assistant Professor in Accounting Thomas Niederkofler are among the 327 accounting educators that contributed to the recently published article. Together, they co-teach the course Management Accounting and Control at Nijmegen School of Management. Nijmegen School of Management spoke about the project with De Meyst and Achhpilia.

‘I came across Wood’s call for contributors to his research on social media and figured it would be a nice project to participate in’, says De Meyst. ‘Especially for Muskan, as this topic is right up her alley.’

Achhpilia started her PhD at the department of Economics & Business Economics in January 2022. Her research is supervised by Department Head Frank Hartmann and Karen De Meyst. Achhpilia studies the ways in which technological innovations, like AI, affect corporate decision-making. ‘So yes, I was excited when Karen told me about this opportunity, she says with a smile.

Results

The 327 accounting educators asked ChatGPT to work out assignments they had created for their specific exams. Together the questions covered all kinds of topics, varying both in difficulty and in type (true or false, multiple choice, short answer, etc.).

The results show that ChatGPT did not outperform the students. It had an overall score of 47.4%, whereas the students obtained 76.7%. ChatGPT struggled particularly with questions that required complex reasoning or knowledge of mathematical processes.

Does this mean accountants have nothing to fear from ChatGPT? Well, there is more to be considered of course. Like the fact that for 11% of the questions, ChatGPT did outperform students. And that the researchers used the original GPT-3 version, whereas OpenAI recently launched its much-improved GPT-4. ChatGPT also showed its ability to learn. When educators explained to it why its answer to a certain question was wrong, it showed understanding and answered the same question correctly one week later.

Using ChatGPT currently forbidden

Now before students get overly excited, it’s perhaps good to note that the use of ChatGPT is currently not allowed for assignments. ‘The examination boards decided to ban its usage’, says De Meyst. ‘We even have to get students to sign forms stating that they won’t use Artificial Intelligence tools like ChatGPT. That’s the situation as it stands at present.’

‘But I don’t think these regulations will ensure a credible commitment from the students,’ Achhpilia adds. Looking at the speed with which the language model ChatGPT is improving, it does seem difficult to imagine it remaining outside the classroom for long. The question is not if it will impact the accounting profession and education, rather how it will affect it. ‘It’s an exogenous shock to the education system and the economy,’ says Achhpilia.

Means to cheat or useful tool

She agrees with the overall tone in the article, which is that ChatGPT’s capabilities are promising and that technology like this might help in shaping the future of accounting education. ‘It could help students solve complex issues and deepen their understanding of accounting’. An obvious concern is that students will use AI technology to cheat, but Achhpilia thinks we shouldn’t look at it that way. ‘A good student will figure out how to use it to his or her advantage.’

Which is not to say that she doesn’t have concerns as well. ‘I think this technology will exacerbate inequality. Stronger students will learn how to use ChatGPT as a tool to increase their own knowledge rather than a means to cheat. Whereas students who struggle with their classes, might use it to pass exams without learning much.’

Looking forward

Another issue is the trust relationship between students and supervisors. Achhpilia: ‘It’s only logical that tutors will pay more and more attention to possible fraud with AI tools. This could lead to an atmosphere of suspicion. And this issue won’t just arise in student-teacher relationships, but in corporations and the economy at large as well.’

In the light of this, Achhpilia is researching how the control-trust relationship between various principal-agents in corporations can be affected by AI. One of the things that is important to consider in this respect is people’s individual preferences towards either algorithm aversion or algorithm appreciation. Some people can’t wait to implement new technological tools, whereas others prefer to be a bit more cautious.

Achhpilia and De Meyst are good examples of this difference themselves. Where Achhpilia is already using AI to help her optimise the tone of voice of her e-mails for instance, De Meyst is not so eager to use AI tools for her work at present. She smiles: ‘I still like writing.’

However, the emergence of ChatGPT has prompted policymakers and accounting educators like Achhpilia and De Meyst to think about responsible ways to incorporate AI technology in their teaching models. Achhpilia: ‘It’s a force that can’t be ignored.’

Photos: Duncan de Fey

Text: Pim Muller

Contact information

Organizational unit: Nijmegen School of Management
About person: Achhpilia, M.P. (Muskan) , Dr Meyst, K.J.L. de (Karen)