How courts decide if it's AI

Share

Students accused of using AI sometimes challenge their penalties in court. An analysis of eleven cases shows that judges, too, have no foolproof way to determine whether a text was written by a human or a machine.

Litigation. Since 2023, a new kind of complaint has been landing on the desks of administrative judges. Students accused of using AI to write essays or during exams are challenging the disciplinary sanctions imposed by their universities. I read eleven rulings (six in Germany, three in France, and two in Belgium) to understand how judges decide whether a text was written by AI.

Obvious. Sometimes, the use of a chatbot is obvious. A Berlin judge found that translation errors (such as writing Billiarde from the English billion instead of the German Milliarde) were evidence of AI use. Among the other pieces of evidence cited by the judge were erratic punctuation and the fact that certain abbreviations were explained multiple times.

Quotes. Judges become suspicious when citations don’t add up. A student at the University of Cologne lost her case because she cited reviews of a film that had been published before the film’s release. AI detectors, meanwhile, were mentioned in four cases, but in three of those four, the judges explicitly noted that such tools are not infallible.

A body of evidence. Most of the time, judges accept that a combination of circumstantial evidence is sufficient. The university does not have to provide definitive proof of cheating. It simply has to show that an abnormal result has no explanation other than the use of AI.

That was the case with a student at Sorbonne Paris Nord who submitted an essay in polished French despite struggling to express herself orally. Or consider a student in Brussels who left no trace of her work on her exam paper other than a perfect answer, even though the university’s professors agreed that the exercise was impossible to complete from memory alone.

Reading these judgments, the students’ apparent audacity can make it hard not to side with the judges. And yet, I have my doubts.

A personal experience. I have my doubts because something similar has happened to me. In 2006, I was a student at the University of Hamburg. My contribution to class was limited, yet I submitted an assignment (in English) I was rather proud of. The professor, unconvinced that I had written it myself, demanded access to my Word document before grading it. Fortunately, generative AI did not exist yet.

The arguments. Sometimes, the judges’ reasoning is hard to follow. The student at the Sorbonne submitted documents to prove that she had indeed conducted research on Mandarin Chinese. But the court ignored them because they were “illegible and untranslated.”

In Kassel, Germany, judges devoted several paragraphs to assessing whether the student’s nervousness might explain why he performed less well orally than in writing. And in Munich, the court asserted that short sentences and a “high density of information” were characteristic of AI-generated writing. I would be inclined to argue the opposite.

Putting them in their place. Reading these judgments, I was struck by the profiles of the accused students. They were often women studying in fields perceived as male-dominated.

The Brussels student mentioned above was taking a computer programming exam. Another student, in Toulouse, was accused of being too good at algebra. (This was one of only two cases in my sample in which the judge ruled in the student’s favor.)

In another German case, a female student was denied her high school diploma because her mathematics performance was deemed too good compared with her usual results. That decision was upheld on appeal.

The weaker party’s voice. In just one case, in Belgium, did a court highlight the absurdity of these proceedings, which require students to prove their innocence while the university merely has to suspect cheating. Even if those suspicions must be grounded in evidence, the difference is substantial.

I am not suggesting that all these students were innocent lambs who suddenly became exceptionally skilled after spending countless nights studying. Of course cheating exists. Nevertheless, a student who is too good at mathematics can always be accused of using AI. She will have nothing but her own good faith to defend herself.

Prejudice. In court, as elsewhere, prejudice plays a role in decision-making. Studies show that men’s testimonies are considered more credible, while those of people perceived as foreign—for example, because they have a different accent—are considered less credible.

These findings come exclusively from experimental studies, since it is impossible to test the truthfulness of a statement under real-world conditions. Nevertheless, when AI use is suspected but cannot be proven, judges are left with nothing but the accounts given by the people involved and their perceived credibility.

And that puts female students at a disadvantage.

European press review

🇳🇱 Netherlands: The government plans to eliminate 25,000 full-time equivalent positions in the public sector and replace them with AI. The police and the military will be exempt. [Rijksoverheid, 26 Oct in Dutch]

🇬🇧 United Kingdom: In June, German media conglomerate Axel Springer bought The Daily Telegraph. They now fired about 100 staff (out of 900) and plan to become “AI-powered”. [The Guardian, 23 Oct]

🇩🇪 Germany: In September, the Poison Information of Göttingen received three times as many inquiries regarding potential mushroom poisoning as it did in the same month last year. According to the center, confidence in chatbots' capacity to identify shrooms are partly responsible for this trend. [NDR, 22 Oct in German]

🇷🇴 Romania: A bill supported by a large majority of members of parliament aims to prohibit chatbots from providing personalized legal advice, such as the drafting of contracts. [Digi24, 21 Oct in Romanian]

Summary from academia

The problem. People involved in the AI ​​economy go by a variety of names. They tend to identify with the platform they work on (e.g., “Turkers” for Mechanical Turk). Academics, for their part, call them gig workers, microworkers, digital workers, and so on. Those with stable contracts, especially in Europe or the United States, see themselves in a different category altogether.

The solution. Sociologist Antonio Casilli proposes calling this social class “data workers.” After all, moderating social media requires analytical and critical thinking skills, just like computer programming. All these people, wherever they are and whatever their contracts, see their work appropriated by Big Tech.

Why it matters. Academics love to name things (to get cited), and perhaps Casilli's proposal will simply be added to the list. But that's not the main point. Naming work is the precondition for any action. Domestic work, for example, has long been denied (“my mother doesn't work”). It is by naming it that we pave the way for demands such as the right to a pension or paid leave. As the author writes, these seemingly semantic debates actually determine the possible forms of solidarity.

Antonio Casilli, The Making of the Data Working Class. Edward Elgar Publishing, 16 Oct

International press review

🇮🇩 Indonesia: Many content creators complained that posts and videos related to the ongoing forest fires are shadowbanned on Instagram (their visibility is limited). The government denies the order came from them. [Detikinet, 21 Oct in Indonesian]

📦 Amazon: An automated system scores warehouse worker based on the likelihood that they will join a union. The system is probably in use in the US, Canada and the UK. [OpenDemocracy, 16 Oct] Reporting from 2020 already showed that the firm used a similar system to rank Whole Foods stores based on their risk of unionizing.

This is fine

Fatigue. A Chinese entrepreneur created an entirely virtual music group consisting of four female avatars. Finding that managing four of them was too exhausting, he had one of the avatars announce that she was going on “maternity leave.”

Backlash. Following the reaction, he issued an apology. The fact that his idiocy went viral didn't help: his music project has only 14 fans on Weibo.

“Chinese AI idol’s ‘pregnancy’ stunt sparks backlash, debate over digital boundaries”. South China Morning Post, 20 Oct