When artificial intelligence has made such inroads, how are we in the media supposed to react to it?

In some circumstances, I have no problem with AI; when it’s used to take care of repetitive tasks so that humans can do what only humans can (IKEA is a good example of wise use of AI, retraining workers whose tasks were taken over instead of laying them off), it’s a good thing. When it’s used to replace humans entirely, especially in creative endeavors (ask me how I feel about AI-generated “art,” especially when it puts talented real artists out of work), it’s not good.

One of the biggest problems opinion editors faced most recently was astroturf: letters and columns that often come through letter-writing campaigns by special interests. When I come across something that sounds too polished or like something I’ve read before, I often will run isolated sentences through Google or a plagiarism checker. Sometimes I’ll find it’s a letter that’s appeared elsewhere (against our rules, and sometimes with the same name but a different town in another state, which is also against our rules, which demand that letter-writers be residents of Arkansas), or was a product of a letter generator for a specific topic (if you really care about a subject, speak from your own heart rather than making use of a form letter).

Recently, though, I received a forwarded message from my boss about a guest column, by a writer with whom I’ve had previous dealings, that someone said an AI detector showed to be nearly 100 percent AI-written.

Well. So I guess we’ve skipped right from, “That was so well-written” to “My AI tool said you didn’t write a lick of this!”

Because I’m the curious sort, I found a different detector tool (free because I’m poor), and entered segments of my work, that of the offending writer, the accuser, and a freelance columnist for the paper. The result? My work came back as 17 percent AI, the offending writer as 55 percent AI, 51 percent for the accuser, and a whopping 76 percent for the freelance columnist.

What does that tell you? Among other things, that it’s hard to put stock in AI detectors at the moment, and I’m not the only one who understands that false positives are a real problem, even with the paid version (GPTZero, I believe) the accuser used. (And hey, I pasted a different piece of my writing in Getsolved.ai’s detector and that of Originality.ai, and came back with 75 percent AI and 100 percent human respectively. For the same two paragraphs I wrote myself. And GetSolved was what I used for that first instance, soooo …)

Pretty much every AI detector, paid or not, advertises itself as the most accurate. So what gives?

AI tools can be helpful, but we have to be realistic. AI large language models (LLMs) are trained using presumably human-generated text, and are only as good as what goes in. Many newspapers have declined such use to protect their work (especially when it’s being scraped up by bots for free). Using the wasteland that is social media would be a horrendous idea, what with all the misinformation, back-biting and general unpleasantness. Those bots are getting their training information (sometimes illicitly) from somewhere, though, so that they can train LLMs to mimic humans and generate copy. No one said the copy would be good.

Given that, it’s of little surprise that the reliability of AI detectors can vary widely, depending on how much copy, the type of copy and, heck, the time of day that you might feed them. Some of the tools do really well with short segments of copy (like a few paragraphs) and struggle on long texts, and others are the opposite. And that in itself is enough reason to temper the enthusiasm for using them as a gotcha.

A December report from NPR highlighted the problems of using AI detectors in schools: “More than 40 percent of surveyed sixth- to 12th-grade teachers used AI detection tools during the last school year, according to a nationally representative poll by the Center for Democracy and Technology, a nonprofit that advocates for civil rights and civil liberties in the digital age. That’s despite numerous research studies showing that AI detection tools are far from reliable.

“‘It’s now fairly well established in the academic integrity field that these tools are not fit for purpose,’ says Mike Perkins, a leading researcher on academic integrity and AI at British University Vietnam.”

One Maryland student was accused three times in two different classes of using AI in assignments she did herself, and was docked points. Now she runs all of her assignments through multiple AI detectors before turning them in, which adds an extra half-hour to each assignment.

A teacher interviewed by NPR uses the tools, but in a way that student’s teachers should have: as a way to open a conversation with the student and figure out if they actually did do the work themselves.

Language and literature teacher John Grady told NPR, “It’s certainly not foolproof. But it gives you something to hang your hat on.”

“Grady teaches at Shaker Heights High School, part of the Shaker Heights City School District outside Cleveland. The district serves roughly 4,400 students, and is paying GPTZero, another AI detection software company, about $5,600 this year for annual licenses for 27 of the district’s teachers. The tool calculates a percentage likelihood that a student’s work is AI-generated.

“Grady says he puts all student essays through GPTZero; if the tool shows more than a 50 percent likelihood AI was used for the assignment, Grady digs deeper. That includes using revision history tools to see how much time a student spent on an assignment, and how many edits they made during the writing process. If it appears that a student made only a few edits and spent hardly any time writing, he’ll check in with that student.

“‘And I’ll say, ‘Hey, this flagged. Can you talk to me about why?’ I’d say the bulk of the time, like 75 percent, if it was AI, they’d be like, ‘Yeah, I did.’ And I’m like, ‘OK, well now you’ve got to rewrite it with less credit,’ Grady says.

“Edward Tian, co-founder and CEO of GPTZero, says this is how educators should be using his company’s tool. ‘We definitely don’t believe this is a punishment tool,’ Tian says. ‘This needs to be a tool in the toolkit and not the final smoking gun.'”

And yet some people trust those tools implicitly, especially when they can lob accusations at others, forgetting that the burden of proof is, or at least should always be, on the accuser.

Matt Lillywhite of The Daily Draft wrote in a Medium essay last month, “Some people are genuinely writing poor-quality and low-effort articles with AI. Many aren’t. Yet, the same evidence gets used for both. People point to em dashes, sentence length, formatting choices, and words that existed long before ChatGPT. So, the result is that innocent writers get treated like suspects, accusations spread faster than proof, and entire groups of strangers reading articles on the toilet end up acting as judge, jury, and executioner without actually knowing what happened.”

You can take my em dashes (and my parentheticals, for that matter) out of my cold, dead hands, and I’m not the only person who feels that way.

Lillywhite did the same thing I did, feeding text into multiple detectors. His text? The Emancipation Proclamation. Based on what the tools showed, he said, “then Abraham Lincoln either invented time travel … or millions of people holding pitchforks are treating guesswork as science.”

Well, if Lincoln could be a vampire hunter, he could obviously also be a time traveler, right?

Lillywhite’s conclusion: “People are treating detector scores as proof even when the tools themselves can’t consistently agree on if it’s human or AI. So, if we’re going to accuse somebody of cheating, lying, or passing off someone else’s work as their own, the burden should probably be on the accuser to prove they’re right.”

At the Arkansas Democrat-Gazette, we edit what goes on the opinion pages ourselves, and what we write, we write ourselves. Some submissions we get are easy edits, partially because some of them go through multiple sets of eyes before they ever get to us (believe me, a college president, for example, does not want to look like an idiot), and some because the submitters are genuinely good writers. Others we might have to massage a bit because not everyone is a natural writer and everyone needs an editor (even editors).

That writer who was accused of using AI didn’t use it; she’s a genuinely good writer, which is one of the reasons I’ve used her on the Voices page and will continue to do so. She just happened to be on the opposite side of an issue from her accuser, who could have easily talked to her first before posting his accusation on social media and in an email to my boss.

It’s a shame the accusation was even made, but it helped make a point: Don’t put too much trust in AI tools.