Always a fan of a counterfactual, I thought I’d give this thought experiment a try: If LLMs were around when I was an academic, how would I have advised students to use them? LLMs weren’t in widespread use about a hundred (four) years ago when I was teaching, so I shall begin with three similar activities and see if I can guess how I would have responded to LLMs.
Issues before LLMs
Copy-paste verbatim from sources
The easiest to spot. I’d often see this before running Turnitin. The usual clues were sections of dodgy, poorly argued and poorly referenced prose (not copy-pasted) and the occasional paragraph of tightly argued, edited, and well-cited prose (maybe copy-pasted?). Sometimes a change in font would give it away. But my “plagiarism checker” was often just Googling the obvious text.
Some students would copy from something on my reading list. I read things on my reading list before putting them there (gawd give me patience). I can’t think of any example where Turnitin revealed that a student was plagiarising. It sometimes helped me find additional examples of copy-paste in an essay that had already elevated my eyebrows.
My advice here would have involved going back to basics on writing essays.
Over-reliance on Wikipedia
Wikipedia is a fantastic resource (and I’m a Wikipedian), but it is often riddled with errors and omissions because it’s written by volunteers like you and me hacking sentences here and there, fixing issues when we happen upon them, generally with little time to do so.
The problem was that some students treated Wikipedia pages as if they were journal articles and cited them. I attempted to fix that problem by doing a live edit in class so students saw how Wikipedia works. The problems with relying on it are then obvious.
I suggested using Wikipedia to get unstuck when doing a literature search, e.g., by looking for terminology ideas to search for, or following a reference cited in an entry and things it cites and are cited by it. Wikipedia can be used to plug into the literature.
Copy-paste from sources and editing
You’re writing an essay on a topic you don’t really understand. So, you grab a collection of good references, copy and paste key paragraphs directly into your essay draft, and move the words around until you think it’s not plagiarism any more.
I’d usually spot this form of plagiarism because students sitting at 3am wired on caffeine and sugar would lose track of which bits they had edited and which bits were pasted from the source.
The alternative approach I offered was this (note: for narrative essays, not systematic reviews!):
- Use a reference manager like Zotero to keep track of your sources.
- Read a bunch of things, highlighting the PDFs and adding quick notes in the margins as you go.
- Let the ideas wash around in your head for a bit.
- Write notes (in a separate doc) on key points that stick, and revisit the papers to fill in details. The aforementioned highlights will help you get back to relevant sections.
The trick is to find the right distance between the verbatim text and the idea it expresses, to find themes across multiple papers rather than relying too much on one. Some specific details should obviously come verbatim from the paper, e.g., don’t let an effect size emerge from your mind.
What does any of this mean for LLMs
Back to the counterfactual. I suspect I’d probably have done something like the following:
- Introduce a topic and ask students to do the readings.
- Get an LLM or two to offer a summary of the topic, with references.
- Discuss any gaps or inaccuracies or weird references.
I think my advice would have been similar to that for Wikipedia: you can use LLMs to get unstuck, to find a way into the literature. But then it is back to research as usual. I’m not sure I’d have any magical way to check if LLMs were used (I see there are checkers and they have problems). I’m curious to know if academics spot the use of LLMs because things aren’t cited properly: pretty obvious if you’re teaching a topic you know well, and highlights the importance of ensuring lecturers are asked to teach a topic that’s close to their research.
The other common use of LLMs, as a fancy grammar checker, doesn’t feel too terrible. Here I’d probably show a few examples live in class and ask students to break into small groups and discuss whether the resulting rewrite was better than the original. Again, LLMs can be helpful for getting unstuck. You have an idea for a sentence and can’t quite make it work. Ask Copilot for a few alternatives. They might all be rubbish but the process can help.
For example, I don’t like the following sentences that I tapped out above (but left as-is):
“I suggested using Wikipedia to get unstuck when doing a literature search, e.g., by looking for terminology ideas to search for, or following a reference cited in an entry and things it cites and are cited by it. Wikipedia can be used to plug into the literature.”
Copilot gave me:
“I recommended using Wikipedia as a resource to overcome obstacles during a literature search. This can include finding new terminology to search for, following references cited in an entry, and exploring the sources those references cite or that cite them. Wikipedia serves as a valuable tool to connect with the broader literature.”
Maybe this is better, but to me it feels a bit bland. I could have edited back to:
“I suggested using Wikipedia to get unstuck when doing a literature search. This can include finding new terminology to search for, looking up references cited in an entry, and exploring the sources those references cite or that cite them. Wikipedia can be used to find relevant literature.”
Here ends the counterfactual. I wonder how close it is to what I actually would have done…