We are witnessing the first generation of students who have full access to generative AI (e.g. to write term papers), and at the same time the first generation of teachers who are bombarded with offers to use AI tools (e.g. to correct or grade term papers). Is this leading to the creative destruction of knowledge ecologies? Are we witnessing the end of collective empowerment through education? Alistar Alexander seeks answers.
*
AI Large Language Models , or LLMs, can do nearly all the things humans can do with language. Except the one thing language was expressly intended to do: convey meaning. Large language models create language quite specifically stripped of all meaning. They can approximate meaning – sometimes uncannily well. But of course their answers actually have no meaning.
This was discussed in a recent article: “When you ask a person, ‘What country is to the south of Rwanda?’ and they answer ‘Burundi,’ they are communicating a fact they believe to be true about the external world. When you pose the question to a language model, what you are really asking is, Given the statistical distribution of words in the vast public corpus of text, what are the words most likely to follow the sequence ‘what country is to the south of Rwanda?’ Even if the system responds with the word ‘Burundi,’ this is a different sort of assertion with a different relationship to reality than the human’s answer.’
Ecologies of knowledge
As LLMs improve and their answers more closely resemble human answers, this actually makes the problem worse; it increases the likelihood of an LLM answer being mistaken for an answer with meaning. So, as the difference between these two types of answer becomes ever more subtle, the impact becomes ever more profound – on all human ecologies of knowledge.
Here, ecologies of knowledge means all the systems and connections humans have for building and sharing knowledge. As LLM generated responses continue to permeate our collective knowledge, we will be less and less able to discern human language from AI language; that is, which language is – or at least should be – meaningful, and which language is definitively meaningless. After all, how many times have you recently wondered: did an AI write that? And how many of those times were you left undecided?
Contamination by AI
In the early 2000s genetically modified foods were highly controversial, with many of the same fears voiced then as now with AI; unproven technologies with unknowable effects, market capture by large corporations, and so on. Scientists then were especially concerned that unproven GM crop strains would escape into ecosystems and interbreed uncontrollably with other strains. This risk was described as ‘contamination,’ and now that GM crops are everywhere, we can see that fear was well-founded.
With the proliferation of AI content we can see our knowledge ecologies also being contaminated. And it may soon be all but impossible to consider any part of our knowledge ecology uncontaminated by AI. Google researchers recently warned AI could “distort collective understanding of socio-political reality or scientific consensus.”
We can see this process especially clearly in the education sector. Public libraries, for example, are filling up with AI generated books. We also know that a huge number of academic papers are being at least partially written with the help of AI. So, it is likely that in the near future, even researchers who chose to write papers without AI, will inadvertently be citing AI sources; as with GM crops, it will soon be impossible to consider any academic paper truly AI free.
Epistemic breakdown
But the impacts of AI on knowledge are far greater than contamination. There is another grave threat from AI: that of epistemic breakdown or knowledge collapse. In other words, that AI will fundamentally and catastrophically degrade our human ecologies of knowledge. We can already see that manifesting in a number of ways.
Perhaps this is most acute in education. Use of AI LLMs is now seemingly ubiquitous among students – sometimes it is easy for teachers to spot, sometimes not. A recent study has shown how this has led to students being less able to form their own ideas. Now, partly in a bid to manage this development, and partly to exploit it, institutions are now actively promoting AI study tools.
At the same time, there is an exploding industry – and (from institutions at least) demand – for AI tools to assist teachers with their workloads. It is claimed these tools will help teachers with admin, and also preparing lessons and marking assignments. These tools are also being used for teaching too. The benefits are all too clear; tailored teaching for different students’ needs, intensive support with no additional overhead, and so on.
Generative AI = degenerative AI
These tools often come integrated in platforms that can monitor success rates and throughput, not just of the students, but the teachers too. These tools are already being used to ‘optimize’ learning outcomes and teachers’ productivity; like Uber drivers, teachers may well find themselves chasing opaque performance metrics ratcheting ever upwards. Schools in the UK and Arizona are taking the next obvious step and offering AI only courses. Notably, The Gates Foundation lobbied for the Arizona scheme’s approval.
This use of AI raises a new set of problems. How can you prevent students using AI when they are also being taught with AI? If their class materials are AI generated, then students are being actively taught not to be able to discern between human and AI content; they may well be even less able to tell the difference than older adults are. What kind of social skills are students going to learn from AI models? And if students become more dependent on AI, and are being taught and marked by AI, then no one is actually learning or teaching anything. Except of course the AI models.
In this way and many others generative AI may be more accurately described as degenerative AI; the AI models are degrading and hollowing out the information ecologies upon which they were originally trained. AI is also having an impact of our perceptions of knowledge itself. In a recent study, students were not only unable to tell the difference between ChatGPT and human poetry from Sylvia Plath and T.S. Eliot, but they actually preferred the ChatGPT poetry. It was rated as ‘more human.’
AI-driven Africa
As with all new technologies with unknowable social impacts, these AI educational tools are being aggressively promoted in Africa by large tech companies and donor organisations. To no-one’s surprise, the Gates Foundation is prominent here too, supporting tools like Khanmigo in African schools. Take a country like Kenya where these AI tools are being rolled out. Currently, they are presented as helping under resourced teachers. But with a growing imbalance in funds between conventional education and tech promoted AI tools, it’s not hard to see that changing, so more and more students will be taught by AI.
Instead of being taught by Kenyan teachers rooted in their community, these students will be taught by AIs exclusively rooted in Silicon Valley – or maybe China. How exactly will a small Kenyan town’s knowledge ecology benefit from having its already small base of teachers inexorably eroded to nothing? Kenya has 50,000 new graduates a year, and as it happens, nearly 50,000 new teacher enrollments. Therefore, we can surmise, a major career path – and a very good career path – for these AI-educated students will be closed off to them; by the AI tools that taught them.
Where will they go?
In countries like Kenya, tech companies like Remotasks and Outlier AI, now employ thousands of highly literate workers to label and process data for AI models. Remotask’s main slack channel was reported to have 461,000 workers worldwide. It turns out that AI automation is astonishingly (human) labor intensive. These workers are low paid, exploited and traumatized. Remotasks and Outlier are subcontractors for Scale AI, a Silicon Valley AI ‘unicorn’ whose 27-year-old founder, Alexander Wang, is being hailed as the world’s youngest ‘self-made’ billionaire. A recent lawsuit filed by workers called Scale AI “the sordid underbelly of the generative AI industry.”
With ever more content being produced by AIs, researchers have noticed another problem: a phenomenon described as “model collapse”; namely, when an AI model is trained on AI data it creates noise and corruption in the outputted data. After only a few iterations the content corrupts completely, and the AI model spurts out gibberish. So it seems once you take out the threads of meaning that hold our shared language together, then the language itself quickly falls apart – and a computer is unable to put it back together. And it may be that humans are unable to do so either.
The leading ‘frontier’ AI models have now ingested literally all the data technically accessible – some of it legally, much of it not – on the web. As AI labs try to improve their models they are increasingly reliant on ‘synthetic’ data created by machines to fill the gaps. Sometimes this seems to work, sometimes it doesn’t, so it brings the prospect of complete model collapse ever closer. And what if epistemic collapse and model collapse become irretrievably entangled; and they end up contaminating each other?
More capable machines rendering humans ever less capable
If model collapse can infect an entire AI model and corrupt its output, and if AI models are contaminating all our human made knowledge – is it not too far fetched to imagine that our entire body of knowledge could be at severe risk from model collapse too? If AI progresses as planned by Silicon Valley, and it becomes so ubiquitous and essential to every aspect of life, then if there was a model collapse, is there anything we could do about it anyway? At that stage would we even be able to tell?
In recent months we have heard a huge amount about ‘Artificial General Intelligence,’ AGI, or even super intelligence being within reach, possibly as soon as 2027. Of course this is eminently possible; as the people that own the AI models are the ones defining AGI, it will be whatever – and therefore when – they decide it to be. The assumption we have is that this extraordinary goal will be reached by producing ever more capable machines. But maybe AGI is reached not through more capable machines, but by rendering humans ever less capable? For some people in AI, this may turn out to be a more feasible – and therefore desirable – outcome.