A new study suggests that ChatGPT, the artificial intelligence chatbot, can sometimes behave like a student, improvising, making mistakes, and refining its answers, when faced with a 2,400-year-old mathematical puzzle first described by Plato.
Education researchers Dr. Nadav Marco from the Hebrew University and Professor Andreas Stylianides of the University of Cambridge asked ChatGPT-4 to solve the “doubling the square” problem, a famous teaching moment in Plato’s dialogue where Socrates guides an uneducated boy to discover that the side of the new square should equal the diagonal of the original.
Instead of simply recalling this well-known solution, ChatGPT initially avoided the geometrical approach that Socrates used and instead attempted an algebraic method, an idea unknown in antiquity.
The researchers described this response as “learner-like,” suggesting that the language model seemed to be trying out its own hypotheses rather than directly repeating pre-learned material.
“When we face a new problem, our instinct is often to try things out based on our past experience,” Marco explained. “In our experiment, ChatGPT seemed to do something similar.”
Intriguingly, the chatbot resisted attempts to repeat the deliberate errors introduced by the researchers, such as the classic mistake of doubling side lengths instead of area. Despite prompting, it stuck to algebra before eventually providing the geometrical solution only after being told its answer was disappointing.
The researchers then posed variations of the puzzle, such as doubling the area of a rectangle or a triangle. In these cases, ChatGPT again defaulted to algebra but occasionally produced mistaken claims—such as incorrectly insisting that no geometrical solution exists for the rectangle case. Marco suggested such errors were unlikely to be drawn directly from its training data, indicating a degree of improvisation.
The study, published in the International Journal of Mathematical Education in Science and Technology, concludes that ChatGPT operates in what the researchers call a “zone of proximal development” (ZPD). It cannot always solve problems independently but may reach solutions with guided prompting.
Stylianides emphasised that this highlights the importance of teaching students not to take AI outputs at face value but to learn how to test and evaluate them, making proof reasoning and critical thinking more central to mathematics education in the AI era.
– Ends