Emily Theophilou, Cansu Koyuturk, Mona Yavari, Sathya Bursic, Gregor Donabauer, Alessia Telari, Alessia Testa, Raffaele Boiano, Davinia Hernandez-Leo, Martin Ruskov, Davide Taibi, Alessandro Gabbiadini and Dimitri Ognibene, Learning to Prompt in the Classroom to Understand AI Limits: A pilot study, AIxIA 2023 – Advances in Artificial Intelligence XXI International Conference of the Italian Association for Artificial Intelligence, Roma, Italy, November 6-9, 2023

Abstract: Artificial intelligence's progress holds great promise in assisting society in addressing health, climate, and other pressing societal issues. In particular Large Language Models (LLM) and the derived chatbots, like ChatGPT, have highly improved the natural language processing capabilities of AI systems allowing them to process an unprecedented amount of unstructured data. The consequent hype has also backfired, raising negative sentiment even after novel AI methods’ surprising contributions (e.g. health and genetics). One of the causes, but also an important issue per se, is the rising and misleading feeling of being able to access and process any form of knowledge to solve problems in any domain with no effort or previous expertise in AI or problem domain, disregarding current LLMs limits, such as hallucinations, limited understanding, and reasoning limits. Acknowledging AI fallibility is crucial to address the impact of dogmatic overconfidence in possibly erroneous suggestions generated by LLMs. At the same time, it can reduce fear and other negative attitudes toward AI. This can be achieved by performing extended AI literacy interventions that allow the public to understand such LLM limits and learn how to use them in a more effective manner, i.e. learning to “prompt”. With this aim, a pilot educational intervention was performed in a high school with 30 students. It involved (i) presenting high-level concepts about intelligence, AI, and LLM, (ii) an initial naive practice with ChatGPT in a non-trivial task, i.e. creating a natural educational conversation, and finally (iii) applying currently-accepted prompting strategies. Encouraging preliminary results have been collected such as students reporting (a) high appreciation of the activity, (b) improved quality of the interaction with the LLM during the educational activity, (c) decreased negative sentiments toward AI, (d) increased understanding of limitations and specifically (d.i) unreliability, (d.ii) limited understanding of commands resulting in unsatisfying or repeating responses, (d.iii) limited presentation flexibility. We aim to study factors that impact AI acceptance and to refine and repeat this activity in more controlled settings.

https://arxiv.org/pdf/2307.01540.pdf