Monday 15th December at 2p.m. Paris time, Nihed Bendahman, Ph.D. Candidate has defended her thesis named “Evaluation and mitigation of hallucinations in automatic summarization in the specific context of legal documents”. Her thesis defense took place at the IRIT Research Laboratory, in Toulouse, France. Take a look at the summary below.
Summary
Legal monitoring is a particularly time-consuming activity: to keep up with a constantly evolving law, experts must read dozens of articles every day, identify new content, write analyses, and produce news articles with their summaries.
This thesis focuses specifically on this last step, namely automatic legal summarizaon. Legal documents and their summaries are characterized by a high concentration of entities of interest, often linked by domain-specific relations. With the fast progress of large language models, the integration of AI tools into the monitoring process is becoming widely possible. However, these models remain subject to “hallucinations”, which can introduce inaccurate and out-of-context information. The aim of this thesis is to study this phenomenon and to propose methods for generating summaries to reduce hallucinations.
To this end, we propose a manually annotated dataset of legal relational triples and explore the use of entities and relational triples, both during training and inference, evaluating several settings of entities and triples driven summarization in order to reduce hallucinations and increase the factuality of the generated summaries.
The jury was composed of:
Examiners:
- Fournier Sébastien – Examiner, Aix-Marseille University
- Cabanac Guillaume – Examiner, University of Toulouse
- Sallaberry Christian – Examiner, University of Pau and Pays de l’Adour
- Névéol Aurélie – Advisor, CNRS Ile-de-France Gif-sur-Yvette
- Calabretto Sylvie – Advisor, INSA Lyon
Supervisors:
- Pinel-Sauvagnat Karen – Director, University of Toulouse
- Gilles Hubert – Co-Director, University of Toulouse
- Billami Mokhtar Boumedyen – Co-Director in the socio-economic field, Berger-Levrault
Guest:
- Bortolaso Christophe – Director of Applied Research, Berger-Levrault
Industrial Perspectives:
The system developed from various experiments will be integrated into the products offered by Berger-Levrault, with the aim of automating the manual writing of legal summaries.
Some of her Scientific Publications:
- (2025) EUR-Lex-Triples: A Legal Relation Extraction Dataset from European Legislation: https://link.springer.com/chapter/10.1007/978-3-032-05409-8_25
- (2025) Not all Hallucinations are Good to Throw Away When it Comes to Legal Abstractive Summarization: https://aclanthology.org/2025.naacl-long.275/
- (2024) IRIT-Berger-Levrault at SemEval-2024: How Sensitive Sentence Embeddings are to Hallucinations?: https://aclanthology.org/2024.semeval-1.86/
