24.10.10 - Text Simplification Experiments Report

I’ve collected all the Italian text simplification corpora I could:

Dataset	Link	Domain	Dimension	Train/test split
SIMPITIKI - Wikipedia	github	Wikipedia + Public Administration	563 Sentences	512/51
PACCSS-IT	italianlp.it	Web	63.000 Sentences Filtered: 9954	8940/994
Terence	italianlp.it	Novels for Children, Educational	1002 Sentences	911 / 101
Admin-it (RS, RD, OP)	github	Public Administration	685 Sentences	636/49
Teacher	italianlp.it	Educational	169 Sentences	153/17

Admin-it (OP) = Simpitiki PA

Totaling with a training set of 11152 documents and a test set of 1212 documents.
PACCSS-IT was filtered by using the difference of Read-IT between the two version so to have the most meaningful pairs possible.

With these aligned sentences was constructed a Dataset $\mathcal{D}$ containing the aligned sentences: the original and the simplified version.

Prompting Experiments

As a baseline, I’ve used two models trained for Italian text generation:

• swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA (From now on ANITA);
• swap-uniba/LLaMAntino-2-7b-hf-dolly-ITA (From now on LLaMAntino);
• iGeniusAI/Italia-9B-Instruct-v0.1 (From now on Italia);

The three models were used in inference and asked to simplify the sentences of the test set.

For all of them was provided a system prompt and a pattern on how to verbalize tasked. Those were followed for both this prompting experiments and for the fine-tuning later on.

ANITA’s prompting function:

Python

def formatting_func(example): text = "Sei un an assistente AI per la lingua Italiana di nome LLaMAntino-3 ANITA " \ "(Advanced Natural-based interaction for the ITAlian language)." \ " Rispondi nella lingua usata per la domanda in modo chiaro, semplice ed esaustivo."\ "Semplifica la seguente frase mantenendo il più possibile intatto il significato. "\ "Rispondi solamente con la frase semplificata: "\ f"### Input:\n{example['input']}\n\n" \ f"### Risposta:\n" return text

Output

LLaMAntino’s prompting function:

Python

def formatting_func(example): text = "Di seguito è riportata un'istruzione che descrive un'attività, accompagnata da un input che aggiunge ulteriore informazione. " \ f"Scrivi una risposta che completi adeguatamente la richiesta.\n\n" \ f"### Istruzione: Semplifica la seguente frase mantenendo il più possibile intatto il significato. "\ "Rispondi solamente con la frase semplificata.\n\n" \ f"### Input:\n{example['input']}\n\n" \ f"### Risposta:\n" return text

Output

Python

SYSTEM_PROMPT = """Il tuo nome è Modello Italia. Tu sei un'intelligenza artificiale, un modello di linguaggio naturale addestrato da iGenius su Leonardo, uno dei supercomputer più potenti al mondo.""" for index, row in df.iterrows(): messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Adesso ti verrà inviato un testo, tu dovrai simplificarne la struttura e il lessico mantenendo intanto il significato. Rispondi solamente col testo semplificato.\n###INPUT:{row['original']}\n###OUTPUT: "}, ] conv_template = tokenizer.apply_chat_template( messages, tokenize=False ) outputs = t_pipeline( conv_template, max_new_tokens=MAX_NEW_TOKENS, do_sample=True, temperature=TEMPERATURE, num_return_sequences=1, )

Output

Then, the models’ predictions were evaluated using SARI, bleu and Bertscore.

Model	SARI	Bleu	Avg BertScore
swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA	39.35	0.07	0.80
swap-uniba/LLaMAntino-2-7b-hf-dolly-ITA	40.99	0.18	0.81
iGeniusAI/Italia-9B-Instruct-v0.1	39.35	0.12	0.79

We can see that they achieve similar results with LLaMAntino being marginally better.

Fine-tuning

To try and have better performance on the task, a LoRA adapter was trained on LLaMAntino using 8 bit quantization using the same formatting function as the one used in pre-training (and before, during prompting analysis).

The model was trained using the full training set for three epochs. At the end the last checkpoint was evaluated and while SARI score increased, the other scores dropped. By looking at the prediction it seemed that the model performed some sort of an identity function on the input.

By looking at the training set, PACCSS-it, which is the majority of the dataset, seemed to have the most sentences where the input was very very similar to the output.
To prevent this, a new version of the model was trained using a smaller version of $\mathcal{D}$ that didn’t have PACCSIT-it (from now on pit) called ${\mathcal{D}}^{\neg pit}$.
The trained model was then evaluated on the original test set (which contains PACCSS-it documents) to comparely justly against the other models.

Model	SARI	Bleu	Avg BertScore
swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA	39.35	0.07	0.80
swap-uniba/LLaMAntino-2-7b-hf-dolly-ITA	40.99	0.18	0.81
iGeniusAI/Italia-9B-Instruct-v0.1	39.35	0.12	0.79
LLaMAntino-2-7b-simp-finetuned-ITA	47.76	0.04	0.72
LLaMAntino-2-7b-simp-finetuned-ITA-no-paccssit	48.07	0.20	0.84
LLaMAntino-3-ANITA-finetuned-ITA-no-paccssit	39.28	0.01	0.53
Italia-9b-finetuned-ita-no-paccssit	35.50	0.002	0.52

This new model seem to work better than any other models from these score perspective.
Fine-tuning ANITA and Italia on ${\mathcal{D}}^{\neg pit}$ was also tested but produced the worst result as of now.

Read-IT Analysis

All the model’s predictions, the original and the simplified dataset was analyzed with the help of Read-IT, a readability metrics that evaluate a text on four different score: one based on raw text feature, one on lexical feature, one on syntactic feature and one overall score.

Read-IT Plots

Original Sentences Read-IT distribution

Gold Simple Sentences Read-IT distribution

LLaMantino Predictions Read-IT distribution

ANITA predictions Read-IT distribution

Italia predictions Read-IT distribution

LLaMAntino Fine-tuned on $\mathcal{D}$

LLaMAntino Fine-tuned on ${\mathcal{D}}^{\neg pit}$

ANITA Fine-tuned on ${\mathcal{D}}^{\neg pit}$

Italia Fine-tuned on ${\mathcal{D}}^{\neg pit}$

Comparative Performance on Read-IT Base

Comparative Performance on Read-IT Lexical

Comparative Performance on Read-IT Syntax

Comparative Performance on Read-IT All

From the Read-IT point-of-view it seems that ANITA and LLaMAntino used in prompting outperforms both fine-tuned LLaMAntino versions, with the one trained on ${\mathcal{D}}^{\neg pit}$ being better than the other.
In fact, they have lower average scores but a bunch more outliers with respect to the model trained on ${\mathcal{D}}^{\neg pit}$.