24.12.28 - New TODO

Significant Repository
Data From here

What has been done and tried:

1. We took all Simplification Corpora available for Italian;

2. We analyzed the performance of various model on their tests set in a zero-shot settings:

   1. LLaMAntino-2-7b
   2. ANITA-9b
   3. Italia-9b

3. We saw taht LLaMAntino was marginally better than the others on various scores like Read-IT, SARI, Bleu, BertScore and sBert.

4. We tried fine-tuning all the previous models (plus Minerva-7B) to see if their performances improved: LLaMAntino performed ok, and improved, the other performed poorly.

5. We took the Italian Wikipedia, and sampled 1000 sentences. We gave them in 0-shot to LLaMAntino-2-7b and, using a decoding technique, we sampled 10 different simplification for each sentence.

6. We analyzed the distribution of the read-it scores on the sentences.

Datasets for the Fine-Tuning

Dataset	Link	Domain	Dimension	Train/test split
SIMPITIKI - Wikipedia	github	Wikipedia + Public Administration	563 Sentences	512/51
PACCSS-IT	italianlp.it	Web	63.000 Sentences Filtered: 9954	8940/994
Terence	italianlp.it	Novels for Children, Educational	1002 Sentences	911 / 101
Admin-it (RS, RD, OP)	github	Public Administration	685 Sentences	636/49
Teacher	italianlp.it	Educational	169 Sentences	153/17

Results

Model	SARI	Bleu	Avg BertScore	Technique
swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA	39.35	0.07	0.80	0-shot
swap-uniba/LLaMAntino-2-7b-hf-dolly-ITA	40.99	0.18	0.81	0-shot
iGeniusAI/Italia-9B-Instruct-v0.1	39.35	0.12	0.79	0-shot
LLaMAntino-2-7b-simp-finetuned-ITA	47.76	0.04	0.72	ft
LLaMAntino-2-7b-simp-finetuned-ITA-no-paccssit	48.07	0.20	0.84	ft
LLaMAntino-3-ANITA-finetuned-ITA-no-paccssit	39.28	0.01	0.53	ft
Italia-9b-finetuned-ita-no-paccssit	35.50	0.002	0.52	ft
Minerva-7B-finetuned-ita-no-paccssit	36.55	0.003	0.51	ft

Useful Commands

taskset --cpu-list X-Y