24.11.19 - Discussion with Felice

Re-analyze all the outputs and the test set with Read-IT, instead of using the document read-it values, do an average of the read-it sentence values.

Human Evaluation

Study how to perform human evaluation of text simplification (or paraphrasing of which TS is a special case of).

We need to evaluate at each level (0-shot, after fine-tuning and after DPO) if the text are simpler than the original. If there is no clear indication whether at one step one works best, than it’s useful to ask comparative questions between two output at different step and ask which is simpler.

It’s also useful to ask if the semantics of the text is maintained trough the simplification or something is missing / it’s added.

To do so, we can present the originals and the output as two sentence without informing the user that the output is produced from the originals, and ask them which is simpler and whether one has more relevant information than the other.

If a DPO preference dataset is built, is then useful to ask humans which of the option is preferred to evaluate whether Read-IT score (that is used to build the preference dataset correlates with human preference).

Building an Italian Target Reading Level Dataset from Scratch

We take the Italian Wikipedia Dataset and we do a zero-shot text-simplification (or paraphrases). For each input we sample ten different outputs (temperature sampling, prompting, decoding technique, this is yet to see).

We use LLaMAntino-2 which is the most “virgin” dataset that have gone trough less steps of training.

Then, we evaluate the ten different outputs using Read-IT and SBERT. The first for having a measure of simplification in order to tag, by binning its score, the sentences at different reading levels, the second to see whether the sentence maintains enough of the semantics of the original text.

To understand what level of similarity is acceptable use SBERT on the training set for text simplification between originals and simplified to see how similar are them.

---

24.11.26

Research about the technical stuff

SBERT information are here.

To install:

pip install -U sentence-transformers

To use as en embedder:

Python

from sentence_transformers import SentenceTransformer sentences = ["This is an example sentence", "Each sentence is converted"] model = SentenceTransformer('sentence-transformers/distiluse-base-multilingual-cased-v1') embeddings = model.encode(sentences) print(embeddings)

Output

To use for sentence similarity:

Python

from sentence_transformers import SentenceTransformer model = SentenceTransformer("multi-qa-mpnet-base-cos-v1") query_embedding = model.encode("How big is London") passage_embeddings = model.encode([ "London is known for its financial district", "London has 9,787,426 inhabitants at the 2011 census", "The United Kingdom is the fourth largest exporter of goods in the world", ]) similarity = model.similarity(query_embedding, passage_embeddings) # => tensor([[0.4659, 0.6142, 0.2697]])

Output

To cite:

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "http://arxiv.org/abs/1908.10084",
}