Optimizing Statistical Machine Translation for Text Simplification

From @xuOptimizingStatisticalMachine2016

The methodology poses text simplification as a paraphrasing problem: given a text, rewrite it under the constraint that the output should be simpler than the input, while preserving as much as the input as possible and maintaining the well-formedness of the input.

Sari

Sari compares system output against references and against the
input sentence. It explicitly measures the goodness of words that are added, deleted and kept by the system.

Addition operations are rewarded where the system output $O$ was not in input $I$ but occurred in any of the references $R$ i.e. $O \cap \overset{ˉ}{I} \cap R$ .

By using $#_{g} (\cdot)$ as a binary operator of occurence of n-grams $g$ in a given text:

#_{g} (O \cap \overset{ˉ}{I}) = ma x (#_{g} (O) - #_{g} (I), 0)

#_{g} (R \cap \overset{ˉ}{I}) = ma x (#_{g} (R) - #_{g} (I), 0)

Then, we can define a n-gram precision $p (n)$ and recall $r (n)$ for addition operations:

p_{a dd} (n) = \frac{\sum _{g \in O} min ( # _{g} ( O \cap I ˉ ) , # _{g} ( R ))}{\sum _{g \in O} # _{g} ( O \cap I ˉ )}

r_{a dd} (n) = \frac{\sum _{g \in O} min ( # _{g} ( O \cap I ˉ ) , # _{g} ( R ))}{\sum _{g \in O} # _{g} ( R \cap I ˉ )}

Also, words that are retained in both the output and references should be rewarded. $R^{'}$ is used to mark the n-grams counts over $R$ with fractions, e.g. if a unigram occurs 2 out of the total $r$ references the its count is weighted by a $\frac{2}{r}$ in computation of the precision and recall:

p_{k ee p} (n) = \frac{\sum _{g \in I} min ( # _{g} ( I \cap O ) , # _{g} ( I \cap R ^{'} ))}{\sum _{g \in I} # _{g} ( I \cap O )}

r_{k ee p} (n) = \frac{\sum _{g \in I} min ( # _{g} ( I \cap O ) , # _{g} ( I \cap R ^{'} ))}{\sum _{g \in I} # _{g} ( I \cap R ^{'} )}

where:

#_{g} (I \cap O) = min (#_{g} (I), #_{g} (O))

#_{g} (I \cap R^{'}) = min (#_{g} (I), #_{g} (R) / r)

For deletion only precision is used because over-deleting hurts readability much more significantly than not deleting:

p_{d e l} (n) = \frac{\sum _{g \in I} min ( # _{g} ( I \cap O ˉ ) , # _{g} ( I \cap R ^{'} ˉ ))}{\sum _{g \in I} # _{g} ( I \cap O ˉ )}

Where:

#_{g} (I \cap \overset{ˉ}{O}) = ma x (#_{g} (I) - #_{g} (O), 0)

#_{g} (I \cap \overset{ˉ}{R^{'}}) = ma x (#_{g} (I) - #_{g} (R) / r, 0)

Then, for add, keep and del the F-Score is calculated through the use of the precisions and recalls.
Finally:

S A R I = d_{1} F_{a dd} + d_{2} F_{k ee p} + d_{3} P_{d e l}

where $d_{1} = d_{2} = d_{3} = 1/3$

Results

SARI correlates highly with human judgments of simplicity, while BLEU exhibits higher correlation on grammaticality and meaning preservation (obviously, since IT’S NOT designed to do simplification).

from @alva-manchegoDataDrivenSentenceSimplification2020
Sari is only useful as a proxy for simplicity gain assessment, limited to lexical simplifications and short distance reordering, despite more text transformations being possible.

📚 Michele's Notes

Explorer

Optimizing Statistical Machine Translation for Text Simplification

Sari

Results

Graph View

Table of Contents

Backlinks