Following An Investigation into the Effect of Control Tokens on Text Simplification it seems that the most effective way to add control token is to add unique tokens to the tokenizer vocabulary.
It also seem that model can handle quite granular control tokens. I’ll start with tokens built like:
<|readability:value|>
e.g.:
<|readability:5|>
To do that however I’ll need to add all 20 possible control tokens to the tokenizer vocabulary.
Then I’ll need to resize the embedding space with
Python
Output
However, since I am using LoRA for the finetuning, all model’s weight will be frozen, and thus the new embeddings can’t be learned.
So I’ll need to unfreeze the embedding and unembedding layer to learn representations for this layers.