Controlled Lexical Simplification idea

The idea is to inject bias into the decoding process in a soft-way. We want to bias the model next-token distribution towards a curated vocabulary of interest.

As a case-study we picked a lexical simplification case with the use of the Vocabolario di Base di Tullio de Mauro.

Instead of hard constraints, we implement a soft probabilistic reallocation of the model’s next token probabilities:

$$p^{\prime }=(1-\alpha )p+\alpha q$$

where:

• $p$ = base model distribution at current step (softmax of logits);
• $q$ = biasing prior distribution over the selected vocabulary;
• $\alpha$ = gating coefficient controlling the strength of the biasing;

The mixing gate $\alpha$ is determined dynamically at each step:

$$\alpha ={\alpha }_{max}\times \sigma (\frac{H(p)-\tau }{s})$$

where:

• $H(p)$ = entropy of the next token distribution;
• $\tau$ = entropy threshold after which the push towards the biased distribution “starts”;
• s = “softness” of the transition;
• $\sigma (\cdot )$ = logistic sigmoid;

i.e.:
Low entropy ($H\ll \tau$) model is confident, $\alpha \approx 0$
High entropy ($H\gg \tau$) model is uncertain, $\alpha \approx a_{max}$

Bias becomes context-sensitive, ensuring fluency, instead of a constant bias.

How to choose the prior $q$

We define the vocabulary containing the tokens we want to bias as $G$, and the tokens $g\in G$ are “green” tokens, whose probability will be biased.

Uniform $q$

$$q_i=\{\begin{array}{ll}\frac{1}{|G|}& i\in G\\ 0,& \text{otherwise}\end{array}$$

This spreads the steering mass equally across all the green tokens in the selected vocabulary . Works well if is small, otherwise each token in $G$ gets a tiny bump in mass, this has then little effect on the generation, especially if greedy decoding.

Re-normalizing model distribution on $G$

$$q_i=\{\begin{array}{ll}\frac{p_i}{{\sum }_{j\in G}{p}_j}& i\in G\\ 0,& \text{otherwise}\end{array}$$

This re-normalizes the distribution of probability that the model assigned to the set of green tokens in $G$, over themselves. This is like if the model only had the green tokens to choose from.

This is “smarter” than uniform, because increases the $\alpha$ of vocabulary tokens dynamically based on how likely the model already wanted to generate them. However, if no vocabulary tokens are in the high-probability region it can have very little effect in biasing.

Top-K Green Tokens

$$q_i=\{\begin{array}{ll}\frac{1}{|K|}& i\in \text{Top-}K\text{ green tokens}\\ 0,& \text{otherwise}\end{array}$$

Here we want to extract the model’s probability for the green tokens, pick the top-$K$ candidates and create a $q$ prior with uniform distribution over just those $K$.

This is similar to the uniform distribution method, but instead of steering all the green tokens in $G$, we just steer the top-$K$ candidates. This reduces the number of tokens where $\alpha$ is spreaded to.

Prior Works I’ve Found

• Logit Biasing (OpenAI API) - Constraints the generation of tokens in a sequence.
• Grid Beam Search - Lexical Constraints over tokens or sub-sequences, the model is forced to include them in their generation;
• GeDi uses a classifier to bias the model towards generating tokens that makes the sequence part of a certain class;
• FUDGE trains a small auxiliary discriminator that, at each step of generation, estimates how likely a continuation with that token will satisfy a target attribute in the future;
• PPLM uses gradient-based perturbations of hidden activations during decoding to steer generation toward an attribute, without fine-tuning the LM itself. Uses a classifier;
• ASR Shallow Fusion Similar to this, but for speech, uses extra model.

Claim

We propose a dynamic soft-mixing of base next-token probabilities and a curated lexicon prior, where mixing strength is gated by the model uncertainty measured as entropy.

Work (venue, year)	Core idea / control signal	Entropy/uncertainty gated?	External prior / classifier?	Hard vs. soft	What it shares with your idea	Key difference from yours
Lexically Constrained Decoding (Grid Beam Search) (ACL 2017)	Beam search that forces inclusion of specified words/phrases.	No	List of constraints	Hard	Decoding-time, lexicon-aware control.	Enforces constraints combinatorially; no probabilistic mixing; no entropy gate. aclanthology.org
Fast Lexically Constrained Decoding (DBA) (NAACL 2018)	Efficient variant (O(1) inconstraints).	No	Constraint list	Hard	As above, decoding-time & lexicon-aware.	Still hard constraints; no ppp/qqq mixing; no entropy gate. aclanthology.org
PPLM (ICLR 2020)	Gradient-based steering of hidden states via attribute model / BoW.	No	Yes (attribute model / BoW)	Soft	Inference-time steering of next-token behavior.	Uses backprop during decoding; not a closed-form ppp–qqq blend; no entropy gate. yosinski.com
FUDGE (NAACL 2021)	Future discriminator reweights next-token probs for target attribute.	No	Yes (aux. discriminator)	Soft	Adjusts next-token distribution at decode time.	Needs learned discriminator; no curated lexicon prior qqq; no entropy gating. aclanthology.org
GeDi (Findings of EMNLP 2021)	Generative discriminator (class-conditional LM) guides decoding.	No	Yes (CC-LM)	Soft	Probability reweighting at decode time.	Requires discriminator LM; no entropy gate; not a fixed lexicon prior. arxiv.org
Shallow-Fusion Contextual Biasing (ASR) (INTERSPEECH 2019)	Log-linear fusion of model with external LM/context bias.	No	Yes (external LM/bias list)	Soft	Conceptually akin to mixing model scores with a context prior.	ASR-oriented; fusion in log-space; no token-entropy gate; not a curated text lexicon GGG. isca-archive.org
Contrastive Decoding (ACL 2023)	Contrast expert vs. amateur LM logits to avoid degeneration.	No	Yes (second LM)	Soft	Modifies next-token distribution pre-sampling.	Objective is contrastive; needs two LMs; no entropy gate; no lexicon prior. aclanthology.org+1
Locally Typical Sampling (TACL 2023)	Info-theoretic sampling near local typicality.	Indirect (info-theory), no gate	No	Soft	Uses information-theoretic criterion at decode time.	Not attribute/lexicon-driven; no external qqq; no entropy-controlled mixing. direct.mit.edu
Uncertainty-Aware Contrastive Decoding (UCD) (Findings of ACL 2025)	Adjusts model contributions based on uncertainty during CD.	Yes (uncertainty-gated)	Yes (two models)	Soft	Same design motif: more control when uncertainty is high.	Uses uncertainty to modulate contrastive objective; not lexicon prior; no ppp–qqq blend. aclanthology.org
Entropy-Based Decoding for RAG (NAACL 2025 long)	Entropy-guided weighting between retrieved contexts vs parametric LM.	Yes (entropy-gated)	Yes (retrieved docs as priors)	Soft	Very close structurally: mixing signals weighted by entropy at decode time.	Aimed at RAG (document ensembles); not a curated token-level lexicon prior qqq. aclanthology.org+1
MCAD-EUC: Multi-Context Adaptive Decoding with Entropy-Based Uncertainty (Expert Systems with Applications, 2025)	Resolves knowledge conflicts by entropy-based uncertainty across contexts at decode time.	Yes (uncertainty-guided)	Yes (multiple contexts)	Soft	Same principle: uncertainty drives intervention strength.	Journal RAG/decoding context; not token-level lexicon prior or ppp–qqq mixing. sciencedirect.com

How to position the contribution

• You’re squarely in the decoding-time, soft control family (like PPLM/FUDGE/GeDi/CD), but with two distinguishing pieces:
  1. a curated lexicon prior $q$ over green tokens;
  2. a smooth entropy-gated mixing coefficient $\alpha$ at each step.
• Among peer-reviewed work, the closest design motif is the entropy/uncertainty-gated line (UCD; Entropy-Based Decoding for RAG; MCAD-EUC), where higher uncertainty ⇒ stronger intervention — but they don’t do lexicon-prior token mixing. Your method fills that gap. sciencedirect.com+3aclanthology.org+3aclanthology.org+3

If you want, I can generate a LaTeX table ready to paste in your paper (with concise bibkeys) or add a short “Related Work (Entropy-Gated Decoding)” paragraph that stitches these together.

Experiments

Basilines: vanilla decoding, logits bias;
variants: uniform vs p-renorm vs top-k
fixed $\alpha$ vs entropy-gated
with and without top-p and temperature

plotting per-step entropy $H(p)$ like greedy sequence, vs green-decoded sequence.