Llamantino can’t learn the new token embedding representation with just 100 examples. Still, even in zero shot without control token, can’t solve the task. He repeats the system prompt, ask himself a new question and answers that completely ignoring the original task.
If given a control prompt, produces gibberish.
With 10000 examples, he actually learns the embeddings somewhat and if prompted with just the control token produces an “explanation” of what those tokens are (mostly topic-related). Still, can’t solve the task, the behaviour is the same: he repeats the system prompt and tries to answer a new question that he provide himself.
Now trying qwen BASE, which I think is the one that didn’t do anything more than pure simple language modeling.