In Examining the Inductive Bias of NLMs with Artificial Languages and its code, to create the language they’ve created a Probabilistic context-free grammar in which the weights assigned to each production were chosen manually through experimentation.
- extracted open-class pseudo-words which are phonotactically plausible in English.
- They’ve inflected the words manually for singular and plural.
- Conjugated the verbs for present and past tense using English rules.
- They manually modeled Morphological Agreement between verbs and subjects numbers.
From the grammar in their code, they define a production as a pattern like:
1 S NP_Subj_P VP_P 1
1 VP_S VP_Past_S
in which the first number is the assigned weight
Then, they define a list of nouns, verbs, pronouns etc. to fill the productions.
Idea. We could extract “productions” from one or more tree-bank, so that we have gold-labels.
In Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models, to create an artificial language they use a Vocabulary filled with integers as the language symbols. Then, they created artificial sentences by approximating in various way the word distributions:
- Uniformly, as a baseline. Where the probability of a token of being sampled is:
- By following Zipf and assuming an :
- These aforementioned methods sample words independently from one another. So they used also a log-linear method, where the tokens in a sentence are drawn from the following probability distribution:
Where is the discourse vector of the sentence and is the word vector of token . The discourse vector represent the of the sentence, and determines the uni-gram distribution over the vocabulary. They initialized the word vectors with a normal distribution, and the discourse vector are drawn from a normal distribution each time they generate a sentence. Both vectors have size 10.
To model latent dependency structures, i.e. morpho-syntactic and syntactic structures within a language, they designed an algorithm that generate structured sentences given a set of token sampled like described before. They pair half of the token (heads) with the other half of the tokens (tails). A pair of head and tail can be represented in right and left brackets with the same integer (e.g. "", and "").
After determining the sentence length they first sample of head and tail and arrange them using one of the following structures:
- Flat Dependency: they arrange the tokens randomly while keeping the right order of the brackets. e.g.: , , , . The dependency arcs can be crossed but relatively to each other the head of the dependency always comes before the tail of the dependency in the sentence.
- Nesting Dependency: Here, brackets are nested hierarchically, like in Natural Language. e.g.: , , , , , .
In On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets, they tested different artificial datasets in pre-training for a RoBERTa. The tested dataset are created as follows:
- Uniform. The artificial dataset is constructed by sampling 29995 integers from the vocabulary at random to form a sentence. The model learns to randomly pick tokens during pre-training.
- Uni-gram. This dataset is created to match the uni-gram distribution on the real-world English tokens calculated on the English Wikipedia.
- Bi-gram. Same as before but with the bi-gram distribution of the tokens.
Then on Explicit Dependencies: - Flat Parenthesis. They sample tokens where is the sequence length. They sample from the integer’s uni-gram distribution. Then each token is duplicated so that it occurs evenly, then the sequence is shuffled. You can view each couple of integers as a pair of depending integers and their distance the length of dependency.
- Nesting Parenthesis. Is a special case of the flat where all the arcs connecting pair of dependencies do not cross each other, forming a hierarchical structure.
Then on Implicit Dependencies: - Shuffle-N: Each sequence in this dataset is formed by concatenating blocks of N integers. In each blocks, N consecutive integers from 0 - 29994 are sampled, then are shuffled to form a block. Integers in different blocks are sampled independently. The model, to solve the MLM task needs to focus on context no longer than tokens (the previous tokens and the next token w.r.t. of the current token). It is not required that each block have non-overlapping integers when constructing the dataset. Each integer could appear in different block in the same sequence, forming an explicit dependency. The dataset is designed to capture the constituent structure of Natural Language, and different sizes can capture the variable length that a constituent may span in the human language.