Two-dimensional text: visualising contextualized language models

We will use the ruwikiruscorpora_tokens_elmo_1024_2019 ELMo model to infer contextualized embeddings for words in your query. Then, for each embeddings, we will find most similar words among 10 000 most frequent other words in this model's vocabulary. Since contextualized architectures do not store non-contextual word embeddings, we generated them beforehand by averaging contextualized embeddings of all occurrences of these words in the training corpus of the ruwikiruscorpora_tokens_elmo_1024_2019 model.

Lexical substitutes are also known as paradigmatic replacements. These are the words which can in theory replace the corresponding word in your input sentence.

Substitutes will change depending on the context. The larger is the substitute font size, the more certain ELMo is about this word.