Models

All the models can be downloaded and used for free, under the CC-BY license (models in bold are available for queries via our web interface and API).

Contextualized embeddings

Static embeddings

The table can (and should) be scrolled horizontally!

Corpora

1. RNC: full Russian National Corpus;
2. Wikipedia: Russian Wikipedia dump for the corresponding date;
3. News: news stream from 1 500 primarily Russian-language news sites (about 30 million documents in total in the recent models);
4. Araneum Russicum Maximum: large web corpus of Russian compiled by Vladimir Benko in 2016 (download lemmatized and tagged corpus);
5. Taiga: open and structured Russian web corpus with morphological and syntax annotation (poetry subcorpus excluded); compiled by Tatyana Shavrina;
6. GeoWAC: a sample of documents in Russian from CommonCrawl dump, geographically balanced; compiled by Jonathan Dunn and Ben Adams;
7. Web: random Russian web pages crawled in December 2014, 9 million documents in total.

Corpus preprocessing

Prior to training, all the corpora were tokenized, split into sentences, lemmatized (except for the models containing "_tokens_" in their identifiers) and PoS-tagged using UDPipe (except for fastText and ELMo models). PoS tags comply to the Universal PoS Tags format (for instance, "печь_NOUN"). The table of conversion from the Mystem tags to the Universal tags is available here. Stop words (punctuation, conjunctions, pronouns, prepositions, particles, etc) were removed, except for the models containing "_tokens_" or "_func" in their identifiers.

Corpus preprocessing is described in detail in our tutorial (in Russian). You can also use ready-made Python scripts reproducing our pre-processing workflow on arbitrary raw Russian text: UDPipe variant and Mystem variant.

Some strong two-word collocations (bigrams) were merged into 1 token with the special character "::" (especially common for proper names), for instance, борис::гребенщиков_PROPN. For comparison, you can also download the model with no bigrams merged (ruwikiruscorpora-nobigrams_upos_skipgram_300_5_2018) and the model in which all the bigrams belonging to productive types were merged, independent of their frequencies (ruwikiruscorpora-superbigrams_skipgram_300_2_2018).

Vectors in the models starting from 2019 are not unit-normed. ELMo models are trained on corpora with minimalist pre-processing: tokenization and (optionally) lemmatization using UDPipe. No stop words were removed, PoS tags were not appended to words.

To work with ELMo models locally, we recommend our easy to use Python library.

Evaluation

Evaluating performance of distributional semantic models is a difficult problem in itself. The best way is arguably to test the models on the downstream tasks for which they are going to be eventually used (extrinsic evaluation). However, if one trains a general-purpose model (like the ones presented at this service), some intrinsic evaluation methods shoud be used, testing the general ability of the models to capture natural language regularities.

We evaluated our models using two well-known approaches:

1. Spearman correlation of model-produced pairwise similarities between words against a manually annotated test set. We used the RuSimLex965 test set based on the Multilingual SimLex999 dataset (our paper about RuSimLex965).
2. Accuracy in solving analogy problems. To test it, we employed the Russian adaptation of Google Analogies Dataset semantic part (translated by Tatyana Kononova; read more about the analogy task).

ELMo models are evaluated on the word sense disambiguation (WSD) task, using Human-Annotated Sense-Disambiguated Word Contexts for Russian test set (a.k.a. RUSSE'18). The final score is macro-F1 across all the test set words. Read more about evaluating ELMo models in our paper. Additionally, we use document pair classification task, with the ParaPhraser project corpus as the golden data source.

Download all our test sets (raw or PoS-tagged).

Who uses the RusVectōrēs models?

1. drafterleo developed a poetic search engine.

2. Boris Orekhov, associate professor at NRU HSE, created "vector rephrasings" of classic Russian literature works.

3. opennota developed a semantic search engine over Pushkin's poems.

4. Sberbank AI employs our models as baseline in their ClassicAI, a shared task on poetry generation.

5. Mikhail Kopotev's group in the Helsinki University automatically extracts constructions based on semantic similarity in the CoCoCo: Colligations, Collocations & Constructions project.

6. Olga Mitrofanova's group at the Saint-Petersburg State University works with the distributional semantic calculator to predict Russian collocations коллокаций with a given set of features.

7. ...

Have something to add to this list? Contact us!