Naeval — comparing quality and performance of NLP systems for Russian language. Naeval is used to evaluate project Natasha components: Razdel, Navec, Slovnet.

Install

Naeval supports Python 3.7+

$ pip install naeval

Documentation

Materials are in Russian:

Naeval page on natasha.github.io
Naeval section of Datafest 2020 talk

Models

Model	Tags	Description
DeepPavlov NER `#`	`ner`	BiLSTM-CRF NER trained on Collection5. Original repo, docs, paper
DeepPavlov BERT NER `#`	`ner`	Current SOTA for Russian language. Docs, video
DeepPavlov Slavic BERT NER `#`	`ner`	DeepPavlov solution for BSNLP-2019. Paper
DeepPavlov Morph `#`	`morph`	Docs
DeepPavlov BERT Morph `#`	`morph`	Docs
DeepPavlov BERT Syntax `#`	`syntax`	BERT + biaffine head. Docs
Slovnet NER `#`	`ner`
Slovnet BERT NER `#`	`ner`
Slovnet Morph `#`	`morph`
Slovnet BERT Morph `#`	`morph`
Slovnet Syntax `#`	`syntax`
Slovnet BERT Syntax `#`	`syntax`
PullEnti `#`	`ner` `morph`	First place on factRuEval-2016, super sophisticated ruled based system
Stanza `#`	`ner` `morph` `syntax`	Tool by Stanford NLP released in 2020. Paper
SpaCy `#`	`token` `sent` `ner` `morph` `syntax`	Uses Russian models trained by @buriy
Texterra `#`	`morph` `syntax` `ner` `token` `sent`	Multifunctional NLP solution by ISP RAS
Tomita `#`	`ner`	GLR-parser by Yandex, only implementation for person names is publicly available
MITIE `#`	`ner`	Engine developed at MIT + third party model for Russian language
RuPosTagger `#`	`morph`	CRF tagger, part of Solarix project
RNNMorph `#`	`morph`	First place solution on morphoRuEval-2017. Post on Habr
Maru `#`	`morph`
UDPipe `#`	`morph` `syntax`	Model trained on SynTagRus
NLTK `#`	`token` `sent`	Multifunctional library, provides model for Russian text segmentation. Docs
MyStem `#`	`token` `morph`	Wrapper for Yandex morphological analyzers
Moses `#`	`token` `sent`	Wrapper for Perl Moses utils
SegTok `#`	`token` `sent`
RuTokenizer `#`	`token`
Razdel `#`	`token` `sent`
Spacy Russian Tokenizer `#`	`token` `sent`	Spacy segmentation pipeline for Russian texts by @aatimofeev
RuSentTokenizer `#`	`sent`	DeepPavlov sentence segmentation

Tokenization

See Razdel evalualtion section for more info.

	corpora		syntag		gicrya		rnc
	errors	time	errors	time	errors	time	errors	time
re.findall(\w+\|\d+\|\p+)	24	0.5	16	0.5	19	0.4	60	0.4
spacy	26	6.2	13	5.8	14	4.1	32	3.9
nltk.word_tokenize	60	3.4	256	3.3	75	2.7	199	2.9
mystem	23	5.0	15	4.7	19	3.7	14	3.9
mosestokenizer	11	2.1	8	1.9	15	1.6	16	1.7
segtok.word_tokenize	16	2.3	8	2.3	14	1.8	9	1.8
aatimofeev/spacy_russian_tokenizer	17	48.7	4	51.1	5	39.5	20	52.2
koziev/rutokenizer	15	1.1	8	1.0	23	0.8	68	0.9
razdel.tokenize	9	2.9	9	2.8	3	2.0	16	2.2

Sentence segmentation

	corpora		syntag		gicrya		rnc
	errors	time	errors	time	errors	time	errors	time
re.split([.?!…])	114	0.9	53	0.6	63	0.7	130	1.0
segtok.split_single	106	17.8	36	13.4	1001	1.1	912	2.8
mosestokenizer	238	8.9	182	5.7	80	6.4	287	7.4
nltk.sent_tokenize	92	10.1	36	5.3	44	5.6	183	8.9
deeppavlov/rusenttokenize	57	10.9	10	7.9	56	6.8	119	7.0
razdel.sentenize	52	6.1	7	3.9	72	4.5	59	7.5

Pretrained embeddings

See Navec evalualtion section for more info.

	type	init, s	get, µs	disk, mb	ram, mb	vocab
hudlit_12B_500K_300d_100q	navec	1.1	21.6	50.6	95.3	500K
news_1B_250K_300d_100q	navec	0.8	20.7	25.4	47.7	250K
ruscorpora_upos_cbow_300_20_2019	w2v	3.3	1.4	220.6	236.1	189K
ruwikiruscorpora_upos_skipgram_300_2_2019	w2v	5.0	1.5	290.0	309.4	248K
tayga_upos_skipgram_300_2_2019	w2v	5.2	1.4	290.7	310.9	249K
tayga_none_fasttextcbow_300_10_2019	fasttext	8.0	13.4	2741.9	2746.9	192K
araneum_none_fasttextcbow_300_5_2018	fasttext	16.4	10.6	2752.1	2754.7	195K

	type	simlex	hj	rt	ae	ae2	lrwc
hudlit_12B_500K_300d_100q	navec	0.310	0.707	0.842	0.931	0.923	0.604
news_1B_250K_300d_100q	navec	0.230	0.590	0.784	0.866	0.861	0.589
ruscorpora_upos_cbow_300_20_2019	w2v	0.359	0.685	0.852	0.758	0.896	0.602
ruwikiruscorpora_upos_skipgram_300_2_2019	w2v	0.321	0.723	0.817	0.801	0.860	0.629
tayga_upos_skipgram_300_2_2019	w2v	0.429	0.749	0.871	0.771	0.899	0.639
tayga_none_fasttextcbow_300_10_2019	fasttext	0.369	0.639	0.793	0.682	0.813	0.536
araneum_none_fasttextcbow_300_5_2018	fasttext	0.349	0.671	0.801	0.706	0.793	0.579

Morphology taggers

See Slovnet evaluation section for more info.

	news	wiki	fiction	social	poetry
slovnet	0.961	0.815	0.905	0.807	0.664
slovnet_bert	0.982	0.884	0.990	0.890	0.856
deeppavlov	0.940	0.841	0.944	0.870	0.857
deeppavlov_bert	0.951	0.868	0.964	0.892	0.865
udpipe	0.918	0.811	0.957	0.870	0.776
spacy	0.964	0.849	0.942	0.857	0.784
stanza	0.934	0.831	0.940	0.873	0.825
rnnmorph	0.896	0.812	0.890	0.860	0.838
maru	0.894	0.808	0.887	0.861	0.840
rupostagger	0.673	0.645	0.661	0.641	0.636

	init, s	disk, mb	ram, mb	speed, it/s
slovnet	1.0	27	115	532.0
slovnet_bert	5.0	475	8087	285.0 (gpu)
deeppavlov	4.0	32	10240	90.0 (gpu)
deeppavlov_bert	20.0	1393	8704	85.0 (gpu)
udpipe	6.9	45	242	56.2
spacy	8.0	140	579	50.0
stanza	2.0	591	393	92.0
rnnmorph	8.7	10	289	16.6
maru	15.8	44	370	36.4
rupostagger	4.8	3	118	48.0

Syntax parser

	news		wiki		fiction		social		poetry
	uas	las	uas	las	uas	las	uas	las	uas	las
slovnet	0.907	0.880	0.775	0.718	0.806	0.776	0.726	0.656	0.542	0.469
slovnet_bert	0.965	0.936	0.891	0.828	0.958	0.940	0.846	0.782	0.776	0.706
deeppavlov_bert	0.962	0.910	0.882	0.786	0.963	0.929	0.844	0.761	0.784	0.691
udpipe	0.873	0.823	0.622	0.531	0.910	0.876	0.700	0.624	0.625	0.534
spacy	0.943	0.916	0.851	0.783	0.901	0.874	0.804	0.737	0.704	0.616
stanza	0.940	0.886	0.815	0.716	0.936	0.895	0.802	0.714	0.713	0.613

	init, s	disk, mb	ram, mb	speed, it/s
slovnet	1.0	27	125	450.0
slovnet_bert	5.0	504	3427	200.0 (gpu)
deeppavlov_bert	34.0	1427	8704	75.0 (gpu)
udpipe	6.9	45	242	56.2
spacy	9.0	140	579	41.0
stanza	3.0	591	890	12.0

NER

See Slovnet evalualtion section for more info.

	factru			gareev		ne5			bsnlp
f1	PER	LOC	ORG	PER	ORG	PER	LOC	ORG	PER	LOC	ORG
slovnet	0.959	0.915	0.825	0.977	0.899	0.984	0.973	0.951	0.944	0.834	0.718
slovnet_bert	0.973	0.928	0.831	0.991	0.911	0.996	0.989	0.976	0.960	0.838	0.733
deeppavlov	0.910	0.886	0.742	0.944	0.798	0.942	0.919	0.881	0.866	0.767	0.624
deeppavlov_bert	0.971	0.928	0.825	0.980	0.916	0.997	0.990	0.976	0.954	0.840	0.741
deeppavlov_slavic	0.956	0.884	0.714	0.976	0.776	0.984	0.817	0.761	0.965	0.925	0.831
pullenti	0.905	0.814	0.686	0.939	0.639	0.952	0.862	0.683	0.900	0.769	0.566
spacy	0.901	0.886	0.765	0.970	0.883	0.967	0.928	0.918	0.919	0.823	0.693
stanza	0.943	0.865	0.687	0.953	0.827	0.923	0.753	0.734	0.938	0.838	0.724
texterra	0.900	0.800	0.597	0.888	0.561	0.901	0.777	0.594	0.858	0.783	0.548
tomita	0.929			0.921		0.945			0.881
mitie	0.888	0.861	0.532	0.849	0.452	0.753	0.642	0.432	0.736	0.801	0.524

	init, s	disk, mb	ram, mb	speed, it/s
slovnet	1.0	27	205	25.3
slovnet_bert	5.0	473	9500	40.0 (gpu)
deeppavlov	5.9	1024	3072	24.3 (gpu)
deeppavlov_bert	34.5	2048	6144	13.1 (gpu)
deeppavlov_slavic	35.0	2048	4096	8.0 (gpu)
pullenti	2.9	16	253	6.0
spacy	8.0	140	625	8.0
stanza	3.0	591	11264	3.0 (gpu)
texterra	47.6	193	3379	4.0
tomita	2.0	64	63	29.8
mitie	28.3	327	261	32.8

Support

Chat — https://t.me/natural_language_processing
Issues — https://github.com/natasha/nerus/issues
Commercial support — https://lab.alexkuk.ru

Development

Dev env

python -m venv ~/.venvs/natasha-naeval
source ~/.venvs/natasha-naeval/bin/activate

pip install -r requirements/dev.txt
pip install -e .

python -m ipykernel install --user --name natasha-naeval

Lint

make lint

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Install

Documentation

Models

Tokenization

Sentence segmentation

Pretrained embeddings

Morphology taggers

Syntax parser

NER

Support

Development

Files

README.md

Latest commit

History

README.md

File metadata and controls

Install

Documentation

Models

Tokenization

Sentence segmentation

Pretrained embeddings

Morphology taggers

Syntax parser

NER

Support

Development