A curated list of resources dedicated to Natural Language Processing

Prominent NLP Research Labs

The Berkeley NLP Group - Notable contributions include a tool to reconstruct long dead languages, referenced here and by taking corpora from 637 languages currently spoken in Asia and the Pacific and recreating their descendant.
Language Technologies Institute, Carnegie Mellon University - Notable projects include Avenue Project, a syntax driven machine translation system for endangered lnaguages like Quechua and Aymara and previously, Noah's Ark which created AQMAR to imporve NLP tools for Arabic.
NLP research group, Columbia University - Responsible for creating BOLT ( interactive error handling for speech translation systems) and an un-named project to characterize laughter in dialogue.
The Center or Language and Speech Processing, John Hopkins University - Recently in the news for developing speech recognition software to create a diagnostic test or Parkinson's Disease, here.
The Stanford Nautral Language Processing Group- One of the top NLP research labs in the world, notable for creating Stanford CoreNLP and their coreference resolution system

Reading Content

216
Hands-On NLTK Tutorial - NLTK Tutorials, Jupyter notebooks
176
NLP in Python - Collection of Github notebooks
Machine Learning 101 from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike
AI Playbook - a16z AI playbook is a great link to forward to your managers or content for your presentations
Ruder's Blog by Sebastian Ruder for commentary on the best of NLP Research
Natural Language Processing by Hal Daumé III

Videos and Online Courses

14019
Deep Natural Language Processing - Lectures series from Oxford
4502
Deep NLP Course by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, language models and so on.
Intro to Artificial Intelligence - Udacity course which touches upon NLP as well
Deep Learning for Natural Language Processing (cs224-n) - Richard Socher and Christopher Manning's Stanford Course
Neural Networks for NLP - Carnegie Mellon Language Technology Institute there

Books

Libraries

13903
spaCy - Industrial strength NLP with Python and Cython :+1:
8777
NLP Compromise - Natural Language processing in the browser
8726
Natural - general natural language facilities for node
6668
AllenNLP - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
6663
Flair - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
5341
Knwl.js - A Natural Language Processor in JS
2923
StarSpace - a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification
2861
NLP.js - An NLP library for building bots
2814
Snips NLU - A production ready library for intent parsing
2265
MIT Information Extraction Toolkit - C, C++, and Python tools for named entity recognition and relation extraction
2164
Twitter-text - A JavaScript implementation of Twitter's text processing library
2015
NLP Architect - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
1791
Retext - Extensible system for analyzing and manipulating natural language
1646
GluonNLP - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks.
1374
PyTorch-NLP - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
1220
textacy - Higher level NLP built on spaCy
981
scattertext - Python library to produce d3 visualizations of how language differs between corpora
785
Kashgari - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
683
Clojure-openNLP - Natural Language Processing in Clojure (opennlp)
602
text2vec - Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
535
MeTA - MeTA : ModErn Text Analysis is a C++ Data Sciences Toolkit that facilitates mining big text data.
465
ReVerb Web-Scale Open Information Extraction
460
Epic - Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models.
435
BigARTM - a fast library for topic modelling
379
PyNLPl - Python Natural Language Processing Library. General purpose NLP library for Python. Also contains some specific modules for parsing common NLP formats, most notably for FoLiA, but also ARPA language models, Moses phrasetables, GIZA++ alignments.
352
CogcompNLP - Core libraries developed in the U of Illinois' Cognitive Computation Group.
316
Word Forms - Word forms can accurately generate all possible forms of an English word
257
whatlang — Natural language recognition library based on trigrams
255
snips-nlu-rs - A production ready library for intent parsing
235
wordVectors - An R package for creating and exploring word2vec and other word embedding models
197
Rosetta - Text processing tools and wrappers (e.g. Vowpal Wabbit)
196
Poplar - A web-based annotation tool for natural language processing (NLP)
194
Chazutsu - A library for downloading&parsing standard NLP research datasets
183
BLLIP Parser - BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
170
Infections-clj - Rails-like inflection library for Clojure and ClojureScript
158
sentiment_classifier - Sentiment Classification using Word Sense Disambiguation and WordNet Reader
143
postagga - A library to parse natural language in Clojure and ClojureScript
121
jProcessing - Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification
114
jPTDP - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
110
RDRPOSTagger - A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages.
91
colibri-core - C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
76
dfr-browser - Creates d3 visualizations for browsing topic models of text in a web browser.
67
OpenRegex An efficient and flexible token-based regular expression language and engine.
58
Saul - Library for developing NLP systems, including built in modules like SRL, POS, etc.
49
frog - Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
42
ucto - Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
41
Multilingual Latent Dirichlet Allocation (LDA) - A multilingual and extensible document clustering pipeline
37
dfrtopics - R package for exploring topic models of text.
33
Lingua A language detection library for Kotlin and Java, suitable for long and short text alike
27
RMallet - R package to interface with the Java machine learning tool MALLET
15
ATR4S - Toolkit with state-of-the-art automatic term recognition methods.
11
libfolia - C++ library for the FoLiA format
11
word2vec-scala - Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy.
6
tm - Implementation of topic modeling based on regularized multilingual PLSA.
2
Kotidgy — an index-based text data generator written in Kotlin
TextBlob - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both :+1:
gensim - Python library to conduct unsupervised semantic modelling from plain text :+1:
CRF++ - Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
CRFsuite - CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
MALLET - MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.

Services

683
Wit-ai - Natural Language Interface for apps and devices
Amazon Comprehend - NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
Google Cloud Natural Language API - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional).
ParallelDots - High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
Textalytic - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more

Annotation Tools

1437
doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
169
Anafora is free and open source, web-based raw text annotation tool
GATE - General Architecture and Text Engineering is 15+ years old, free and open source
brat - brat rapid annotation tool is an online environment for collaborative text annotation
tagtog, costs $
prodigy is an annotation tool powered by active learning, costs $
LightTag - Hosted and managed text annotation tool for teams, costs $
rstWeb - open source local or online tool for discourse tree annotations

Sentence and Language Model Based Word Embeddings

sense2vec - on word sense disambiguation
Skip Thought Vectors - word representation method
Adaptive skip-gram - similar approach, with adaptive properties
Sequence to Sequence Learning - word vectors for machine translation

Question Answering and Knowledge Extraction

3297
DrQA - Open Domain Question Answering work by Facebook Research on Wikipedia data
294
Document-QA - Simple and Effective Multi-Paragraph Reading Comprehension by AllenAI

Datasets

3155
nlp-datasets great collection of nlp datasets

Multilingual NLP Frameworks

218
NLP-Cube : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
183
UDPipe is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing.

Libraries

KoNLPy - Python package for Korean natural language processing.
Mecab (Korean) - C++ library for Korean NLP
KoalaNLP - Scala library for Korean Natural Language Processing.
KoNLP - R package for Korean Natural language processing

Datasets

KAIST Corpus - A corpus from the Korea Advanced Institute of Science and Technology in Korean.
Chosun Ilbo archive - dataset in Korean from one of the major newspapers in South Korea, the Chosun Ilbo.

Libraries

40
goarabic - Go package for Arabic text processing
15
jsastem - Javascript for Arabic stemming
PyArabic - Python libraries for Arabic

Datasets

82
Arabic Stopwords - A list of Arabic stopwords from various resources
63
Multidomain Datasets - Largest Available Multi-Domain Resources for Arabic Sentiment Analysis
15
LABR - LArge Arabic Book Reviews dataset

Libraries

4208
SnowNLP - Python package for Chinese NLP
2116
FudanNLP - Java library for Chinese text processing
jieba - Python package for Words Segmentation Utilities in Chinese

NLP in German

30
German-NLP - Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

Data

Word and Sentence Embeddings

Data, Corpora and Treebanks

22
Hindi Movie Reviews Dataset ~1k Samples, 3 polarity classes
4
IIT Patna Hindi ABSA Dataset 5.4k Samples, 12 Domains, 4k aspect terms, aspect and sentence level polarity in 4 classes
Hindi Dependency Treebank - A multi-representational multi-layered treebank for Hindi and Urdu
Parallel Universal Dependencies Treebank in Hindi - A smaller part of the above-mentioned treebank.
NLTK Corpus 60k Words POS Tagged, Bangla, Hindi, Marathi, Telugu
Bangla ABSA 5.5k Samples, 2 Domains, 10 aspect terms
IIT Patna Movie Review Sentiment Dataset 2k Samples, 3 polarity labels
BBC News Hindi Dataset 4.3k Samples, 14 classes

Corpora/Datasets that need a login/access can be gained via email

SAIL 2015 Twitter and Facebook labelled sentiment samples in Hindi, Bengali, Tamil, Telugu.
IIT Bombay NLP Resources Sentiwordnet, Movie and Tourism parallel labelled corpora, polarity labelled sense annotated corpus, Marathi polarity labelled corpus.

Language Models and Word Embeddings

Libraries and Tooling

339
iNLTK - A Natural Language Toolkit for Indic Languages (Indian subcontinent languages) built on top of Pytorch/Fastai, which aims to provide out of the box support for common NLP tasks.
194
Anoop Kunchukuttan 18 Languages, whole host of features from tokenization to translation
1
Multi-Task Deep Morphological Analyzer Deep Network based Morphological Parser for Hindi and Urdu
SivaReddy's Dependency Parser Dependency Parser and Pos Tagger for Kannada, Hindi and Telugu. Python3 Port

Libraries

367
PyThaiNLP - Thai NLP in Python Package
137
CutKum - Word segmentation with deep learning in TensorFlow
14
JTCC - A character cluster library in Java
Thai Language Toolkit - Based on a paper by Wirote Aroonmanakun in 2002 with included dataset
SynThai - Word segmentation and POS tagging using deep learning in Python

Data

Inter-BEST - A text corpus with 5 million words with word segmentation
Prime Minister 29 - Dataset containing speeches of the current Prime Minister of Thailand

NLP in Danish

10
DaNLP - NLP resources in Danish

Libraries

574
underthesea - Vietnamese NLP Toolkit
176
vn.vitk - A Vietnamese Text Processing Toolkit
148
VnCoreNLP - A Vietnamese natural language processing toolkit

Data

15
UD_Vietnamese - Vietnamese Universal Dependency Treebank
Vietnamese treebank - 10,000 sentences for the constituency parsing task
BKTreeBank - a Vietnamese Dependency Treebank
VIVOS - a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab
VNTQcorpus(big).txt - 1.75 million sentences in news

Datasets

38
IDN for PoS tagging: This corpus contains 10K sentences and 250K word tokens
9
IndoSum for text summarization and classification both
PANL10N for PoS tagging: 39K sentences and 900K word tokens
Wordnet-Bahasa - large, free, semantic dictionary

Libraries & Embedding

Комментарии

comments powered by Disqus