Search⌘ K
AI Features
Home
Courses
Performing Natural Language Processing with R

Understanding quanteda

Explore the quanteda package to perform fast and efficient text preprocessing, create document-feature matrices, and conduct advanced text analyses including sentiment and topic modeling. Understand its terminology and how it compares with tm to better apply NLP concepts in R.

We'll cover the following...

What is quanteda?

The quanteda package is similar to the tm package in that both are frameworks for natural language processing. The quanteda package provides a larger set of tools and has greater capabilities, but requires a deeper understanding of NLP. The quanteda package is an R package for quantitative text analysis that provides a suite of tools for text preprocessing, document-feature matrix construction, text visualization, and statistical analysis of text data. It’s designed to be efficient for handling large corpora of text data, and provides a flexible framework for researchers to explore and analyze text data for a wide range of research questions. The quanteda package also supports various text analysis techniques, including topic modeling, sentiment analysis, and network analysis.

The quanteda package is also a more complex package and not a beginner framework to use when first learning NLP. However, now that we’ve spent time understanding the NLP basics and have an understanding of how to implement these concepts with the tm package, quanteda will be much more approachable.

Speed and efficiency

The quanteda package is generally faster and more memory-efficient than tm, particularly when dealing with large datasets.

Easy-to-use functions

The quanteda package has a more streamlined and user-friendly syntax for common text analysis tasks, making text analysis easier. It follows an easy-to-understand syntax:

quanteda Function List

Object

Related Functions

`corpus`

corpus_group( )

corpus_reshape( )

corpus_sample( )

corpus_segment( )

corpus_subset( )

corpus_trim( )

`char`

char_segment( )

char_trim( )

char_ngrams( )

char_wordstem()

char_select()

char_remove()

char_keep()

`tokens`

tokens_chunk()

tokens_compound()

tokens_group()

tokens_lookup()

tokens_ngrams()

tokens_skipgrams()

tokens_replace()

tokens_sample()

tokens_select()

tokens_remove()

tokens_keep()

tokens_split()

tokens_subset()

tokens_tolower()

tokens_toupper()

tokens_wordstem()

`dfm`

dfm_wordstem()

dfm_compress()

dfm_group()

dfm_lookup()

dfm_match()

dfm_replace()

dfm_sample()

dfm_select()

dfm_remove()

dfm_keep()

dfm_sort()

dfm_subset()

dfm_tfidf()

dfm_tolower()

dfm_toupper()

dfm_trim()

dfm_weight()

dfm_smooth()

Notice each function is prefixed with the object it accepts. This table isn’t a complete list of quanteda functions but illustrates the logic for naming and selecting functions based on the target object.

Terminology

Most NLP packages (such as tm) refer to tokens, which are typically words but can be lines, sentences, paragraphs, n-grams, or custom aggregations of words. quanteda prefers the term “features” instead of “tokens.” As a result, we’ll find the term “DFM” instead of “DTM” or “TDM.” DFM is short for document-feature matrix. In tm, DTM is short for document-term matrix, TDM is short for term-document matrix. Functionally, these are all the same thing.

The quanteda package proves to be a robust environment for text mining and natural language processing, offering a wealth of tools and capabilities to support our tasks in these fields.