Classifying Text Content with TF-IDF

In Natural Language Processing (NLP), word vectorization offers several options for assigning numerical values to text data. It is this process that allows us to build models that can reliably identify characteristics from a given document. We will explore how Term Frequency-Inverse Document Frequency (TF-IDF) vectorization can be applied to distinguish patterns in a document and help us classify where text may have originated from given its content.

Background for Experiment:
We will be using TF-IDF to help us classify content from Reddit posts to see if a model can identify which subreddit a post came from. For our purposes here, we have selected the Movies and Gaming subreddits. Both of these subreddits are distinct enough, but also overlap in that they both refer to entertainment topics. This can be sufficiently challenging for a classification model and an opportunity to see what TF-IDF can do for us.

Reddit Text Data:
We used the PRAW API for retrieving the text data from the respective subreddits. In advance of vectorization we created two DataFrames for the Movies (dfm), and Gaming (dfg) subreddits. Below is an example of our text data and target to be later classified for the “movies” subreddit:

‘alltext’ was engineered to combine both the title and body text of the post.

TF-IDF: Background & Application
Vectorizing the words demands a few prior considerations. Do we leave in or remove stop words? (and, the, to, on etc.), What can we learn from the most common N-gram ranges in our data? N-grams are groupings of N amount of words that themselves can reveal interesting patterns about text data. TF-IDF assigns a score to particular words or N-grams based on whether it occurs more or less frequently in a document. We can think of a TF-IDF score as a measure of the relative importance of a word vector when comparing between different document types. Another discription of this is “document similarity” with similarity here being measured by how common or “important” certain words are in one document vs. another.

Vectorization & Modeling:
Can TF-IDF help us make a robust model for classifying between different Subreddits? Lets find out. We’ll be using Scikit-learn’s Pipeline and GridSearchCV to fully utilize the TF-IDF transformer. We will be using a basic Logistic Regression model to classify the documents:

# import libraries
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

After fitting our model with GridSearch, we can create and evaluate the classification metrics. We’ll take a look at the Confusion martix and ROC Curve.

from sklearn.metrics import confusion_matrix, plot_confusion_matrix, plot_roc_curve
Our model did very well at distinguishing the two classes with minimal counts for both false positives and false negatives

We have encouraging results from the Confusion Matrix from TF-IDF being paired with Logistic Regression. Let’s see if this is also validated by the ROC Curve:

plt.title('Logistic ROC Curve')
This is a great performance by the model, with a possible max AUC score of 1

For good measure, let’s also take a look at the other ratios of true/false positive/negative rates:

print('precision: ',tp / (tp + fp))
print('sensitivity: ',tp / (tp + fn))
print('specificity: ',tn / (tn + fp))
print('accuracy: ', (tn + tp) / (tn + fp + fn + tp))
Great scores all across the board

What did we find was the best hyperpeter combination for TF-IDF?

Limiting max_features to 4,000 instead of 5,000 proved to be more useful

We can definitely appreciate the nuances offered by TF-IDF. It intuitevely makes sense that different words would have more “importance” in different documents and our transformation of that importance into scores has delivered a resonably good classifier. We can proceed with different types of classification models following this to see if our results could be even better.

Link to PRAW documentation:

Data professional with a passion for numbers.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store