Note
Click here to download the full example code or to run this example in your browser via Binder
Dirty categories: learning with non normalized strings¶
Including strings that represent categories often calls for much data preparation. In particular categories may appear with many morphological variants, when they have been manually input, or assembled from diverse sources.
Including such a column in a learning pipeline as a standard categorical colum leads to categories with very high cardinalities and can loose information on which categories are similar.
Here we look at a dataset on wages [1] where the column Employee Position Title contains dirty categories.
We investigate encodings to include such compare different categorical encodings for the dirty column to predict the Current Annual Salary, using gradient boosted trees. For this purpose, we use the dirty-cat library ( https://dirty-cat.github.io ).
The data¶
Data Importing and preprocessing¶
We first download the dataset:
from dirty_cat.datasets import fetch_employee_salaries
employee_salaries = fetch_employee_salaries()
print(employee_salaries['DESCR'])
Out:
Annual salary information including gross pay and overtime pay for all active, permanent employees of Montgomery County, MD paid in calendar year 2016. This information will be published annually each year.
Downloaded from openml.org.
Then we load it:
import pandas as pd
df = employee_salaries['data'].copy()
df
Now, let’s carry out some basic preprocessing:
First we extract the target
target_column = 'Current Annual Salary'
y = df[target_column].values.ravel()
Assembling a machine-learning pipeline that encodes the data¶
The learning pipeline¶
To build a learning pipeline, we need to assemble encoders for each column, and apply a supvervised learning model on top.
The categorical encoders¶
A encoder is needed to turn a categorical column into a numerical representation
from sklearn.preprocessing import OneHotEncoder
one_hot = OneHotEncoder(handle_unknown='ignore', sparse=False)
We assemble these to be applied on the relevant columns. The column transformer is created by specifying a set of transformers alongside with the column names on which to apply them
from sklearn.compose import make_column_transformer
encoder = make_column_transformer(
(one_hot, ['gender', 'department_name', 'assignment_category']),
('passthrough', ['Year First Hired']),
# Last but not least, our dirty column
(one_hot, ['employee_position_title']),
remainder='drop',
)
Pipelining an encoder with a learner¶
We will use a HistGradientBoostingRegressor, which is a good predictor for data with heterogeneous columns (for scikit-learn 0.24 we need to require the experimental feature)
from sklearn.experimental import enable_hist_gradient_boosting
# now you can import normally from ensemble
from sklearn.ensemble import HistGradientBoostingRegressor
# We then create a pipeline chaining our encoders to a learner
from sklearn.pipeline import make_pipeline
pipeline = make_pipeline(encoder, HistGradientBoostingRegressor())
The pipeline can be readily applied to the dataframe for prediction
pipeline.fit(df, y)
Out:
Pipeline(steps=[('columntransformer',
ColumnTransformer(transformers=[('onehotencoder-1',
OneHotEncoder(handle_unknown='ignore',
sparse=False),
['gender', 'department_name',
'assignment_category']),
('passthrough', 'passthrough',
['Year First Hired']),
('onehotencoder-2',
OneHotEncoder(handle_unknown='ignore',
sparse=False),
['employee_position_title'])])),
('histgradientboostingregressor',
HistGradientBoostingRegressor())])
Dirty-category encoding¶
The one-hot encoder is actually not well suited to the ‘Employee Position Title’ column, as this columns contains 400 different entries.
We will now experiments with encoders for dirty columns
from dirty_cat import SimilarityEncoder, TargetEncoder, MinHashEncoder,\
GapEncoder
similarity = SimilarityEncoder(similarity='ngram')
target = TargetEncoder(handle_unknown='ignore')
minhash = MinHashEncoder(n_components=100)
gap = GapEncoder(n_components=100)
encoders = {
'one-hot': one_hot,
'similarity': SimilarityEncoder(similarity='ngram'),
'target': target,
'minhash': minhash,
'gap': gap}
We now loop over the different encoding methods, instantiate each time a new pipeline, fit it and store the returned cross-validation score:
from sklearn.model_selection import cross_val_score
import numpy as np
all_scores = dict()
for name, method in encoders.items():
encoder = make_column_transformer(
(one_hot, ['gender', 'department_name', 'assignment_category']),
('passthrough', ['Year First Hired']),
# Last but not least, our dirty column
(method, ['employee_position_title']),
remainder='drop',
)
pipeline = make_pipeline(encoder, HistGradientBoostingRegressor())
scores = cross_val_score(pipeline, df, y)
print('{} encoding'.format(name))
print('r2 score: mean: {:.3f}; std: {:.3f}\n'.format(
np.mean(scores), np.std(scores)))
all_scores[name] = scores
Out:
one-hot encoding
r2 score: mean: 0.776; std: 0.028
similarity encoding
r2 score: mean: 0.923; std: 0.014
target encoding
r2 score: mean: 0.842; std: 0.030
minhash encoding
r2 score: mean: 0.919; std: 0.012
gap encoding
r2 score: mean: 0.909; std: 0.011
Plotting the results¶
Finally, we plot the scores on a boxplot:
import seaborn
import matplotlib.pyplot as plt
plt.figure(figsize=(4, 3))
ax = seaborn.boxplot(data=pd.DataFrame(all_scores), orient='h')
plt.ylabel('Encoding', size=20)
plt.xlabel('Prediction accuracy ', size=20)
plt.yticks(size=20)
plt.tight_layout()
The clear trend is that encoders that use the string form of the category (similarity, minhash, and gap) perform better than those that discard it.
SimilarityEncoder is the best performer, but it is less scalable on big data than MinHashEncoder and GapEncoder. The most scalable encoder is the MinHashEncoder. GapEncoder, on the other hand, has the benefit that it provides interpretable features (see sphx_glr_auto_examples_04_feature_interpretation_gap_encoder.py)
An easier way: automatic vectorization¶
The code to assemble the column transformer is a bit tedious. We will now explore a simpler, automated, way of encoding the data.
Let’s start again from the raw data:
X = employee_salaries['data'].copy()
y = employee_salaries['target']
We’ll drop a few columns we don’t want
X.drop([
'Current Annual Salary', # Too linked with target
'full_name', # Not relevant to the analysis
'2016_gross_pay_received', # Too linked with target
'2016_overtime_pay', # Too linked with target
'date_first_hired' # Redundant with "year_first_hired"
], axis=1, inplace=True)
We still have a complex and heterogeneous dataframe:
The SuperVectorizer can to turn this dataframe into a form suited for
machine learning.
Using the SuperVectorizer in a supervised-learning pipeline¶
Assembling the SuperVectorizer in a pipeline with a powerful learner,
such as gradient boosted trees, gives a machine-learning method that
can be readily applied to the dataframe.
# The supervectorizer requires dirty_cat 0.2.0a1. If you have an older
# version, you can install the alpha release with
#
# pip install -pre dirty_cat==0.2.0a1
#
from dirty_cat import SuperVectorizer
pipeline = make_pipeline(
SuperVectorizer(auto_cast=True),
HistGradientBoostingRegressor()
)
Let’s perform a cross-validation to see how well this model predicts
Out:
scores=array([0.87651907, 0.87186239, 0.90534537, 0.9120328 , 0.92163719])
mean=0.897479364762644
std=0.0197627927771675
The prediction perform here is pretty much as good as above but the code here is much simpler as it does not involve specifying columns manually.
Analyzing the features created¶
Let us perform the same workflow, but without the Pipeline, so we can analyze its mechanisms along the way.
sup_vec = SuperVectorizer(auto_cast=True)
We split the data between train and test, and transform them:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.15, random_state=42
)
X_train_enc = sup_vec.fit_transform(X_train, y_train)
X_test_enc = sup_vec.transform(X_test)
The encoded data, X_train_enc and X_test_enc are numerical arrays:
Out:
array([[1.0449171656292862, 0.7205922491346307, 0.39571982605966466, ...,
0.07790133090254987, 0.0866859328077334, 2007],
[0.05211638296647561, 29.96795271024839, 0.06107061378494277, ...,
0.08273956019725892, 0.0597482324987131, 2005],
[0.06000891970972247, 0.057149413239210564, 0.07738515680926894,
..., 0.08811993992371518, 0.05843022597056747, 2009],
...,
[0.09535114306092987, 0.06932818336882285, 0.16664185835837403,
..., 0.0827813134238929, 0.06117155771841887, 1990],
[0.08348536132941996, 0.06641634058081916, 0.07844733009530419,
..., 0.0836319932541453, 0.056200358797388036, 2012],
[0.060952392787100976, 0.07522578586235484, 0.055811123562746645,
..., 0.08273956019725892, 0.0597482324987131, 2014]], dtype=object)
They have more columns than the original dataframe, but not much more:
Out:
(7843, 56)
Inspecting the features created¶
The SuperVectorizer assigns a transformer for each column. We can inspect this
choice:
Out:
[('high_card_str', GapEncoder(), ['division', 'employee_position_title', 'underfilled_job_title']), ('low_card_cat', OneHotEncoder(), ['gender', 'assignment_category']), ('high_card_cat', GapEncoder(), ['department', 'department_name']), ('remainder', 'passthrough', [7])]
This is what is being passed to the ColumnTransformer under the hood.
If you’re familiar with how the later works, it should be very intuitive.
We can notice it classified the columns “gender” and “assignment_category”
as low cardinality string variables.
A OneHotEncoder will be applied to these columns.
The vectorizer actually makes the difference between string variables
(data type object and string) and categorical variables
(data type category).
Next, we can have a look at the encoded feature names.
Before encoding:
Out:
['gender', 'department', 'department_name', 'division', 'assignment_category', 'employee_position_title', 'underfilled_job_title', 'year_first_hired']
After encoding (we only plot the first 8 feature names):
feature_names = sup_vec.get_feature_names()
feature_names[:8]
Out:
['division: supports, twinbrook, compliance', 'division: health, emergency, school', 'division: technology, traffic, safety', 'division: gaithersburg, transit, silver', 'division: assignment, assessment, programs', 'division: administration, operations, battalion', 'division: accountability, development, planning', 'division: management, facilities, maintenance']
As we can see, it created a new column for each unique value.
This is because we used SimilarityEncoder on the column “division”,
which was classified as a high cardinality string variable.
(default values, see SuperVectorizer’s docstring).
In total, we have reasonnable number of encoded columns.
len(feature_names)
Out:
56
Feature importance in the statistical model¶
In this section, we will train a regressor, and plot the feature importances
First, let’s train the RandomForestRegressor,
from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor()
regressor.fit(X_train_enc, y_train)
Out:
RandomForestRegressor()
Retreiving the feature importances
importances = regressor.feature_importances_
std = np.std(
[
tree.feature_importances_
for tree in regressor.estimators_
],
axis=0
)
indices = np.argsort(importances)[::-1]
Plotting the results:
import matplotlib.pyplot as plt
plt.figure(figsize=(12, 9))
plt.title("Feature importances")
n = 20
n_indices = indices[:n]
labels = np.array(feature_names)[n_indices]
plt.barh(range(n), importances[n_indices], color="b", yerr=std[n_indices])
plt.yticks(range(n), labels, size=15)
plt.tight_layout(pad=1)
plt.show()
We can deduce from this data that the three factors that define the most the salary are: being hired for a long time, being a manager, and having a permanent, full-time job :).
Total running time of the script: ( 3 minutes 14.261 seconds)
