Gain Statistical Insights into Your DataTable

Woodwork provides methods on DataTable to allow users to utilize the typing information inherent in a DataTable to better understand their data.

Let’s walk through how to use describe and mutual_information on a retail DataTable so that we can see the full capabilities of the functions.

[1]:

import pandas as pd
from woodwork import DataTable
from woodwork.demo import load_retail

dt = load_retail()
dt

[1]:
Physical Type Logical Type Semantic Tag(s)
Data Column
order_product_id category Categorical ['index']
order_id category Categorical ['category']
product_id category Categorical ['category']
description string NaturalLanguage []
quantity Int64 Integer ['numeric']
order_date datetime64[ns] Datetime ['time_index']
unit_price float64 Double ['numeric']
customer_name category Categorical ['category']
country category Categorical ['category']
total float64 Double ['numeric']
cancelled boolean Boolean []

DataTable.describe

We use dt.describe() to calculate statistics for the Data Columns in a DataTable in the format of a Pandas DataFrame with the relevant calculations done for each Data Column.

[2]:

dt.describe()

[2]:
order_id product_id description quantity order_date unit_price customer_name country total cancelled
physical_type category category string Int64 datetime64[ns] float64 category category float64 boolean
logical_type Categorical Categorical NaturalLanguage Integer Datetime Double Categorical Categorical Double Boolean
semantic_tags {category} {category} {} {numeric} {time_index} {numeric} {category} {category} {numeric} {}
count 401604 401604 401604 401604 401604 401604 401604 401604 401604 401604
nunique 22190 3684 NaN 436 20460 620 4372 37 3952 NaN
nan_count 0 0 0 0 0 0 0 0 0 0
mean NaN NaN NaN 12.1833 2011-07-10 12:08:23.848567552 5.73221 NaN NaN 34.0125 NaN
mode 576339 85123A WHITE HANGING HEART T-LIGHT HOLDER 1 2011-11-14 15:27:00 2.0625 Mary Dalton United Kingdom 24.75 False
std NaN NaN NaN 250.283 NaN 115.111 NaN NaN 710.081 NaN
min NaN NaN NaN -80995 2010-12-01 08:26:00 0 NaN NaN -277975 NaN
first_quartile NaN NaN NaN 2 NaN 2.0625 NaN NaN 7.0125 NaN
second_quartile NaN NaN NaN 5 NaN 3.2175 NaN NaN 19.305 NaN
third_quartile NaN NaN NaN 12 NaN 6.1875 NaN NaN 32.67 NaN
max NaN NaN NaN 80995 2011-12-09 12:50:00 64300.5 NaN NaN 277975 NaN
num_true NaN NaN NaN NaN NaN NaN NaN NaN NaN 8872
num_false NaN NaN NaN NaN NaN NaN NaN NaN NaN 392732

There are a couple things to note in the above dataframe:

  • The DataTable’s index, order_product_id, is not included

  • We provide each Data Column’s typing information according to Woodwork’s typing system

  • Any statistic that cannot be calculated for a Data Column, say num_false on a Datetime, will be filled with NaN.

  • Null values would not get counted in any of the calculations other than nunique

DataTable.value_counts

We can use dt.value_counts() to calculate the most frequent values for each Data Columns that has category as a standard tag. This returns a dictionary where each Data Column is associated with a sorted list of dictionaries. Each dictionary contains value and count.

[3]:

dt.value_counts()

[3]:

{'order_product_id': [{'value': 401603, 'count': 1},
  {'value': 133859, 'count': 1},
  {'value': 133861, 'count': 1},
  {'value': 133862, 'count': 1},
  {'value': 133863, 'count': 1},
  {'value': 133864, 'count': 1},
  {'value': 133865, 'count': 1},
  {'value': 133866, 'count': 1},
  {'value': 133867, 'count': 1},
  {'value': 133868, 'count': 1}],
 'order_id': [{'value': '576339', 'count': 542},
  {'value': '579196', 'count': 533},
  {'value': '580727', 'count': 529},
  {'value': '578270', 'count': 442},
  {'value': '573576', 'count': 435},
  {'value': '567656', 'count': 421},
  {'value': '567183', 'count': 392},
  {'value': '575607', 'count': 377},
  {'value': '571441', 'count': 364},
  {'value': '570488', 'count': 353}],
 'product_id': [{'value': '85123A', 'count': 2065},
  {'value': '22423', 'count': 1894},
  {'value': '85099B', 'count': 1659},
  {'value': '47566', 'count': 1409},
  {'value': '84879', 'count': 1405},
  {'value': '20725', 'count': 1346},
  {'value': '22720', 'count': 1224},
  {'value': 'POST', 'count': 1196},
  {'value': '22197', 'count': 1110},
  {'value': '23203', 'count': 1108}],
 'customer_name': [{'value': 'Mary Dalton', 'count': 7812},
  {'value': 'Dalton Grant', 'count': 5898},
  {'value': 'Jeremy Woods', 'count': 5128},
  {'value': 'Jasmine Salazar', 'count': 4459},
  {'value': 'James Robinson', 'count': 2759},
  {'value': 'Bryce Stewart', 'count': 2478},
  {'value': 'Vanessa Sanchez', 'count': 2085},
  {'value': 'Laura Church', 'count': 1853},
  {'value': 'Kelly Alvarado', 'count': 1667},
  {'value': 'Ashley Meyer', 'count': 1640}],
 'country': [{'value': 'United Kingdom', 'count': 356728},
  {'value': 'Germany', 'count': 9480},
  {'value': 'France', 'count': 8475},
  {'value': 'EIRE', 'count': 7475},
  {'value': 'Spain', 'count': 2528},
  {'value': 'Netherlands', 'count': 2371},
  {'value': 'Belgium', 'count': 2069},
  {'value': 'Switzerland', 'count': 1877},
  {'value': 'Portugal', 'count': 1471},
  {'value': 'Australia', 'count': 1258}]}

DataTable.mutual_information()

dt.mutual_information will calculate the mutual information between all pairs of relevant Data Columns. Certain types such as strings cannot have mutual information calculated.

The mutual information between columns A and B can be understood as the amount of knowlege we can have about column A if we have the values of column B. The more mutual information there is between A and B, the less uncertainty there is in A knowing B or vice versa.

If we call dt.mutual_information(), we’ll see that order_date will be excluded from the resulting dataframe.

[4]:

dt.mutual_information()

[4]:
column_1 column_2 mutual_info
0 order_id customer_name 0.886411
1 order_id product_id 0.475745
2 product_id unit_price 0.426383
3 order_id order_date 0.391906
4 product_id customer_name 0.361855
5 order_date customer_name 0.187982
6 quantity total 0.184497
7 customer_name country 0.155593
8 product_id total 0.152183
9 order_id total 0.129882
10 order_id country 0.126048
11 order_id quantity 0.114714
12 unit_price total 0.103210
13 customer_name total 0.099530
14 product_id quantity 0.088663
15 quantity customer_name 0.085515
16 quantity unit_price 0.082515
17 order_id unit_price 0.077681
18 product_id order_date 0.057175
19 total cancelled 0.044032
20 unit_price customer_name 0.041308
21 quantity cancelled 0.035528
22 product_id country 0.028569
23 country total 0.025071
24 order_id cancelled 0.022204
25 quantity country 0.021515
26 order_date country 0.010361
27 customer_name cancelled 0.006456
28 product_id cancelled 0.003769
29 country cancelled 0.003607
30 order_date unit_price 0.003180
31 order_date total 0.002625
32 unit_price country 0.002603
33 quantity order_date 0.002146
34 unit_price cancelled 0.001677
35 order_date cancelled 0.000199

Available Parameters

dt.mutual_information provides two parameters for tuning the mutual information calculation.

  • num_bins - In order to calculate mutual information on continuous data, we bin numeric data into categories. This parameter allows users to choose the number of bins with which to categorize data.

    • Defaults to using 10 bins

    • The more bins there are, the more variety a column will have. The number of bins used should accurately portray the spread of the data.

  • nrows - If nrows is set at a value below the number of rows in the DataTable, that number of rows will be randomly sampled from the underlying data

    • Defaults to using all the available rows.

    • Decreasing the number of rows can speed up the mutual information calculation on a DataTable with many rows, though care should be taken that the number being sampled is large enough to accurately portray the data.

Now we’ll explore changing the number of bins. Note that this will only impact numeric Data Columns quantity and unit_price. We’re going to increase the number of bins from 10 to 50, only showing the impacted columns.

[5]:

mi = dt.mutual_information()
mi[mi['column_1'].isin(['unit_price', 'quantity']) | mi['column_2'].isin(['unit_price', 'quantity'])]

[5]:
column_1 column_2 mutual_info
2 product_id unit_price 0.426383
6 quantity total 0.184497
11 order_id quantity 0.114714
12 unit_price total 0.103210
14 product_id quantity 0.088663
15 quantity customer_name 0.085515
16 quantity unit_price 0.082515
17 order_id unit_price 0.077681
20 unit_price customer_name 0.041308
21 quantity cancelled 0.035528
25 quantity country 0.021515
30 order_date unit_price 0.003180
32 unit_price country 0.002603
33 quantity order_date 0.002146
34 unit_price cancelled 0.001677 
[6]:

mi = dt.mutual_information(num_bins = 50)
mi[mi['column_1'].isin(['unit_price', 'quantity']) | mi['column_2'].isin(['unit_price', 'quantity'])]

[6]:
column_1 column_2 mutual_info
2 product_id unit_price 0.528865
4 unit_price total 0.405555
7 quantity total 0.349243
10 order_id quantity 0.157188
13 product_id quantity 0.143938
14 order_id unit_price 0.140257
16 quantity customer_name 0.113431
17 quantity unit_price 0.105052
18 quantity cancelled 0.081334
19 unit_price customer_name 0.078942
24 quantity country 0.023758
27 order_date unit_price 0.011905
30 unit_price country 0.006311
31 quantity order_date 0.004170
34 unit_price cancelled 0.001671
Woodwork Global Configuration Options Using Woodwork with Dask and Koalas DataFrames