Start

In this guide, we will walk through an example of creating a Woodwork DataTable, and will show how to update and remove logical types and semantic tags. We will also demonstrate how to use the typing information to select subsets of data.

import woodwork as ww

data = ww.demo.load_retail(nrows=100)
data.head(5)

	order_id	product_id	description	quantity	order_date	unit_price	customer_name	country	total	cancelled
0	536365	85123A	WHITE HANGING HEART T-LIGHT HOLDER	6	2010-12-01 08:26:00	4.2075	Andrea Brown	United Kingdom	25.245	False
1	536365	71053	WHITE METAL LANTERN	6	2010-12-01 08:26:00	5.5935	Andrea Brown	United Kingdom	33.561	False
2	536365	84406B	CREAM CUPID HEARTS COAT HANGER	8	2010-12-01 08:26:00	4.5375	Andrea Brown	United Kingdom	36.300	False
3	536365	84029G	KNITTED UNION FLAG HOT WATER BOTTLE	6	2010-12-01 08:26:00	5.5935	Andrea Brown	United Kingdom	33.561	False
4	536365	84029E	RED WOOLLY HOTTIE WHITE HEART.	6	2010-12-01 08:26:00	5.5935	Andrea Brown	United Kingdom	33.561	False

As we can see, this is a dataframe containing several different data types, including dates, categorical values, numeric values and natural language descriptions. Let’s use Woodwork to create a DataTable from this data.

Creating a DataTable

Creating a Woodwork DataTable is as simple as passing in a dataframe with the data of interest during initialization. An optional name parameter can be specified to label the DataTable.

dt = ww.DataTable(data, name="retail")
dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
description	string	NaturalLanguage	{}
quantity	Int64	WholeNumber	{numeric}
order_date	datetime64[ns]	Datetime	{}
unit_price	float64	Double	{numeric}
customer_name	string	NaturalLanguage	{}
country	string	NaturalLanguage	{}
total	float64	Double	{numeric}
cancelled	boolean	Boolean	{}

Using just this simple call, Woodwork was able to infer the logical types present in our data by analyzing the dataframe dtypes as well as the information contained in the columns. In addition, Woodwork also added semantic tags to some of the columns based on the logical types that were inferred.

Updating Logical Types

If the initial inference was not to our liking, the logical type can be changed to a more appropriate value. Let’s change some of the columns to a different logical type to illustrate this process. Below we will set the logical type for the quantity, customer_name and country columns to be Categorical.

dt.set_logical_types({
    'quantity': 'Categorical',
    'customer_name': 'Categorical',
    'country': 'Categorical'
})
dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
description	string	NaturalLanguage	{}
quantity	category	Categorical	{category}
order_date	datetime64[ns]	Datetime	{}
unit_price	float64	Double	{numeric}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}
total	float64	Double	{numeric}
cancelled	boolean	Boolean	{}

If we now inspect the information in the types output, we can see that the Logical type for the three columns has been updated with the Categorical logical type we specified.

Selecting Columns

Now that we have logical types we are happy with, we can select a subset of the columns based on their logical types. Let’s select only the columns that have a logical type of WholeNumber or Double:

numeric_dt = dt.select_ltypes(['WholeNumber', 'Double'])
numeric_dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
unit_price	float64	Double	{numeric}
total	float64	Double	{numeric}

This selection process has returned a new DataTable containing only the columns that match the logical types we specified. After we have selected the columns we want, we can also access a dataframe containing just those columns if we need it for additional analysis.

numeric_dt.to_pandas()

	order_id	unit_price	total
0	536365	4.2075	25.245
1	536365	5.5935	33.561
2	536365	4.5375	36.300
3	536365	5.5935	33.561
4	536365	5.5935	33.561
...	...	...	...
95	536378	4.2075	25.245
96	536378	0.6930	83.160
97	536378	0.9075	21.780
98	536378	0.9075	21.780
99	536378	0.9075	21.780

100 rows × 3 columns

Note

Accessing the dataframe associated with a DataTable by using dt.to_pandas() will return a reference to the dataframe. Modifications to the returned dataframe can cause unexpected results. If you need to modify the dataframe, you should use dt.to_pandas(copy=True) to return a copy of the stored dataframe that can be safely modified without impacting the DataTable behavior.

Adding Semantic Tags

Next, let’s add semantic tags to some of the columns. We will add the tag of product_details to the description column and tag the total column with currency.

dt.set_semantic_tags({'description':'product_details', 'total': 'currency'})
dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
description	string	NaturalLanguage	{product_details}
quantity	category	Categorical	{category}
order_date	datetime64[ns]	Datetime	{}
unit_price	float64	Double	{numeric}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}
total	float64	Double	{numeric, currency}
cancelled	boolean	Boolean	{}

We can also select columns based on a semantic tag. Perhaps we want to only select the columns tagged with category:

category_dt = dt.select_semantic_tags('category')
category_dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
product_id	category	Categorical	{category}
quantity	category	Categorical	{category}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}

We can also select columns using mutiple semantic tags, or even a mixture of semantic tags and logical types:

category_numeric_dt = dt.select_semantic_tags(['numeric', 'category'])
category_numeric_dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
quantity	category	Categorical	{category}
unit_price	float64	Double	{numeric}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}
total	float64	Double	{numeric, currency}

mixed_dt = dt.select(['Boolean', 'product_details'])
mixed_dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
description	string	NaturalLanguage	{product_details}
cancelled	boolean	Boolean	{}

If we wanted to select an individual column, we just need to specify the column name. We can then get access to the data in the DataColumn using the series attribute:

dc = dt['total']
dc

<DataColumn: total (Physical Type = float64) (Logical Type = Double) (Semantic Tags = {'numeric', 'currency'})>

dc.series

0     25.245
1     33.561
2     36.300
3     33.561
4     33.561
       ...
95    25.245
96    83.160
97    21.780
98    21.780
99    21.780
Name: total, Length: 100, dtype: float64

You can also access multiple columns by supplying a list of column names:

multiple_cols_dt = dt[['product_id', 'total', 'unit_price']]
multiple_cols_dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
product_id	category	Categorical	{category}
total	float64	Double	{numeric, currency}
unit_price	float64	Double	{numeric}

Removing Semantic Tags

We can also remove specific semantic tags from a column if they are no longer needed. Let’s remove the product_details tag from the description column:

dt.remove_semantic_tags({'description':'product_details'})
dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
description	string	NaturalLanguage	{}
quantity	category	Categorical	{category}
order_date	datetime64[ns]	Datetime	{}
unit_price	float64	Double	{numeric}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}
total	float64	Double	{numeric, currency}
cancelled	boolean	Boolean	{}

Notice how the product_details tag has now been removed from the description column. If we wanted to remove all user-added semantic tags from all columns, we can also do that:

dt.reset_semantic_tags()
dt.types

	Physical Type	Logical Type	Semantic Tag(s)
Data Column
order_id	Int64	WholeNumber	{numeric}
product_id	category	Categorical	{category}
description	string	NaturalLanguage	{}
quantity	category	Categorical	{category}
order_date	datetime64[ns]	Datetime	{}
unit_price	float64	Double	{numeric}
customer_name	category	Categorical	{category}
country	category	Categorical	{category}
total	float64	Double	{numeric}
cancelled	boolean	Boolean	{}

Install API Reference