Get Started

In this guide, you walk through examples where you initialize Woodwork on a DataFrame and on a Series. Along the way, you learn how to update and remove logical types and semantic tags. You also learn how to use typing information to select subsets of data.

Types and Tags

Woodwork relies heavily on the concepts of physical types, logical types and semantic tags. These concepts are covered in detail in Working with Types and Tags, but we provide brief definitions here for reference:

  • Physical Type: defines how the data is stored on disk or in memory.

  • Logical Type: defines how the data should be parsed or interpreted.

  • Semantic Tag(s): provides additional data about the meaning of the data or how it should be used.

Start learning how to use Woodwork by reading in a dataframe that contains retail sales data.

[1]:
import pandas as pd

df = pd.read_csv("https://api.featurelabs.com/datasets/online-retail-logs-2018-08-28.csv")
df['order_product_id'] = range(df.shape[0])
df.head(5)
[1]:
order_id product_id description quantity order_date unit_price customer_name country total cancelled order_product_id
0 536365 85123A WHITE HANGING HEART T-LIGHT HOLDER 6 2010-12-01 08:26:00 4.2075 Andrea Brown United Kingdom 25.245 False 0
1 536365 71053 WHITE METAL LANTERN 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 1
2 536365 84406B CREAM CUPID HEARTS COAT HANGER 8 2010-12-01 08:26:00 4.5375 Andrea Brown United Kingdom 36.300 False 2
3 536365 84029G KNITTED UNION FLAG HOT WATER BOTTLE 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 3
4 536365 84029E RED WOOLLY HOTTIE WHITE HEART. 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 4

As you can see, this is a dataframe containing several different data types, including dates, categorical values, numeric values, and natural language descriptions. Next, initialize Woodwork on this DataFrame.

Initializing Woodwork on a DataFrame

Importing Woodwork creates a special namespace on your DataFrames, DataFrame.ww, that can be used to set or update the typing information for the DataFrame. As long as Woodwork has been imported, initializing Woodwork on a DataFrame is as simple as calling .ww.init() on the DataFrame of interest. An optional name parameter can be specified to label the data.

[2]:
import woodwork as ww

df.ww.init(name="retail")
df.ww
[2]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name category Categorical ['category']
country category Categorical ['category']
total float64 Double ['numeric']
cancelled bool Boolean []
order_product_id int64 Integer ['numeric']

Using just this simple call, Woodwork was able to infer the logical types present in the data by analyzing the DataFrame dtypes as well as the information contained in the columns. In addition, Woodwork also added semantic tags to some of the columns based on the logical types that were inferred.

Warning

Woodwork uses a weak reference for maintaining a reference from the accessor to the DataFrame. Because of this, chaining a Woodwork call onto another call that creates a new DataFrame or Series object can be problematic.

Instead of calling pd.DataFrame({'id':[1, 2, 3]}).ww.init(), first store the DataFrame in a new variable and then initialize Woodwork:

df = pd.DataFrame({'id':[1, 2, 3]})
df.ww.init()

All Woodwork methods and properties can be accessed through the ww namespace on the DataFrame. DataFrame methods called from the Woodwork namespace will be passed to the DataFrame, and whenever possible, Woodwork will be initialized on the returned object, assuming it is a Series or a DataFrame.

As an example, use the head method to create a new DataFrame containing the first 5 rows of the original data, with Woodwork typing information retained.

[3]:
head_df = df.ww.head(5)
head_df.ww
[3]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name category Categorical ['category']
country category Categorical ['category']
total float64 Double ['numeric']
cancelled bool Boolean []
order_product_id int64 Integer ['numeric']
[4]:
head_df
[4]:
order_id product_id description quantity order_date unit_price customer_name country total cancelled order_product_id
0 536365 85123A WHITE HANGING HEART T-LIGHT HOLDER 6 2010-12-01 08:26:00 4.2075 Andrea Brown United Kingdom 25.245 False 0
1 536365 71053 WHITE METAL LANTERN 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 1
2 536365 84406B CREAM CUPID HEARTS COAT HANGER 8 2010-12-01 08:26:00 4.5375 Andrea Brown United Kingdom 36.300 False 2
3 536365 84029G KNITTED UNION FLAG HOT WATER BOTTLE 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 3
4 536365 84029E RED WOOLLY HOTTIE WHITE HEART. 6 2010-12-01 08:26:00 5.5935 Andrea Brown United Kingdom 33.561 False 4

Note

Once Woodwork is initialized on a DataFrame, it is recommended to go through the ww namespace when performing DataFrame operations to avoid invalidating Woodwork’s typing information.

Updating Logical Types

If the initial inference was not to our liking, the logical type can be changed to a more appropriate value. Let’s change some of the columns to a different logical type to illustrate this process. In this case, set the logical type for the order_product_id and country columns to be Categorical and set customer_name to have a logical type of PersonFullName.

[5]:
df.ww.set_types(logical_types={
    'customer_name': 'PersonFullName',
    'country': 'Categorical',
    'order_product_id': 'Categorical'
})
df.ww.types
[5]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name string PersonFullName []
country category Categorical ['category']
total float64 Double ['numeric']
cancelled bool Boolean []
order_product_id category Categorical ['category']

Inspect the information in the types output. There, you can see that the Logical type for the three columns has been updated with the logical types you specified.

Selecting Columns

Now that you’ve prepared logical types, you can select a subset of the columns based on their logical types. Select only the columns that have a logical type of Integer or Double.

[6]:
numeric_df = df.ww.select(['Integer', 'Double'])
numeric_df.ww
[6]:
Physical Type Logical Type Semantic Tag(s)
Column
quantity int64 Integer ['numeric']
unit_price float64 Double ['numeric']
total float64 Double ['numeric']

This selection process has returned a new Woodwork DataFrame containing only the columns that match the logical types you specified. After you have selected the columns you want, you can use the DataFrame containing just those columns as you normally would for any additional analysis.

[7]:
numeric_df
[7]:
quantity unit_price total
0 6 4.2075 25.2450
1 6 5.5935 33.5610
2 8 4.5375 36.3000
3 6 5.5935 33.5610
4 6 5.5935 33.5610
... ... ... ...
401599 12 1.4025 16.8300
401600 6 3.4650 20.7900
401601 4 6.8475 27.3900
401602 4 6.8475 27.3900
401603 3 8.1675 24.5025

401604 rows × 3 columns

Adding Semantic Tags

Next, let’s add semantic tags to some of the columns. Add the tag of product_details to the description column, and tag the total column with currency.

[8]:
df.ww.set_types(semantic_tags={'description':'product_details', 'total': 'currency'})
df.ww
[8]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category', 'product_details']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name string PersonFullName []
country category Categorical ['category']
total float64 Double ['currency', 'numeric']
cancelled bool Boolean []
order_product_id category Categorical ['category']

Select columns based on a semantic tag. Only select the columns tagged with category.

[9]:
category_df = df.ww.select('category')
category_df.ww
[9]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['product_details', 'category']
country category Categorical ['category']
order_product_id category Categorical ['category']

Select columns using multiple semantic tags or a mixture of semantic tags and logical types.

[10]:
category_numeric_df = df.ww.select(['numeric', 'category'])
category_numeric_df.ww
[10]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['product_details', 'category']
quantity int64 Integer ['numeric']
unit_price float64 Double ['numeric']
country category Categorical ['category']
total float64 Double ['currency', 'numeric']
order_product_id category Categorical ['category']
[11]:
mixed_df = df.ww.select(['Boolean', 'product_details'])
mixed_df.ww
[11]:
Physical Type Logical Type Semantic Tag(s)
Column
description category Categorical ['product_details', 'category']
cancelled bool Boolean []

To select an individual column, specify the column name. Woodwork will be initialized on the returned Series and you can use the Series for additional analysis as needed.

[12]:
total = df.ww['total']
total.ww
[12]:
<Series: total (Physical Type = float64) (Logical Type = Double) (Semantic Tags = {'currency', 'numeric'})>
[13]:
total
[13]:
0         25.2450
1         33.5610
2         36.3000
3         33.5610
4         33.5610
           ...
401599    16.8300
401600    20.7900
401601    27.3900
401602    27.3900
401603    24.5025
Name: total, Length: 401604, dtype: float64

Select multiple columns by supplying a list of column names.

[14]:
multiple_cols_df = df.ww[['product_id', 'total', 'unit_price']]
multiple_cols_df.ww
[14]:
Physical Type Logical Type Semantic Tag(s)
Column
product_id category Categorical ['category']
total float64 Double ['currency', 'numeric']
unit_price float64 Double ['numeric']

Removing Semantic Tags

Remove specific semantic tags from a column if they are no longer needed. In this example, remove the product_details tag from the description column.

[15]:
df.ww.remove_semantic_tags({'description':'product_details'})
df.ww
[15]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name string PersonFullName []
country category Categorical ['category']
total float64 Double ['currency', 'numeric']
cancelled bool Boolean []
order_product_id category Categorical ['category']

Notice how the product_details tag has been removed from the description column. If you want to remove all user-added semantic tags from all columns, you can do that, too.

[16]:
df.ww.reset_semantic_tags()
df.ww
[16]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime []
unit_price float64 Double ['numeric']
customer_name string PersonFullName []
country category Categorical ['category']
total float64 Double ['numeric']
cancelled bool Boolean []
order_product_id category Categorical ['category']

Set Index and Time Index

At any point, you can designate certain columns as the Woodwork index or time_index with the methods set_index and set_time_index. These methods can be used to assign these columns for the first time or to change the column being used as the index or time index.

Index and time index columns contain index and time_index semantic tags, respectively.

[17]:
df.ww.set_index('order_product_id')
df.ww.index
[17]:
'order_product_id'
[18]:
df.ww.set_time_index('order_date')
df.ww.time_index
[18]:
'order_date'
[19]:
df.ww
[19]:
Physical Type Logical Type Semantic Tag(s)
Column
order_id category Categorical ['category']
product_id category Categorical ['category']
description category Categorical ['category']
quantity int64 Integer ['numeric']
order_date datetime64[ns] Datetime ['time_index']
unit_price float64 Double ['numeric']
customer_name string PersonFullName []
country category Categorical ['category']
total float64 Double ['numeric']
cancelled bool Boolean []
order_product_id category Categorical ['index']

Using Woodwork with a Series

Woodwork also can be used to store typing information on a Series. There are two approaches for initializing Woodwork on a Series, depending on whether or not the Series dtype is the same as the physical type associated with the LogicalType. For more information on logical types and physical types, refer to Working with Types and Tags.

If your Series dtype matches the physical type associated with the specified or inferred LogicalType, Woodwork can be initialized through the ww namespace, just as with DataFrames.

[20]:
series = pd.Series([1, 2, 3], dtype='int64')
series.ww.init(logical_type='Integer')
series.ww
[20]:
<Series: None (Physical Type = int64) (Logical Type = Integer) (Semantic Tags = {'numeric'})>

In the example above, we specified the Integer LogicalType for the Series. Because Integer has a physical type of int64 and this matches the dtype used to create the Series, no Series dtype conversion was needed and the initialization succeeds.

In cases where the LogicalType requires the Series dtype to change, a helper function ww.init_series must be used. This function will return a new Series object with Woodwork initialized and the dtype of the series changed to match the physical type of the LogicalType.

To demonstrate this case, first create a Series, with a string dtype. Then, initialize a Woodwork Series with a Categorical logical type using the init_series function. Because Categorical uses a physical type of category, the dtype of the Series must be changed, and that is why we must use the init_series function here.

The series that is returned will have Woodwork initialized with the LogicalType set to Categorical as expected, with the expected dtype of category.

[21]:
string_series = pd.Series(['a', 'b', 'a'], dtype='string')
ww_series = ww.init_series(string_series, logical_type='Categorical')
ww_series.ww
[21]:
<Series: None (Physical Type = category) (Logical Type = Categorical) (Semantic Tags = {'category'})>

As with DataFrames, Woodwork provides several methods that can be used to update or change the typing information associated with the series. As an example, add a new semantic tag to the series.

[22]:
series.ww.add_semantic_tags('new_tag')
series.ww
[22]:
<Series: None (Physical Type = int64) (Logical Type = Integer) (Semantic Tags = {'new_tag', 'numeric'})>

As you can see from the output above, the specified tag has been added to the semantic tags for the series.

You can also access Series properties methods through the Woodwork namespace. When possible, Woodwork typing information will be retained on the value returned. As an example, you can access the Series shape property through Woodwork.

[23]:
series.ww.shape
[23]:
(3,)

You can also call Series methods such as sample. In this case, Woodwork typing information is retained on the Series returned by the sample method.

[24]:
sample_series = series.ww.sample(2)
sample_series.ww
[24]:
<Series: None (Physical Type = int64) (Logical Type = Integer) (Semantic Tags = {'new_tag', 'numeric'})>
[25]:
sample_series
[25]:
0    1
2    3
dtype: int64

List Logical Types

Retrieve all the Logical Types present in Woodwork. These can be useful for understanding the Logical Types, as well as how they are interpreted.

[26]:
from woodwork.type_sys.utils import list_logical_types

list_logical_types()
[26]:
name type_string description physical_type standard_tags is_default_type is_registered parent_type
0 Address address Represents Logical Types that contain address ... string {} True True None
1 Age age Represents Logical Types that contain whole nu... int64 {numeric} True True Integer
2 AgeFractional age_fractional Represents Logical Types that contain non-nega... float64 {numeric} True True Double
3 AgeNullable age_nullable Represents Logical Types that contain whole nu... Int64 {numeric} True True IntegerNullable
4 Boolean boolean Represents Logical Types that contain binary v... bool {} True True BooleanNullable
5 BooleanNullable boolean_nullable Represents Logical Types that contain binary v... boolean {} True True None
6 Categorical categorical Represents Logical Types that contain unordere... category {category} True True None
7 CountryCode country_code Represents Logical Types that use the ISO-3166... category {category} True True Categorical
8 Datetime datetime Represents Logical Types that contain date and... datetime64[ns] {} True True None
9 Double double Represents Logical Types that contain positive... float64 {numeric} True True None
10 EmailAddress email_address Represents Logical Types that contain email ad... string {} True True None
11 Filepath filepath Represents Logical Types that specify location... string {} True True None
12 IPAddress ip_address Represents Logical Types that contain IP addre... string {} True True None
13 Integer integer Represents Logical Types that contain positive... int64 {numeric} True True IntegerNullable
14 IntegerNullable integer_nullable Represents Logical Types that contain positive... Int64 {numeric} True True None
15 LatLong lat_long Represents Logical Types that contain latitude... object {} True True None
16 NaturalLanguage natural_language Represents Logical Types that contain text or ... string {} True True None
17 Ordinal ordinal Represents Logical Types that contain ordered ... category {category} True True Categorical
18 PersonFullName person_full_name Represents Logical Types that may contain firs... string {} True True None
19 PhoneNumber phone_number Represents Logical Types that contain numeric ... string {} True True None
20 PostalCode postal_code Represents Logical Types that contain a series... category {category} True True Categorical
21 SubRegionCode sub_region_code Represents Logical Types that use the ISO-3166... category {category} True True Categorical
22 Timedelta timedelta Represents Logical Types that contain values s... timedelta64[ns] {} True True None
23 URL url Represents Logical Types that contain URLs, wh... string {} True True None
24 Unknown unknown Represents Logical Types that cannot be inferr... string {} True True None