Understanding Types and Tags

Using Woodwork effectively requires a good understanding of physical types, logical types, and semantic tags, all concepts that are core to Woodwork. This guide provides a detailed overview of types and tags, as well as how to work with them.

Definitions of Types and Tags

Woodwork has been designed to allow users to easily specify additional information about data contained in a DataTable while providing interfaces to access the underlying data based on that additional information. Because a single DataTable might store various types of data like numbers, text, or dates in different columns, the additional information is defined on a per-column basis.

There are 3 main ways that Woodwork stores additional information about user data:

  • Physical Type: defines how the data is stored on disk or in memory.

  • Logical Type: defines how the data should be parsed or interpreted.

  • Semantic Tag(s): provides additional data about the meaning of the data or how it should be used.

Physical Types

Physical types define how the data is stored on disk or in memory. You might also see the physical type for a column referred to as the column’s dtype.

For example, typical Pandas dtypes often used include object, int64, float64 and datetime64[ns], though there are many more. In Woodwork, there are 7 different physical types that are used, each corresponding to a Pandas dtype. When a DataTable is created, the dtype of the underlying data is converted to one of these values, if it isn’t already one of these types:

  • boolean

  • category

  • datetime64[ns]

  • float64

  • Int64

  • string

  • timedelta64[ns]

The physical type conversion is done based on the LogicalType that has been specified or inferred for a given column.

Logical Types

Logical types define how data should be interpreted or parsed. Logical types provide an additional level of detail beyond the physical type. Some columns might share the same physical type, but might have different parsing requirements depending on the information that is stored in the column.

For example, email addresses and phone numbers would typically both be stored in a data column with a physical type of string. However, when reading and validating these two types of information, different rules apply. For email addresses, the presence of the @ symbol is important. For phone numbers, you might want to confirm that only a certain number of digits are present, and special characters might be restricted to +, -, ( or ). In this particular example Woodwork defines two different logical types to separate these parsing needs: EmailAddress and PhoneNumber.

There are many different logical types defined within Woodwork. To get a complete list of all the available logical types, you can use the list_logical_types function.

[1]:
from woodwork import list_logical_types
list_logical_types()
[1]:
name type_string description physical_type standard_tags is_default_type is_registered parent_type
0 Boolean boolean Represents Logical Types that contain binary v... boolean {} True True None
1 Categorical categorical Represents Logical Types that contain unordere... category {category} True True None
2 CountryCode country_code Represents Logical Types that contain categori... category {category} True True Categorical
3 Datetime datetime Represents Logical Types that contain date and... datetime64[ns] {} True True None
4 Double double Represents Logical Types that contain positive... float64 {numeric} True True None
5 EmailAddress email_address Represents Logical Types that contain email ad... string {} True True NaturalLanguage
6 Filepath filepath Represents Logical Types that specify location... string {} True True NaturalLanguage
7 FullName full_name Represents Logical Types that may contain firs... string {} True True NaturalLanguage
8 IPAddress ip_address Represents Logical Types that contain IP addre... string {} True True NaturalLanguage
9 Integer integer Represents Logical Types that contain positive... Int64 {numeric} True True None
10 LatLong lat_long Represents Logical Types that contain latitude... object {} True True None
11 NaturalLanguage natural_language Represents Logical Types that contain text or ... string {} True True None
12 Ordinal ordinal Represents Logical Types that contain ordered ... category {category} True True Categorical
13 PhoneNumber phone_number Represents Logical Types that contain numeric ... string {} True True NaturalLanguage
14 SubRegionCode sub_region_code Represents Logical Types that contain codes re... category {category} True True Categorical
15 Timedelta timedelta Represents Logical Types that contain values s... timedelta64[ns] {} True True None
16 URL url Represents Logical Types that contain URLs, wh... string {} True True NaturalLanguage
17 ZIPCode zip_code Represents Logical Types that contain a series... category {category} True True Categorical

In the table, notice that each logical type has a specific pandas_dtype value associated with it. Any time a logical type is set for a column, the physical type of the underlying data is converted to the type shown in the pandas_dtype column. There is only one physical type associated with each logical type.

Semantic Tags

Semantic tags provide more context about the meaning of a data column. This could directly affect how the information contained in the column is interpreted. Unlike physical types and logical types, semantic tags are much less restrictive. A column might contain many semantic tags or none at all. Regardless, when assigning semantic tags, users should take care to not assign tags that have conflicting meanings.

As an example of how semantic tags can be useful, consider a dataset with 2 date columns: a signup date and a user birth date. Both of these columns have the same physical type (datetime64[ns]), and both have the same logical type (Datetime). However, semantic tags can be used to differentiate these columns. For example, you might want to add the date_of_birth semantic tag to the user birth date column to indicate this column has special meaning and could be used to compute a user’s age. Computing an age from the signup date column would not make sense, so the semantic tag can be used to differentiate between what the dates in these columns mean.

Standard Semantic Tags

As you can see from the table generated with the list_logical_types function above, Woodwork has some standard tags that are applied to certain columns by default. Woodwork adds a standard set of semantic tags to columns with LogicalTypes that fall under certain predefined categories.

The standard tags are as follows:

  • 'numeric' - The tag applied to numeric Logical Types.

    • Integer

    • Double

  • 'category' - The tag applied to Logical Types that represent categorical variables.

    • Categorical

    • CountryCode

    • Ordinal

    • SubRegionCode

    • ZIPCode

There are also 2 tags that get added to index columns. If no index columns have been specified, these tags are not present:

  • 'index' - on the index column, when specified

  • 'time_index' on the time index column, when specified

The application of standard tags, excluding the index and time_index tags, which have special meaning, can be controlled by the user. This is discussed in more detail in the Working with Semantic Tags section. There are a few different semantic tags defined within Woodwork. To get a list of the standard, index, and time index tags, you can use the list_semantic_tags function.

[2]:
from woodwork import list_semantic_tags
list_semantic_tags()
[2]:
name is_standard_tag valid_logical_types
0 category True [Categorical, CountryCode, Ordinal, SubRegionC...
1 numeric True [Double, Integer]
2 index False [Integer, Double, Categorical, Datetime]
3 time_index False [Datetime]
4 date_of_birth False [Datetime]

Working with Logical Types

When creating a DataTable, users have the option to specify the logical types for all, some, or none of the columns in the underlying dataframe. If logical types are defined for all of the columns, these logical types are used directly, provided the data is compatible with the specified logical type. You can’t, for example, use a logical type of Integer on a column that contains text values that can’t be converted to integers.

If users don’t supply any logical type information during creation of the DataTable, Woodwork infers the logical types based on the physical type of the column and the information contained in the columns. If the user passes information for some of the columns, the logical types are inferred for any columns not specified.

These scenarios are illustrated in this section. To start, create a simple dataframe to use for this example.

[3]:
import pandas as pd
from woodwork import DataTable

data = pd.DataFrame({
    'integers': [-2, 30, 20],
    'bools': [True, False, True],
    'names': ["Jane Doe", "Bill Smith", "John Hancock"]
})
data
[3]:
integers bools names
0 -2 True Jane Doe
1 30 False Bill Smith
2 20 True John Hancock

Now that you’ve created the data to use for the example, create a DataTable object, assigning logical type values to each of the columns. Then view the types stored for each column by using the DataTable.types property.

[4]:
logical_types = {
    'integers': 'Integer',
    'bools': 'Boolean',
    'names': 'FullName'
}

dt = DataTable(data, logical_types=logical_types)
dt
[4]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names string FullName []

As you can see, the logical types that you specified have been assigned to each of the columns. Now assign only one logical type value, and let Woodwork infer the types for the other columns.

[5]:
logical_types = {
    'names': 'FullName',
}
dt = DataTable(data, logical_types=logical_types)
dt
[5]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names string FullName []

With that input, you get the same results. Woodwork used the FullName logical type you assigned to the names column and then correctly inferred the logical types for the integers and bools columns.

Next, look at what happens if we do not specify any logical types.

[6]:
dt = DataTable(data)
dt
[6]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names category Categorical ['category']

In this case, Woodwork correctly inferred type for the integers and bools columns, but failed to recognize the names column should have a logical type of FullName. In situations like this, Woodwork provides users the ability to change the logical type.

Update the logical type of the names column to be FullName.

[7]:
dt = dt.set_types(logical_types={'names': 'FullName'})
dt
[7]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names string FullName []

If you look carefully at the output, you can see that several things happened to the names column. First, the correct FullName logical type has been applied. Second, the physical type of the column has changed from category to string to match the standard physical type for the FullName logical type. Finally, the standard tag of category that was previously set for the names column has been removed because it no longer applies.

When setting the LogicalType for a column, the type can be specified by passing a string representing the camel-case name of the LogicalType class as you have done in previous examples. Alternatively, you can pass the class directly instead of a string or the snake-case name of the string. All of these would be valid values to use for setting the FullName Logical type: FullName, "FullName" or "full_name".

Note—in order to use the class name, firs you have to import the class.

Working with Semantic Tags

Woodwork provides several methods for working with semantic types. You can add and remove specific tags, or you can reset the tags to their default values. In this section, you learn how to use those methods.

Note

When calling any of the methods that modify the logical type or semantic tags for a column, a new DataTable object is returned and the original DataTable remains unchanged. If you need to use the modified table, assign the returned DataTable to a variable that allows access to it.

Standard Tags

As mentioned above, Woodwork applies default semantic tags to columns by default, based on the logical type that was specified or inferred. If this behavior is undesirable, it can be controlled by setting the parameter use_standard_tags to False when creating the DataTable.

[8]:
dt = DataTable(data, use_standard_tags=False)
dt
[8]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer []
bools boolean Boolean []
names category Categorical []

As can be seen in the table above, when creating a DataTable with the use_standard_tags set to False, all semantic tags are empty. The only exception to this is if the index or time index column were set. We discuss that in more detail later on.

Create a new table with the standard tags, and specify some additional user-defined semantic tags during creation.

[9]:
semantic_tags = {
    'bools': 'user_status',
    'names': 'legal_name'
}
dt = DataTable(data, semantic_tags=semantic_tags)
dt
[9]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean ['user_status']
names category Categorical ['category', 'legal_name']

Woodwork has applied the tags we specified along with any standard tags to the columns in our DataTable.

After creating the table, you have changed your mind and decided you don’t like the tag of user_status that you applied to the bools column. Now you want to remove it. You can do that with the remove_semantic_tags method.

[10]:
dt = dt.remove_semantic_tags({'bools':'user_status'})
dt
[10]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names category Categorical ['category', 'legal_name']

The user_status tag has been removed.

You can also add multiple tags to a column at once by passing in a list of tags, rather of a single tag. Similarly, multiple tags can be removed at once by passing a list of tags.

[11]:
dt = dt.add_semantic_tags({'bools':['tag1', 'tag2']})
dt
[11]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean ['tag2', 'tag1']
names category Categorical ['category', 'legal_name']
[12]:
dt = dt.remove_semantic_tags({'bools':['tag1', 'tag2']})
dt
[12]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names category Categorical ['category', 'legal_name']

All tags can be reset to their default values by using the reset_semantic_tags methods. If use_standard_tags is True, the tags are reset to the standard tags. Otherwise, the tags are reset to be empty sets.

[13]:
dt = dt.reset_semantic_tags()
dt
[13]:
Physical Type Logical Type Semantic Tag(s)
Data Column
integers Int64 Integer ['numeric']
bools boolean Boolean []
names category Categorical ['category']

In this case, since you created our DataTable with the default behavior of using standard tags, calling reset_semantic_tags resulted in all of our semantic tags being reset to the standard tags for each column.

Index and Time Index Tags

When creating a DataTable, you have the option to specify which column represents the index and which column represents the time index. If these columns are specified, semantic tags of index and time_index are applied to the specified columns. Behind the scenes, Woodwork is performing additional validation checks on the columns to make sure they are appropriate. For example, index columns must be unique, and time index columns must contain datetime values or values that can be converted to datetimes.

Because of the need for these validation checks, you can’t set the index or time_index tags directly on a column. In order to designate a column as the index, the set_index method should be used. Similarly, in order to set the time index column, the set_time_index method should be used. Optionally, these can be specified when initially creating the DataTable.

Setting or changing the index

Create a new sample data set that contains columns that can be used as index and time index columns by using the index or time_index parameters.

[14]:
data = pd.DataFrame({
    'index': [0, 1, 2],
    'id': [1, 2, 3],
    'times': pd.to_datetime(['2020-09-01', '2020-09-02', '2020-09-03']),
    'numbers': [10, 20, 30]
})

dt = DataTable(data)
dt
[14]:
Physical Type Logical Type Semantic Tag(s)
Data Column
index Int64 Integer ['numeric']
id Int64 Integer ['numeric']
times datetime64[ns] Datetime []
numbers Int64 Integer ['numeric']

Without specifying an index or time index column during creation of the DataTable, Woodwork has inferred that the index and id columns are integers and the numeric semantic tag has been applied. You can now set the index column with the set_index method.

[15]:
dt = dt.set_index('index')
dt
[15]:
Physical Type Logical Type Semantic Tag(s)
Data Column
index Int64 Integer ['index']
id Int64 Integer ['numeric']
times datetime64[ns] Datetime []
numbers Int64 Integer ['numeric']

Inspecting the types now reveals that the index semantic tag has been added to the index column, and the numeric standard tag has been removed. You can also check that the index has been set correctly by checking the value of the DataTable.index attribute.

[16]:
dt.index
[16]:
'index'

If you want to change the index column to be the id column instead, you can do that with another call to set_index.

[17]:
dt = dt.set_index('id')
dt
[17]:
Physical Type Logical Type Semantic Tag(s)
Data Column
index Int64 Integer []
id Int64 Integer ['index']
times datetime64[ns] Datetime []
numbers Int64 Integer ['numeric']

The index tag has been removed from the index column and added to the id column. The numeric standard tag that was originally present on the index column has not been added back. If this tag is desired, the user must currently add it back using the add_semantic_tags method.

Setting or changing the time index

Setting the time index works similarly to setting the index. You can now set the time index for the DataTable with the set_time_index method.

[18]:
dt = dt.set_time_index('times')
dt
[18]:
Physical Type Logical Type Semantic Tag(s)
Data Column
index Int64 Integer ['numeric']
id Int64 Integer ['index']
times datetime64[ns] Datetime ['time_index']
numbers Int64 Integer ['numeric']

After calling set_time_index, the time_index semantic tag has been added to the times column of the DataTable.