Using Woodwork effectively requires a good understanding of physical types, logical types, and semantic tags, all concepts that are core to Woodwork. This guide provides a detailed overview of types and tags, as well as how to work with them.
Woodwork has been designed to allow users to easily specify additional information about data contained in a DataTable while providing interfaces to access the underlying data based on that additional information. Because a single DataTable might store various types of data like numbers, text, or dates in different columns, the additional information is defined on a per-column basis.
There are 3 main ways that Woodwork stores additional information about user data:
Physical Type: defines how the data is stored on disk or in memory.
Logical Type: defines how the data should be parsed or interpreted.
Semantic Tag(s): provides additional data about the meaning of the data or how it should be used.
Physical types define how the data is stored on disk or in memory. You might also see the physical type for a column referred to as the column’s dtype.
dtype
For example, typical Pandas dtypes often used include object, int64, float64 and datetime64[ns], though there are many more. In Woodwork, there are 7 different physical types that are used, each corresponding to a Pandas dtype. When a DataTable is created, the dtype of the underlying data is converted to one of these values, if it isn’t already one of these types:
object
int64
float64
datetime64[ns]
boolean
category
Int64
string
timedelta64[ns]
The physical type conversion is done based on the LogicalType that has been specified or inferred for a given column.
LogicalType
Logical types define how data should be interpreted or parsed. Logical types provide an additional level of detail beyond the physical type. Some columns might share the same physical type, but might have different parsing requirements depending on the information that is stored in the column.
For example, email addresses and phone numbers would typically both be stored in a data column with a physical type of string. However, when reading and validating these two types of information, different rules apply. For email addresses, the presence of the @ symbol is important. For phone numbers, you might want to confirm that only a certain number of digits are present, and special characters might be restricted to +, -, ( or ). In this particular example Woodwork defines two different logical types to separate these parsing needs: EmailAddress and PhoneNumber.
@
+
-
(
)
EmailAddress
PhoneNumber
There are many different logical types defined within Woodwork. To get a complete list of all the available logical types, you can use the list_logical_types function.
list_logical_types
[1]:
from woodwork import list_logical_types list_logical_types()
In the table, notice that each logical type has a specific pandas_dtype value associated with it. Any time a logical type is set for a column, the physical type of the underlying data is converted to the type shown in the pandas_dtype column. There is only one physical type associated with each logical type.
pandas_dtype
Semantic tags provide more context about the meaning of a data column. This could directly affect how the information contained in the column is interpreted. Unlike physical types and logical types, semantic tags are much less restrictive. A column might contain many semantic tags or none at all. Regardless, when assigning semantic tags, users should take care to not assign tags that have conflicting meanings.
As an example of how semantic tags can be useful, consider a dataset with 2 date columns: a signup date and a user birth date. Both of these columns have the same physical type (datetime64[ns]), and both have the same logical type (Datetime). However, semantic tags can be used to differentiate these columns. For example, you might want to add the date_of_birth semantic tag to the user birth date column to indicate this column has special meaning and could be used to compute a user’s age. Computing an age from the signup date column would not make sense, so the semantic tag can be used to differentiate between what the dates in these columns mean.
Datetime
date_of_birth
As you can see from the table generated with the list_logical_types function above, Woodwork has some standard tags that are applied to certain columns by default. Woodwork adds a standard set of semantic tags to columns with LogicalTypes that fall under certain predefined categories.
The standard tags are as follows:
'numeric' - The tag applied to numeric Logical Types.
'numeric'
Integer
Double
'category' - The tag applied to Logical Types that represent categorical variables.
'category'
Categorical
CountryCode
Ordinal
SubRegionCode
ZIPCode
There are also 2 tags that get added to index columns. If no index columns have been specified, these tags are not present:
'index' - on the index column, when specified
'index'
'time_index' on the time index column, when specified
'time_index'
The application of standard tags, excluding the index and time_index tags, which have special meaning, can be controlled by the user. This is discussed in more detail in the Working with Semantic Tags section. There are a few different semantic tags defined within Woodwork. To get a list of the standard, index, and time index tags, you can use the list_semantic_tags function.
index
time_index
list_semantic_tags
[2]:
from woodwork import list_semantic_tags list_semantic_tags()
When creating a DataTable, users have the option to specify the logical types for all, some, or none of the columns in the underlying dataframe. If logical types are defined for all of the columns, these logical types are used directly, provided the data is compatible with the specified logical type. You can’t, for example, use a logical type of Integer on a column that contains text values that can’t be converted to integers.
If users don’t supply any logical type information during creation of the DataTable, Woodwork infers the logical types based on the physical type of the column and the information contained in the columns. If the user passes information for some of the columns, the logical types are inferred for any columns not specified.
These scenarios are illustrated in this section. To start, create a simple dataframe to use for this example.
[3]:
import pandas as pd from woodwork import DataTable data = pd.DataFrame({ 'integers': [-2, 30, 20], 'bools': [True, False, True], 'names': ["Jane Doe", "Bill Smith", "John Hancock"] }) data
Now that you’ve created the data to use for the example, create a DataTable object, assigning logical type values to each of the columns. Then view the types stored for each column by using the DataTable.types property.
DataTable.types
[4]:
logical_types = { 'integers': 'Integer', 'bools': 'Boolean', 'names': 'FullName' } dt = DataTable(data, logical_types=logical_types) dt
As you can see, the logical types that you specified have been assigned to each of the columns. Now assign only one logical type value, and let Woodwork infer the types for the other columns.
[5]:
logical_types = { 'names': 'FullName', } dt = DataTable(data, logical_types=logical_types) dt
With that input, you get the same results. Woodwork used the FullName logical type you assigned to the names column and then correctly inferred the logical types for the integers and bools columns.
FullName
names
integers
bools
Next, look at what happens if we do not specify any logical types.
[6]:
dt = DataTable(data) dt
In this case, Woodwork correctly inferred type for the integers and bools columns, but failed to recognize the names column should have a logical type of FullName. In situations like this, Woodwork provides users the ability to change the logical type.
Update the logical type of the names column to be FullName.
[7]:
dt = dt.set_types(logical_types={'names': 'FullName'}) dt
If you look carefully at the output, you can see that several things happened to the names column. First, the correct FullName logical type has been applied. Second, the physical type of the column has changed from category to string to match the standard physical type for the FullName logical type. Finally, the standard tag of category that was previously set for the names column has been removed because it no longer applies.
When setting the LogicalType for a column, the type can be specified by passing a string representing the camel-case name of the LogicalType class as you have done in previous examples. Alternatively, you can pass the class directly instead of a string or the snake-case name of the string. All of these would be valid values to use for setting the FullName Logical type: FullName, "FullName" or "full_name".
"FullName"
"full_name"
Note—in order to use the class name, firs you have to import the class.
Woodwork provides several methods for working with semantic types. You can add and remove specific tags, or you can reset the tags to their default values. In this section, you learn how to use those methods.
Note
When calling any of the methods that modify the logical type or semantic tags for a column, a new DataTable object is returned and the original DataTable remains unchanged. If you need to use the modified table, assign the returned DataTable to a variable that allows access to it.
As mentioned above, Woodwork applies default semantic tags to columns by default, based on the logical type that was specified or inferred. If this behavior is undesirable, it can be controlled by setting the parameter use_standard_tags to False when creating the DataTable.
use_standard_tags
False
[8]:
dt = DataTable(data, use_standard_tags=False) dt
As can be seen in the table above, when creating a DataTable with the use_standard_tags set to False, all semantic tags are empty. The only exception to this is if the index or time index column were set. We discuss that in more detail later on.
Create a new table with the standard tags, and specify some additional user-defined semantic tags during creation.
[9]:
semantic_tags = { 'bools': 'user_status', 'names': 'legal_name' } dt = DataTable(data, semantic_tags=semantic_tags) dt
Woodwork has applied the tags we specified along with any standard tags to the columns in our DataTable.
After creating the table, you have changed your mind and decided you don’t like the tag of user_status that you applied to the bools column. Now you want to remove it. You can do that with the remove_semantic_tags method.
user_status
remove_semantic_tags
[10]:
dt = dt.remove_semantic_tags({'bools':'user_status'}) dt
The user_status tag has been removed.
You can also add multiple tags to a column at once by passing in a list of tags, rather of a single tag. Similarly, multiple tags can be removed at once by passing a list of tags.
[11]:
dt = dt.add_semantic_tags({'bools':['tag1', 'tag2']}) dt
[12]:
dt = dt.remove_semantic_tags({'bools':['tag1', 'tag2']}) dt
All tags can be reset to their default values by using the reset_semantic_tags methods. If use_standard_tags is True, the tags are reset to the standard tags. Otherwise, the tags are reset to be empty sets.
reset_semantic_tags
True
[13]:
dt = dt.reset_semantic_tags() dt
In this case, since you created our DataTable with the default behavior of using standard tags, calling reset_semantic_tags resulted in all of our semantic tags being reset to the standard tags for each column.
When creating a DataTable, you have the option to specify which column represents the index and which column represents the time index. If these columns are specified, semantic tags of index and time_index are applied to the specified columns. Behind the scenes, Woodwork is performing additional validation checks on the columns to make sure they are appropriate. For example, index columns must be unique, and time index columns must contain datetime values or values that can be converted to datetimes.
Because of the need for these validation checks, you can’t set the index or time_index tags directly on a column. In order to designate a column as the index, the set_index method should be used. Similarly, in order to set the time index column, the set_time_index method should be used. Optionally, these can be specified when initially creating the DataTable.
set_index
set_time_index
Create a new sample data set that contains columns that can be used as index and time index columns by using the index or time_index parameters.
[14]:
data = pd.DataFrame({ 'index': [0, 1, 2], 'id': [1, 2, 3], 'times': pd.to_datetime(['2020-09-01', '2020-09-02', '2020-09-03']), 'numbers': [10, 20, 30] }) dt = DataTable(data) dt
Without specifying an index or time index column during creation of the DataTable, Woodwork has inferred that the index and id columns are integers and the numeric semantic tag has been applied. You can now set the index column with the set_index method.
id
[15]:
dt = dt.set_index('index') dt
Inspecting the types now reveals that the index semantic tag has been added to the index column, and the numeric standard tag has been removed. You can also check that the index has been set correctly by checking the value of the DataTable.index attribute.
numeric
DataTable.index
[16]:
dt.index
If you want to change the index column to be the id column instead, you can do that with another call to set_index.
[17]:
dt = dt.set_index('id') dt
The index tag has been removed from the index column and added to the id column. The numeric standard tag that was originally present on the index column has not been added back. If this tag is desired, the user must currently add it back using the add_semantic_tags method.
add_semantic_tags
Setting the time index works similarly to setting the index. You can now set the time index for the DataTable with the set_time_index method.
[18]:
dt = dt.set_time_index('times') dt
After calling set_time_index, the time_index semantic tag has been added to the times column of the DataTable.
times