Pandas was developed in the context of financial modeling, so as you might expect, it contains a fairly extensive set of tools for working with dates, times, and time-indexed data. Date and time data comes in a few flavors, which we will discuss here:
In this section, we will introduce how to work with each of these types of date/time data in Pandas. This short section is by no means a complete guide to the time series tools available in Python or Pandas, but instead is intended as a broad overview of how you as a user should approach working with time series. We will start with a brief discussion of tools for dealing with dates and times in Python, before moving more specifically to a discussion of the tools provided by Pandas. After listing some resources that go into more depth, we will review some short examples of working with time series data in Pandas.
The Python world has a number of available representations of dates, times, deltas, and timespans. While the time series tools provided by Pandas tend to be the most useful for data science applications, it is helpful to see their relationship to other packages used in Python.
datetime
and dateutil
Python's basic objects for working with dates and times reside in the built-in datetime
module.
Along with the third-party dateutil
module, you can use it to quickly perform a host of useful functionalities on dates and times.
For example, you can manually build a date using the datetime
type:
datetime.datetime(2015, 7, 4, 0, 0)
Or, using the dateutil
module, you can parse dates from a variety of string formats:
datetime.datetime(2015, 7, 4, 0, 0)
Once you have a datetime
object, you can do things like printing the day of the week:
'Saturday'
In the final line, we've used one of the standard string format codes for printing dates ("%A"
), which you can read about in the strftime section of Python's datetime documentation.
Documentation of other useful date utilities can be found in dateutil's online documentation.
A related package to be aware of is pytz
, which contains tools for working with the most migrane-inducing piece of time series data: time zones.
The power of datetime
and dateutil
lie in their flexibility and easy syntax: you can use these objects and their built-in methods to easily perform nearly any operation you might be interested in.
Where they break down is when you wish to work with large arrays of dates and times:
just as lists of Python numerical variables are suboptimal compared to NumPy-style typed numerical arrays, lists of Python datetime objects are suboptimal compared to typed arrays of encoded dates.
datetime64
The weaknesses of Python's datetime format inspired the NumPy team to add a set of native time series data type to NumPy.
The datetime64
dtype encodes dates as 64-bit integers, and thus allows arrays of dates to be represented very compactly.
The datetime64
requires a very specific input format:
array(datetime.date(2015, 7, 4), dtype='datetime64[D]')
Once we have this date formatted, however, we can quickly do vectorized operations on it:
array(['2015-07-04', '2015-07-05', '2015-07-06', '2015-07-07', '2015-07-08', '2015-07-09', '2015-07-10', '2015-07-11', '2015-07-12', '2015-07-13', '2015-07-14', '2015-07-15'], dtype='datetime64[D]')
Because of the uniform type in NumPy datetime64
arrays, this type of operation can be accomplished much more quickly than if we were working directly with Python's datetime
objects, especially as arrays get large
(we introduced this type of vectorization in Computation on NumPy Arrays: Universal Functions).
One detail of the datetime64
and timedelta64
objects is that they are built on a fundamental time unit.
Because the datetime64
object is limited to 64-bit precision, the range of encodable times is times this fundamental unit.
In other words, datetime64
imposes a trade-off between time resolution and maximum time span.
For example, if you want a time resolution of one nanosecond, you only have enough information to encode a range of nanoseconds, or just under 600 years. NumPy will infer the desired unit from the input; for example, here is a day-based datetime:
numpy.datetime64('2015-07-04')
Here is a minute-based datetime:
numpy.datetime64('2015-07-04T12:00')
Notice that the time zone is automatically set to the local time on the computer executing the code. You can force any desired fundamental unit using one of many format codes; for example, here we'll force a nanosecond-based time:
numpy.datetime64('2015-07-04T12:59:59.500000000')
The following table, drawn from the NumPy datetime64 documentation, lists the available format codes along with the relative and absolute timespans that they can encode:
Code | Meaning | Time span (relative) | Time span (absolute) |
---|---|---|---|
Y | Year | ± 9.2e18 years | [9.2e18 BC, 9.2e18 AD] |
M | Month | ± 7.6e17 years | [7.6e17 BC, 7.6e17 AD] |
W | Week | ± 1.7e17 years | [1.7e17 BC, 1.7e17 AD] |
D | Day | ± 2.5e16 years | [2.5e16 BC, 2.5e16 AD] |
h | Hour | ± 1.0e15 years | [1.0e15 BC, 1.0e15 AD] |
m | Minute | ± 1.7e13 years | [1.7e13 BC, 1.7e13 AD] |
s | Second | ± 2.9e12 years | [ 2.9e9 BC, 2.9e9 AD] |
ms | Millisecond | ± 2.9e9 years | [ 2.9e6 BC, 2.9e6 AD] |
us | Microsecond | ± 2.9e6 years | [290301 BC, 294241 AD] |
ns | Nanosecond | ± 292 years | [ 1678 AD, 2262 AD] |
ps | Picosecond | ± 106 days | [ 1969 AD, 1970 AD] |
fs | Femtosecond | ± 2.6 hours | [ 1969 AD, 1970 AD] |
as | Attosecond | ± 9.2 seconds | [ 1969 AD, 1970 AD] |
For the types of data we see in the real world, a useful default is datetime64[ns]
, as it can encode a useful range of modern dates with a suitably fine precision.
Finally, we will note that while the datetime64
data type addresses some of the deficiencies of the built-in Python datetime
type, it lacks many of the convenient methods and functions provided by datetime
and especially dateutil
.
More information can be found in NumPy's datetime64 documentation.
Pandas builds upon all the tools just discussed to provide a Timestamp
object, which combines the ease-of-use of datetime
and dateutil
with the efficient storage and vectorized interface of numpy.datetime64
.
From a group of these Timestamp
objects, Pandas can construct a DatetimeIndex
that can be used to index data in a Series
or DataFrame
; we'll see many examples of this below.
For example, we can use Pandas tools to repeat the demonstration from above. We can parse a flexibly formatted string date, and use format codes to output the day of the week:
Timestamp('2015-07-04 00:00:00')