pyfolio no longer supports pandas < 0.20

[jscotkin]

Pyfolio crashes (see below) on pandas 0.18 (or 0.19). This causes a bit of trouble since Zipline doesn't support pandas > 0.18. Running them in separate environments works.

The crash log below is on current git head. Same notebook works fine with pandas 0.20. I'm guessing it was introduced with merge #395 , which had some datetime updates related to pandas, but could be wrong.

Thank you!

pf.create_full_tear_sheet(returns, positions, transactions, slippage=0, round_trips=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
ValueError: Error parsing datetime string "1/1/1970" at position 1

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
<ipython-input-3-09e416a0ebcc> in <module>()
----> 1 pf.create_full_tear_sheet(returns, positions, transactions, slippage=0, round_trips=True)
      2 
      3 #pf.create_returns_tear_sheet(returns, live_start_date='2015-01-01')

~\Anaconda3\envs\zipfolio2\lib\site-packages\pyfolio\tears.py in create_full_tear_sheet(returns, positions, transactions, market_data, benchmark_rets, slippage, live_start_date, sector_mappings, bayesian, round_trips, estimate_intraday, hide_positions, cone_std, bootstrap, unadjusted_returns, risk, style_factor_panel, sectors, caps, shares_held, volumes, percentile, set_context)
    159 
    160     if benchmark_rets is None:
--> 161         benchmark_rets = utils.get_symbol_rets('SPY')
    162 
    163     if (unadjusted_returns is None) and (slippage is not None) and\

~\Anaconda3\envs\zipfolio2\lib\site-packages\pyfolio\utils.py in get_symbol_rets(symbol, start, end)
    517     return SETTINGS['returns_func'](symbol,
    518                                     start=start,
--> 519                                     end=end)
    520 
    521 

~\Anaconda3\envs\zipfolio2\lib\site-packages\pyfolio\utils.py in default_returns_func(symbol, start, end)
    286         end = _1_bday_ago()
    287 
--> 288     start = get_utc_timestamp(start)
    289     end = get_utc_timestamp(end)
    290 

~\Anaconda3\envs\zipfolio2\lib\site-packages\pyfolio\utils.py in get_utc_timestamp(dt)
    120     """
    121 
--> 122     dt = pd.to_datetime(dt)
    123     try:
    124         dt = dt.tz_localize('UTC')

~\Anaconda3\envs\zipfolio2\lib\site-packages\pandas\util\decorators.py in wrapper(*args, **kwargs)
     89                 else:
     90                     kwargs[new_arg_name] = new_arg_value
---> 91             return func(*args, **kwargs)
     92         return wrapper
     93     return _deprecate_kwarg

~\Anaconda3\envs\zipfolio2\lib\site-packages\pandas\tseries\tools.py in to_datetime(arg, errors, dayfirst, yearfirst, utc, box, format, exact, coerce, unit, infer_datetime_format)
    422         return _convert_listlike(arg, box, format)
    423 
--> 424     return _convert_listlike(np.array([arg]), box, format)[0]
    425 
    426 # mappings for assembling units

~\Anaconda3\envs\zipfolio2\lib\site-packages\pandas\tseries\tools.py in _convert_listlike(arg, box, format, name, tz)
    393                     dayfirst=dayfirst,
    394                     yearfirst=yearfirst,
--> 395                     require_iso8601=require_iso8601
    396                 )
    397 

pandas/tslib.pyx in pandas.tslib.array_to_datetime (pandas\tslib.c:43270)()

pandas/tslib.pyx in pandas.tslib.array_to_datetime (pandas\tslib.c:41765)()

pandas/tslib.pyx in pandas.tslib.array_to_datetime (pandas\tslib.c:41001)()

pandas/src/datetime.pxd in datetime._string_to_dts (pandas\tslib.c:87106)()

SystemError: <class 'str'> returned a result with an error set

[eigenfoo]

@gusgordon sounds like a pretty big compatibility issue

[twiecki]

CC @richafrank

[twiecki]

@jscotkin Which pandas version is that? The error is pretty confusing, what does pd.to_datetime('1/1/1970') do?

[yankees714]

@jscotkin I just merged support for pandas>=0.18.1 in #443, and haven't been able to reproduce this issue. Try updating to master, if you're still seeing issues feel free to open a more specific bug report.

[fpaboim]

I installed from git, the pandas dependency is OK but I'm still getting the same error as the OP

[twiecki]

@fpaboim Can you post the exception again, just to make sure?

[fpaboim]

Sorry, couldn't reproduce this. I've messed a lot with pip and python installations, and got it to work with pandas 0.18, don't know how the issue started..

[twiecki]

We fixed the issue :).

[cyniphile]

Seems like this is still a bug in python3 pandas <0.19.1. See https://stackoverflow.com/a/42600828/3835466

I'm still getting this issue using latest install of pyfolio in python3.

[twiecki]

Can you post the code to reproduce the issue along with the exception?

[cyniphile]

from zipline.api import symbol, order, record
from IPython import get_ipython
import pyfolio as pf
import pandas as pd


def initialize(context):
    pass


def handle_data(context, data):
    order(symbol('AAPL'), 10)


ipython = get_ipython()
ipython.magic('load_ext zipline')
algo = "zipline --start 2010-1-1 --end 2014-1-1 -o results_py3_test.pickle"
ipython.magic(algo)


results = pd.read_pickle('results_py3_test.pickle')
returns, positions, transactions = pf.utils.extract_rets_pos_txn_from_zipline(results)
pf.create_full_tear_sheet(returns, positions=positions, transactions=transactions,
                          live_start_date='2013-6-1')

Run in ipython.

Throws error with pandas 0.18.1 (zipline default), and pandas 0.22 (pyfolio default, on read_pickle). Works with pandas==0.19.2 manually installed. All py3.

[tdrobbin]

I seem to be getting the same issue, with almost identical code to @cyniphile

I'm using one environment with both zipline[v1.1.1] and pyfoli[v0.8.0] (both installed with pip) installed and have pandas[v0.18.1] installed.

[AdrianTeng]

I believe this bug exist when you have pandas < 0.19.2 (incl 0.18.1), and python 3.6 installed.

The root problem seems to be this throwing exception (when it should just return a Timestamp):
pd.to_datetime('01/01/2007')

This works if your python is <=3.5, regardless of pandas version

FWIW hopefully we could have pandas 0.22 working with zipline soon, then we can all move away from 0.18.1

[twiecki]

Thanks for figuring this out @AdrianTeng. Is there any work-around you can think of which helps for now?

[AdrianTeng]

@twiecki I've moved back to python 3.5 for now (sadly!) Once zipline is moved to pandas 0.22 I would move back to python 3.6

[jcrben]

Note that zipline does seem to be working with pandas 0.22.

[jcrben]

Not sure how/why I unassigned @yankees714