Any example to use pipeline offline?

[taoluo]

this question was originally posted on zipline's google group:
There are some posts on quantopian.com introducing pipeline under the online Algorithm environment, while I am wondering how to modify the algorithm to run pipeline offline.

can anyone provide an short example ? I really appreciate any help you could provide.

[ssanderson]

@mickeydonkey the best place to look for examples on how the pipeline machinery works is the pipeline test suite, which lives in tests/pipeline.

There isn't a great short answer to the question of "how do I use Pipeline with real data", because the Pipeline API exists primarily for simplifying computations on large point-in-time datasets, and there aren't many such datasets freely available for public use. The most promising one that I'm aware of is Quandl's WIKI dataset, which contains a couple thousand assets and includes dividends and splits. I have a branch lying around somewhere that started building machinery for creating a zipline-compatible asset database from there.

The long answer to your question is that, to run an algorithm using the Pipeline machinery, you need need to write a function, get_loader, that takes a pipeline dataset column, (e.g. USEquityPricing.close) and returns an object implementing a method named load_adjusted_array whose signature is

def load_adjusted_array(self, columns, dates, sids, mask):

load_adjusted_array should return a dictionary mapping the entries in columns to instances of AdjustedArray containing data for the requested dates and sids (sids is a term for asset_ids in zipline for historical reasons).

If the dataset you want to use is small enough to hold in memory all at once, then you can use the built-in DataFrameLoader class from zipline.pipeline.loaders.frame for your loaders. The docstring for that class describes its functionality fairly well:

    """
    A PipelineLoader that reads its input from DataFrames.

    Mostly useful for testing, but can also be used for real work if your data
    fits in memory.

    Parameters
    ----------
    column : zipline.pipeline.data.BoundColumn
        The column whose data is loadable by this loader.
    baseline : pandas.DataFrame
        A DataFrame with index of type DatetimeIndex and columns of type
        Int64Index.  Dates should be labelled with the first date on which a
        value would be **available** to an algorithm.  This means that OHLCV
        data should generally be shifted back by a trading day before being
        supplied to this class.

    adjustments : pandas.DataFrame, default=None
        A DataFrame with the following columns:
            sid : int
            value : any
            kind : int (zipline.pipeline.loaders.frame.ADJUSTMENT_TYPES)
            start_date : datetime64 (can be NaT)
            end_date : datetime64 (must be set)
            apply_date : datetime64 (must be set)

        The default of None is interpreted as "no adjustments to the baseline".
    """

The adjustments frame is used to represent events that retroactively change our view of history. Most commonly, these are splits and dividends, which apply a backward-looking multiplier to the baseline array. If your dataset already uses "adjusted" prices/volumes, then you probably just want to pass None here.

[taoluo]

@ssanderson Thanks for the pointer, I am on my way of inspecting and running tests. I think it will take me some time to fully understand your long answer.

[fredfortier]

Given this recommendation: "The most promising one that I'm aware of is Quandl's WIKI dataset, which contains a couple thousand assets and includes dividends and splits." in relation to its implementation as a data bundle in 1.0, was the intent to enable Pipeline computations in zipline? If so, my naive attempt at running a Quantopian-tested algorithm did not seem to initialize the Pipeline.

[fredfortier]

I got a bit further. Guided by the zipline source code, I instantiated TradingAlgorithm() instead of calling run_algorithm(). I gave a get_pipeline_loader parameter to its constructor => get_pipeline_loader=lambda column: pipeline_loader, where pipeline_loader comes from USEquityPricingLoader.from_files(path, path).

However, this seems to only factor in securities which I explicitly referenced using symbols(), not the broader universe. It would be great if I could download complete data bundles by sector and use that in the pipeline.

[llllllllll]

run_algorithm should setup the pipeline loader, not doing so is a bug which we will try to fix soon in a 1.0.1 release. When you say that the loader can only reference things seen in a symbols call, what do you mean. What is the shape of the result of pipeline_output?

[fredfortier]

Great! I hesitate to add much more info to this issue as I might have loaded the data incorrectly. I do not want to add noise to your issues. As soon as 1.0.1 is out, or this particular fix is committed to the master, I will try again using the natural method and give a more detailed account should I continue to experience the behavior.

[npezolano]

Is it possible to extend DataFrameLoader to accept multiple columns or a full dataset as an input?

See:
https://github.com/quantopian/zipline/blob/master/tests/pipeline/test_frameload.py#L51-L53

[qinghanmeng]

what's the relationship between data bundle and pipeline.data ?