Skip to content

Add caching mechanism and rework remote loader? #438

@vitorbaptista

Description

@vitorbaptista

Originally by @femtotrader on trickvi/datapackage#61:

Hello,

I think datapackage should provide a cache mechanism.

For this (if user want to have this cache mechanism 2 optional dependencies could be requests and requests-cache)

One possible use could be :

import datapackage
import requests_cache
import datetime
session = requests_cache.CachedSession(cache_name='cache', backend='sqlite', expire_after=datetime.timedelta(days=60))
datapkg = datapackage.DataPackage('http://data.okfn.org/data/cpi/', session=session)

Default value of parameter session should beNone`.
This session should be stored as a member of DataPackage.

When session is not None request will be performed using

self.session.get(url)

Kind regards

PS : a similar approach was used in https://github.com/femtotrader/pandas_datareaders_unofficial

edit: and is now (oct 2015) used in official "pandas-datareader" https://github.com/pydata/pandas-datareader/

see also pydata/pandas-datareader#48

Metadata

Metadata

Assignees

No one assigned

    Labels

    generalGeneral improvements

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions