[WIP] introduce object shelving by aabadie · Pull Request #619 · joblib/joblib

aabadie · 2018-01-26T17:22:04Z

This PR fixes #593 but is still WIP.

Note that this basic shelving can only be used with a script using a single python process because the futures returned by the shelf are only referenced in this python process. This means that it may not work as expected if using Parallel with loky or multiprocessing backend. But it should work with threading.

There are also some tests for the basic functionalities and top-level functions exposed to users:

shelving object from the standard library: dict, list, string, numbers
shelving numpy arrays
verifying that folders containing the shelved data are deleted when expected and not deleted when not expected (most important!)

Data are deleted in the following cases:

a shelved data is deleted from disk only when its last future reference is deleted
if the shelf object is deleted, the global shelf directory is fully deleted only when all futures are derefenced. This ensure that a future doesn't point to an already removed data.

Here are examples showing how to use this new feature:

>>> from joblib import shelve
>>> data = "A very big data"
>>> future = shelve(data)
>>> # Now the data can be removed
>>> del data
>>> future
<joblib.shelf.JoblibShelfFuture at 0x7fc15fad7a90>
>>> # The data can be retrieved from the future
>>> print(future.result())
A very big data

Here is the version with memmap:

>>> import numpy as np
>>> from joblib import shelve_mmap
>>> data = np.ones((10, 10))
>>> mmap = shelve_mmap(data)
>>> # the initial data can be removed
>>> del data
>>> # but can still be retrieved from the mmap
>>> mmap
memmap([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])

aabadie · 2018-01-26T19:00:35Z

I updated the initial comment with a bit more information and a better wording

codecov · 2018-01-26T19:05:46Z

Codecov Report

Merging #619 into master will increase coverage by 0.26%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #619      +/-   ##
==========================================
+ Coverage   95.02%   95.29%   +0.26%     
==========================================
  Files          39       41       +2     
  Lines        5427     5586     +159     
==========================================
+ Hits         5157     5323     +166     
+ Misses        270      263       -7

Impacted Files	Coverage Δ
joblib/__init__.py	`100% <100%> (ø)`	⬆️
joblib/test/test_shelf.py	`100% <100%> (ø)`
joblib/memory.py	`95.38% <100%> (+0.05%)`	⬆️
joblib/shelf.py	`100% <100%> (ø)`
joblib/test/test_parallel.py	`96.31% <0%> (+0.52%)`	⬆️
joblib/_parallel_backends.py	`95.68% <0%> (+1.72%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1507e2...7338508. Read the comment docs.

GaelVaroquaux · 2018-01-26T21:45:16Z

Can you add an example while you work on this PR. I find this very useful, because it helps thinking of the use case.

GaelVaroquaux · 2018-01-26T21:45:52Z

joblib/__init__.py

 from .parallel import register_parallel_backend
 from .parallel import parallel_backend
 from .parallel import effective_n_jobs
+from .shelf import JoblibShelf, shelve, shelve_mmap


Should we expose to the user JoblibShelf? It seems to me that it should be internal.

Indeed, will change that

GaelVaroquaux · 2018-01-26T21:56:08Z

joblib/memory.py

+        All values are cached on the filesystem, in a deep directory
+        structure.
+
+        see :ref:`memory_reference`


Hum, did we loose the docstring?

It was already like this IIRC. The __init__ docstring has moved to StoreBase

joblib/shelf.py

aabadie · 2018-01-27T21:57:42Z

Can you add an example while you work on this PR. I find this very useful, because it helps thinking of the use case.

I added a couple of examples in each function docstring. But maybe you are talking of a sphinx-gallery example ? If you have good ideas of examples, I buy them

GaelVaroquaux · 2018-03-14T10:21:16Z

Aside from the example, what remains to be done here?

GaelVaroquaux · 2018-03-14T10:27:04Z

joblib/shelf.py

+    memory. The future, a light-weight object, can be used later to reload the
+    initial object.
+
+    During the life of the future, the input object is kept written on a store


I would rather say "The input object is kept in a store (by default a file on a disk) as long as the future object exists (technically: as long as there is a reference on the future)".

GaelVaroquaux · 2018-03-14T10:54:32Z

joblib/shelf.py

+    return _active_shelf.put(input_object)
+
+
+def shelve_mmap(input_array):


I think that the variable should rather be called "input_object" (here and in the docstring below, the word "array" should often be replaced by "object").

This function is only meant to be used with numpy arrays, since it returns a future on a mmap. That's why I think it's important to keep the 'array' in the variable name

aabadie · 2018-04-19T16:37:43Z

Aside from the example, what remains to be done here?

@ogrisel suggested that the shelve_mmap could directly return the np.memmap object instead of the future. The future will be used internally to flush the array on disk when no more reference exists on the memmap.

This will allow a transparent use of this function with Parallel calls: no need to use the result() method of the future to retrieve the memmap.

aabadie · 2018-04-19T17:20:40Z

@ogrisel, added the mmap change in e35aa4a
As expected, it works with the threading backend but not with loky (multiprocessing)

GaelVaroquaux · 2018-04-19T21:42:34Z

+def shelve_mmap(input_array): This function is only meant to used with numpy array, since it return a future on a mmap. That's why I think it's important to keep the 'array' in the variable name

It's not limited to arrays. It should be able to take any object as an input.

aabadie · 2018-04-19T22:01:19Z

It's not limited to arrays. It should be able to take any object as an input.

I updated the input parameter and the shelve_mmap function documentation a bit. Now it directly returns a memory mmap.

aabadie · 2022-11-01T16:43:05Z

Is this still of interest to joblib ? I'd like to close it if possible :) And I'm not sure if it's in a rebasable state.

GaelVaroquaux reviewed Jan 26, 2018

View reviewed changes

aabadie commented Jan 26, 2018

View reviewed changes

joblib/shelf.py Outdated Show resolved Hide resolved

aabadie commented Jan 26, 2018

View reviewed changes

joblib/shelf.py Outdated Show resolved Hide resolved

aabadie force-pushed the object_store branch from a0e7ada to 37202fe Compare January 27, 2018 11:14

aabadie changed the title ~~WIP: introduce object shelving~~ [WIP] introduce object shelving Jan 27, 2018

aabadie force-pushed the object_store branch 8 times, most recently from cbb7361 to 999feab Compare January 27, 2018 21:19

GaelVaroquaux reviewed Mar 14, 2018

View reviewed changes

aabadie force-pushed the object_store branch from 999feab to f3fa3fe Compare April 19, 2018 16:00

aabadie added 4 commits April 24, 2018 15:35

WIP: introduce object shelving

07502cf

fix remaining merge issue

4bb3185

trying to improve shelve_mmap function

f3b7457

improve example

32550b0

aabadie added 5 commits April 24, 2018 15:35

flake8

aaa4d88

improve object shelving with memory map

a3503bd

update shelve_mmap documentation

72a15bf

add more tests, polish shelving

775b586

flake8

7338508

aabadie force-pushed the object_store branch from 72e2f5f to 7338508 Compare April 24, 2018 13:36

tueda mentioned this pull request Jan 24, 2020

Distributing large input data over worker processes? #996

Open

Nanored4498 mentioned this pull request Feb 12, 2026

FEA shelving #1779

Open

		return _active_shelf.put(input_object)


		def shelve_mmap(input_array):

Conversation

aabadie commented Jan 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Jan 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GaelVaroquaux commented Jan 26, 2018

Uh oh!

GaelVaroquaux Jan 26, 2018

Choose a reason for hiding this comment

Uh oh!

aabadie Jan 26, 2018

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux Jan 26, 2018

Choose a reason for hiding this comment

Uh oh!

aabadie Jan 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aabadie commented Jan 27, 2018

Uh oh!

GaelVaroquaux commented Mar 14, 2018

Uh oh!

GaelVaroquaux Mar 14, 2018

Choose a reason for hiding this comment

Uh oh!

aabadie Apr 19, 2018

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux Mar 14, 2018

Choose a reason for hiding this comment

Uh oh!

aabadie Apr 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabadie commented Apr 19, 2018

Uh oh!

aabadie commented Apr 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GaelVaroquaux commented Apr 19, 2018 via email

Uh oh!

aabadie commented Apr 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Nov 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aabadie commented Jan 26, 2018 •

edited

Loading

aabadie commented Jan 26, 2018 •

edited

Loading

codecov bot commented Jan 26, 2018 •

edited

Loading

aabadie Jan 26, 2018 •

edited

Loading

aabadie Apr 19, 2018 •

edited

Loading

aabadie commented Apr 19, 2018 •

edited

Loading

aabadie commented Apr 19, 2018 •

edited

Loading