I think this method is interesting but I think the class organisation needs a rework:

• the current MiniBatchSparsePCA does minibatches in the feature space, I think this is an abuse. At least it's not consistent with other use of the MiniBatch prefix in scikit-learn. I would rather have called this BlockedCoordinateSparsePCA or something similar.
• the new method would implement Sparse PCA with mini-batches in the sample space (via ENet projections of the components after each minibatch update if I understand correctly) but is currently implemented as part of the online dictionary learning code base for code reuse purpose.

However sparse PCA is not dictionary learning as the activations / code are not sparse. From a user point of you I find this really confusing. I think we should find a way to refactor the code to have the estimator name reflect the model name (the class of objective function with either sparse components or sparse code) and not the kind of solver used internally.

Out of curiosity what is the speed impact of using the python version of the Elastic Net projection? If the use of the the cython version is critical for speed, I think we should just write tests for that version and completely drop the Python version from the code base. There is no point in keeping 2 versions if one is always sub optimal.

We gain a factor ~40 in performance on the face dataset. I kept the python version in the database for testing, but it should be included in a test file.

Actually the online SPCA method presented in this PR can also enforce code sparsity : however it is often the case that you do not want both sparse code and sparse dictionary.

I think we could refactor as follow :

• Allow user selection of code computation algorithm, ie {'cd', 'lars', 'ols'}, such allowing him to use a non sparse code. This would involve a parameter alpha controlling code sparsity.
• Allow user selection of dictionary constraint, i.e. {'ordinary', 'sparse'}, with a parameter l1_ratio controlling dictionary sparsity.

MiniBatchDictionaryLearning would then default to sparse code, dense dictionary, while MiniBatchSparsePCA would default to sparse dictionary, sparse code, which is its expected behaviour as for version 0.16.

We could keep a class MiniBatchFeatureSparsePCA (or what not...) that would keep computing a sparse code and a dense dictionary by transposing X.

I still need to do some benchmarking to see if we do not loose considerable performance depending on X shape.

One of the problems with this approach is that the output of fit using new MiniBatchDictionaryLearning would be different using the same parameter, as there is no obvious correspondence between regulation parameters l1_ratio and alpha (which is used today to regulate code sparsity and thus component_ sparsity in MiniBatchSparsePCA).

Sorry for the multiple commits, to sum up I added a tolerance based early stopping, using the surrogate function $\hat f (t)$ described in above reference, and I changed the API so we can control code learning algorithm ('lars, 'cd' (not working on master, but working using PR #4775 ), 'ols'), and dictionary learning constraints (either euclidian or enet, which is structure inducing).

I tweaked the cython code so that it can release the gil. The for loop in _update_dict_enet is embarissingly parallel, are we allowed to use prange or should we always stick to joblib ?

prange cannot be currently used in scikit-learn because:

• not all supported compilers (e.g. clang on OSX or MSVC 2008 on windows) implement OpenMP
• the OpenMP implementation of gcc (libgomp) is still crashing when used in a program that uses multiprocessing somewhere else: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035

However you can either use joblib with backend="threading" if you do not do many calls to parallel in a row (otherwise the thread pool creation and destruction will be killing the performance). Alternatively you can maintain you own pool (the use of contextlib is necessary to use the ThreadPool as a context manager in Python 2.6):

from contextlib import closing
from multiprocessing.pool import ThreadPool

with closing(ThreadPool(n_jobs)) as p:
    for X_batch in minibatch_iterator:
        # use the pool to apply a function in parallel for instance p.map

Note that we could also work to make joblib.Parallel a context manager with __enter__ and __exit__ methods to keep a managed pool internally for successive jobs. This requires a lot more work and would not bring any benefit fro the threading backend case. It could be useful for the multiprocessing backend though.

Since we are only using a ThreadPool for enet version, I explicitly close at the end of dict_learning_online. I also tried to use the constructed Pool in sparse_encode, without gain in performance (I guess we need a multiprocessing Pool for that).

I changed this PR to [WIP] as we need to resolve the API issue. I suggest we introduce classes with name IncrementalDictionaryLearning and IncrementalSparsePCA to host the implementation of the cases where we have a partial_fit method and keep DictionaryLearning and SparsePCA for the cases where we cannot implement a partial_fit method (even if the fit method does some kind of internal 'online' optimization in the axis of the features).

We could then deprecate the current MiniBatchDictionaryLearning and MiniBatchSparsePCA as they are introducing confusion as they are not consistent.

@arthurmensch when you rebase and squash commit, please cleanup the full commit message: they are old phony messages included in the commit message for 0fbc54b.

The major issue here is now a naming problem : this PR code could be used to change MiniBatchSparsePCA into a real online estimator.

However, we would have to keep present MiniBatchSparsePCA for the next few versions and deprecate it, while creating another estimator (IncrementalSparsePCA ?)

Output from example :

Dataset consists of 400 faces
Extracting the top 30 Online Dictionary learning...
Iteration   0 (elapsed time:   0s,  0.0mn)
Iteration  10 (elapsed time:   0s,  0.0mn)
Iteration  20 (elapsed time:   0s,  0.0mn)
Iteration  30 (elapsed time:   0s,  0.0mn)
Iteration  40 (elapsed time:   0s,  0.0mn)
Iteration  50 (elapsed time:   1s,  0.0mn)
Iteration  60 (elapsed time:   1s,  0.0mn)
Iteration  70 (elapsed time:   1s,  0.0mn)
Iteration  80 (elapsed time:   1s,  0.0mn)
Iteration  90 (elapsed time:   2s,  0.0mn)
done in 2.416s
Online Dictionary learning - Component density
0.0601806640625
Code density
1.0

I am closing this PR for the moment as it needs serious rework from my side

ok thanks for working on this :)