Ping @IssamLaradji

Thanks. Looks great. I will have mlp and elm tested here. :-)

I know you knew I'd ask this but: should we be keeping this much duplicate code between bench_mnist and bench_covtype? I know the benchmarks already have a lot of duplicate code...

Good question. My opinion is that you will always tidbit differences that will matter. For instance, you will want to give different sets of parameters for each classifier in each benchmark. Prep-processing of the dataset will be different.

I also like the fact that the benchmark is straightforward and self-contained.

Travis highlights again an omp cv heisen failure.

I know I said +1 to the covtype changes, but in order to be particularly useful for playing with parameters, it should be usable in interactive mode. This means making a function like def main(classifiers, order='C', n_jobs=1, random_state=None), rather than having it all under if __name__ == '__main__'.

Then:

$ ipython -i benchmarks/bench_mnist.py
In [1]: main(my_classifiers)

That wouldn't be hard to do. I could define an overall benchmark method which would take an estimator as parameter.

What would expect as output of your main function?

As a main() it can just print to stdout.

On 14 August 2014 21:12, Arnaud Joly notifications@github.com wrote:

What would expect as output of your main function?

—
Reply to this email directly or view it on GitHub
#3562 (comment)
.

@jnothman I agree that making the benchmarks more interactive would be interesting. However, I think that this is already useful as it is. And for my usecase, a static script is already pretty good.

Coverage remained the same when pulling 535d7fc on arjoly:bench-mnist into 5024503 on scikit-learn:master.

LGTM +1 for merge as it is. We can always improve this later (there is no public API issue for benchmark scripts).

Did you mean to add changes to two other files?

Did you look at the changes I did in #3939?
Gamma in the kernel approximation was buggy and needs to be changed for master. I also made the output prettier.

@jnothman Yes, the modification in ``benchmarks/bench_covertype.py` solved potential bugs.

@amueller I have tried to pull your improvements.

I think it would be nice to have the output including all algorithms in the docstring. Otherwise LGTM

@amueller I have taken into account your last remark.

Coverage remained the same when pulling 5bb3cbf on arjoly:bench-mnist into 454aee9 on scikit-learn:master.

Coverage remained the same when pulling 5bb3cbf on arjoly:bench-mnist into 454aee9 on scikit-learn:master.

Thanks :)