In my limited experience with LBFGS, the number of function calls is greater than the number of iterations.

The impact of this bug is that with solver='lbfgs' is probably not doing as many iterations as it should in master although I am not sure it matters that much in practice.

To get an idea how much funtion calls differ from iterations, I tweaked examples/neural_networks/plot_mnist_filters.py to be able to run for a few hundred iterations:

mlp = MLPClassifier(hidden_layer_sizes=(100,), max_iter=1000, alpha=1e-4,
                    solver='lbfgs', verbose=10, tol=1e-16, random_state=1,
                    learning_rate_init=.1)

The result: 393 iterations and 414 function calls.

Not sure whether we nest to test this, and how to test it, suggestions more than welcome!

• add a whats_new entry once there is agreement