Our test time has increased from 25 mins to more than a couple of hours on occasion, timing out the build in travis.

Faster Tests:

Tasks for improving speed:

• Profile tests; and flag tests that individually take longer than 10 minutes.
• Separate out the slow running integration tests via travis matrix.
• Explore parallelizing batches of tests using pytest.

Explored (but discarded):

• Have travis run the tests in parallel (will only be helpful if none of the tests individually is a bottleneck). See [Do not merge] Run tests in parallel #116

Not explored (on hold):

• explore caching of data folder (https://docs.travis-ci.com/user/caching/) so that we do not need to download the MNIST dataset each time.

Less flaky tests:

• Identify flaky tests (there are, from what I can see, two of them); for now let us increase the tolerance until we find a longer term solution (@fritzo suggested a good strategy in Regression test system (for stochastic computations) #101).
• Make unit tests deterministic.
• Mark flaky integration tests so that these do not fail the build.