That only decreases the error by a factor of sqrt(10). Could bite the bullet and set the rng seed.

Agreed, just set the seed so that this problem doesn't occur - our tests should not need to test general statistics!

Should we leave it to the doctest to explicitly set the seed, or make it a general feature of refcheck before each new code run?

Should we leave it to the doctest to explicitly set the seed, or make it a general feature of refcheck before each new code run?

Hmm. If it is in the doctest it would work for cut-and-paste, OTOH, making it a feature of refcheck might be a good insurance policy and allow repeatability. In this case I think the checks for mean and variance are part of the problem, maybe just print the values with an explanatory note that results may vary. If they are close to the expected values, that should be good enough.

If setting the seed in refcheck would reset the value for every test, that might slow things down. If there is a quarantee of the order in which the tests in each example are performed, it might be better to set the seed in those tests that need it.

Agreed with @charris that in this case the tests are not that good, and that it is probably more instructure to write: "Verify that the mean is close to 0 and the standard deviation close to 1" and then just print those (with a doctest: +SKIP).

Perhaps that could be the general policy as well: just don't do tests on random data in doctests, and in regular tests be sure to set a seed.

The # may vary directive in refguide is suitable for these scenarios too. It is already present in many of our docstrings.

Yes, that seems better. I clearly should look more closely at refguide - we should probably start using it in astropy as well...

Yes, that seems better. I clearly should look more closely at refguide - we should probably start using it in astropy as well...

good idea. at some point we should bit the bullet and make it a self-contained thing that people can then either depend on or vendor, rather than keep copying it around.

Hmm. If it is in the doctest it would work for cut-and-paste, OTOH, making it a feature of refcheck might be a good insurance policy and allow repeatability

If seeds are forgotten, it is probably better just to let the test fail. otherwise the results are stable for us but not for users, which will in the end be more confusing for them

Added a # may vary comment.