MAINT: Replace WarningManager with catch_warnings and other numpy 1.14.0 issues by thequackdaddy · Pull Request #4205 · statsmodels/statsmodels

thequackdaddy · 2018-01-13T13:32:37Z

I think bumpy's WarningManager has gone the way of the Monty Python parrot.

This is causing AppVeyor to not run.

coveralls · 2018-01-13T14:14:55Z

Coverage decreased (-0.0009%) to 82.129% when pulling 9c020e0 on thequackdaddy:warnings into a222cec on statsmodels:master.

coveralls · 2018-01-13T14:14:56Z

Coverage decreased (-0.0009%) to 82.129% when pulling 9c020e0 on thequackdaddy:warnings into a222cec on statsmodels:master.

coveralls · 2018-01-13T14:44:23Z

Coverage decreased (-0.0009%) to 82.129% when pulling 9c020e0 on thequackdaddy:warnings into a222cec on statsmodels:master.

thequackdaddy · 2018-01-13T15:13:27Z

Hmmm... looks like appveyor is still unhappy.

I think it may have something to do with the upgraded numpy... I’ll take a peek.

josef-pkt · 2018-01-13T15:35:07Z

Are there a lot of failures with the latest numpy 1.14?
AFAICS, there are 22 failures which are mostly/all due to an upgraded numpy.

thequackdaddy · 2018-01-13T17:17:40Z

statsmodels/genmod/tests/results/results_glm.py

        filename = os.path.join(os.path.dirname(os.path.abspath(__file__)),
            "stata_lbw_glm.csv")
-        data=np.recfromcsv(open(filename, 'rb'), converters={4: lambda s: s.strip(asbytes("\""))})
+        data=np.recfromcsv(open(filename, 'rb'))


Not sure why, but when you have converters, numpy 1.14.0 drops the column. So I think this is an easy workaround. Probably not perfect.

See the linked numpy issue I opened.

thequackdaddy · 2018-01-13T17:18:48Z

statsmodels/stats/tests/test_contingency_tables.py


    s = str(rslt1)
-    assert_equal(s.startswith('A 5x5 contingency table with counts:\n[[ 8.'), True)
+    assert_equal(s.startswith('A 5x5 contingency table with counts:'), True)


numpy 1.14.0 seems to remove the leading space between the \n[[ and the 8.. I'm just going to check the text matches and the first element of the 2 dimensional array matches.

thequackdaddy · 2018-01-13T17:42:52Z

Ugh... I got the annoying test_simulate error... re-pushing to re-run. Sorry.

coveralls · 2018-01-13T17:48:09Z

Coverage decreased (-0.0006%) to 82.13% when pulling dc5196a on thequackdaddy:warnings into a222cec on statsmodels:master.

coveralls · 2018-01-13T18:24:40Z

Coverage decreased (-0.0006%) to 82.13% when pulling 815edd0 on thequackdaddy:warnings into a222cec on statsmodels:master.

thequackdaddy · 2018-01-13T21:12:58Z

Hmm... there's another rec.array change.

it seems you can no longer do data[['a_field', 'b_field']].view((float, 2)). A brief look at the docs has led me to think this is intentional design change.

My guess is that it would just be easier to use pandas for this... Otherwise, I'd have to do something like

np.hstack((a['a_field'], b['b_field']), dtype=float)...

I'll play with that shortly.

josef-pkt · 2018-01-13T21:23:21Z

I'm giving up on our rec array usage! We have to fix something with each numpy version (or new numpy/pandas interaction for them)
If you can make any fixes more easily by switching to pandas, then you can do so already now.

I would like to drop rec array usage and support completely in 0.10, but if some changes that we need for compatibility fixes are easier by dropping them right away, then we can do so. This is especially true for changes in our "old" testing code.
If we don't fix the code, then recarrays will only be supported for numpy != 1.4.0 or < 1.4.0, as long as we get the unit tests green.

coveralls · 2018-01-13T23:48:19Z

Coverage decreased (-0.0006%) to 82.13% when pulling cd003f3 on thequackdaddy:warnings into a222cec on statsmodels:master.

coveralls · 2018-01-13T23:48:19Z

Coverage decreased (-0.0006%) to 82.13% when pulling cd003f3 on thequackdaddy:warnings into a222cec on statsmodels:master.

codecov-io · 2018-01-14T00:03:07Z

Codecov Report

Merging #4205 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4205      +/-   ##
==========================================
- Coverage   79.52%   79.51%   -0.01%     
==========================================
  Files         561      561              
  Lines       83693    83690       -3     
  Branches     9552     9552              
==========================================
- Hits        66553    66550       -3     
  Misses      14955    14955              
  Partials     2185     2185

Impacted Files	Coverage Δ
statsmodels/formula/tests/test_formula.py	`100% <100%> (ø)`	⬆️
statsmodels/stats/tests/test_contingency_tables.py	`100% <100%> (ø)`	⬆️
statsmodels/tsa/tests/test_stattools.py	`97.32% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a222cec...bb8fba9. Read the comment docs.

thequackdaddy · 2018-01-14T00:27:57Z

Ok this looks decent to me. The coveralls fail is irrelevant.

Thoughts?

Let me know if you want me to squash.

josef-pkt · 2018-01-14T00:38:59Z

Looks very good to me, it's all in unit tests. Squashing would make it nicer.

@thequackdaddy Thanks a lot for working through this

Used np.vectorize to strip the excess string identfiers and switch from string check to integer check for counts. Replaced rec.array with pd.DataFrame... life is easier.

coveralls · 2018-01-14T01:21:59Z

Coverage decreased (-0.0006%) to 82.13% when pulling bb8fba9 on thequackdaddy:warnings into a222cec on statsmodels:master.

josef-pkt · 2018-01-14T06:20:40Z

merging

thequackdaddy force-pushed the warnings branch 2 times, most recently from b983160 to 9c020e0 Compare January 13, 2018 13:34

thequackdaddy mentioned this pull request Jan 13, 2018

Possible regression with numpy 1.14.0 and recfromcsv numpy/numpy#10394

Closed

thequackdaddy commented Jan 13, 2018

View reviewed changes

thequackdaddy changed the title ~~MAINT: Replace WarningManager with catch_warnings~~ MAINT: Replace WarningManager with catch_warnings and other numpy 1.14.0 issues Jan 13, 2018

thequackdaddy force-pushed the warnings branch from dc5196a to 815edd0 Compare January 13, 2018 17:36

Replace WarningManager with catch_warnings

bb8fba9

Used np.vectorize to strip the excess string identfiers and switch from string check to integer check for counts. Replaced rec.array with pd.DataFrame... life is easier.

thequackdaddy force-pushed the warnings branch from cd003f3 to bb8fba9 Compare January 14, 2018 00:40

josef-pkt merged commit 4151898 into statsmodels:master Jan 14, 2018

thequackdaddy deleted the warnings branch January 14, 2018 08:09

josef-pkt added this to the 0.9 milestone Apr 15, 2018

josef-pkt added type-test maintenance labels Apr 27, 2018

Conversation

thequackdaddy commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thequackdaddy commented Jan 13, 2018

Uh oh!

josef-pkt commented Jan 13, 2018

Uh oh!

thequackdaddy Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thequackdaddy Jan 13, 2018

Choose a reason for hiding this comment

Uh oh!

thequackdaddy commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thequackdaddy commented Jan 13, 2018

Uh oh!

josef-pkt commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018

Uh oh!

coveralls commented Jan 13, 2018

Uh oh!

codecov-io commented Jan 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

thequackdaddy commented Jan 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josef-pkt commented Jan 14, 2018

Uh oh!

coveralls commented Jan 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josef-pkt commented Jan 14, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Jan 13, 2018 •

edited

Loading

thequackdaddy Jan 13, 2018 •

edited

Loading

coveralls commented Jan 13, 2018 •

edited

Loading

coveralls commented Jan 13, 2018 •

edited

Loading

codecov-io commented Jan 14, 2018 •

edited

Loading

thequackdaddy commented Jan 14, 2018 •

edited

Loading

coveralls commented Jan 14, 2018 •

edited

Loading