I get a consistent error, but this also happens on the current develop head (tested locally):

   def test_get_runs_list_by_tag(self):
        # TODO: comes from live, no such lists on test
        openml.config.server = self.production_server
        runs = openml.runs.list_runs(tag='curves')
>       self.assertGreaterEqual(len(runs), 1)
E       AssertionError: 0 not greater than or equal to 1

These seem to be server issues. is my best bet. If I try to filter runs with the tag through the website then over 6k results are returned quickly, but through the API I got the following response:


@janvanrijn any clue?

edit: requesting the response through my webbrowser is similarly slow (and has the same result).

Seems like joining to the run tag column is extremely slow (I guess due to the large size).

This could be solved by adding an index on the tag name, at the expense of additional disk cost. We currently have 23M run tag records, mostly indicating stuff that can be obtained through other ways as well. I think we should review the utility of (run) tags, and when a library automatically adds a tag to a run.

1. But the website loads them extremely quickly, e.g. runs with 'curves'?
2. It seemed to have worked before? git blame tells me this code was last edited 2017-03-02 14:34:54. And we have definitely had this test passed lots of times (though I realize there are more runs over time, and there could have been a proverbial final drop).

website runs on ES, which is an index build upon the database.

I am surprised that the query returns no results. It should either crash or return 6000+ results.

It does seem odd. Either way, I think we're best served by skipping this unit test for now as it is not an issue with the Python code. So I'll go ahead and do that (also @mfeurer so he knows what's up too).

@PGijsbers feel free to open an issue about this in the main tracker, with tags 'CoreSystem' and 'highest priority' assigned to me. I can't solve it this week (ECML deadline) but it will be high on my priority list

Codecov Report

Merging #654 into develop will decrease coverage by 0.02%.
The diff coverage is 90.19%.

@@             Coverage Diff             @@
##           develop     #654      +/-   ##
===========================================
- Coverage    90.96%   90.94%   -0.03%     
===========================================
  Files           32       32              
  Lines         3388     3391       +3     
===========================================
+ Hits          3082     3084       +2     
- Misses         306      307       +1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b081c5...c9ee562. Read the comment docs.