Codecov Report

Merging #527 (8ad7445) into master (ccb9982) will increase coverage by 0.01%.
The diff coverage is 98.55%.

@@            Coverage Diff             @@
##           master     #527      +/-   ##
==========================================
+ Coverage   99.49%   99.50%   +0.01%     
==========================================
  Files          80       82       +2     
  Lines        5340     5458     +118     
==========================================
+ Hits         5313     5431     +118     
  Misses         27       27              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ccb9982...8ad7445. Read the comment docs.

It's working now (at least for the tfidf backend) but pretty slow - at least an order of magnitude slower than the Snowball analyzer. I think some batching must be used to make it more efficient, but that requires changes to the Analyzer API as well as to individual backends.

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

Rebased and force-pushed.

There is a new release of spaCy (3.2.0) available, should test that.

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

Rebased on current master, fixed conflicts and force-pushed.

Ready for review!

Things I'm a bit unsure about:

• Does this actually work with all relevant backends? Especially YAKE, where I adjusted the way the backend calls the analyzer.
• Are the Dockerfile changes sensible? I.e. adding spacy as a default extension and downloading spacy models based on a build argument.

Added wiki documentation:

• https://github.com/NatLibFi/Annif/wiki/Analyzers#spacy-analyzer
• https://github.com/NatLibFi/Annif/wiki/Optional-features-and-dependencies#spacy-analyzer
• https://github.com/NatLibFi/Annif/wiki/Usage-with-Docker#customizing-spacy-models

Things I'm a bit unsure about:

* Does this actually work with all relevant backends? Especially YAKE, where I adjusted the way the backend calls the analyzer.

* Are the Dockerfile changes sensible? I.e. adding spacy as a default extension and downloading spacy models based on a build argument.

Works with YAKE, and Spacy analyzer also somewhat improved evaluation results compared to Snowball analyzer (on JYU test set F1@5 0.1706 -> 0.1870).

Dockerfile looks good to me. (The three tries for timeouts in downloading ntlk data was originally added for builds by/in Drone, as there were some network problems in Drone at that time, but I think the situation has improved now.)

Just one point to consider: if Spacy model has not been loaded, a bit lengthy traceback is shown ending OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory. The ending is quite self-explanatory, so no big deal.

I compared this to Snowball using the Annif-tutorial yso-nlf data sets and the three backend configurations (tfidf, mllm, omikuji-parabel) used in the tutorial. Results (best score for each backend type highlighted):

Observations:

• spaCy is much much slower than Snowball (not entirely unexpected as it is a lot more sophisticated)
• train memory usage was a bit higher for spaCy (vs Snowball) but the differences were quite small
• eval memory usage of spaCy was much higher (compared to Snowball) for tfidf and omikuji, but not much higher for MLLM - not sure why but maybe it will become clear after some thinking
• the eval results for tfidf and mllm were worse than Snowball, but with Omikuji there was a small improvement (and the lowercased version was better than plain)

The point of these experiments was to check that the analyzer works reasonably well with those backends, not that the results are necessarily better in terms of F1 scores etc. spaCy has other advantages, especially the many languages it supports.

I also tested svc and fasttext backends using the 20news data set in Annif-corpora. Results:

Observations:

• spaCy was again much slower than Snowball
• memory usage wasn't dramatically higher
• results improved a bit for SVC, but were worse for fastText

I'd say this is good enough, I will check a few final things (including the error shown when a model doesn't exist, thanks @juhoinkinen!) and then merge this.

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication