Consistent handling of no token lemmas#930
Conversation
Deploying asbplayer with
|
| Latest commit: |
3c58f8c
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://51c82787.asbplayer.pages.dev |
| Branch Preview URL: | https://no-lemmas-fix.asbplayer.pages.dev |
killergerbah
left a comment
There was a problem hiding this comment.
Would it be more consistent to just consider any token with no lemmas as having itself as a lemma? Or is the intention to only let these types of tokens exist in DB as local tokens picked by the user?
|
I considered just always appending the token if no lemmas but I didn't want it for the other use cases outside of
No we should generally not allow non dictionary entries into the database, at least now there is much reason to. So that when we parse Anki cards or user imports we can ignore these words. But we can allow it when they manually collect it on a subtitle for better UX essentially.
Yes that's the goal. Only local tokens that the user specifically picked out. |
|
I see, the entire diff including the last commit became a lot simpler with this change |
After thinking about it some more there is some inconsistencies on how no lemmas are being handled. In #929 we need to always build the exact cache in case we are in a new session where the user hasn't collected the token yet, not just for token refreshes.
The Anki database was allowing junk tokens by allowing ones without lemmas to be stored. Even if these tokens were valid it would only match with the exact strategy and not the others which could lead to confusion. So it's best to just ignore these. I don't think it's worth trying to prune the existing entries for users as these shouldn't appear in subtitles and will naturally get pruned over time if the card/note is ever modified.