[Feature] JavaScript Addons Support - Reviewer Addons#9232
[Feature] JavaScript Addons Support - Reviewer Addons#9232krmanik wants to merge 12 commits intoankidroid:mainfrom
Conversation
david-allison
left a comment
There was a problem hiding this comment.
First quick look over - tests are failing
To discuss with the others @mikehardy @Arthur-Milchior: I don't think we should be implementing a second type of concurrency primitive in this PR. As much as I dislike AsyncTask, until we actively pick the new direction, we shouldn't fragment how we implement concurrency
AnkiDroid/src/androidTest/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/androidTest/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/androidTest/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
| return CompletableFuture.supplyAsync(() -> { | ||
| mTaskListener.addonShowProgressBar(); | ||
| try { | ||
| ObjectMapper mapper = new ObjectMapper() |
There was a problem hiding this comment.
Won't this fail if the NPM schema adds additional fields (which should be backwards compatible)?
There was a problem hiding this comment.
It will fail if any fields in json change by NPM org.
The only field that depend this PR is getting tarball url to download .tgz file.
That file only can be downloaded from https://registry.npmjs.org/
Other fields like ankidroidJsApi and addonType distinguish from other npm packages.
So, for backward compatible JSONObject will be used to parse only required fields when above fails.
There was a problem hiding this comment.
I don't get your discussion.
Since you set the configuration not to fail on unknown property, it should work even if the properties change in the future
There was a problem hiding this comment.
Yes, it would work. Because in JSONObject implementation that can not be solved. If these fields in top of package.json then it will work.
The PR is implemented according to this https://docs.npmjs.com/cli/v7/configuring-npm/package-json .
There was a problem hiding this comment.
It feels that the issue is that NPM are already at v7, they'll happily go to v8, and when they do, it forces all AnkiDroid clients to update, or they'll be broken.
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
| init(); | ||
| shadowOf(getMainLooper()).idle(); | ||
|
|
||
| CompletableFuture<String> result1 = mNpmPackageDownloader.getTarball(validAddonName); |
There was a problem hiding this comment.
What's our opinion on having tests depend on internet resources? I'm against this as-is (as it exposes us to failures from systems that we can't control).
I'd be OK if these were informational, rather than failing the build (maybe a separate test category + warning on GitHub, rather than a straight test failure).
There was a problem hiding this comment.
I forget to remove this. Because CompletableFuture is removed from the class but not from test. I will remove it.
There was a problem hiding this comment.
I thought that we already depended a lot on web, as we download things. I assume that the website we download from can afford it, even if its not efficient. Is github putting any limitation?
I'd clearly love to be as realistic as possible, so I find it cool to be tested actually.
I'm okay with informational failure, but I fear that it will be missed in the quantity of information, the ignored test, the long text printed.
There was a problem hiding this comment.
The website will handle it, because this PR are not scrapping the website. In this PR user visit the npm package page as they visit in browser, then click 'Install Addon', here package.json downloaded for the package and tarball link extracted. The tarball taken from registry page of npm. http://registry.npmjs.org/
The same thing happen in nodejs cli in PC.
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/DownloadAddonBackgroundTask.java
Outdated
Show resolved
Hide resolved
|
This PR is finally implemented using TaskDelegate with test. |
|
Added It will works as follows
|
Arthur-Milchior
left a comment
There was a problem hiding this comment.
You wrote
to all deck templates
I don't understand it. Do you mean "card type's template"?
In your video, how is the user supposed to know what to enter and in which folder to go?
In particular, it's confusing to call it the name of the add-on while it's a command line.
Ideally, if there is a simpler way to deal with it, I'd prefer you take a real name. If it can't be done, then call it something else than "name".
Can you edit the first message, so that I know the current state of the code without having to check the latest comment.
I'd be great if you can rework the commit history. It seems that it's useless to read the sequence of commit one by one right now, and the whole change is so big it's hard to review. I probably forgot many comments,
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
| init(); | ||
| shadowOf(getMainLooper()).idle(); | ||
|
|
||
| CompletableFuture<String> result1 = mNpmPackageDownloader.getTarball(validAddonName); |
There was a problem hiding this comment.
I thought that we already depended a lot on web, as we download things. I assume that the website we download from can afford it, even if its not efficient. Is github putting any limitation?
I'd clearly love to be as realistic as possible, so I find it cool to be tested actually.
I'm okay with informational failure, but I fear that it will be missed in the quantity of information, the ignored test, the long text printed.
The contents of addon in this code javascript code added to reviewer to any decks. i.e. any card templates.
I will edit the first commit. I have implemented this PR with
I will rewrite the commit history. The PR is started from |
|
It seems hard to remove the older commit without removing reviewed message. |
|
I beg your pardon, you quote two questions I asked and give a single answer. It seems to ignore my second question, can you please answer it? Also, what is the relation between deck and card template? Usually, there are none
In this case, why does the name in your screen capture example start by "name -i"?
Sorry, in this case, I was not speaking of commit message. I understand the ambiguity. I was mentionning the first PR message. That is, what we see at the top of #9232 that starts by "Pull Request template" By the way, it's not a problem if we loose some code comment in github. Either they are adressed and it's great. Or they are not, and we can redo them when we see the problem still exists.
THanks |
The same command copy from npmjs.org pasted inside edit text. But in later implementation, I added a menu to Custom tabs so, user do not need to copy it. They just have to visit the npmjs package then click 'Install Addon` from options menu.
I have written so that user may find what should be entered from npmjs package. But it will be removed and replaced with Custom tabs menu. It is more simpler than copying command from npmjs.
I have explained the commit message, also added test cases for almost all functions. |
Arthur-Milchior
left a comment
There was a problem hiding this comment.
I've just reviewed the firs commit, I may go onto other commits later, but it's already big and hard to review.
I would really really love smaller commits, which introduce a single feature, such as extracting tgz or getting the name of add-on. This way, if we merge those features quickly, it mean the remaining of the PR is quick to review, as I'll only review the parts that we could not merge quickly first
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/test/java/com/ichi2/anki/jsaddons/NpmPackageDownloaderTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
|
Oops, sorry, as I review commit by commit, I missed that you added test. I would appreciate if you could add tests in the same commit than the code they test. If you want, we can take an hour at some point in time and pair code, so that at the very least I can discuss what I'd expect of commits. Because while they are split, they are not split in the way we usually do |
|
Also, I'd really love you to update the screen captured in the top of this page. Because I'd like to see the current version of the implementation. My questions were really related to the fact that your screen capture in the PR description contains "npm i" |
AnkiDroid/src/androidTest/java/com/ichi2/anki/jsaddons/NpmPackageTgzExtractTest.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.java
Outdated
Show resolved
Hide resolved
|
This is an important feature that I really want, but no feature with 99 comments and +1500 lines of code in a single commit is okay. That is just not okay. I actively fear even looking at this. I avoid it. This must be separated into even smaller things to be digestible - from my/our perspective there are more than 70 open PRs right now that all need attention, and 2 students working on Google Summer of Code that need lots of attention, along with regular release cycles and bugs and user feedback etc. Reviewing a +1500 line commit that gets pushed multiple times per day is just not possible, I can't do it. I wish I could but I cannot. When I see this text: I think of commits (maybe completely separate PRs) in chunks like:
Each of those is a reasonably separate feature, at the very least easy to reason about as a separate commit |
|
Yes, This PR is big. It needs split further. Currently I am doing that. I will mark it as draft for now. When PR split is done. I will mark it for Review. Thanks |
|
I have added contents to |
david-allison
left a comment
There was a problem hiding this comment.
Right now, the UX on the download could do with improving. It's very hard to find the "install addon" button.
Once we find the "install" button, most of the listed addons on NPM don't install,
Is there a way to filter NPM to only addons which are available, then by star rating (/etc..) to avoid addons which don't work?
We might need to use the same technique that the "Shared Decks Download" uses here.
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageTgzExtract.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageTgzExtract.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageTgzExtract.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageTgzExtract.java
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/DownloadAddonBroadcastReceiver.kt
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.kt
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.kt
Outdated
Show resolved
Hide resolved
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.kt
Outdated
Show resolved
Hide resolved
|
Hi @krmanik, I noticed you are constructing |
|
Hi @TarekkMA Also the PR needs review, so you can point out more suggestions. Thanks |
|
I will try to have a look tonight or tomorrow. |
TarekkMA
left a comment
There was a problem hiding this comment.
These are the things that I noticed right away, will try to give it another look sometime.
Thank you for the great work ❤️
AnkiDroid/src/main/java/com/ichi2/anki/jsaddons/NpmPackageDownloader.kt
Outdated
Show resolved
Hide resolved
| val mapper = ObjectMapper() | ||
| .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) | ||
|
|
||
| val addonModel = mapper.readValue(File(file, "package/package.json"), AddonModel::class.java) |
There was a problem hiding this comment.
Using jackson kotlin module, you can omit the last parameter.
| val addonModel = mapper.readValue(File(file, "package/package.json"), AddonModel::class.java) | |
| val addonModel: AddonModel = mapper.readValue(File(file, "package/package.json")) |
There was a problem hiding this comment.
You need to use jackson-module-kotlin library,
There was a problem hiding this comment.
Don't forget to register it in AnkiSerilization class
val mapper = ObjectMapper().registerModule(KotlinModule())There was a problem hiding this comment.
I understood it but now I think I should split this PR into multiple PR for faster review. Please review the other PR I will create.
| fun getJson(url: URL): MutableList<AddonModel>? { | ||
| try { | ||
| val mapper = AnkiSerialization.objectMapper | ||
| return mapper?.readValue(url, object : TypeReference<MutableList<AddonModel>>() {}) |
There was a problem hiding this comment.
Also here you can use kotlin module to be more elegant looking.
| return mapper?.readValue(url, object : TypeReference<MutableList<AddonModel>>() {}) | |
| return mapper?.readValue(url) |
There was a problem hiding this comment.
|
I will mark this PR as draft. Because it becomes harder to manage the code. |
|
Hello 👋, this PR has been opened for more than 2 months with no activity on it. If you think this is a mistake please comment and ping a maintainer to get this merged ASAP! Thanks for contributing! You have 7 days until this gets closed automatically |
|
I know you are watching the stale bot and keeping things that are "alive" live @krmanik - your patience is always appreciated |
|
This PR is split and implemented in parts. So this PR should be closed. But the PR will be used as reference for suggestions and review other smaller PR.
Yes, The priority should be GSoC, backend code then scope migration. Because I have more better idea for using backend code in add-on support. |
|
There are some merged (split) PR and more will be merged soon. |


Pull Request template
Purpose / Description
Using AnkiDroid JS API a deck can be extended to implement features like progress bar or custom card layouts. But it requires user to manually add the
html/css/jsto card templates. This is implementation for a way to inject js code to all card templates when for activated addons.This addon will be implemented in three part
Fixes
Fixes #7959
Approach
This implemented using TaskDelegate and jackson.
Jackson used to fetch latest package json then it parse to
AddoInfo. There isisValidAnkiDroidAddonboolean function will check the addons is valid or not.'Addons' menu to side navigation drawer
Added Addon Browser activity, it will be open from side nav menu
'Install Addons' button will be added at bottom for valid addona
Onlick 'Install Addon', confirmation alert shown
Download, extract and copied to addons folder
Implemented .tgz file extract and its test
This implemented using android.googlesource.com/TarUtil.java. There are test added by adding .tgz files to assets folder in androidTest. Using the functions to extract to temp folder and check if extracted files exists and match to original files.
Implemented AddonModel for mapping package.json from npmjs and its test
This is implemented using jackson
This is sample package.json of an addon. The
jackson ObjectMappermap this toAddonModel.The
isvalidAnkiDroidAddonin next commits in this PR check ifname,ankidroidJsApi,keywordsandaddonTypepresent or not. These fields distinguish from other npm packages.{ "name": "ankidroid-js-addon-progress-bar", "version": "1.0.0", "author": "https://github.com/infinyte7", "homepage": "https://github.com/infinyte7/Anki-Custom-Card-Layout", "ankidroidJsApi": "0.0.1", "addonType": "reviewer", "keywords": ["ankidroid-js-addon"] }There are two types of addon
reviewerandnote_editor.In this PR, first part of reviewer addon is implemented.
NpmUtils for extracting package name from url and its test
These are sample url of addons. The
getAddonNameFromUrlfunction inNpmUtilsextractpackage namefrom url. It replacehttps://www.npmjs.prg/packagethen check if/or '?' then split and get first element in split array.View NpmUtilsTest for more.
The extracted package name used by
Install Addonsin Custom tabs.Implemented Download addon, extract and copy to addons folder and its test
In this commit, files downloaded using TaskDelegate in background then extracted using previous
tar extractimplementation then copied toAnkiDroid/addonsfoldertest for isvalidAnkiDroidAddon using to .tgz file (valid & invalid .tgz files)
This is used for testing if current addon extracted in addons directory is valid or not. It will be helpful when user manually put the extracted files to addons folder. So, this function check before listing in AddonBrowser screen.
How Has This Been Tested?
Tested on emulator and devices
Demo
In this demo following tested
buttons added for only valid addon <br>Note: This option only added for URL starts withhttps://www.npmjs.com`Install Addonon npm package search URL i.e.https://www.npmjs.com/search?q=keywords:ankidroid-js-addon. It toasts withinvalid addonmessage because no package can be extracted from search page.Now
DownloadAddonclass implemented usingTaskDelegatedownload and do the following:-package.jsonusingjackson ObjectMappertoAddonModelname,ankidroidJsApi,keywordsandaddonTypepresent or noturl fromdist:{}` fieldHttpFetcher.downloadFileToSdCardMethodandtarballurl download .tgz filesNpmPackageTgzExtract.extractTarGzipToAddonFolderextract toAnkiDroid/addonsfolder, it show toast with messageinvalid addons` and returnLearning (optional, can help others)
https://junit.org/junit5/
https://github.com/FasterXML/jackson
https://github.com/FasterXML/jackson-docs
https://github.com/ankidroid/Anki-Android/wiki/Asynchronous-computation
Checklist
Please, go through these checks before submitting the PR.
ifstatements)