/cc @mAAdhaTTah @LeaVerou @Golmote

It works decently with TypeScript’s checkJs, although that doesn’t yet support @memberof.

I looked a little into this and there don't seem to be any workarounds for this, are there?

The only solution would be if TypeScript would add support for @memberof or if we would migrate portions of this project to TypeScript.

Well, there is an issue so the hope is there...

I'm actually quite a fan of TS but I think that:

1. Porting the project for the sake of JS-Check isn't worth it.
2. TS isn't necessary for Prism.

Also, TS is not as popular as JS, so some people might get held back from contributing.

That's a good idea. I'll change all similar occurrences.

Ok. It's done.

After this comment, I gave it another try. This time with the goal of just getting something that works even if it barely works.

The main problem with marrying TS and JSDoc was that TS couldn't do namespaces and wasn't smart enough to figure out some things about the structure of Prism instances. That was a year ago. TS has since become a bit smarter and we had a PR that made it easier for TS to figure out Prism. So TS could now understand Prism. One problem solved.
Now namespaces. Screw namespaces! I just didn't use them. Without using namespaces, the generated doc is somewhat odd because all classes and type definitions are part of the global namespace. This isn't really scalable documentation-wise but TS can understand it, so sure why not. We only have 5 types in global space right now, so it isn't a big deal.

After some tinkering and hacking, I finally had something that worked with both TS and JSDoc.
I also used a different theme that is a lot more configurable and just plain better than the previous one.

It finally works and it even looks kinda nice. Demo.

The hacking

I still have a small CSS file to change/fix the theme but it's better now.

The real hacking comes with JSDoc's parser. It can't deal with some TS syntax (e.g. intersection types and type imports) and will throw an error.
To fix that, I added a handler for JSDoc parser events that will just remove all type imports via a small regex and hopes that it works then. This is nothing more than a hack but it works and I don't plan to change anything about it because the JSDoc team doesn't intend to support this syntax, so this is the best I can do.

TS

One of the main intentions was to get good type checking, so did that turnout?

Amazing. It works really well. You have to kinda work around some issue but it's doable. My goal of moving Prism toward strict typing via doc comments is almost fulfilled. There are only 2 type problems left in Core. For that alone, I want to see this merged.

Other nice things

Documentation is opt-in. You can add doc comments however you like and for whatever you like. Only if you mark a doc comment as @public will it be part of the generated documentation site.

Thanks to the new theme, we now have a search bar and we can add your own links to the nav.

Not so nice things

The doc site includes a "jump to source" function. This means that every change to Core will result in changes in the doc site. To mitigate this a little, the gulp docs task is not part of the default build task. If you make changes to the doc, you have to rebuild the site manually. Not nice, but not too bad either.

Closing thoughts

It's not perfect but it works. You can write some doc and it will generate a nice site for you.

I criticize JSDoc and my solutions to all the problems I had to deal with a lot but I'm actually really happy with this result.

Let me know what you think.
Awaiting review.

In another project, I actually wrote everything in JS and managed to generate .d.ts files from them. If we're going to put all this work into writing & maintaining the documentation, is it possible to do that here? Not a requirement for this PR, by any means, just throwing it out there.

is it possible to do that here?

Should be possible. Let's make an issue of this after this PR is through.

I'm generally ok with punting off defining TokenStream to a future PR and not blocking this on it.

Well, if we really narrow down the definition of a Token after this, then that's a breaking change.

Actually, let's just take the current definition of Token and TokenStream as they are in here right now. This will resolve #2071.

I'm definitely on board with these API docs in general. I would love to eventually get to a world where we generate both types of docs into a single cohesive website, but this works for now.

I should probably point out that we now have API doc, but I haven't actually linked to it yet... Let just me do that real quick.

Minor change: I narrowed down the type definition of a Token (reason).
I'm gonna merge now.