Can you give a recap over the way that the contents of the manifest are used by the JSR-233 implementation? What does a sample manifest look like? Are we following a convention established by other JSR-223 implementations, or have we created our own convention?

I've been looking at 3a30af1#diff-58f7fb77a943765504dc8346cf02d0bdR125 for some insights, but that commit seems a bit short on commentary.

I've found some more documentation in your ScalaDays slidedeck: http://jscl-meditor.sourceforge.net/scalaint-slides.pdf

I also now understand that the Name: com/acme/Bar.class attribute is part of the JAR file standard. It is usually seen in the manifest of signed JARs, but is valid in any JAR and can be used to set attributes on a per-class or per-package basis.

The problems of packages not being enumerable is also present in the toolbox compiler in runtime reflection: https://issues.scala-lang.org/browse/SI-6393. This isn't surprising, as ToolBox is basically a proprietery API for the same sorts of use cases as JSR-223. But it is concerning that these two features don't share solutions to the problem.

The problem with URLClassLoader is when URLs have a different protocol than file. For instance with java web start the protocol is http and contents is cached in a java proprietary way, and basically impossible to retrieve in any other fashion than through the ClassLoader. You also have problems with web containers, whose security generally forbids to directly open arbitrary jar files. OSGI is yet another matter, as it will isolate classloaders from one another, making jars in one bundle unavailable to others and to the script engine for that matter.

Yep, it is a rather annoying limitation. Perhaps we should work towards a best-effort hybrid solution, both in JSR scripting and in Toolbox: uses info from the Manifest class listing when available, and otherwise attempt to listing file:// classpath entries JARs when permissible.

With regards to this PR and its expanded manifest in scala-library: I feel like we should be uniform, rather than just doing this for scala-library.jar. scala/scala also publishes scala-{reflect, compiler}.jar; and we also have the modules.

Yes, but to be really uniform, any possible jar on the planet (or maybe just sonatype) should be jarlisted. Instead, I think we should document properly how to jarlist a library (and I started to do that with the man page) and let the users do it as needed. Regarding scala-{reflect, compiler}, the point is that they are rarely needed in common cases. Making them available to the script engine is akin to using the standard scala repl in power mode : not everyone needs it (and moreover, it is dangerous). I would special-case scala-library.jar , because it allows to run the script engine out of the box with jrunscript. And most importantly, it allows to include it in tests. But I can understand your concern about uniformity, that is why I separated the commits, and we can perfectly drop the inclusion in the build for the moment.

retargeting for 2.11.8, unless @retronym wants to fast-track it

/rebuild

re-review by @retronym?

@retronym , I have added an option to get the classpath from the URLClassloader if available. I think it could be used from ToolBox but, as far as I can tell, it would have to be rewritten to use the normal Global and not ReflectGlobal. I don't know what was the rationale behind the creation of the latter.

Echoing comments from #4733 to gently nudge @retronym: moving this along would facilitate work in on JSR-223 compliance.

Answering myself, I think @odersky introduced ReflectGlobal in e1a9fd9 , and he might have overlooked this relative import problem, and that to "use reflection to get class infos, instead of reading class or source files" is not enough. Perhaps Martin, you would care to comment on this issue, and give us some advice/direction ?

I think the inability to enumerate the classpath was just taken as a fundamental limitation of a compiler based purely on the vanilla Classloader API / Java reflection, but despite this, runtime compilation was still useful enough to include in the experimental reflection module.

I hadn't heard of the technique to read the list of classes from the manifest until I looked at this PR. It seems a clean use of a standard metadata format, but is somewhat limited as it requires library authors to opt in when publishing, or, failing that, added complexity in your own build chain using something like jarlister that you've also provided.

I suggested the approach of downcasting to a URLClassLoader to @xeno-by a while back. I felt this was likely to work in a number of sitatuations.

The Toolbox compiler can be started wth an explicit classpath, using mkToolbox("-classpath foo.jar:bar.jar"). This might provide a means of using the results of your classloader probing in this context.

Stepping back for a second, scala-reflect remains experimental, so we have to timebox our efforts in this area, to make sure the non-experimental parts of the compiler and library get enough attention. That's why I've been a bit so slow in reviewing this PR. While it all seems like an improvement, there are a lot of moving parts in reflection, the interpreter, classpaths and classloaders, and we don't have a lot of testing around these.

I do feel that jarlister tool ought to be a third party project. There is a wide selection of similar tools in the build tool plugin ecosystem that follow this model. The tool would also need integrations into popular build tools (SBT, Maven, Gradle).

The second part of this PR, scala.useurlcp, looks promising. I have a few discussion points around this:

• Make it available via a compiler option, not just a system property
• What about classloader heirarchies? Do we need to look up the chain?
• Is using the thread context classloader always what users need? By contrast, the embeddedDefaults method of populating the classpath (used by SBT to pass a list of classpath entries to a compiler instance created in an application that runs under sbt runMain MyApp) takes a classloader as an parameter.

What about classloader heirarchies? Do we need to look up the chain?

As prior art: The twitter-eval module also calculated the "implied classpath" by probing the classloader, and this traverses up the classloader heirarchy, and also attempts to honour the "Class-Path" entry in the manifest.

Jason,

I have created a new PR for the URLClassLoader mechanism. I have addressed your remarks 1 and 2. I am currently investigating the third one (embeddedDefaults), but according to my extensive tests at the time of the manifest classpath mechanism, the context classloader was working fine in a lot of challenging/embedded/restricted situations like applets, servets and so on.
Regarding this PR (jarlister), what I can do is:

• draw the code out of Scala and create a standalone project (done)
• publish it to maven (to do)
• have the scala ant script to download it (to do)
• use it from the build to jarlist library.jar (done)
• exercise the manifest classpath mechanism in the tests (done)

That way, Scala would be free of the jarlister but the manifest classpath mechanism would still be exercized (it is important for me as it was a huge piece of work). Is it ok for you ?

I do feel that jarlister tool ought to be a third party project

I agree. It is valuable — but this isn't the right repo for it. Hence, closing this PR.