Scala compiler classpath gets confused by directory ending in .zip #12019
Description
we compile scala class files at runtime. the scala files we compile are configuration files in a custom dsl, mainly used for spark. for this we use the scala compiler in scala.tools.nsc.Global for scala 2.12.10
recently we started tested spark cluster deploy mode (so our code now runs on cluster in a jvm not directly created by us instead of in client jvm). as it turns out the jvm on the cluster had an unusual classpath including a directory named __spark_conf__.zip. this confused the classpath code used by the scala compiler, which tries to read this directory on the classpath as a file (presumably because its ending in .zip? maybe adding a simple File.isFile() check will solve this?)
below is a redacted stacktrace.
best,
koert
2020-05-26 13:14:47 ERROR yarn.ApplicationMaster: User class threw exception: scala.reflect.internal.FatalError: Error accessing /data/8/hadoop/nm/usercache/koert/filecache/722/__spark_conf__.zip
scala.reflect.internal.FatalError: Error accessing /data/8/hadoop/nm/usercache/koert/filecache/722/__spark_conf__.zip
at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:99)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:87)
at scala.tools.nsc.util.ClassPath.list(ClassPath.scala:36)
at scala.tools.nsc.util.ClassPath.list$(ClassPath.scala:36)
at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:30)
at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:284)
at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:230)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1542)
at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:257)
at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:74)
at scala.tools.nsc.Global.rootMirror(Global.scala:72)
at scala.tools.nsc.Global.rootMirror(Global.scala:44)
at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:295)
at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:295)
at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1480)
at scala.tools.nsc.Global$Run.<init>(Global.scala:1199)
<redacted>
Caused by: java.io.IOException: Error accessing /data/8/hadoop/nm/usercache/koert/filecache/722/__spark_conf__.zip
at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:141)
at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:179)
at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:176)
at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:212)
at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:212)
at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76)
at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63)
at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62)
at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:56)
at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:91)
... 35 more
Caused by: java.io.FileNotFoundException: /data/8/hadoop/nm/usercache/koert/filecache/722/__spark_conf__.zip (Is a directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:225)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.zip.ZipFile.<init>(ZipFile.java:169)
at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:138)
... 44 more