This cuts our full compile time in half (around 1000 profiles), the checkStringIndex is roughly 10-15% and should be generic for all usage (given enough profiles); the retrievePanSource caching is 35-40% gain (given enough profiles and usage of includedirs (5 in our case, via cluster.build.includes) and LOADPATH (2 in our case))
YMMV

Just tested this at RAL:

aquilon

With a small test sandbox this does seem to reduce the compile time slightly, but more importantly introduces no unusual changes to the profiles.

scdb

No measurable speed-up, but also no changes to profiles.

@jrha heh, i'm surprised and a bit dissapointed. if you find the time, you could try to connect viusalvm to the compiler (just start visualvm, it will see any java proc) and do cpu profile dump (it will slow things down). i can have a look and see if anything is different in aquilon.

I'm not sure the aquilon test was large enough to draw any conclusions from, I can profile the scdb build, but I don't know what I'm doing!

@jrha just connect visualvm, start cpu profiing, stop cpu profiling and send me the file 😄

Ok sure, which file? 😇

Quick testing shows that based on the "Total time" reported by panc itself, there's about 10% speedup when generating uncompressed JSON (nice), and about 20% slowdown when generating uncompressed XML (ouch!), both compared to 10.2. Even in the XML case, some batches are slower and some are quicker, so I'll need to dig a bit deeper into where the time is spent. In real life, compressed output speed is what matters for us, but using compression makes comparing the output between two runs more difficult - I'll try that later if I find the time.

The only other difference I see is some Unicode characters, which were previously replaced by '??', now are appearing in the output.

@gombasg thanks for testing, but can you also compare to current master https://jenkins0.ugent.be/job/panc/1822/org.quattor.pan$panc/

i do not know where the xml slowdown could come from; but i haven't tested xml output myself. i also do not know what you use to compare, but there shouldn't be a real issue with using gzipped files (we use the following to compare gzipped json diff -u <(zcat old.json.gz) <(zcat new.json.gz)) (well wrapped in script etc https://github.com/stdweird/quattor-SCDB-ugent/blob/master/zdiff.sh)

@gombasg jar files from latest master are in jenkins, eg https://jenkins0.ugent.be/job/panc/lastSuccessfulBuild/org.quattor.pan$panc/

Closing this PR. will be replaced by smaller PRs, hoping this makes merging easier