Hi @SethTisue and @lrytz not sure who to ping. Any thoughts? Thanks!

the UIDs should be 3L not 2L

Note that this PR targets 2.12.x

I think we remain serialization compatible between minor releases, the test t8549.scala (SerializationStabilityTest.scala in 2.13.x) checks that (at least to some degree).

Java's default serialization always includes the serial version UID. If not specified explicitly (using an annotation in Scala, a field in Java), the value is calculated based on the class structure. When the UID doesn't match on deserialization, an InvalidClassException is thrown.

Therefore I don't think we can just set the serial version UID to 2 (or 3) for classes that currently don't have an annotation, as it will break serialization compatibility.

What we can do however is make the calculated UID explicit and put it into an annotation. The serialver command line tool displays calculated UIDs:

$> serialver -classpath $(cs fetch -p org.scala-lang:scala-library:2.11.12) 'scala.collection.immutable.BitSet$BitSet2'
scala.collection.immutable.BitSet$BitSet2:    private static final long serialVersionUID = 5802956648923438352L;

Let me add that adding an explicit @SerialVersionUID is very much recommended, because the one that's calculated automatically is sensitive to changes that don't actually affect serialization compatibility, such as adding a public method. Ideally this should come with a test that checks the stability of the serialized form.

See scala/scala-dev#237.

Note that this PR targets 2.12.x

gah, I always miss that. sorry!

@regadas interested in continuing with this?

Hi @SethTisue yeah sorry for the delay! I was on vacation recently and I was still catching up! I'll have a look at the comments! Thx for the poke!

@regadas closing for inactivity, but we can reopen anytime

I'm confused. Is this going to be fixed in 2.12? I'm running into this problem with Spark 3.0.1 on Scala 2.12. Spark 3.0.1 seems to bundle 2.12.10, which is not compatible with 2.12.12 due to this issue. I believe the incompatibility was introduced by b8226e1.

@mallman since we haven't heard from @regadas in a while, you (or another contributor) are free to submit a new PR on the same theme

I don't know how to fix this. Revert the commit?

I don't know how to fix this.

see @lrytz's #9166 (comment) — followup questions welcome

Let's make sure we're on the same page. Is this serialization issue a bug?

yes, it is scala/bug#5046, which was fixed for 2.13 but not for 2.12

Hi @SethTisue Sorry for this, tbh i completely forgot to follow up on this! Let me revive this this week as I have some time now /cc @mallman

I'm going to stick with 2.12.10 for now. I don't have the time to work on a PR for this.

Hi @SethTisue Sorry for this, tbh i completely forgot to follow up on this! Let me revive this this week as I have some time now /cc @mallman

Thank you @regadas.

If we merge this after 2.12.13, we should update these two UIDs to their 2.12.13 value.

Sorry, maybe I misunderstand. To be clear, will 2.12.14 be fully serialization-compatible with 2.12.10?

will 2.12.14 be fully serialization-compatible with 2.12.10?

I don't think so, there are classes without @SerialVersionUID that changed between 2.12.10 and 2.12.14, so instances serialized in 2.12.10 cannot be de-serialized in 2.12.14. De-serialization will fail because the (calculated) SerialVersionUID is different.

The best we can do is to add @SerialVersionUID annotations that correspond to the calculated value of the last release, which is what this PR does (currently, this PR sets the values of 2.12.12). However, there are two cases where the calculated value changes between 2.12.12 and 2.12.13, so we need to update this PR to use the 2.12.13 values.

An example of how to get the SerialVersionUID: #8778 (comment).

Hello 👋 I can update this to reflect the recent 2.12.13 release. /cc @lrytz @dwijnand

The best we can do is to add @serialversionuid annotations that correspond to the calculated value of the last release, which is what this PR does (currently, this PR sets the values of 2.12.12). However, there are two cases where the calculated value changes between 2.12.12 and 2.12.13, so we need to update this PR to use the 2.12.13 values.

Let me give some more background around my query. Spark 3.0.1 ships with Scala 2.12.10. When I try to run a Spark job with Scala 2.12.12 I encounter a serialization failure:

java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID = 3456489343829468865, local class serialVersionUID = 1028182004549731694

From a practical perspective, I'm hoping to see Spark 3.0.1 (+) become compatible with maintenance releases of Scala 2.12 moving forward, so we are not stuck with Scala 2.12.10. I know that the OP referenced Flink compatibility issues as well. Is that a shared goal here, and is it achievable?

(MiMa part needs rebase because of #9436)

@mallman sorry for my late reply!

We probably could set the explicit SerialVersionUIDs in this PR to the values that were valid in 2.12.10, but that would be quite an arbitrary choice (current Spark is using it, but other projects might be on a different version). So I believe that the best way forward is to use the values of our latest release (2.12.13). Once there's a spark release built with 2.12.13 out, a WrappedArray$ofRef serialized on a newer minor Scala releases (2.12.13+) will work.

Does that sound OK?

@regadas do you have time to rebase this to fix the merge conflicts, and to update the annotations to the 2.12.13 values?

@mallman sorry for my late reply!

We probably could set the explicit SerialVersionUIDs in this PR to the values that were valid in 2.12.10, but that would be quite an arbitrary choice (current Spark is using it, but other projects might be on a different version). So I believe that the best way forward is to use the values of our latest release (2.12.13). Once there's a spark release built with 2.12.13 out, a WrappedArray$ofRef serialized on a newer minor Scala releases (2.12.13+) will work.

Does that sound OK?

@regadas do you have time to rebase this to fix the merge conflicts, and to update the annotations to the 2.12.13 values?

Hi @lrytz. Your perspective makes sense—let's not break things that were built against 2.12.12 or 2.12.13.

I'm not sure if the Spark team is aware of this issue. I didn't find anything from performing a cursory search of their issue tracker.

Their current master pom.xml still lists Scala 2.12.10, and Spark 3.1.1 is at rc2 at this point. I will try to raise an issue and submit a PR to upgrade to Spark to Scala 2.12.13, but I think it's unlikely it will be merged in time for Spark 3.1.1.

Thank you.

I will try to raise an issue and submit a PR to upgrade to Spark to Scala 2.12.13

That upgrade was previously tried and reverted due to a bug: apache/spark#31223 (comment) (the fix for that bug will be in 2.12.14).

I will try to raise an issue and submit a PR to upgrade to Spark to Scala 2.12.13

That upgrade was previously tried and reverted due to a bug: apache/spark#31223 (comment) (the fix for that bug will be in 2.12.14).

Thank you for pointing that out.

@lrytz I'll spend some time on this today.

@lrytz I think I addressed everything let me know If I missed smth.