[Failing Test]: Java_GCP_IO_Direct on Github Actions is very flaky due to OOMs when running FhirIOSearchIT tests #28635
Description
What happened?
Looks like both testFhirIOSearchWithGenericParameters and testFhirIOSearch are failing due to OOM errors. Seeing these errors very frequently when running Java_GCP_IO_Direct on Github Actions but not so much for the Jenkins jobs.
Example runs: https://github.com/apache/beam/actions/runs/6281472530/job/17059883679?pr=28634, https://github.com/apache/beam/actions/runs/6278950376/job/17053724245?pr=28631
Not to say it doesn't happen on Jenkins (see this failing test), just not seeing it as much
Stacktrace dump for testFhirIOSearch:
com.google.gson.JsonParseException: Failed parsing JSON source: JsonReader at line 1 column 518561 path $[121].resource.item[1].adjudication[1].category.coding[0].code to Json
at com.google.gson.JsonParser.parseReader(JsonParser.java:106)
at com.google.gson.JsonParser.parseReader(JsonParser.java:70)
at com.google.gson.JsonParser.parseString(JsonParser.java:51)
at org.apache.beam.sdk.io.gcp.healthcare.JsonArrayCoder.decode(JsonArrayCoder.java:46)
at org.apache.beam.sdk.io.gcp.healthcare.JsonArrayCoder.decode(JsonArrayCoder.java:28)
at org.apache.beam.sdk.coders.Coder.decode(Coder.java:154)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:84)
at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:37)
at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:142)
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:102)
at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:96)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:168)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions(MutationDetectors.java:139)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified(MutationDetectors.java:128)
at org.apache.beam.runners.direct.ImmutabilityEnforcementFactory$ImmutabilityCheckingEnforcement.verifyUnmodified(ImmutabilityEnforcementFactory.java:146)
at org.apache.beam.runners.direct.ImmutabilityEnforcementFactory$ImmutabilityCheckingEnforcement.afterElement(ImmutabilityEnforcementFactory.java:131)
at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:175)
at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at com.google.gson.stream.JsonReader.nextQuotedValue(JsonReader.java:1014)
at com.google.gson.stream.JsonReader.nextName(JsonReader.java:798)
at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:783)
at com.google.gson.internal.bind.TypeAdapters$28.read(TypeAdapters.java:725)
at com.google.gson.internal.Streams.parse(Streams.java:49)
at com.google.gson.JsonParser.parseReader(JsonParser.java:102)
Issue Failure
Failure: Test is flaky
Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner