The cloud documentation suggests ..."Bigtable can scale to billions of rows and thousands of columns" so this diagonal structured key-value might not be the typical use case for Bigtable

We possibly need a numColumns in Configuration, default set to 1 and make the column key String.format("value%09d", index%numColumn)

Not sure if this is the reason of apache/beam#27022. Also going to investigate it

I tried to set multiple columns as you suggested, it didn't help with the issue.

I find the cause is

• cell does not have set timestamp, so it default to epoch (1970-01-01)
• the createTable has a garbage collection policy of 1h, so large amount data written triggers GC and some records get deleted

We need to use .setTimestampMicros(java.time.Instant.now().toEpochMilli()*1000) for Mutation.SetCell

Thank you!

createTable has a default maxAge of 1h. This causes cells without setting timestamp (default to epoch) gets garbage collected. I think it should be never by default. What do you think @pranavbhandari24

I've updated the javadoc to make it clearer, please check

That's interesting. Yeah, I'd agree that the data should never be garbage collected (if that is an option)
Will look into this.