Validate timestamps when creating windowed or timestamped values by kennknowles · Pull Request #16749 · apache/beam

kennknowles · 2022-02-04T23:31:01Z

In debugging a problem with out-of-bounds timestamps, some bad timestamps are caught when a DoFn outputs, when it would be better to catch it early as possible.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

kennknowles · 2022-02-04T23:31:09Z

R: @laraschmidt

kennknowles · 2022-02-08T19:00:23Z

Run Java PreCommit

kennknowles · 2022-02-08T22:38:55Z

green!

y1chi · 2022-02-11T00:33:49Z

R: @laraschmidt

laraschmidt · 2022-02-11T01:35:21Z

This is going to break some internal tests as is because of the bug we found on the portable runner. We may have to wait for that to be fixed and rolled out first.

kennknowles · 2022-02-11T19:09:21Z

On the other hand, we know from those tests that the portable runner is broken, so would sickbaying the tests work?

laraschmidt · 2022-02-11T19:28:57Z

That issue is only affecting a very small subset of jobs because of where the check is located (e.g. it requires allowed skew be infinite). I'm pretty sure this would affect all jobs with impulse.

kennknowles · 2022-02-16T14:24:52Z

run dataflow validatesrunner

kennknowles · 2022-02-16T14:26:07Z

Run Java Dataflow V2 ValidatesRunner

kennknowles · 2022-02-16T14:26:14Z

Run Java Dataflow V2 ValidatesRunner Streaming

aaltay · 2022-02-25T00:52:53Z

What is the next step on this PR?

laraschmidt · 2022-02-25T01:05:18Z

Needs to wait for an internal dataflow fix to roll out, otherwise it will break Dataflow. :)

github-actions · 2022-04-26T12:55:33Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

kennknowles · 2022-04-26T17:04:34Z

I'm thinking this might be ready now.

kennknowles · 2022-04-26T17:26:19Z

Run Dataflow ValidatesRunner

kennknowles · 2022-04-26T18:53:36Z

known pulsar & fnapi flakes

kennknowles · 2022-04-26T18:53:42Z

Run Java PreCommit

kennknowles · 2022-04-27T15:45:16Z

Run Java Dataflow V2 ValidatesRunner

kennknowles · 2022-04-27T15:49:53Z

Run Java Dataflow V2 ValidatesRunner Streaming

kennknowles · 2022-04-27T20:52:06Z

Yup, looks good now. PTAL.

kennknowles · 2022-04-28T01:59:40Z

Run Java Dataflow V2 ValidatesRunner Streaming

kennknowles · 2022-04-28T22:20:04Z

@dpcollins-google

aaltay · 2022-05-12T21:41:17Z

@dpcollins-google - could you please review this?

aaltay · 2022-05-27T03:04:07Z

@dpcollins-google - ping?

dpcollins-google · 2022-05-27T11:25:56Z

This seems to be a different issue than the one I'm running into. In particular, its validating that the timestamp is in bounds [MIN, MAX] while my issue is that impulse is not returning MIN. LGTM though

kennknowles · 2022-05-31T15:25:21Z

Taking a look at whether runner v2 is still producing invalid timestamps.

kennknowles · 2022-05-31T19:26:48Z

Seeing Provided timestamp -290308-12-21T19:59:05.224Z must be within bounds [-290308-12-21T19:59:05.225Z, 294247-01-10T04:00:54.775Z]. Unless I am reading it wrong, that timestamp is within bounds, so the check added here is buggy.

aaltay · 2022-06-09T20:18:25Z

Last ping from me on this PR :)

What is the next step on this PR? Should this be closed?

kennknowles · 2022-06-16T15:08:09Z

No need to ping. Just my limited bandwidth. I will get to it.

github-actions · 2023-06-20T12:43:21Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-06-27T12:45:21Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions · 2023-08-29T12:41:39Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-09-06T12:41:47Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

hpvd · 2024-04-24T11:48:29Z

added this to [Parent issue] Support for Apache Pulsar #31078

github-actions bot added the java label Feb 4, 2022

kennknowles force-pushed the timestampedvalue branch from c5077ec to 83e5908 Compare February 7, 2022 22:12

github-actions bot added direct runners labels Feb 7, 2022

kennknowles force-pushed the timestampedvalue branch 3 times, most recently from 9153479 to 63c6723 Compare February 8, 2022 17:00

laraschmidt mentioned this pull request Feb 17, 2022

Fix final allowskew error to properly handle a large allowedSkew #16885

Merged

4 tasks

github-actions bot added the stale label Apr 26, 2022

kennknowles removed the stale label Apr 26, 2022

kennknowles force-pushed the timestampedvalue branch from 63c6723 to 774e8e0 Compare April 26, 2022 17:05

dpcollins-google approved these changes May 27, 2022

View reviewed changes

Validate timestamps when creating windowed or timestamped values

43f95b3

kennknowles force-pushed the timestampedvalue branch from 774e8e0 to 43f95b3 Compare May 31, 2022 15:27

github-actions bot added the stale label Jun 20, 2023

github-actions bot closed this Jun 27, 2023

kennknowles reopened this Jun 28, 2023

github-actions bot removed the stale label Jun 29, 2023

github-actions bot added the stale label Aug 29, 2023

github-actions bot closed this Sep 6, 2023

kennknowles deleted the timestampedvalue branch January 4, 2024 20:35

hpvd mentioned this pull request Apr 24, 2024

[Parent issue] Support for Apache Pulsar #31078

Open

Conversation

kennknowles commented Feb 4, 2022

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

kennknowles commented Feb 4, 2022

Uh oh!

kennknowles commented Feb 8, 2022

Uh oh!

kennknowles commented Feb 8, 2022

Uh oh!

y1chi commented Feb 11, 2022

Uh oh!

laraschmidt commented Feb 11, 2022

Uh oh!

kennknowles commented Feb 11, 2022

Uh oh!

laraschmidt commented Feb 11, 2022

Uh oh!

kennknowles commented Feb 16, 2022

Uh oh!

kennknowles commented Feb 16, 2022

Uh oh!

kennknowles commented Feb 16, 2022

Uh oh!

aaltay commented Feb 25, 2022

Uh oh!

laraschmidt commented Feb 25, 2022

Uh oh!

github-actions bot commented Apr 26, 2022

Uh oh!

kennknowles commented Apr 26, 2022

Uh oh!

kennknowles commented Apr 26, 2022

Uh oh!

kennknowles commented Apr 26, 2022

Uh oh!

kennknowles commented Apr 26, 2022

Uh oh!

kennknowles commented Apr 27, 2022

Uh oh!

kennknowles commented Apr 27, 2022

Uh oh!

kennknowles commented Apr 27, 2022

Uh oh!

kennknowles commented Apr 28, 2022

Uh oh!

kennknowles commented Apr 28, 2022

Uh oh!

aaltay commented May 12, 2022

Uh oh!

aaltay commented May 27, 2022

Uh oh!

dpcollins-google commented May 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kennknowles commented May 31, 2022

Uh oh!

kennknowles commented May 31, 2022

Uh oh!

aaltay commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kennknowles commented Jun 16, 2022

Uh oh!

github-actions bot commented Jun 20, 2023

Uh oh!

github-actions bot commented Jun 27, 2023

Uh oh!

github-actions bot commented Aug 29, 2023

Uh oh!

github-actions bot commented Sep 6, 2023

Uh oh!

hpvd commented Apr 24, 2024

Uh oh!

Reviewers

Assignees

`ValidatesRunner` compliance status (on master branch)

dpcollins-google commented May 27, 2022 •

edited

Loading

aaltay commented Jun 9, 2022 •

edited

Loading