Incremented major version because batch_size, interval, and initial_interval are moved from package level to data-stream level.

No longer relevant as the configs are moved back to package level.

This doesn't look like it takes units into account. The API expects this to be in days, but we accept h/m/s (and others). Should we say the value is given in days or calculate days from the duration?

The intention was to change the initial_interval format. I updated it to default of 1 from existing 24h. I also updated the description to mention it should be an integer in days.

Ah, I missed that because of the change in the package manifest. I think this is likely to cause confusion.

I think it would be nice to be able to use the same syntax here, to avoid confusion and the conversion from a time.Duration to a day count is now a solved problem, so it should be easy to do.

Updated in 4a4f1ea. Also moved batch_size, interval, and initial_interval back to package level. Removing the need for breaking change.

Tested with 2 scenarios:

1. Initial Interval: 48h:

   • First request: {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:2\"}. This returns last event's lastSeen as 1734500477.
   • Second request (after interval): {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}

2. Initial Interval: 4h:

   • First request: {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}. This returns last event's lastSeen as 1734500477.
   • Second request (after interval): {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}

I think this should have an [[else]], probably reaching back initial_interval.

I'm not sure this calculation is right. What we have is equivalent to (now.Unix - .cursor.last_event_ts)/86400. This will truncate the value meaning that we will potentially lose data; if the cursor value indicates a non-integer day, the time interval calculated in the numerator will be the fraction less than a full day and the remainder of that day will be truncated by the division.

Also, is .cursor.last_event_ts a type that results in epoch seconds when converted to an integer? From the docs, and the code below this does not look like a time value. It looks like it is very hard to hold this API correctly.

I think this should have an [[else]], probably reaching back initial_interval

We have the default block, but I also think it would make sense to avoid template error. I will add the [[else]] case.

I'm not sure this calculation is right.
Also, is .cursor.last_event_ts a type that results in epoch seconds when converted to an integer?

Based on the samples ingested, the value of .cursor.last_event_ts which is derived from .last_event.lastSeen is always in epoch seconds. I verified that lastSeen in the response of each object is epoch seconds. I am not sure if this can make cursor value a non-integer day.
I will remove the int conversion of .cursor.last_event_ts as it is not required.

Based on the samples ingested, the value of .cursor.last_event_ts which is derived from .last_event.lastSeen is always in epoch seconds.

Is the documentation for this?

I am not sure if this can make cursor value a non-integer day.

If the value from the document has come from any time other than 00:00, it will result in a non-integer day before integer truncation, and the truncation will result in the issue that I raised.

Updated the logic as per ceil division to avoid data loss: a7055b9

Here are the results:

1. Re-verified documents counts between exiting package version 1.27.0 (Initial Interval: 48h) and new change version 2.0.0 (Initial Interval: 2). The counts match.
2. Verified from tracers the lastSeen is correctly getting applied (Initial Interval: 2):
   • First request: {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:2\"}. This returns last event's lastSeen as 1734500477.
   • Second request (after interval): {\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}