[Response Ops][Reporting] Scheduled Reports - Task (merging into feature branch) by ymao1 · Pull Request #219770 · elastic/kibana

ymao1 · 2025-04-30T16:51:13Z

Towards #216313

Note

This PR will be merged into a feature branch

Summary

This PR adds the scheduled_report task, which generates reports on a recurring cadence.

maxConcurrency of this task is 1 so only one can run at a time
this task shares concurrency with the existing report:execute task, so only one of either task type can run at a time
this task requires an API key to run so it will only run when ES security and API keys are enabled and a permanent encryption key for Kibana is set
when the task runs, it creates a new ReportSource document in the .kibana-reporting index that stores the final base64 encoded generated output. This ReportSource matches the data model of existing reports with the addition of a scheduled_report_id field which links to the scheduled report saved object ID.

Verify

Use the dev console to schedule some reports.
Verify the the reports get generated at the expected cadence
Verify that the correct time range is used for each scheduled report

Note, the UI has not been updated to differentiate between scheduled and single run reports. They will look the same from in the report listing, except the scheduled report will have a timestamp appended to the report name.

In the cloud deployment for this PR, we have an example scheduled report that is running every day at 4:45 EDT: https://kibana-pr-219770.kb.us-west2.gcp.elastic-cloud.com/app/management/insightsAndAlerting/reporting

…apping'

…-fix'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

…te --fix'

… src/core/server/integration_tests/ci_checks'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

… src/core/server/integration_tests/ci_checks'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

elasticmachine · 2025-05-20T18:55:25Z

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #37 / console app misc console behavior keyboard shortcuts open documentation should open documentation when Ctrl+/ is pressed
[job] [logs] FTR Configs #37 / console app misc console behavior keyboard shortcuts open documentation should open documentation when Ctrl+/ is pressed

History

💚 Build #301110 succeeded 4ca09af
💛 Build #299874 was flaky ea37895
💛 Build #299433 was flaky 9a7bda6
💛 Build #299078 was flaky e20f1bf
💚 Build #299027 succeeded 4a4e0c5

…heduled-reports-task

pmuellr · 2025-05-28T01:45:55Z

Took me a bit to figure out, so saving for posterity. curl command to schedule the web logs dashboard to generate a report every hour:

curl $KB_URL/internal/reporting/schedule/printablePdfV2 \
  -H "kbn-xsrf: foo" \
  -H "x-elastic-internal-origin: kibana" \
  -H "content-type: application/json" \
  -d '
{
  "jobParams":"(browserTimezone:America/New_York,layout:(dimensions:(height:2220,width:1335),id:preserve_layout),locatorParams:!((id:DASHBOARD_APP_LOCATOR,params:(dashboardId:edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b,preserveSavedFilters:!t,timeRange:(from:now-7d/d,to:now),useHash:!f,viewMode:edit))),objectType:dashboard,title:LogsWebTraffic)",
  "schedule": { "rrule": { "freq": 3, "interval": 1, "byhour": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23] } }
}
'

pmuellr · 2025-05-29T13:00:50Z

I tried creating a scheduled report in a non-default space, via my incantation above, but with a /s/test prefix, and saw the following logged when the report finally ran:

[ERROR][plugins.reporting.runTask] Error: Saved object [scheduled_report/ebcfc3e8-13ad-405c-b486-66b4bcc5ea0e] not found
    at Function.createGenericNotFoundError (saved_objects_error_helpers.ts:259:28)
    at performGet (get.ts:73:36)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at SavedObjectsRepository.get (repository.ts:371:12)
    at RunScheduledReportTask.prepareJob (run_scheduled_report.ts:44:24)
    at Object.run (run_report.ts:443:15)

I actually referencede a dashboard that WASN'T in this space, which I realized later, so I wouldn't have expected it to actually generate a report. But it looks like it had a problem loading the scheduled report saved object.

pmuellr · 2025-05-29T13:02:57Z

Given the error I saw, mentioned in the comment above, I noticed it did NOT get retried. I think this is by design, but probably something we want to figure out, since it sometimes happens that a retry will get the search indices warmed up to get a long-running report to succeed after an initial failure.

…heduled-reports-task

ymao1 · 2025-05-29T13:57:11Z

I tried creating a scheduled report in a non-default space, via my incantation above, but with a /s/test prefix, and saw the following logged when the report finally ran:

Good catch! Updated to be space aware in 1f46d45

ymao1 · 2025-05-29T13:58:30Z

Given the error I saw, mentioned in the comment above, I noticed it did NOT get retried. I think this is by design, but probably something we want to figure out, since it sometimes happens that a retry will get the search indices warmed up to get a long-running report to succeed after an initial failure.

Yea, we typically don't perform retries for recurring tasks so there's no built in retry. I think we'd have to update task manager to allow retries for recurring tasks if we want that ability. I can open an issue to discuss

… scheduled-reports-task

pmuellr · 2025-05-30T15:14:47Z

Yea, we typically don't perform retries for recurring tasks so there's no built in retry. I think we'd have to update task manager to allow retries for recurring tasks if we want that ability. I can open an issue to discuss

Ya, let's open an issue. Given Tim's previous note that "subsequent attempts often succeed", I think we'll have to do something. Doesn't seem like a blocker to get the basic function out. Wondering if we'd just want to do an "inline retry" of these, which would complicate the task itself, but not require any new tasks, or changes to TM ...

ymao1 · 2025-05-30T15:27:22Z

Ya, let's open an issue. Given Tim's previous note that "subsequent attempts often succeed", I think we'll have to do something. Doesn't seem like a blocker to get the basic function out. Wondering if we'd just want to do an "inline retry" of these, which would complicate the task itself, but not require any new tasks, or changes to TM ...

Issue: #222079

pmuellr

LGTM, was able to generate PDF reports as usual in default and a test space, as well as schedule a report in both spaces, and have them run successfully.

pmuellr · 2025-05-27T21:20:08Z

x-pack/platform/plugins/private/reporting/server/core.ts

+    }
+
+    // otherwise use internal repository
+    return savedObjects.createInternalRepository([SCHEDULED_REPORT_SAVED_OBJECT_TYPE]);


I wonder if this could be confusing in the future, since it returns either a scoped or internal client. And complicates the typing (minor concern). Would it be better to have separate methods? It seems pretty obvious to me in the code, right now ...

Split into separate functions in ad65b91

elasticmachine · 2025-05-30T18:49:28Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/reporting-common`	90	93	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.7MB	1.7MB	+252.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`reporting`	49.7KB	49.9KB	+194.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/reporting-common`	100	103	+3

History

💛 Build #303439 was flaky 035d994
💔 Build #303399 failed a3ed61d
💚 Build #302662 succeeded 3955376
💚 Build #302071 succeeded d60bbff
💚 Build #301395 succeeded 4750222

cc @ymao1

Resolves #216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - #219771 * Scheduled report task runner - #219770 * List and disable API - #220922 * Audit logging - #221846 * Send scheduled report emails - #220539 * Commit to check license - f5f9d9d * Update to list API response format - #224262 --------- Co-authored-by: Ersin Erdal <ersin.erdal@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com>

Resolves elastic#216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - elastic#219771 * Scheduled report task runner - elastic#219770 * List and disable API - elastic#220922 * Audit logging - elastic#221846 * Send scheduled report emails - elastic#220539 * Commit to check license - elastic@f5f9d9d * Update to list API response format - elastic#224262 --------- Co-authored-by: Ersin Erdal <ersin.erdal@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> (cherry picked from commit a409627) # Conflicts: # src/platform/packages/private/kbn-reporting/common/routes.ts # x-pack/platform/plugins/private/canvas/server/feature.test.ts # x-pack/platform/plugins/private/reporting/server/core.ts # x-pack/platform/plugins/private/reporting/server/features.ts # x-pack/platform/plugins/shared/features/server/__snapshots__/oss_features.test.ts.snap # x-pack/platform/test/api_integration/apis/features/features/features.ts # x-pack/test_serverless/api_integration/test_suites/chat/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/observability/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/search/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/security/platform_security/authorization.ts

# Backport This will backport the following commits from `main` to `8.19`: - [[Response Ops][Reporting] Scheduled Reports (#221028)](#221028)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

ersin-erdal and others added 30 commits April 9, 2025 18:38

Support rrule for task scheduling

a74e49d

remove redundant return

6163f7e

add functional tests

20ecece

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

c29a08e

…apping'

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

de20db1

…-fix'

fix linting

65f0960

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

2dde5f6

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

fix linter

717cadb

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

72b108e

…te --fix'

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

38e69ef

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

c9e71ba

… src/core/server/integration_tests/ci_checks'

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

5f6fad9

Allowing shared concurrency task types

d05d9c8

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

4a04b90

set second to [0]

249f14c

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

4c60d8e

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

testing schedule is more resilient

4a14142

remove "only"

4ec1bca

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

1275f1e

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

8aff2c9

Merge branch 'main' into shared-concurrency-tasks

04b70a2

Merge branch 'main' into shared-concurrency-tasks

c9547ae

throw when interval is not provided for the next run calculation

850ec64

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

10c40e5

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

af75dfb

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

95cf333

… src/core/server/integration_tests/ci_checks'

update read me

d5227aa

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

2a1482e

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

Merge branch 'main' into shared-concurrency-tasks

372d4c7

update read me

62fe072

Merge branch 'scheduled-reports' of github.com:elastic/kibana into sc…

4750222

…heduled-reports-task

ymao1 self-assigned this May 21, 2025

ymao1 marked this pull request as ready for review May 21, 2025 17:17

ymao1 requested a review from a team as a code owner May 21, 2025 17:17

ymao1 requested review from pmuellr and umbopepato May 21, 2025 17:18

ymao1 changed the title ~~[Response Ops][Reporting] Scheduled Reports - Task~~ [Response Ops][Reporting] Scheduled Reports - Task (merging into feature branch) May 21, 2025

ymao1 mentioned this pull request May 21, 2025

[Response Ops][Reporting] Scheduled Reports #221028

Merged

ymao1 added 2 commits May 23, 2025 09:50

Merge branch 'scheduled-reports' of github.com:elastic/kibana into sc…

d60bbff

…heduled-reports-task

Merge branch 'scheduled-reports' of github.com:elastic/kibana into sc…

3955376

…heduled-reports-task

ymao1 added 2 commits May 29, 2025 09:22

Merge branch 'scheduled-reports' of github.com:elastic/kibana into sc…

5a2c076

…heduled-reports-task

Fixing space access

1f46d45

kibanamachine and others added 3 commits May 29, 2025 14:07

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

a3ed61d

Fixing types

d95012c

Merge branch 'scheduled-reports-task' of github.com:ymao1/kibana into…

035d994

… scheduled-reports-task

pmuellr approved these changes May 30, 2025

View reviewed changes

PR feedback

ad65b91

ymao1 merged commit 9cf25dd into elastic:scheduled-reports May 30, 2025
10 checks passed

Conversation

ymao1 commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verify

Uh oh!

elasticmachine commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

History

Uh oh!

pmuellr commented May 28, 2025

Uh oh!

pmuellr commented May 29, 2025

Uh oh!

pmuellr commented May 29, 2025

Uh oh!

ymao1 commented May 29, 2025

Uh oh!

ymao1 commented May 29, 2025

Uh oh!

pmuellr commented May 30, 2025

Uh oh!

ymao1 commented May 30, 2025

Uh oh!

pmuellr left a comment

Choose a reason for hiding this comment

Uh oh!

pmuellr May 27, 2025

Choose a reason for hiding this comment

Uh oh!

ymao1 May 30, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ymao1 commented Apr 30, 2025 •

edited

Loading

elasticmachine commented May 20, 2025 •

edited

Loading

elasticmachine commented May 30, 2025 •

edited

Loading