[One Workflow] Implement outputs, workflow.output and workflow.fail steps #253700
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements workflow outputs functionality, enabling workflows to declare structured output fields, emit values at runtime, and terminate execution with specific status codes.
Changes:
- Added
outputstop-level field to workflow schema with validation for string, number, boolean, choice, and array types - Implemented
workflow.outputandworkflow.failstep types with execution engine support - Enhanced editor autocomplete to suggest declared output fields within
workflow.outputsteps'with:blocks
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| validate_workflow_outputs.ts | Delegates output validation to shared field validator |
| validate_workflow_outputs.test.ts | Comprehensive validation tests for all output types and constraints |
| validate_workflow_fields.ts | Generic Zod-based validator shared between inputs and outputs |
| generate_connector_snippet.ts | Updated to use centralized isBuiltInStepType helper |
| generate_builtin_step_snippet.ts | Added snippet generation for workflow.output and workflow.fail steps |
| get_workflow_outputs_suggestions.ts | Context-aware autocomplete for output field names in workflow.output steps |
| get_suggestions.ts | Refactored suggestion logic to support workflow output autocomplete |
| get_connector_type_suggestions.ts | Updated to pass workflow outputs for dynamic snippet generation |
| variable.test.ts | Added integration tests for workflow output autocomplete |
| get_completion_item_provider.ts | Suppresses YAML provider suggestions inside workflow.output with blocks |
| autocomplete.types.ts | Changed workflowDefinition type to WorkflowYaml for better type safety |
| get_workflow_context_schema.ts | Builds output schema from declared outputs for variable autocomplete |
| validate_workflow_outputs_in_yaml.ts | Validates workflow.output steps against declared outputs schema |
| use_yaml_validation.ts | Integrates workflow output validation into YAML validation pipeline |
| workflow_execution_state.ts | Fixed merge logic to avoid overwriting defined fields with undefined |
| workflow_execution_runtime_manager.ts | Added methods for setting outputs, status, and completing ancestor steps |
| step_execution_runtime.ts | Made workflowExecution getter public and improved type safety |
| build_workflow_context.ts | Added output field to workflow context |
| workflow_output_step_impl.ts | Core implementation of workflow.output step with validation and termination logic |
| nodes_factory.ts | Added factory method for workflow.output node creation |
| workflow_fail.test.ts | Integration tests for workflow.fail step execution |
| schema.ts | Added WorkflowOutput, WorkflowOutputStep, and WorkflowFailStep schemas |
| schema.test.ts | Schema validation tests for workflow output step types |
| generate_yaml_schema_from_connectors.ts | Registered new step types in YAML schema generation |
| base.ts | Added WorkflowOutputGraphNode type |
| build_execution_graph.ts | Added graph building logic for workflow.output and workflow.fail steps |
...ws_management/public/widgets/workflow_yaml_editor/lib/validation/validate_workflow_fields.ts
Outdated
Show resolved
Hide resolved
...hared/workflows_execution_engine/server/workflow_context_manager/workflow_execution_state.ts
Outdated
Show resolved
Hide resolved
...red/workflows_execution_engine/server/step/workflow_output_step/workflow_output_step_impl.ts
Outdated
Show resolved
Hide resolved
rosomri
left a comment
There was a problem hiding this comment.
Nice work on the outputs implementation! 👑
One design observation: outputs currently only support the legacy array format ([{ name, type, required }]), while inputs already support both legacy and JSON Schema formats with automatic normalization.
If outputs followed the same pattern (normalize to JSON Schema at parse time via .transform() like inputs do), the existing utilities (convertLegacyInputToJsonSchemaProperty, normalizeInputsToJsonSchema, convertJsonSchemaToZod) could replace the three custom output-to-Zod implementations (makeOutputValidator, makeWorkflowFieldsValidator, outputToSchema/buildOutputsSchema), cutting ~400 lines of new code and keeping inputs/outputs architecturally consistent.
That said, if adding JSON Schema support for outputs is a larger design change that you see as out of scope for this PR, I’m fine moving forward as-is. I just wanted to flag the opportunity - aligning with the input pipeline would give you JSON Schema support and shared utilities essentially for free down the line.
...ws_management/public/widgets/workflow_yaml_editor/lib/validation/validate_workflow_fields.ts
Outdated
Show resolved
Hide resolved
|
|
||
| const stepsSeq = stepsNode as YAMLSeq; | ||
|
|
||
| for (const stepNode of stepsSeq.items) { |
There was a problem hiding this comment.
Note, this only iterates top-level steps items. workflow.output steps nested inside if or foreach blocks will not be validated by the editor
There was a problem hiding this comment.
Added recursion in validate_workflow_outputs_in_yaml.ts: collectFromStepsSequence() traverses nested steps and else in if/foreach.
| @@ -0,0 +1,314 @@ | |||
| /* | |||
There was a problem hiding this comment.
There are thorough integration tests for workflow.fail, but no integration test for workflow.output with status: 'completed' (the happy path). The schema tests validate parsing, but they don't verify runtime behavior: outputs stored in context.output, workflow status is COMPLETED, subsequent steps are NOT executed.
Consider adding a workflow_output.test.ts alongside this file covering: basic output emission, output validation against declared schema, early termination (steps after workflow.output should not run), and type-preserving template expressions.
There was a problem hiding this comment.
Added workflow_output.test.ts (basic emission, early termination, templates, workflow.output inside if).
|
|
||
| // Fail the step with the error | ||
| // Note: failStep() already sets the workflow error, so we don't need to call setWorkflowError() separately | ||
| this.stepExecutionRuntime.failStep(failureError); |
There was a problem hiding this comment.
failStep() only sets the step-level error (on the step execution document), not the workflow-level error. This means workflowExecution.error won't be populated with the failure message.
The validation failure path (line ~180) and catch block (line ~274) both call setWorkflowError() explicitly after failStep(). This branch should do the same for consistency:
this.stepExecutionRuntime.failStep(failureError);
this.workflowExecutionRuntime.setWorkflowError(failureError);Also, the comment on line 234 should be updated since it's incorrect.
@VladimirFilonov let me know if I'm misunderstanding the flow here - is there maybe error propagation happening elsewhere that covers this case?
There was a problem hiding this comment.
failStep() already calls updateWorkflowExecution({ error }). I removed redundant setWorkflowError() from validation and catch paths;
Acknowledged as future improvement - will create follow-up ticket for that |
...red/workflows_execution_engine/server/step/workflow_output_step/workflow_output_step_impl.ts
Fixed
Show fixed
Hide fixed
…d schema validations
…r template syntax
…e proper step completion and error reporting
4a6049f to
5e59574
Compare
…low-output-and-fail
...red/workflows_execution_engine/server/step/workflow_output_step/workflow_output_step_impl.ts
Outdated
Show resolved
Hide resolved
...red/workflows_execution_engine/server/step/workflow_output_step/workflow_output_step_impl.ts
Outdated
Show resolved
Hide resolved
...red/workflows_execution_engine/server/step/workflow_output_step/workflow_output_step_impl.ts
Outdated
Show resolved
Hide resolved
...agement/public/features/validate_workflow_yaml/lib/use_monaco_markers_changed_interceptor.ts
Show resolved
Hide resolved
...agement/public/widgets/workflow_yaml_editor/lib/autocomplete/get_completion_item_provider.ts
Show resolved
Hide resolved
src/platform/packages/shared/kbn-workflows/spec/lib/build_zod_schema_from_fields.ts
Outdated
Show resolved
Hide resolved
src/platform/packages/shared/kbn-workflows/spec/lib/build_zod_schema_from_fields.ts
Outdated
Show resolved
Hide resolved
src/platform/plugins/shared/workflows_management/common/lib/json_schema_to_zod.ts
Show resolved
Hide resolved
src/platform/plugins/shared/workflows_execution_engine/server/utils/templates.ts
Fixed
Show fixed
Hide fixed
rosomri
left a comment
There was a problem hiding this comment.
Several call sites need explicit casts:
- load_workflows_thunk.ts:
workflow.definition?.inputs as NormalizableFieldSchema - validate_json_schema_defaults.ts:
inputs as NormalizableFieldSchema - workflow_execute_modal.tsx:
inputs as NormalizableFieldSchema - workflow_output_step_impl.ts:
declaredOutputs as NormalizableFieldSchema
This suggests the upstream types (WorkflowYaml.inputs, WorkflowYaml.outputs) don't align with NormalizableFieldSchema. Consider widening the WorkflowYaml type to make inputs and outputs typed as NormalizableFieldSchema | undefined so these casts become unnecessary.
src/platform/packages/shared/kbn-workflows/spec/lib/build_fields_zod_validator.test.ts
Show resolved
Hide resolved
| const stepType = getStepType(stepNode); | ||
| if (stepType === 'workflow.output') { | ||
| collectWorkflowOutputStepFromNode(stepNode, items, model); | ||
| } else if (stepType === 'foreach' || stepType === 'if') { |
There was a problem hiding this comment.
Not all cases are covered - what about on-failure/timeout/while steps?
Also, each new step would need to be explicitly added here, which is fragile at best and essentially broken.
Instead, could the steps be flattened somehow, instead of traversing the DAG? The graph structure doesn’t really matter anyway.
There was a problem hiding this comment.
https://github.com/elastic/kibana/pull/253700/changes#r2925014657 - can fix it as well
| export function makeWorkflowOutputsValidator(outputs: NormalizableFieldSchema) { | ||
| return makeWorkflowFieldsValidator(outputs); | ||
| } |
There was a problem hiding this comment.
nit - Why not call makeWorkflowFieldsValidator directly?
| @@ -54,7 +57,9 @@ export const generateSchema = ({ workflow }: { workflow: WorkflowDetailDto }): z | |||
| } | |||
|
|
|||
| // Normalize inputs to the new JSON Schema format (handles backward compatibility) | |||
| const normalizedInputs = normalizeInputsToJsonSchema(workflow.definition.inputs); | |||
| const normalizedInputs = normalizeFieldsToJsonSchema( | |||
| workflow.definition.inputs as NormalizableFieldSchema | |||
There was a problem hiding this comment.
needed? you already overcome it
| workflow.definition.inputs as NormalizableFieldSchema | |
| workflow.definition.inputs |
|
|
| * Reads the outputs section from the raw YAML document as a plain object. | ||
| * Used when workflowDefinition.outputs is missing (e.g. schema parse stripped it). |
There was a problem hiding this comment.
I don't really get the usecase and why a fallback is needed instead of relying on the 'outputs' only, but anyway, the logic only recognizes the JSON Schema format. If outputs are declared in the legacy array format (which is valid per WorkflowOutputSchema = WorkflowInputSchema), this fallback won't detect them
There was a problem hiding this comment.
-
Documented the fallback use case in JSDoc - we use it when workflowDefinition.outputs is missing (e.g. parse failure or definition not yet available) so we can still validate workflow.output steps from the raw YAML.
I think it's valid, sinceworkflow.outputsallow any amount for extra keys, not described in schema. So it's fare to assume that it can be none + extra fields -
Fallback now supports legacy array format as well: if the raw document has outputs: [{ name, type, ... }], we return it and normalizeFieldsToJsonSchema handles it. Added a test for the legacy fallback path.
| /** | ||
| * Recursively collects workflow.output steps from a steps sequence (including nested if/foreach). | ||
| */ | ||
| function collectFromStepsSequence( |
There was a problem hiding this comment.
The manual recursive traversal (getStepType, findNestedStepsSequence, collectFromStepsSequence, collectWorkflowOutputSteps) reimplements logic that already exists in common/lib/yaml/get_step_nodes_with_type.ts. That utility uses visit() to walk the entire YAML AST and collect all step nodes at any nesting depth - it already handles if, foreach, while, etc.
You could replace the ~60-80 lines of traversal code with:
import { getStepNodesWithType } from '../../../../common/lib/yaml/get_step_nodes_with_type';
const allStepNodes = getStepNodesWithType(yamlDocument);
const outputStepNodes = allStepNodes.filter((node) => {
const typePair = node.items.find(
(item) => isPair(item) && isScalar(item.key) && item.key.value === 'type'
);
return isPair(typePair) && isScalar(typePair.value) && typePair.value.value === 'workflow.output';
});
| @@ -0,0 +1,269 @@ | |||
| /* | |||
There was a problem hiding this comment.
The functions findStepIndexAtPosition, getStepTypeFromIndex, stepHasWithProperty, and isInWorkflowOutputWithBlock manually scan YAML lines to determine if the cursor is inside a workflow.output step's with: block. But focusedStepInfo from the autocomplete context already provides this - it's derived from the YAML AST and handles nesting correctly.
This whole chain could be replaced with:
focusedStepInfo?.stepType === 'workflow.output'>
This is the same approach used in get_workflow_inputs_suggestions.ts (line 117), which checks focusedStepInfo?.stepType === 'workflow.execute' without any custom line scanning.
Bonus: findStepIndexAtPosition only checks top-level steps, so autocomplete inside nested blocks likely won't trigger. focusedStepInfo handles those correctly.
| ) : ( | ||
| <> | ||
| {selectedTabId === 'output' && ( | ||
| <> |
There was a problem hiding this comment.
You sure these are not needed anymore?
There was a problem hiding this comment.
Yes, this is wrong callout for Output: "You can reference these values using {{ inputs. }}"
It's just an accidental leftover
…low-output-and-fail
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Any counts in public APIs
Async chunks
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
History
|
…teps (elastic#253700) ## Summary Adds support for **workflow outputs**, the **`workflow.output`** step, and the **`workflow.fail`** step — enabling workflows to declare structured outputs, emit values at runtime, and terminate with a failure status. This is part of [Workflow Composition](elastic/security-team#14633), extracted as a standalone deliverable without dependency on `workflow.execute`. ### What's included **Schema & Types (`@kbn/workflows`)** - New `outputs` top-level field on the workflow definition schema (`WorkflowOutput[]`) - New `WorkflowOutputStepSchema` (`workflow.output`) and `WorkflowFailStepSchema` (`workflow.fail`) step types - `output` field added to `WorkflowContextSchema` and `DynamicWorkflowContextSchema` for variable access (`{{ output.fieldName }}`) - `WorkflowSchemaForAutocomplete` updated to include `outputs` for editor tooling **Execution Engine** - `workflow.output` step implementation that emits structured outputs and terminates execution - `workflow.fail` step (internally transformed to `workflow.output` with `status: 'failed'`) - `setWorkflowOutputs`, `setWorkflowStatus`, and `completeAncestorSteps` on the runtime manager for early termination - Execution graph builder handles `workflow.output` and `workflow.fail` node types **Editor & Autocomplete** - Context-aware autocomplete for the `with:` block of `workflow.output` steps — suggests declared output field names from the workflow's `outputs` definition - Position-based fallback detection for cases where the YAML parser returns an empty path (cursor on blank lines inside `with:`) - YAML provider suppression when inside `workflow.output`'s `with:` block to prevent generic JSON Schema suggestions from leaking through - Dynamic snippet generation for `workflow.output` (pre-fills `with` block from declared outputs) and `workflow.fail` - `output` variable autocomplete (`{{ output.fieldName }}`) with type-aware schema derived from declared outputs **Validation** - `validateWorkflowOutputs` — validates declared `outputs` definitions (types, constraints, naming) - `validateWorkflowOutputsInYaml` — validates `workflow.output` steps' `with:` blocks against declared outputs schema - Generic `validateWorkflowFields` utility for reusable Zod-based field validation **Tests** - Schema tests for `WorkflowOutputSchema`, `WorkflowOutputStepSchema`, `WorkflowFailStepSchema` - Integration tests for `workflow.fail` execution - Validation tests for `validateWorkflowOutputs` (352 lines covering all output types and constraints) - Autocomplete integration tests for `workflow.output` `with:` block suggestions ## Test plan - [ ] Verify `outputs` field is accepted in workflow YAML definitions - [ ] Verify `workflow.output` step emits outputs and terminates execution - [ ] Verify `workflow.fail` step terminates with failed status - [ ] Verify autocomplete suggests declared output names inside `workflow.output`'s `with:` block - [ ] Verify autocomplete filters out already-provided output keys - [ ] Verify `{{ output.fieldName }}` variable autocomplete works with correct types - [ ] Verify validation errors for invalid output definitions - [ ] Verify validation errors when `workflow.output` `with:` block doesn't match declared outputs - [ ] Run `yarn test:jest src/platform/plugins/shared/workflows_management/ --no-coverage` - [ ] Run `yarn test:jest src/platform/packages/shared/kbn-workflows/ --no-coverage` ## Example ```yaml name: Fail enabled: true description: This is a new workflow triggers: - type: manual inputs: - name: fail type: boolean default: false outputs: - name: result type: string steps: - name: if condition: "intpus.fail: true" steps: - name: success type: workflow.output with: result: "success" else: - name: fail type: workflow.fail with: message: "Ooops" ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…teps (elastic#253700) ## Summary Adds support for **workflow outputs**, the **`workflow.output`** step, and the **`workflow.fail`** step — enabling workflows to declare structured outputs, emit values at runtime, and terminate with a failure status. This is part of [Workflow Composition](elastic/security-team#14633), extracted as a standalone deliverable without dependency on `workflow.execute`. ### What's included **Schema & Types (`@kbn/workflows`)** - New `outputs` top-level field on the workflow definition schema (`WorkflowOutput[]`) - New `WorkflowOutputStepSchema` (`workflow.output`) and `WorkflowFailStepSchema` (`workflow.fail`) step types - `output` field added to `WorkflowContextSchema` and `DynamicWorkflowContextSchema` for variable access (`{{ output.fieldName }}`) - `WorkflowSchemaForAutocomplete` updated to include `outputs` for editor tooling **Execution Engine** - `workflow.output` step implementation that emits structured outputs and terminates execution - `workflow.fail` step (internally transformed to `workflow.output` with `status: 'failed'`) - `setWorkflowOutputs`, `setWorkflowStatus`, and `completeAncestorSteps` on the runtime manager for early termination - Execution graph builder handles `workflow.output` and `workflow.fail` node types **Editor & Autocomplete** - Context-aware autocomplete for the `with:` block of `workflow.output` steps — suggests declared output field names from the workflow's `outputs` definition - Position-based fallback detection for cases where the YAML parser returns an empty path (cursor on blank lines inside `with:`) - YAML provider suppression when inside `workflow.output`'s `with:` block to prevent generic JSON Schema suggestions from leaking through - Dynamic snippet generation for `workflow.output` (pre-fills `with` block from declared outputs) and `workflow.fail` - `output` variable autocomplete (`{{ output.fieldName }}`) with type-aware schema derived from declared outputs **Validation** - `validateWorkflowOutputs` — validates declared `outputs` definitions (types, constraints, naming) - `validateWorkflowOutputsInYaml` — validates `workflow.output` steps' `with:` blocks against declared outputs schema - Generic `validateWorkflowFields` utility for reusable Zod-based field validation **Tests** - Schema tests for `WorkflowOutputSchema`, `WorkflowOutputStepSchema`, `WorkflowFailStepSchema` - Integration tests for `workflow.fail` execution - Validation tests for `validateWorkflowOutputs` (352 lines covering all output types and constraints) - Autocomplete integration tests for `workflow.output` `with:` block suggestions ## Test plan - [ ] Verify `outputs` field is accepted in workflow YAML definitions - [ ] Verify `workflow.output` step emits outputs and terminates execution - [ ] Verify `workflow.fail` step terminates with failed status - [ ] Verify autocomplete suggests declared output names inside `workflow.output`'s `with:` block - [ ] Verify autocomplete filters out already-provided output keys - [ ] Verify `{{ output.fieldName }}` variable autocomplete works with correct types - [ ] Verify validation errors for invalid output definitions - [ ] Verify validation errors when `workflow.output` `with:` block doesn't match declared outputs - [ ] Run `yarn test:jest src/platform/plugins/shared/workflows_management/ --no-coverage` - [ ] Run `yarn test:jest src/platform/packages/shared/kbn-workflows/ --no-coverage` ## Example ```yaml name: Fail enabled: true description: This is a new workflow triggers: - type: manual inputs: - name: fail type: boolean default: false outputs: - name: result type: string steps: - name: if condition: "intpus.fail: true" steps: - name: success type: workflow.output with: result: "success" else: - name: fail type: workflow.fail with: message: "Ooops" ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>






Summary
Adds support for workflow outputs, the
workflow.outputstep, and theworkflow.failstep — enabling workflows to declare structured outputs, emit values at runtime, and terminate with a failure status.This is part of Workflow Composition, extracted as a standalone deliverable without dependency on
workflow.execute.What's included
Schema & Types (
@kbn/workflows)outputstop-level field on the workflow definition schema (WorkflowOutput[])WorkflowOutputStepSchema(workflow.output) andWorkflowFailStepSchema(workflow.fail) step typesoutputfield added toWorkflowContextSchemaandDynamicWorkflowContextSchemafor variable access ({{ output.fieldName }})WorkflowSchemaForAutocompleteupdated to includeoutputsfor editor toolingExecution Engine
workflow.outputstep implementation that emits structured outputs and terminates executionworkflow.failstep (internally transformed toworkflow.outputwithstatus: 'failed')setWorkflowOutputs,setWorkflowStatus, andcompleteAncestorStepson the runtime manager for early terminationworkflow.outputandworkflow.failnode typesEditor & Autocomplete
with:block ofworkflow.outputsteps — suggests declared output field names from the workflow'soutputsdefinitionwith:)workflow.output'swith:block to prevent generic JSON Schema suggestions from leaking throughworkflow.output(pre-fillswithblock from declared outputs) andworkflow.failoutputvariable autocomplete ({{ output.fieldName }}) with type-aware schema derived from declared outputsValidation
validateWorkflowOutputs— validates declaredoutputsdefinitions (types, constraints, naming)validateWorkflowOutputsInYaml— validatesworkflow.outputsteps'with:blocks against declared outputs schemavalidateWorkflowFieldsutility for reusable Zod-based field validationTests
WorkflowOutputSchema,WorkflowOutputStepSchema,WorkflowFailStepSchemaworkflow.failexecutionvalidateWorkflowOutputs(352 lines covering all output types and constraints)workflow.outputwith:block suggestionsTest plan
outputsfield is accepted in workflow YAML definitionsworkflow.outputstep emits outputs and terminates executionworkflow.failstep terminates with failed statusworkflow.output'swith:block{{ output.fieldName }}variable autocomplete works with correct typesworkflow.outputwith:block doesn't match declared outputsyarn test:jest src/platform/plugins/shared/workflows_management/ --no-coverageyarn test:jest src/platform/packages/shared/kbn-workflows/ --no-coverageExample