Skip to content

Make nodes runnable standalone, prompting the user for inputs#1111

Merged
phil-opp merged 10 commits intomainfrom
interactive-node-api
Oct 8, 2025
Merged

Make nodes runnable standalone, prompting the user for inputs#1111
phil-opp merged 10 commits intomainfrom
interactive-node-api

Conversation

@phil-opp
Copy link
Copy Markdown
Collaborator

Adds a new interactive mode to the Rust node API, which automatically activates when no DORA_NODE_CONFIG env variable is set. When run in interactive mode, the user is prompted for inputs on the command line. Input data can be given as string or as JSON.

This feature makes it possible to try a node with some inputs and see how it reacts, without the need to set up a full dataflow for it. This can be useful during development.

Partially resolves #1019

Adds a new interactive mode to the Rust node API, which automatically activates when no `DORA_NODE_CONFIG` env variable is set. When run in interactive mode, the user is prompted for inputs on the command line. Input data can be given as string or as JSON.

This feature makes it possible to try a node with some inputs and see how it reacts, without the need to set up a full dataflow for it. This can be useful during development.
@phil-opp phil-opp force-pushed the interactive-node-api branch from 8f62811 to b8b5abe Compare September 3, 2025 12:10
@phil-opp phil-opp added the waiting-for-review Pull request is waiting for a review from maintainers. label Sep 3, 2025
@haixuanTao
Copy link
Copy Markdown
Collaborator

Could you add example and documentation on how to use this?

We have some Python tests that try to initialize a dora node. We don't want to run in interactive mode in this case because the executable is run by `pytest`. So there is no way to supply interactive input. By checking if stdin is a terminal, we can continue to throw an error in this case.
@phil-opp
Copy link
Copy Markdown
Collaborator Author

I added some docs with an example in 57237d2.

@phil-opp
Copy link
Copy Markdown
Collaborator Author

I also just tried it with the Python template generated by dora new and it also worked as expected:

> dora new test_python_project --lang python --internal-create-with-path-dependencies
> cd test_python_project
> uv venv --seed -p 3.12
> uv pip install -e ../apis/python/node
> uv run talker-1/talker_1/main.py
Starting node in interactive mode as DORA_NODE_CONFIG env variable is not set
2025-09-16T15:43:35.058724Z DEBUG opentelemetry-otlp:  name="MetricsTonicChannelBuilding"
2025-09-16T15:43:35.058777Z DEBUG opentelemetry-otlp:  name="TonicChannelBuilt" endpoint="http://localhost:4317" timeout_in_millisecs=10000 compression="None" headers="[]"
2025-09-16T15:43:35.058786Z DEBUG opentelemetry-otlp:  name="TonicsMetricsClientBuilt"
2025-09-16T15:43:35.058790Z DEBUG opentelemetry-otlp:  name="MetricExporterBuilt"
2025-09-16T15:43:35.058854Z DEBUG opentelemetry_sdk:  name="MeterProvider.Building" builder="MeterProviderBuilder { resource: None, readers: [PeriodicReader], views: 0 }"
2025-09-16T15:43:35.058871Z DEBUG opentelemetry_sdk:  name="MeterProvider.Built"
2025-09-16T15:43:35.058877Z  INFO opentelemetry:  name="MeterProvider.GlobalSet" Global meter provider is set. Meters can now be created using global::meter() or global::meter_with_scope().
2025-09-16T15:43:35.058891Z DEBUG opentelemetry_sdk:  name="MeterProvider.NewMeterCreated" meter_name="1301dfe7-43b0-4f42-8a01-92b9d400a50d/"
2025-09-16T15:43:35.058889Z DEBUG opentelemetry_sdk:  name="PeriodReaderThreadStarted" interval_in_millisecs=60000
2025-09-16T15:43:35.058906Z DEBUG opentelemetry_sdk:  name="PeriodReaderThreadLoopAlive" Next export will happen after interval, unless flush or shutdown is triggered. interval_in_millisecs=60000
2025-09-16T15:43:35.059459Z DEBUG opentelemetry_sdk:  name="InstrumentCreated" instrument_name="process.cpu.usage"
2025-09-16T15:43:35.059480Z DEBUG opentelemetry_sdk:  name="InstrumentCreated" instrument_name="process.cpu.utilization"
2025-09-16T15:43:35.059497Z DEBUG opentelemetry_sdk:  name="InstrumentCreated" instrument_name="process.memory.usage"
2025-09-16T15:43:35.059510Z DEBUG opentelemetry_sdk:  name="InstrumentCreated" instrument_name="process.memory.virtual"
2025-09-16T15:43:35.059524Z DEBUG opentelemetry_sdk:  name="InstrumentCreated" instrument_name="process.disk.io"
Node asks for next input
? Input ID  
[empty input ID to stop]

@haixuanTao
Copy link
Copy Markdown
Collaborator

yeah I meant like markdown based documentation and / or examples within the example folder on how to use this. It's not obvious to me what the input format should look like

@phil-opp
Copy link
Copy Markdown
Collaborator Author

A markdown example/documentation is within the docs for init_interactive:

Example

Run any node that uses init_interactive or init_from_env directly
from a terminal. The node will then start in "interactive mode" and prompt you for the next
input:

> cargo build -p rust-dataflow-example-node
> target/debug/rust-dataflow-example-node
hello
Starting node in interactive mode as DORA_NODE_CONFIG env variable is not set
Node asks for next input
? Input ID
[empty input ID to stop]

The rust-dataflow-example-node expects a tick input, so let's set the input ID to
tick. Tick messages don't have any data, so we leave the "Data" empty when prompted:

Node asks for next input
> Input ID tick
> Data
tick 0, sending 0x943ed1be20c711a4
node sends output random with data: PrimitiveArray<UInt64>
[
10682205980693303716,
]
Node asks for next input
? Input ID
[empty input ID to stop]

We see that both the stdout output of the node and also the output messages that it sends
are printed to the terminal. Then we get another prompt for the next input.

If you want to send an input with data, you can either send it as text (for string data)
or as a JSON object (for struct data). Other data types are not supported currently.

Empty input IDs are interpreted as stop instructions:

> Input ID
given input ID is empty -> stopping
Received stop
Node asks for next input
event channel was stopped -> returning empty event list
node reports EventStreamDropped
node reports closed outputs []
node reports OutputsDone

In addition to the node output, we see log messages for the different events that the node
reports. After OutputsDone, the node should exit.

I can put this also somewhere else if you like, e.g. the web docs at dora-rs.ai?

examples within the example folder on how to use this

This should basically work with any of the existing nodes, so creating a separate example feels a bit duplicated. Given that it's interactive, it's difficult to test this through an automated example. The only limitation right now is that only string or struct inputs (as JSON) are currently supported.

@haixuanTao
Copy link
Copy Markdown
Collaborator

Yeah we really need an individual markdown file for this on dora-rs.ai at least.

Also could you provide an example using json?

@phil-opp
Copy link
Copy Markdown
Collaborator Author

phil-opp commented Oct 8, 2025

Yeah we really need an individual markdown file for this on dora-rs.ai at least.

I opened dora-rs/dora-rs.github.io#46 for this.

Also could you provide an example using json?

Added in d2092ee.

@haixuanTao
Copy link
Copy Markdown
Collaborator

haixuanTao commented Oct 8, 2025

Sorry for being annoying but could add a simpler example with just a simple integer array or simple string?

I tried to run the example with a simple integer specified array but it seems to be automatically converted into uint8 array:

> Input ID 123
> Data [123, 123]
Ignoring unexpected input 123 data: `ArrowData(PrimitiveArray<UInt8>
[
  91,
  49,
  50,
  51,
  44,
  32,
  49,
  50,
  51,
  93,
])`

Same for string:

> Input ID abc
> Data abcd
Ignoring unexpected input abc data: `ArrowData(PrimitiveArray<UInt8>
[
  97,
  98,
  99,
  100,
])`

@phil-opp
Copy link
Copy Markdown
Collaborator Author

phil-opp commented Oct 8, 2025

No worries, thanks for the examples!

Right now only JSON objects are handled as JSON, everything else is interpreted as a string. And it turns out that I used the wrong arrow type for string representation (I stored it as an u8 array).

Let me try to push a new version that always go through the json schema inference to see how that works.

@phil-opp
Copy link
Copy Markdown
Collaborator Author

phil-opp commented Oct 8, 2025

@haixuanTao Could you try again with 1de895e?

Copy link
Copy Markdown
Collaborator

@haixuanTao haixuanTao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks!

@phil-opp phil-opp merged commit e5f55a5 into main Oct 8, 2025
50 checks passed
@phil-opp phil-opp deleted the interactive-node-api branch October 8, 2025 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-for-review Pull request is waiting for a review from maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A command for simplifying node testing

2 participants