A first internal version of tedge flows has been merged: #3650

We have now to move forward a public release. A lot of details have to be fixed here and there!

Feedback

Road test the core functionality with real-world use-cases (ideally from real customers, or customer proxies).

Questions to ask about the development experience:

• What is easy to develop?
• What is the testing experience like? Do I need thin-edge.io to test everything, or can unit tests be written without thin-edge.io?
• What is the debugging experience like, how to spot errors?
• Can AI assist users to create flows for them?

Open design questions

• Explore the idea of context à la NodeRed
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (review)
  • How to pass the context to each steps
    • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Finalize the idea of meta topics
  • Is this redundant with the concept of Context? Complementary?
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (review)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Explore how flows can be used by legacy mappers
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Research how to splice in common logic, e.g. How to implement a rate limiter for the cloud communication easily without having to modify each existing flow, and modify the tedge-mapper-c8y topics.
• Combine JS and Rust filters
• Consider to pass and return JSON values instead of messages with a stringified payload

Fixes

• Don't trigger ticks only when this is exactly tick time, better to be a bit late rather than missing a tick
• Simplify unique module names
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Improve script timeouts
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Reduce boilerplate code related to flow stats
• Remove the hack on the interval default value
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)

Improvements

• Handle binary payloads
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Combine sources and sinks of different types
  • Not only MQTT in, MQTT out
  • named pipes or unix sockets
  • process output (stdout) which is executed periodically by the runtime engine
• Improve error handling
  • Add an error topic to each flow, where exceptions thrown by steps are published to
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• [ ] Mitigate infinite loops
  • Add expected output topics to each pipeline: discarding any messages sent outside these topics
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Extend statistics
  • Rolling windows of stats, e.g. only show stats over the last X hours?
  • Publish stats over MQTT
• Configuration
  • Use relative paths in flow .toml files and on tedge flows test cli
    • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • Only support one configuration value at the top-level which is accessible from every flow?
  • Ability to configure stats frequency
• Testing flows
  • Improve tedge flows test help docs
    • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
  • Ability to set a timestamp on test cases pushed to tedge flows test
  • Passing a step config to a test
  • Consider to rename --final-interval
    • feat: tedge flows, a configurable mapper using JavaScript #3650 (comment)
• Logging / debugging
  • Allow logging the input and output of step functions to assist debugging deployed flows
    and to prevent users having to put "console.log()" statements everywhere within the JavaScript
• [ ] Allow setting retain flags and QoS levels for outgoing MQTT messages

Lifecycle management

Full flow lifecycle management to allow users to easily develop, share, deploy and monitor flows.
A strong focus on device-side observability is key for gaining trust that the flow is doing the right thing.

• Implement flow management, push, pull images, and to add/remove instances of those images
• Flow meta information
  • Description, user configuration/parametrization, changelog?
• Deploying flows as a non-developer / treat a flow like software,
  it can be configured and deployed without knowing the internals
• How can a flow declare its configuration that users can change
• Can users set limits on flows? e.g. max number of messages allowed to be published by a flow for a given time period? (to protect against bad flows), or time limits
• Pause/unpause flow (but retain the flow's configuration).
  If easy, otherwise postpone as the file can be renamed to {file}.disable

Polish

Final polishing of the product so it can be exposed to users

• User Documentation
  • Mapper description
  • Development tutorial / guideline
• Example repositories
• How to discover community flows (publish on website, github repo labels)