Skip to content

feat: allow operation handlers to match on json payloads#3144

Merged
rina23q merged 4 commits intothin-edge:mainfrom
rina23q:feature/3095/extend-c8y-custom-operation-handler
Oct 4, 2024
Merged

feat: allow operation handlers to match on json payloads#3144
rina23q merged 4 commits intothin-edge:mainfrom
rina23q:feature/3095/extend-c8y-custom-operation-handler

Conversation

@rina23q
Copy link
Copy Markdown
Member

@rina23q rina23q commented Sep 27, 2024 •
edited
Loading

Proposed changes

The extension of custom operation handler makes it easy to use c8y custom operations in thin-edge. Users no longer have to create custom SmartREST templates.

Changes:

  • custom operation handler can accept JSON over MQTT topic
  • command field accepts arguments from JSON payload, same format as operation workflow
  • supporting 504-506 messages to update operation status
  • added skip_status_update = true option

Todo:

  • add skip_status_update option
  • error handling
  • unit/system test

Will be done in follow-up:

Example usage:

Install c8y-command-plugin and modify /etc/tedge/operations/c8y/c8y_Command as below.

[exec]
topic = "c8y/devicecontrol/notifications"
on_fragment = "c8y_Command"
command = "/usr/bin/c8y-command ${.payload.c8y_Command.text}"
skip_status_update = true

Then, try Shell tab in c8y.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#3095 "Drop 1"

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@reubenmiller reubenmiller added the theme:c8y Theme: Cumulocity related topics label Oct 1, 2024
@codecov
Copy link
Copy Markdown

codecov bot commented Oct 2, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 86.61202% with 49 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/extensions/c8y_mapper_ext/src/converter.rs 67.88% 22 Missing and 13 partials ⚠️
crates/core/c8y_api/src/smartrest/operations.rs 91.83% 7 Missing and 1 partial ⚠️
crates/extensions/c8y_mapper_ext/src/tests.rs 95.74% 0 Missing and 6 partials ⚠️
Additional details and impacted files

📢 Thoughts on this report? Let us know!

@rina23q rina23q force-pushed the feature/3095/extend-c8y-custom-operation-handler branch from 844609b to 55446c8 Compare October 2, 2024 20:40
@rina23q rina23q temporarily deployed to Test Pull Request October 2, 2024 20:40 — with GitHub Actions Inactive
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 2, 2024 •
edited
Loading

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
506 0 2 506 100 1h26m49.943781999s

@rina23q rina23q force-pushed the feature/3095/extend-c8y-custom-operation-handler branch from 55446c8 to af45393 Compare October 2, 2024 21:16
@rina23q rina23q temporarily deployed to Test Pull Request October 2, 2024 21:16 — with GitHub Actions Inactive
@rina23q rina23q temporarily deployed to Test Pull Request October 2, 2024 21:55 — with GitHub Actions Inactive
@rina23q rina23q marked this pull request as ready for review October 2, 2024 22:20
didier-wenzek
didier-wenzek reviewed Oct 3, 2024
View reviewed changes
didier-wenzek
didier-wenzek reviewed Oct 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still some required work.

crates/core/c8y_api/src/smartrest/operations.rs
Comment on lines 23 to 28
/// Operations are derived by reading files subdirectories per cloud /etc/tedge/operations directory
/// Each operation is a file name in one of the subdirectories
/// The file name is the operation name

#[derive(Debug, Clone, Deserialize, Eq, PartialEq)]
#[serde(rename_all = "snake_case")]
pub struct OnMessageExec {
Copy link
Copy Markdown
Contributor

@didier-wenzek didier-wenzek Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc comment is attached to the wrong struct.

I would move the struct Operation declaration here. i.e. before the struct OnMessageExec definition.
Even better: Operations first.

  1. Operations
  2. Operation
  3. OnMessageExec

Copy link
Copy Markdown
Member Author

@rina23q rina23q Oct 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but I prefer to do as you suggested after this PR, otherwise moving struct and changes will be mixed. Or really before merging this PR with a single dedicated commit for the refactoring.

Copy link
Copy Markdown
Contributor

@didier-wenzek didier-wenzek Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would help. The file is currently organized upside down, from the details to the main point.

@rina23q rina23q temporarily deployed to Test Pull Request October 3, 2024 16:55 — with GitHub Actions Inactive
didier-wenzek
didier-wenzek reviewed Oct 4, 2024
View reviewed changes
@rina23q rina23q temporarily deployed to Test Pull Request October 4, 2024 10:11 — with GitHub Actions Inactive
@rina23q rina23q temporarily deployed to Test Pull Request October 4, 2024 13:45 — with GitHub Actions Inactive
didier-wenzek
didier-wenzek approved these changes Oct 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

rina23q and others added 4 commits October 4, 2024 16:35
@rina23q
feat: extend custom operation handler
79111fb 
The extension of custom operation handler makes easy to use c8y custom
operations in thin-edge. Users no longer have to create custom SmartREST
templates.

Changes:
- custom operation handler can accept JSON over MQTT topic
- command field accepts arguments from JSON payload, same format as
operation workflow

Signed-off-by: Rina Fujino <rina.fujino.23@gmail.com>
@Ruadhri17 @rina23q
test: add system tests for custom operation handler
d8aa961 
Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
@rina23q
Add skip_status_update option to custom operation handler
486a369 
If this key is set to `true` (default `false`), c8y-mapper will execute
the command, but it'll not send operation status update messages to c8y

Signed-off-by: Rina Fujino <rina.fujino.23@gmail.com>
@rina23q
Validate custom operation handler contents on load in c8y-mapper
9262b33 
Signed-off-by: Rina Fujino <rina.fujino.23@gmail.com>
@rina23q rina23q force-pushed the feature/3095/extend-c8y-custom-operation-handler branch from 024813d to 9262b33 Compare October 4, 2024 14:36
@rina23q rina23q temporarily deployed to Test Pull Request October 4, 2024 14:36 — with GitHub Actions Inactive
@rina23q rina23q changed the title feat: extend c8y custom operation handler feat: allow operation handlers to match on json payloads Oct 4, 2024
@rina23q rina23q added this pull request to the merge queue Oct 4, 2024
@rina23q rina23q mentioned this pull request Oct 4, 2024
Open
@rina23q
Copy link
Copy Markdown
Member Author

rina23q commented Oct 4, 2024

If there is a custom operation handlers with invalid contents, you may see multiple times of the same warning message as below.

WARN c8y_api::smartrest::operations: ''command' is missing in the JSON custom operation handler mapping 'my_CustomOperation'

This is caused by too many loadings of the /etc/tedge/operations/c8y directory by c8y-mapper. Reported in #3160.

Merged via the queue into thin-edge:main with commit 952b71b Oct 4, 2024
@rina23q rina23q mentioned this pull request Oct 4, 2024
Closed
@rina23q rina23q deleted the feature/3095/extend-c8y-custom-operation-handler branch October 4, 2024 15:56
@rina23q rina23q mentioned this pull request Oct 4, 2024
Merged
11 tasks
@gligorisaev
Copy link
Copy Markdown
Contributor

gligorisaev commented Oct 8, 2024

Review done, implemented as described, the tests/RobotFramework/tests/cumulocity/custom_operation/custom_operation_command/custom_operation_command.robot
is doing what it is intended for. Checked for flakines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@didier-wenzek didier-wenzek didier-wenzek approved these changes

@reubenmiller reubenmiller Awaiting requested review from reubenmiller reubenmiller is a code owner

@Bravo555 Bravo555 Awaiting requested review from Bravo555 Bravo555 is a code owner

@albinsuresh albinsuresh Awaiting requested review from albinsuresh albinsuresh is a code owner

@jarhodes314 jarhodes314 Awaiting requested review from jarhodes314 jarhodes314 is a code owner

@gligorisaev gligorisaev Awaiting requested review from gligorisaev

Assignees

@gligorisaev gligorisaev

Labels

theme:c8y Theme: Cumulocity related topics

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@rina23q @gligorisaev @reubenmiller @didier-wenzek @Ruadhri17