Skip to content

#1650 provide WoT TM validation#1936

Merged
thjaeckle merged 27 commits intoeclipse-ditto:masterfrom
beyonnex-io:wot-tm-validation
Aug 26, 2024
Merged

#1650 provide WoT TM validation#1936
thjaeckle merged 27 commits intoeclipse-ditto:masterfrom
beyonnex-io:wot-tm-validation

Conversation

@thjaeckle
Copy link
Member

@thjaeckle thjaeckle commented May 6, 2024 •
edited
Loading

  • restructuring of "ditto-wot" module to enable re-usability of non-pekko/non-Ditto specifics
  • adding "validator" concept and first sample implementation
  • using a json-schema-validation library to create validation exception details
  • applying validator to:
    • thing attribute updates
    • feature property updates
    • thing actions (inputs and outputs)
    • feature actions (inputs and outputs)
    • thing events
    • feature events
  • each aspect of validation is fully configurable
    • whether to enable/disable
    • also dynamically, e.g. excluding aspects for certain models from validation
  • adding traces in order to determine how much this validation costs "extra" in a productive setup

Open points:

  • Provide actual validation of TMs for all hierarchy levels of changes to Things
  • Optimize conversion from Ditto-Json to Jackson
  • Provide validation for WoT actions and their responses
  • Provide validation for WoT events
  • Add support for WoT oneOf for DataSchema - validation must only match "one of" the provided schemas
  • Provide advanced configuration options to be able to "skip" validation on certain conditions, e.g.:
    • when auth-subject is "xyz"
    • when model URL matches a configured regex
  • Add (enforcement) metrics in order to be able to inspect how much time is spent for validation
  • Provide tests
  • Provide Helm configuration
  • Provide documentation

Resolves: #1650

@thjaeckle thjaeckle added this to the 3.6.0 milestone May 6, 2024
@thjaeckle thjaeckle self-assigned this May 6, 2024
@thjaeckle thjaeckle force-pushed the wot-tm-validation branch 8 times, most recently from 7230638 to 4a8e4b6 Compare May 7, 2024 14:10
@thjaeckle thjaeckle force-pushed the wot-tm-validation branch 2 times, most recently from 45ae2b5 to 49b206e Compare June 24, 2024 07:50
@thjaeckle thjaeckle added the WoT Web of Things related enhancements label Jun 26, 2024
@thjaeckle thjaeckle force-pushed the wot-tm-validation branch 3 times, most recently from c0ab94d to c05c867 Compare July 17, 2024 05:33
thjaeckle added 14 commits July 29, 2024 13:31
@thjaeckle
eclipse-ditto#1650 provide WoT TM validation
7d08a2b 
* restructuring of "ditto-wot" module to enable re-usability of non-pekko/non-Ditto specifics
* adding "validator" concept and first sample implementation (WIP)

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle
eclipse-ditto#1650 prepare validation of features as Submodels of a T…
2de51cd 
…hingModel

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle
eclipse-ditto#1650: adding implementation for feature properties and …
f6a4028 
…desired properties validation

* taking care of e.g. "ditto:category" defined
* first validation error will provide the error output (which is for now simpler and results in quicker obtained validation errors)

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle
eclipse-ditto#1650: adding support for:
909dbe8 
* validating features and properties as part of "modify thing"
* handling "ditto:category" correctly
* ensuring completeness of defined features
* validation on creation/modification
  * of thing
  * of attributes
  * on attribute
  * on features
  * on feature

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle
eclipse-ditto#1650: added validation of feature properties + feature …
99b0a39 
…property

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
@thjaeckle
eclipse-ditto#1650 enforce "MergeThing" command
56d1eb9 
* enforce modifying thing definition + feature definition
* enhance configuration options
@thjaeckle
eclipse-ditto#1650 fix path handling, always using absolute paths to …
35c7074 
…the thing for validationError paths
@thjaeckle
eclipse-ditto#1650 provided javadoc for validation interfaces
05cd1e9
@thjaeckle
eclipse-ditto#1650 first steps of (thing) message (action) validation
2e1491c
@thjaeckle
eclipse-ditto#1650 update dependencies
4745aee
@thjaeckle
eclipse-ditto#1650 added validation for actions / inbox messages and …
b4e3233 
…their responses

* also updated some dependencies
* simplified DefaultWotThingModelValidator a little
@thjaeckle
eclipse-ditto#1650 provided validation for events / outbox messages
75ee80b
@thjaeckle
eclipse-ditto#1650 added SingleDataSchema implementation without prov…
d2bf636 
…iding "type" (valid for e.g. "oneOf") use

* update json-schema-validator library
@thjaeckle
eclipse-ditto#1650 refactored too big class DefaultWotThingModelValid…
5472a78 
…ation in smaller pieces

* strict split between config reading in DefaultWotThingModelValidation and static functionality in Internal* classes
thjaeckle added 11 commits July 29, 2024 13:32
@thjaeckle
eclipse-ditto#1650 provide dynamic configuration of WoT validation
9cfd1b2 
* based on either DittoHeaders or on ThingDefinition/FeatureDefinition URLs
@thjaeckle
eclipse-ditto#1650 added tracing for wot_validation
5069f4d 
* fixed and harmonized other traces while doing that in order to get correct parent hierarchy of spans
@thjaeckle
eclipse-ditto#1650 fix BackgroundSyncActorTest
7228a8a
@thjaeckle
eclipse-ditto#1650 added validations when deleting parts of a thing b…
2ed2641 
…ased on the WoT model

* restructured some too long methods
* fixed remaining TODOs
@thjaeckle
eclipse-ditto#1650 added unit tests covering the added validation fun…
9c2ed96 
…ctionality
@thjaeckle
eclipse-ditto#1650 provided Helm configuration for WoT TM validation
d0c3051 
* also for dynamic part
* changed config structure a bit in order to better fit Helm YAML config
@thjaeckle
eclipse-ditto#1650 hardened validation implementation, supporting mor…
48ffa81 
…e corner cases

* e.g. deeply nested updates, removing definition via PATCH
* enabled WoT based validation by default
@thjaeckle
eclipse-ditto#1650 provided more unit tests
1f0d538
@thjaeckle
eclipse-ditto#1650 fix merge thing when updating definition
99a5311 
* first choosing potentially new definition and only as fall-back the old one
@thjaeckle
eclipse-ditto#1650 fix skipping required properties check
9069625
@thjaeckle
eclipse-ditto#1650 added documentation for new WoT TM based validation
608894f
@thjaeckle thjaeckle force-pushed the wot-tm-validation branch from dbc03b1 to 608894f Compare July 29, 2024 11:46
@thjaeckle thjaeckle marked this pull request as ready for review July 29, 2024 11:46
@thjaeckle
Copy link
Member Author

thjaeckle commented Jul 29, 2024

FMPOV ready to be reviewed :)

It is of course a massively huge PR - let's discuss how to review it most efficiently @alstanchev @kalinkostashki

@thjaeckle thjaeckle requested a review from alstanchev July 31, 2024 07:15
@thjaeckle
Copy link
Member Author

thjaeckle commented Aug 26, 2024

@alstanchev did you manage to have a look? would want to merge this PR soon ..

@alstanchev
Copy link
Contributor

alstanchev commented Aug 26, 2024 •
edited
Loading

yes, LGTM 👍

@thjaeckle thjaeckle merged commit e4ce12e into eclipse-ditto:master Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alstanchev alstanchev Awaiting requested review from alstanchev

Assignees

@thjaeckle thjaeckle

Labels

WoT Web of Things related enhancements

Projects

Ditto Planning
Status: Done

Milestone

3.6.0

Development

Successfully merging this pull request may close these issues.

Add option to enforce validation based on referenced WoT model

2 participants

@thjaeckle @alstanchev