Skip to content

Add decoding of sparkplug payload#582

Merged
thomasnordquist merged 13 commits into
thomasnordquist:masterfrom
dubyte:add_sparkplug
Feb 27, 2022
Merged

Add decoding of sparkplug payload#582
thomasnordquist merged 13 commits into
thomasnordquist:masterfrom
dubyte:add_sparkplug

Conversation

@dubyte

@dubyte dubyte commented Aug 8, 2021

Copy link
Copy Markdown
Contributor

The pull request is to try to decode sparkplug in case json fails to parse. it works in the raw format and diff viewers.

@marktoddgea

Copy link
Copy Markdown

This is awesome! I hope this gets approved ASAP! Much needed.

@thomasnordquist thomasnordquist left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution @dubyte.
Really nice and clean work.

Two things:

  • Do you know where I can get example sparkplugb messages? (I want to write a short test and check that the definition is narrow enough that other valid strings are not interpreted as sparkplug message)
  • ProtobufJS should be able to read and compile .proto files, is there an official source for this proto definition? We could then get rid of the sparkplug.ts file, or generate it during build.
    I just found https://github.com/Cirrus-Link/Sparkplug/blob/master/sparkplug_b/sparkplug_b.proto

Personally I prefer not having generated code in a project if it can be avoided.

Comment thread app/src/components/TopicPlot.tsx Outdated
Co-authored-by: Thomas Nordquist <thomasnordquist@users.noreply.github.com>
@thomasnordquist

Copy link
Copy Markdown
Owner

Could be as easy as using the webpack file-loader for .proto files.
I am not entirely sure about the inline-loader syntax.

Adapted from https://github.com/protobufjs/protobuf.js#examples

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!../../path/to/file/sparkplug-b.proto';

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

    // Exemplary payload
    var payload = { awesomeField: "AwesomeString" };

    // Verify the payload if necessary (i.e. when possibly incomplete or invalid)
    var errMsg = AwesomeMessage.verify(payload);
    if (errMsg)
        throw Error(errMsg);

    // Create a new message
    var message = AwesomeMessage.create(payload); // or use .fromObject if conversion is necessary

    // Encode a message to an Uint8Array (browser) or Buffer (node)
    var buffer = AwesomeMessage.encode(message).finish();
    // ... do something with buffer

    // Decode an Uint8Array (browser) or Buffer (node) to a message
    var message = AwesomeMessage.decode(buffer);
    // ... do something with message

    // If the application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.

    // Maybe convert the message back to a plain object
    var object = AwesomeMessage.toObject(message, {
        longs: String,
        enums: String,
        bytes: String,
        // see ConversionOptions
    });
});

Webpack

https://webpack.js.org/concepts/loaders/#inline

@dubyte

dubyte commented Aug 11, 2021

Copy link
Copy Markdown
Contributor Author

That is cool, I will try to do that.

@dubyte

dubyte commented Aug 12, 2021

Copy link
Copy Markdown
Contributor Author

This is mock so you can test locally: https://github.com/dubyte/sparkplugbmock/releases/tag/v0.1.0

The payload of sparkplugb is binary so it is hard to see: but this is an example:

This one is taking hex string and parsing it to protojson , you can get the decoder from https://github.com/dubyte/sparkplugclidecoder/releases/tag/v0.1.1
echo 08E5ACBDBDB32F12120A056576656E7418E5ACBDBDB32F200B70001800 | sparkplugclidecoder

After xxd that is how it looks in a terminal
echo 08E5ACBDBDB32F12120A056576656E7418E5ACBDBDB32F200B70001800 | xxd -r -p

@dubyte

dubyte commented Aug 12, 2021

Copy link
Copy Markdown
Contributor Author

Could be as easy as using the webpack file-loader for .proto files.
I am not entirely sure about the inline-loader syntax.

Adapted from https://github.com/protobufjs/protobuf.js#examples

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!../../path/to/file/sparkplug-b.proto';

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

    // Exemplary payload
    var payload = { awesomeField: "AwesomeString" };

    // Verify the payload if necessary (i.e. when possibly incomplete or invalid)
    var errMsg = AwesomeMessage.verify(payload);
    if (errMsg)
        throw Error(errMsg);

    // Create a new message
    var message = AwesomeMessage.create(payload); // or use .fromObject if conversion is necessary

    // Encode a message to an Uint8Array (browser) or Buffer (node)
    var buffer = AwesomeMessage.encode(message).finish();
    // ... do something with buffer

    // Decode an Uint8Array (browser) or Buffer (node) to a message
    var message = AwesomeMessage.decode(buffer);
    // ... do something with message

    // If the application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.

    // Maybe convert the message back to a plain object
    var object = AwesomeMessage.toObject(message, {
        longs: String,
        enums: String,
        bytes: String,
        // see ConversionOptions
    });
});

Webpack

https://webpack.js.org/concepts/loaders/#inline

In this example I don't get it, how I can expose Payload if it is created inside a callback?

Sorry I am not used to this syntax

What I want is be able to load the proto during the start of the app, then with an import be able to call decode and fromJson.

I can pass from:

`
// ./backend/src/Model/SparkplugB.ts
const protobuf = require('protobufjs')

// Using webpacks file-loader to emit an url to the proto file, changed webpack config to support proto
import sparkplugBProto from '../../../res/sparkplug_b.proto'

protobuf.load(sparkplugBProto).then(function (root: any) {
const Payload = root.lookupType('com.cirruslink.sparkplug.protobuf.Payload')
})
`

@dubyte

dubyte commented Aug 12, 2021

Copy link
Copy Markdown
Contributor Author

Yesterday I had been trying to load the proto, but for some reason I can find the type.

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!./../../../../res/sparkplug_b.proto';   // I put this file in /res 

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var Payload = root.lookupType("com.cirruslink.sparkplug.protobuf.Payload"); // <- the code fails to find out the type :(

@dubyte

dubyte commented Aug 15, 2021

Copy link
Copy Markdown
Contributor Author

I make it work by loading the proto instead of use a generated code.
I am not sure Payload = undefined is a best practice could you give me a recomendation about that?

Thanks

@dubyte

dubyte commented Aug 23, 2021

Copy link
Copy Markdown
Contributor Author

@thomasnordquist could you give me a review on this, I did the changes you suggested.
Thanks

@martinscheffler

Copy link
Copy Markdown

Just wanted to say it is great that you guys are workin on this, this will be very useful indeed!

@marktoddgea

Copy link
Copy Markdown

Just wanted to say it is great that you guys are workin on this, this will be very useful indeed!

I agree, I have been using MQTT FX because it does decode sparkplugb, but it isn't supported anymore and has some very serious bugs. Sparkplugb is really a great light-weight messaging system and I hope more people start using it. MQTT Explorer is such an outstanding tool. Thanks for all the hard work!

@vdwpsmt

vdwpsmt commented Oct 22, 2021

Copy link
Copy Markdown

Wow, why didn't I find this mqtt client earlier? awesome this topic tree explorer auto-updating etc...
Any idea when this sparkplug decoding feature could be released?

@dubyte

dubyte commented Jan 5, 2022

Copy link
Copy Markdown
Contributor Author

Any update about the PR? could you review? Thanks

@danwale

danwale commented Jan 24, 2022

Copy link
Copy Markdown

I'm also very keen on the Sparkplug B decoding functionality being added, I was previously using MQTT FX but considering it uses a vulnerable version of log4j and isn't getting any updates to the free version anymore I want to move to using this tool instead to do the Sparkplug B decoding easily.

@hobbes1069

Copy link
Copy Markdown

Adding my +1 to this PR.

@aott33

aott33 commented Feb 12, 2022

Copy link
Copy Markdown

Adding my +1 as well

@thomasnordquist thomasnordquist merged commit 72400af into thomasnordquist:master Feb 27, 2022
@thomasnordquist

Copy link
Copy Markdown
Owner

Trying to do a release the next few days.

@hobbes1069

Copy link
Copy Markdown

Just checking in, if I understand correctly it looks like the CI tests are failing?

@AaronAutomated

Copy link
Copy Markdown

Bumping this. Would love to see Sparkplug decoding added as a feature as this is by far the best MQTT client for testing and development.

@Ekion-1

Ekion-1 commented Jul 11, 2022

Copy link
Copy Markdown

Agreed. This would be a game changer.

@tord-v

tord-v commented Sep 23, 2022

Copy link
Copy Markdown

Bumping. All I want for Christmas is this.

@iansebryk

Copy link
Copy Markdown

shameless bump. this would alter the landscape forever. please keep going on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.