Create new MQTT notify plugin by numericOverflow · Pull Request #2624 · Flexget/Flexget

numericOverflow · 2020-03-13T22:07:58Z

Motivation for changes:

Enable notification to an MQTT broker, because a plugin didn't exist yet to do so

Detailed changes:

New MQTT client notify using paho MQTT library
Allows user/pass authentication
Allows SSL/TLS certificate authentication

Addressed issues:

None

Implemented feature requests:

None, just my own needs so contribute to code base

Config usage if relevant (new plugin or updated schema):

    Example::
      notify:
        entries:
          via:
            - mqtt:
                broker_address: "iot.eclipse.org"
                topic: "flexget/notifications"
                [broker_port: 1883]
                [broker_timeout: 30]
                [broker_transport: ['tcp','websockets'] ]
                [broker_protocol: ['MQTTv31', 'MQTTv311'] ]
                [username: yourUsernameHere]
                [password: yourPasswordHere]
                [enable_encrypted_communication: True/False]
                [certificates:
                    broker_ca_cert: /path/to/pem/encoded/broker_ca_certificate.crt
                    client_cert: /path/to/pem/encoded/client_certificate.crt
                    client_key: /path/to/pem/encoded/client_certificate.key
                    validate_broker_cert: True/False
                    tls_version: ['tlsv1.2', 'tlsv1.1', 'tlsv1', '']
                ]
                [qos: [0,1,2] ]
                [retain: True/False]

Log and/or tests output (preferably both):

Test config (redacted servers)

            - mqtt:
                broker_address: '5.196.95.208'
                topic: 'flexget/notifications/latest'
                broker_port: 1883
                broker_protocol: 'MQTTv31' #the version of the MQTT protocol to use. Use either MQTTv31 or MQTTv311.
                username: 'mqttclient3'
                password: 'password'
                qos: 1
                retain: True        
            - mqtt:
                broker_address: "mqtt.domain.com"
                topic: "flexget/notifications/latest"
                broker_port: 8883
                certificates: 
                    broker_ca_cert: "C:/Users/User1/Documents/mqtt_test_certs/test_intermediateCA.ca.chain.crt"
                    client_cert: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.crt"
                    client_key: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.key"
                    validate_broker_cert: False
                qos: 1
                retain: True
            - mqtt:
                broker_address: "mqtt.domain.com"
                topic: "flexget/notifications/by_series/{% if series_name is defined %}{{series_name}}{% endif %}/{% if ((series_season is defined) and (series_season is number)) %}S{{ '%02d' % series_season }}{% endif %}{% if ((series_episode is defined) and (series_episode is number)) %}E{{ '%02d' % series_episode }}{% endif %}"
                broker_port: 8883
                certificates: 
                    broker_ca_cert: "C:/Users/User1/Documents/mqtt_test_certs/test_intermediateCA.ca.chain.crt"
                    client_cert: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.crt"
                    client_key: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.key"
                    validate_broker_cert: False
                qos: 1
                retain: True
            - mqtt:
                broker_address: "mqtt.domain.com"
                topic: "flexget/notifications/by_timestamp/{{now.strftime('%Y.%m.%d_%H.%M.%S')}}/{% if series_name is defined %}{{series_name}}{% endif %} {% if ((series_season is defined) and (series_season is number)) %}S{{ '%02d' % series_season }}{% endif %}{% if ((series_episode is defined) and (series_episode is number)) %}E{{ '%02d' % series_episode }}{% endif %}"
                broker_port: 8883
                certificates: 
                    broker_ca_cert: "C:/Users/User1/Documents/mqtt_test_certs/test_intermediateCA.ca.chain.crt"
                    client_cert: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.crt"
                    client_key: "C:/Users/User1/Documents/mqtt_test_certs/mqttclient3.local.key"
                    validate_broker_cert: False
                qos: 1
                retain: True

Log file output (redacted servers a bit)

020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Credential passwords s are redacted to protect the innocent...
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Auth credentials: username=[mqttclient3] password sha256 hash is "0b14d501a594442a01c6859541bcb3e816d183d32937b851835442f69d5c94e"
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows You can validate them yourself by calculating the sha256 hex digest of your password string (google is your friend if you do not know how to do this)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Note: a password that is not provided (i.e. blank) will hash to "e3b0c44298fc1c149afbf4c8996fb92427a41e4649b934ca495991b7852b855"
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connecting to 5.196.95.208:1883
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending CONNECT (u0, p0, wr0, wq0, wf0, c1, k30) client_id=b''
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connected to MQTT broker
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Publishing message [Episode Aired: S99E01   {Series Title Redacted} - Episode Title:   {Episode Title Redacted} air date:               2020.03.08] to topic [aewadafsfe/latest]
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending PUBLISH (d0, q1, r1, m1), 'b'flexget/notifications/latest'', ... (100 bytes)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Notification sent to broker, waiting for callback response to confirm publishing success - rc=(0, 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Received CONNACK (0, 0)
020-03-13 16:27:21 VERBOSE  notify        task_trakt_shows Successfully sent a notification to `mqtt`
020-03-13 16:27:21 VERBOSE  mqtt                          Received PUBACK (Mid: 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS version is _SSLMethod.PROTOCOL_TLSv1_2
020-03-13 16:27:21 VERBOSE  mqtt                          MQTT on_publish callback -  message was successfully published to broker as messageID=1
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS cert_required=VerifyMode.CERT_NONE
020-03-13 16:27:21 VERBOSE  mqtt                          Sending DISCONNECT
020-03-13 16:27:21 VERBOSE  mqtt                          MQTT on_disconnect callback - disconnected with result code 0 [Connection Accepted]
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connecting to mqtt.domain.com:8883
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending CONNECT (u0, p0, wr0, wq0, wf0, c1, k30) client_id=b'4WaybXA3OXZWt7JxbbY3g9'
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connected to MQTT broker
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Publishing message [Episode Aired: S99E01   {Series Title Redacted} - Episode Title:   {Episode Title Redacted} air date:               2020.03.08] to topic [flexget/notifications/latest]
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending PUBLISH (d0, q1, r1, m1), 'b'flexget/notifications/latest'', ... (100 bytes)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Notification sent to broker, waiting for callback response to confirm publishing success - rc=(0, 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Received CONNACK (0, 0)
020-03-13 16:27:21 VERBOSE  notify        task_trakt_shows Successfully sent a notification to `mqtt`
020-03-13 16:27:21 VERBOSE  mqtt                          Received PUBACK (Mid: 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS version is _SSLMethod.PROTOCOL_TLSv1_2
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS cert_required=VerifyMode.CERT_NONE
020-03-13 16:27:21 VERBOSE  mqtt                          MQTT on_publish callback -  message was successfully published to broker as messageID=1
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connecting to mqtt.domain.com:8883
020-03-13 16:27:21 VERBOSE  mqtt                          Sending DISCONNECT
020-03-13 16:27:21 VERBOSE  mqtt                          MQTT on_disconnect callback - disconnected with result code 0 [Connection Accepted]
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending CONNECT (u0, p0, wr0, wq0, wf0, c1, k30) client_id=b'5jIiSJc06SepuskMjpidGA'
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Connected to MQTT broker
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Publishing message [Episode Aired: S99E01   {Series Title Redacted} - Episode Title:   {Episode Title Redacted} air date:               2020.03.08] to topic [flexget/notifications/by_series/{Series Title Redacted}/S99E01]
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Sending PUBLISH (d0, q1, r1, m1), 'b"flexget/notifications/by_series/{Series Title Redacted}/S99E01"', ... (10 bytes)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Notification sent to broker, waiting for callback response to confirm publishing success - rc=(0, 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows Received CONNACK (0, 0)
020-03-13 16:27:21 VERBOSE  notify        task_trakt_shows Successfully sent a notification to `mqtt`
020-03-13 16:27:21 VERBOSE  mqtt                          Received PUBACK (Mid: 1)
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS version is _SSLMethod.PROTOCOL_TLSv1_2
020-03-13 16:27:21 VERBOSE  mqtt                          MQTT on_publish callback -  message was successfully published to broker as messageID=1
020-03-13 16:27:21 VERBOSE  mqtt          task_trakt_shows TLS cert_required=VerifyMode.CERT_NONE
020-03-13 16:27:22 VERBOSE  mqtt                          Sending DISCONNECT
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Connecting to mqtt.domain.com:8883
020-03-13 16:27:22 VERBOSE  mqtt                          MQTT on_disconnect callback - disconnected with result code 0 [Connection Accepted]
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Sending CONNECT (u0, p0, wr0, wq0, wf0, c1, k30) client_id=b'4I34j21VA4XARV8uos2Gyu'
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Connected to MQTT broker
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Publishing message [Episode Aired: S99E01   {Series Title Redacted} - Episode Title:   {Episode Title Redacted} air date:               2020.03.08] to topic [flexget/notifications/by_timestamp/2020.03.13_16.27.21/{Series Title Redacted} S99E01]
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Sending PUBLISH (d0, q1, r1, m1), 'b"flexget/notifications/by_timestamp/2099.01.01_01.01.01/{Series Title Redacted} S99E01"', ... (100 bytes)
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Notification sent to broker, waiting for callback response to confirm publishing success - rc=(0, 1)
020-03-13 16:27:22 VERBOSE  mqtt          task_trakt_shows Received CONNACK (0, 0)
020-03-13 16:27:22 VERBOSE  notify        task_trakt_shows Successfully sent a notification to `mqtt`
020-03-13 16:27:22 VERBOSE  mqtt                          Received PUBACK (Mid: 1)
020-03-13 16:27:22 VERBOSE  mqtt                          MQTT on_publish callback -  message was successfully published to broker as messageID=1
020-03-13 16:27:22 VERBOSE  mqtt                          Sending DISCONNECT
020-03-13 16:27:22 VERBOSE  mqtt                          MQTT on_disconnect callback - disconnected with result code 0 [Connection Accepted]

To Do:

More validation with other brokers?
Current code is non-blocking, but there's an option to make the MQTT publishing calls blocking. input on which method to use?
Tweak code to fit project as needed/suggested by maintainers

Alpha level coding of MQTT notification plugin

liiight

Thanks for this! A few changed requested

flexget/components/notify/notifiers/mqtt.py

…this commit)

liiight

Please install pre-commit as we use it for a bunch of auto style and import order formatting

flexget/components/notify/notifiers/mqtt.py

ianstalk

Cool idea, and the code looks pretty good too!

numericOverflow · 2020-12-27T01:25:39Z

@liiight - Sorry I disappeared for so long. Figured I should finally clean this up and merge to project. I think I've addressed all the requests, let me know if there's other issues/suggestions you want me to address.

flexget/components/notify/notifiers/mqtt.py

paranoidi · 2020-12-27T18:54:58Z

@liiight - Sorry I disappeared for so long. Figured I should finally clean this up and merge to project. I think I've addressed all the requests, let me know if there's other issues/suggestions you want me to address.

just couple cleanups, welcome back :)

…convention

numericOverflow

Comments have been addressed with a couple additional commits

numericOverflow · 2020-12-27T21:06:02Z

@paranoidi - requested changes made

liiight · 2021-02-22T06:17:00Z

@paranoidi LGTM

paranoidi · 2021-02-22T08:57:23Z

flexget/components/notify/notifiers/mqtt.py

+                publish_info,
+            )
+        except Exception as e:
+            raise PluginWarning(f'Error publishing to MQTT broker: {e}')


Should probably use log.exception .. this catches everything unknown and with this we don't get proper tracebacks.

paranoidi · 2021-02-22T08:59:15Z

@paranoidi LGTM

Yeah, just one thing about error logging caught my eye. Other than that. Seems good!

@numericOverflow thanks!

soloam · 2021-04-20T18:19:15Z

Looking forward for this plugin

paranoidi · 2021-04-21T10:07:05Z

flexget/components/notify/notifiers/mqtt.py

+                    tls_version: ['tlsv1.2', 'tlsv1.1', 'tlsv1']
+                ]
+                [qos: [0,1,2] ]
+                [retain: True/False]


We use yes/no in yaml as it is more natural language.

paranoidi · 2021-04-21T10:07:33Z

flexget/components/notify/notifiers/mqtt.py

+                    'client_cert': {'type': 'string'},
+                    'client_key': {'type': 'string'},
+                    'validate_broker_cert': {'type': 'boolean', 'default': True},
+                    'tls_version': {'type': 'string', 'enum': ['tlsv1.2', 'tlsv1.1', 'tlsv1', '']},


Why is empty string allowed?

paranoidi · 2021-04-21T10:11:58Z

flexget/components/notify/notifiers/mqtt.py

+                logger.debug('TLS secure cert mode enabled. Broker cert will be validated')
+
+        # Handle user/pass authentication
+        if config.get('username'):


When checking key presence in dict the pythonic way is if "username" in config:, same applies to few other places in the code.

github-actions · 2021-09-18T12:54:30Z

This PR is stale because it has been open 150 days with no activity. Remove stale label or comment or this will be closed in 60 days.

github-actions · 2022-02-18T12:53:25Z

This PR is stale because it has been open 150 days with no activity. Remove stale label or comment or this will be closed in 60 days.

numericOverflow · 2022-03-30T01:20:52Z

not stale, TODO eventually

github-actions · 2022-08-27T03:03:49Z

This PR is stale because it has been open 150 days with no activity. Remove stale label or comment or this will be closed in 60 days.

paranoidi · 2022-08-30T08:32:02Z

Let's merge this in, it's been pending too long.

paranoidi · 2022-08-30T08:34:12Z

Awesome job, sorry it took so long to merge this

paranoidi · 2022-08-30T10:39:47Z

Maybe I spoke too soon, it doesn't seem to work.

@numericOverflow Issue #3533

Can you help a bit with that?

numericOverflow and others added 2 commits March 9, 2020 10:41

Initial commit

dc03217

Alpha level coding of MQTT notification plugin

Get mqtt callbacks working and user/pass broker authentication

ce92486

liiight requested changes Mar 14, 2020

View reviewed changes

NumericOverflow added 10 commits March 14, 2020 13:18

Rework logger messages to remove extraneous .format() calls

1ae4cbe

Move stdlib imports to module level

92edfa8

Remove explicit uses of None as default for dict.get()

b90192c

Remove blank defaults and blank2None lambda function

b7881f7

Use oneliner for cert_required as suggested by @liiight

b50e01d

Fix looger display hash when password is None

49aba90

Remove blank defaults and blank2None lambda function (correctly with …

bb17b96

…this commit)

remove schema string defaults, again..

0868a05

Also remove empty string default on tls_version in schema definition

859d39e

refactor code by removing unneeded PublishMQTT class to simplify code

964dba7

liiight requested changes Mar 15, 2020

View reviewed changes

ianstalk reviewed Apr 12, 2020

View reviewed changes

NumericOverflow added 9 commits December 26, 2020 15:40

Be a little less defensive when accessing certificates

09fba9c

remove excess str() calls when passing args to logger

52ffd12

Use fstring syntax on exceptions

5b23add

Remove MQTT_proto_map and tidy up logic

80d1b42

Tidy up with another fstring instead of format()

ca2b843

cleanup stray useless comment

5faf460

Add empty string option in tls_version enum for automatic selection

79bec10

Cleanup redundant defaults on config.get()

4d0db68

Added a schema dependency and simplify user/pass requirement logic

826ff3d

paranoidi reviewed Dec 27, 2020

View reviewed changes

flexget/components/notify/notifiers/mqtt.py Outdated Show resolved Hide resolved

paranoidi reviewed Dec 27, 2020

View reviewed changes

flexget/components/notify/notifiers/mqtt.py Outdated Show resolved Hide resolved

NumericOverflow added 2 commits December 27, 2020 14:31

Refactor code formatting with Black

793d9e5

Drop the enable prefix on "enable_encrypted_communication" to follow …

088d270

…convention

numericOverflow commented Dec 27, 2020

View reviewed changes

Remove extraneous import of json package

aaf5190

liiight approved these changes Feb 22, 2021

View reviewed changes

paranoidi reviewed Feb 22, 2021

View reviewed changes

paranoidi reviewed Apr 21, 2021

View reviewed changes

github-actions bot added the Stale label Sep 18, 2021

paranoidi removed the Stale label Sep 21, 2021

github-actions bot added the Stale label Feb 18, 2022

github-actions bot removed the Stale label Mar 30, 2022

github-actions bot added the Stale label Aug 27, 2022

paranoidi merged commit 67c68ca into Flexget:develop Aug 30, 2022

Conversation

numericOverflow commented Mar 13, 2020

Motivation for changes:

Detailed changes:

Addressed issues:

Implemented feature requests:

Config usage if relevant (new plugin or updated schema):

Log and/or tests output (preferably both):

To Do:

Uh oh!

liiight left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liiight left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ianstalk left a comment

Choose a reason for hiding this comment

Uh oh!

numericOverflow commented Dec 27, 2020

Uh oh!

Uh oh!

Uh oh!

paranoidi commented Dec 27, 2020

Uh oh!

numericOverflow left a comment

Choose a reason for hiding this comment

Uh oh!

numericOverflow commented Dec 27, 2020

Uh oh!

liiight commented Feb 22, 2021

Uh oh!

paranoidi Feb 22, 2021

Choose a reason for hiding this comment

Uh oh!

paranoidi commented Feb 22, 2021

Uh oh!

soloam commented Apr 20, 2021

Uh oh!

paranoidi Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

paranoidi Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

paranoidi Apr 21, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 18, 2021

Uh oh!

github-actions bot commented Feb 18, 2022

Uh oh!

numericOverflow commented Mar 30, 2022

Uh oh!

github-actions bot commented Aug 27, 2022

Uh oh!

paranoidi commented Aug 30, 2022

Uh oh!

paranoidi commented Aug 30, 2022

Uh oh!

paranoidi commented Aug 30, 2022

Uh oh!