Issue 3806 - Add ability to work with rTorrent named custom fields by jasonlyle88 · Pull Request #3807 · Flexget/Flexget

jasonlyle88 · 2023-07-30T12:21:37Z

Motivation for changes:

rTorrent has the ability to work with named custom fields, but flexget only exposes the builtin custom1...custom5 fields. This PR is to expose the functionality of named custom fields to flexget for both the rtorrent and from_rtorrent plugins.

Detailed changes:

Modify existinf functions of rtorrent client to work with custom fields

Addressed issues/feature requests:

Implements #3806

Config usage if relevant (new plugin or updated schema):

from_rtorrent

from_rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom_fields:
        - foo
        - bar
        - foobar

rtorrent

rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom1: 'ubuntu'
      custom_fields:
        foo: 'This is a test value from a variables file: {? test_set.foo ?}'
        bar: 'down total: {{down_total|default(-1)}}'
        foobar: 'foobar_value_new'

Log and/or tests output (preferably both):

config.yml

variables: "variables.yml"

tasks:
  rtorrent_output_plugin_test_add:
    priority: 10
    verify_ssl_certificates: no
    disable:
      - seen
      - seen_info_hash
    mock:
      - {title: 'ubuntu-23.04-live-server-amd64.iso', url: 'https://releases.ubuntu.com/23.04/ubuntu-23.04-live-server-amd64.iso.torrent'}
      - {title: 'ubuntu-23.04-desktop-amd64.iso', url: 'https://releases.ubuntu.com/23.04/ubuntu-23.04-desktop-amd64.iso.torrent'}
    accept_all: yes
    rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom1: 'ubuntu'
      custom_fields:
        foo: 'This is a test value from a variables file: {? test_set.foo ?}'
        bar: 'down total: {{down_total|default(-1)}}'
        foobar: 'foobar_value_new'

  rtorrent_input_plugin_test_after_add:
    priority: 20
    verify_ssl_certificates: no
    disable:
      - seen
      - seen_info_hash
    from_rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom_fields:
        - foo
        - bar
        - foobar
        - foobar2
    accept_all: yes
    exec: |
      printf '\n'
      printf 'TEST %-16s: "%s"\n' 'title' "{{title}}"
      printf 'TEST %-16s: "%s"\n' 'hash' "{{torrent_info_hash}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foo' "{{foo}}"
      printf 'TEST %-16s: "%s"\n' 'custom-bar' "{{bar}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foobar' "{{foobar}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foobar2' "{{foobar2}}"

  rtorrent_output_plugin_test_update:
    priority: 30
    verify_ssl_certificates: no
    disable:
      - seen
      - seen_info_hash
    from_rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom_fields:
        - foo
        - bar
        - foobar
        - foobar2
    accept_all: yes
    rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      action: update
      custom_fields:
        bar: 'down total (updated): {{down_total}}'
        foobar: 'foobar_value_updated'
        foobar2: 'foobar2_value_new'

  rtorrent_input_plugin_test_after_update:
    priority: 40
    verify_ssl_certificates: no
    disable:
      - seen
      - seen_info_hash
    from_rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom_fields:
        - foo
        - bar
        - foobar
        - foobar2
    accept_all: yes
    exec: |
      printf '\n'
      printf 'TEST %-16s: "%s"\n' 'title' "{{title}}"
      printf 'TEST %-16s: "%s"\n' 'hash' "{{torrent_info_hash}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foo' "{{foo}}"
      printf 'TEST %-16s: "%s"\n' 'custom-bar' "{{bar}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foobar' "{{foobar}}"
      printf 'TEST %-16s: "%s"\n' 'custom-foobar2' "{{foobar2}}"

  rtorrent_input_plugin_delete_server_download:
    priority: 99
    verify_ssl_certificates: no
    disable:
      - seen
      - seen_info_hash
    from_rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      custom_fields:
        - foo
        - bar
        - foobar
    if:
      - "custom1 == 'ubuntu'": accept
    rtorrent:
      uri: '{? rtorrent.uri ?}'
      username: '{? rtorrent.username ?}'
      password: '{? rtorrent.password ?}'
      action: 'delete'

Test output

2023-07-31 22:34:08 VERBOSE  task_queue                    There are 5 tasks to execute. Shutdown will commence when they have completed.
2023-07-31 22:34:08 VERBOSE  details       rtorrent_output_plugin_test_add Produced 2 entries.
2023-07-31 22:34:08 VERBOSE  task          rtorrent_output_plugin_test_add ACCEPTED: `ubuntu-23.04-live-server-amd64.iso` by accept_all plugin
2023-07-31 22:34:08 VERBOSE  task          rtorrent_output_plugin_test_add ACCEPTED: `ubuntu-23.04-desktop-amd64.iso` by accept_all plugin
2023-07-31 22:34:08 INFO     download      rtorrent_output_plugin_test_add Downloading: ubuntu-23.04-live-server-amd64.iso
2023-07-31 22:34:09 INFO     download      rtorrent_output_plugin_test_add Downloading: ubuntu-23.04-desktop-amd64.iso
2023-07-31 22:34:09 VERBOSE  details       rtorrent_output_plugin_test_add Summary - Accepted: 2 (Rejected: 0 Undecided: 0 Failed: 0)
2023-07-31 22:34:10 INFO     rtorrent      rtorrent_output_plugin_test_add ubuntu-23.04-live-server-amd64.iso added to rtorrent
2023-07-31 22:34:11 INFO     rtorrent      rtorrent_output_plugin_test_add ubuntu-23.04-desktop-amd64.iso added to rtorrent
2023-07-31 22:34:12 VERBOSE  details       rtorrent_input_plugin_test_after_add Produced 2 entries.
2023-07-31 22:34:12 VERBOSE  task          rtorrent_input_plugin_test_after_add ACCEPTED: `ubuntu-23.04-live-server-amd64.iso` by accept_all plugin
2023-07-31 22:34:12 VERBOSE  task          rtorrent_input_plugin_test_after_add ACCEPTED: `ubuntu-23.04-desktop-amd64.iso` by accept_all plugin
2023-07-31 22:34:12 VERBOSE  details       rtorrent_input_plugin_test_after_add Summary - Accepted: 2 (Rejected: 0 Undecided: 0 Failed: 0)
2023-07-31 22:34:12 VERBOSE  exec          rtorrent_input_plugin_test_after_add Executing: printf '\n'
printf 'TEST %-16s: "%s"\n' 'title' "ubuntu-23.04-live-server-amd64.iso"
printf 'TEST %-16s: "%s"\n' 'hash' "D6B4535BA8F2B34012BC633569F113E77017E032"
printf 'TEST %-16s: "%s"\n' 'custom-foo' "This is a test value from a variables file: a value from a variable"
printf 'TEST %-16s: "%s"\n' 'custom-bar' "down total: -1"
printf 'TEST %-16s: "%s"\n' 'custom-foobar' "foobar_value_new"
printf 'TEST %-16s: "%s"\n' 'custom-foobar2' ""
2023-07-31 22:34:12 INFO     exec          rtorrent_input_plugin_test_after_add Stdout:
TEST title           : "ubuntu-23.04-live-server-amd64.iso"
TEST hash            : "D6B4535BA8F2B34012BC633569F113E77017E032"
TEST custom-foo      : "This is a test value from a variables file: a value from a variable"
TEST custom-bar      : "down total: -1"
TEST custom-foobar   : "foobar_value_new"
TEST custom-foobar2  : ""
2023-07-31 22:34:12 VERBOSE  exec          rtorrent_input_plugin_test_after_add Executing: printf '\n'
printf 'TEST %-16s: "%s"\n' 'title' "ubuntu-23.04-desktop-amd64.iso"
printf 'TEST %-16s: "%s"\n' 'hash' "443C7602B4FDE83D1154D6D9DA48808418B181B6"
printf 'TEST %-16s: "%s"\n' 'custom-foo' "This is a test value from a variables file: a value from a variable"
printf 'TEST %-16s: "%s"\n' 'custom-bar' "down total: -1"
printf 'TEST %-16s: "%s"\n' 'custom-foobar' "foobar_value_new"
printf 'TEST %-16s: "%s"\n' 'custom-foobar2' ""
2023-07-31 22:34:12 INFO     exec          rtorrent_input_plugin_test_after_add Stdout:
TEST title           : "ubuntu-23.04-desktop-amd64.iso"
TEST hash            : "443C7602B4FDE83D1154D6D9DA48808418B181B6"
TEST custom-foo      : "This is a test value from a variables file: a value from a variable"
TEST custom-bar      : "down total: -1"
TEST custom-foobar   : "foobar_value_new"
TEST custom-foobar2  : ""
2023-07-31 22:34:12 VERBOSE  details       rtorrent_output_plugin_test_update Produced 2 entries.
2023-07-31 22:34:12 VERBOSE  task          rtorrent_output_plugin_test_update ACCEPTED: `ubuntu-23.04-live-server-amd64.iso` by accept_all plugin
2023-07-31 22:34:12 VERBOSE  task          rtorrent_output_plugin_test_update ACCEPTED: `ubuntu-23.04-desktop-amd64.iso` by accept_all plugin
2023-07-31 22:34:12 VERBOSE  details       rtorrent_output_plugin_test_update Summary - Accepted: 2 (Rejected: 0 Undecided: 0 Failed: 0)
2023-07-31 22:34:13 VERBOSE  rtorrent      rtorrent_output_plugin_test_update Updated ubuntu-23.04-live-server-amd64.iso (D6B4535BA8F2B34012BC633569F113E77017E032) in rtorrent
2023-07-31 22:34:13 VERBOSE  rtorrent      rtorrent_output_plugin_test_update Updated ubuntu-23.04-desktop-amd64.iso (443C7602B4FDE83D1154D6D9DA48808418B181B6) in rtorrent
2023-07-31 22:34:14 VERBOSE  details       rtorrent_input_plugin_test_after_update Produced 2 entries.
2023-07-31 22:34:14 VERBOSE  task          rtorrent_input_plugin_test_after_update ACCEPTED: `ubuntu-23.04-live-server-amd64.iso` by accept_all plugin
2023-07-31 22:34:14 VERBOSE  task          rtorrent_input_plugin_test_after_update ACCEPTED: `ubuntu-23.04-desktop-amd64.iso` by accept_all plugin
2023-07-31 22:34:14 VERBOSE  details       rtorrent_input_plugin_test_after_update Summary - Accepted: 2 (Rejected: 0 Undecided: 0 Failed: 0)
2023-07-31 22:34:14 VERBOSE  exec          rtorrent_input_plugin_test_after_update Executing: printf '\n'
printf 'TEST %-16s: "%s"\n' 'title' "ubuntu-23.04-live-server-amd64.iso"
printf 'TEST %-16s: "%s"\n' 'hash' "D6B4535BA8F2B34012BC633569F113E77017E032"
printf 'TEST %-16s: "%s"\n' 'custom-foo' "This is a test value from a variables file: a value from a variable"
printf 'TEST %-16s: "%s"\n' 'custom-bar' "down total (updated): 0"
printf 'TEST %-16s: "%s"\n' 'custom-foobar' "foobar_value_updated"
printf 'TEST %-16s: "%s"\n' 'custom-foobar2' "foobar2_value_new"
2023-07-31 22:34:14 INFO     exec          rtorrent_input_plugin_test_after_update Stdout:
TEST title           : "ubuntu-23.04-live-server-amd64.iso"
TEST hash            : "D6B4535BA8F2B34012BC633569F113E77017E032"
TEST custom-foo      : "This is a test value from a variables file: a value from a variable"
TEST custom-bar      : "down total (updated): 0"
TEST custom-foobar   : "foobar_value_updated"
TEST custom-foobar2  : "foobar2_value_new"
2023-07-31 22:34:14 VERBOSE  exec          rtorrent_input_plugin_test_after_update Executing: printf '\n'
printf 'TEST %-16s: "%s"\n' 'title' "ubuntu-23.04-desktop-amd64.iso"
printf 'TEST %-16s: "%s"\n' 'hash' "443C7602B4FDE83D1154D6D9DA48808418B181B6"
printf 'TEST %-16s: "%s"\n' 'custom-foo' "This is a test value from a variables file: a value from a variable"
printf 'TEST %-16s: "%s"\n' 'custom-bar' "down total (updated): 0"
printf 'TEST %-16s: "%s"\n' 'custom-foobar' "foobar_value_updated"
printf 'TEST %-16s: "%s"\n' 'custom-foobar2' "foobar2_value_new"
2023-07-31 22:34:14 INFO     exec          rtorrent_input_plugin_test_after_update Stdout:
TEST title           : "ubuntu-23.04-desktop-amd64.iso"
TEST hash            : "443C7602B4FDE83D1154D6D9DA48808418B181B6"
TEST custom-foo      : "This is a test value from a variables file: a value from a variable"
TEST custom-bar      : "down total (updated): 0"
TEST custom-foobar   : "foobar_value_updated"
TEST custom-foobar2  : "foobar2_value_new"
2023-07-31 22:34:14 VERBOSE  details       rtorrent_input_plugin_delete_server_download Produced 2 entries.
2023-07-31 22:34:14 VERBOSE  task          rtorrent_input_plugin_delete_server_download ACCEPTED: `ubuntu-23.04-live-server-amd64.iso` by if plugin because matched requirement: custom1 == 'ubuntu'
2023-07-31 22:34:14 VERBOSE  task          rtorrent_input_plugin_delete_server_download ACCEPTED: `ubuntu-23.04-desktop-amd64.iso` by if plugin because matched requirement: custom1 == 'ubuntu'
2023-07-31 22:34:14 VERBOSE  details       rtorrent_input_plugin_delete_server_download Summary - Accepted: 2 (Rejected: 0 Undecided: 0 Failed: 0)
2023-07-31 22:34:14 VERBOSE  rtorrent      rtorrent_input_plugin_delete_server_download Deleted ubuntu-23.04-live-server-amd64.iso (D6B4535BA8F2B34012BC633569F113E77017E032) in rtorrent
2023-07-31 22:34:15 VERBOSE  rtorrent      rtorrent_input_plugin_delete_server_download Deleted ubuntu-23.04-desktop-amd64.iso (443C7602B4FDE83D1154D6D9DA48808418B181B6) in rtorrent

for more information, see https://pre-commit.ci

…flexget into rtorrent_custom_fields

for more information, see https://pre-commit.ci

jasonlyle88 · 2023-08-01T02:39:42Z

@gazpachoking I ran with your comments on my issue and got this PR all set. This is my first PR for Flexget, so unsure if there is a preferred method of letting you and the team know this is ready for you to review... so this is ready! Let me know if you need anything from me!

gazpachoking

I made a few mostly superficial suggestions, but it looks pretty good to me! I don't use rTorrent, or have any intimate knowledge about how the API works, so I'm trusting that the actual functionality works as intended.

flexget/plugins/clients/rtorrent.py

gazpachoking · 2023-08-01T17:28:36Z

flexget/plugins/clients/rtorrent.py

+        # Set custom fields
+        for key, val in custom_fields.items():
+            # Values must be escaped if within params
+            params.append(f'd.custom.set={re.escape(str(key))},{re.escape(str(val))}')


I'm not sure what escaping rules are needed for the rtorrent api, but I doubt it this is the exact right way. re.escape is meant to escape characters that have special meanings in regex. A quick peek at the api guide makes me think maybe just surrounding the key and value with double quotes is enough?

Suggested change

params.append(f'd.custom.set={re.escape(str(key))},{re.escape(str(val))}')

params.append(f'd.custom.set="{key}","{val}"')

I've got the re.escape used in a couple places. I did the escaping that way because that is what was used already for the fields, and thought I would keep it consistent. However, I will test with just quoting on my parts and see how this goes. If everything works out, then I will commit that.

@gazpachoking I have updated all the of the re.escape calls used for custom fields to be quoted and to escape double quotes with a \" character. I believe this should do the trick. I did leave the re.escape from the fields code that was there before I started my merge since I didn't want to change how things currently work. However, I am happy to make that change if you think it is appropriate.

Thanks for the help!

flexget/plugins/clients/rtorrent.py

Co-authored-by: Chase Sterling <chase.sterling@gmail.com>

jasonlyle88 · 2023-08-01T19:50:53Z

@gazpachoking Thank you very much for the helpful suggestions (and the catch on the entry vs config ordering). I am not very well versed in python, so I will take all the suggestions you are willing to offer!

for more information, see https://pre-commit.ci

gazpachoking · 2023-08-03T14:30:49Z

Looks good to me, thanks! Can you update the wiki with the new options?

jasonlyle88 · 2023-08-03T14:45:45Z

@gazpachoking Thank you very much! I have updated the wiki with the documentation and an example for this functionality!

Jason Lyle and others added 17 commits July 30, 2023 08:14

Empty commit

87638e1

Add crash report logs to git ignore

4583373

Add in work for rTorrent named custom fields

d652be3

[pre-commit.ci] auto fixes from pre-commit.com hooks

06a72d4

for more information, see https://pre-commit.ci

Attempt to fix test_add

159b7a9

Fix missing comma

09c64f8

Update output plugin tests to include empty custom_fields

a65ec2d

[pre-commit.ci] auto fixes from pre-commit.com hooks

b78dadb

for more information, see https://pre-commit.ci

Finish fixing output tests

6b23a30

Fix input plugin tests

6815b86

Modify test_add to add with custom fields

4d8d6c9

[pre-commit.ci] auto fixes from pre-commit.com hooks

6641fab

for more information, see https://pre-commit.ci

Add custom fields for test_add_torrent_set

34c6795

Merge branch 'rtorrent_custom_fields' of personal.github:jasonlyle88/…

42dd90f

…flexget into rtorrent_custom_fields

Add values for remaining custom_fields tests

5c79343

Add missing commas

b31f790

[pre-commit.ci] auto fixes from pre-commit.com hooks

855bb25

for more information, see https://pre-commit.ci

jasonlyle88 marked this pull request as ready for review August 1, 2023 02:36

gazpachoking reviewed Aug 1, 2023

View reviewed changes

Apply suggestions from code review

6c1e1ed

Co-authored-by: Chase Sterling <chase.sterling@gmail.com>

Jason Lyle and others added 2 commits August 1, 2023 17:16

Add missing =, use appropriate escaping for custom fields

f275f44

[pre-commit.ci] auto fixes from pre-commit.com hooks

6d357fb

for more information, see https://pre-commit.ci

gazpachoking merged commit 23ef2bb into Flexget:develop Aug 3, 2023

jasonlyle88 deleted the rtorrent_custom_fields branch August 3, 2023 14:45

jasonlyle88 mentioned this pull request Aug 3, 2023

[Feature Request] rTorrent custom field enhancement #3806

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 3806 - Add ability to work with rTorrent named custom fields#3807

Issue 3806 - Add ability to work with rTorrent named custom fields#3807
gazpachoking merged 20 commits intoFlexget:developfrom
jasonlyle88:rtorrent_custom_fields

jasonlyle88 commented Jul 30, 2023 •

edited

Loading

Uh oh!

jasonlyle88 commented Aug 1, 2023

Uh oh!

gazpachoking left a comment

Uh oh!

Uh oh!

Uh oh!

gazpachoking Aug 1, 2023

Uh oh!

jasonlyle88 Aug 1, 2023

Uh oh!

jasonlyle88 Aug 1, 2023

Uh oh!

Uh oh!

Uh oh!

jasonlyle88 commented Aug 1, 2023

Uh oh!

gazpachoking commented Aug 3, 2023

Uh oh!

jasonlyle88 commented Aug 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	params.append(f'd.custom.set={re.escape(str(key))},{re.escape(str(val))}')
	params.append(f'd.custom.set="{key}","{val}"')

Conversation

jasonlyle88 commented Jul 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for changes:

Detailed changes:

Addressed issues/feature requests:

Config usage if relevant (new plugin or updated schema):

Log and/or tests output (preferably both):

Uh oh!

jasonlyle88 commented Aug 1, 2023

Uh oh!

gazpachoking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gazpachoking Aug 1, 2023

Choose a reason for hiding this comment

Uh oh!

jasonlyle88 Aug 1, 2023

Choose a reason for hiding this comment

Uh oh!

jasonlyle88 Aug 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jasonlyle88 commented Aug 1, 2023

Uh oh!

gazpachoking commented Aug 3, 2023

Uh oh!

jasonlyle88 commented Aug 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jasonlyle88 commented Jul 30, 2023 •

edited

Loading