Skip to content

[add] rutracker.org topic pages urlrewrite#2609

Merged
liiight merged 1 commit intoFlexget:developfrom
Niluge-KiWi:dev-rutracker_urlrewrite
Jun 15, 2020
Merged

[add] rutracker.org topic pages urlrewrite#2609
liiight merged 1 commit intoFlexget:developfrom
Niluge-KiWi:dev-rutracker_urlrewrite

Conversation

@Niluge-KiWi
Copy link
Copy Markdown
Contributor

Motivation for changes:

Get magnet url from rutracker.org topics.

Detailed changes:

  • get torrent hash from topic id: uses https://api.t-ru.org to get topic data
  • use it to reconstruct a magnet link

Config usage if relevant (new plugin or updated schema):

rutracker.org has an atom feed: http://feed.rutracker.cc/atom/f/0.atom

Log and/or tests output (preferably both):

I added a test in tests/test_urlrewriting.py. Not sure it's the best way to do it (and it requires online access: don't know if we want to mock the network call nor how to do it.

To Do:

  • maybe deprecate the old rutracker_auth plugin: I'm not sure it's that useful since torrents are public there, and my plugin creates magnet link without needing to really download .torrent files from rutracker.org.
  • maybe rename the file from site_rutracker.py to rutracker.py if the old plugin is removed, or merge both?

@Niluge-KiWi Niluge-KiWi force-pushed the dev-rutracker_urlrewrite branch 3 times, most recently from 8b7dbcd to 34b2743 Compare February 29, 2020 18:17
@githubbla
Copy link
Copy Markdown

githubbla commented Mar 31, 2020

Could you please provide sample how to configure rutracker with magnet?

@Niluge-KiWi Niluge-KiWi force-pushed the dev-rutracker_urlrewrite branch 4 times, most recently from 7ebfa2f to 85a32b8 Compare April 13, 2020 22:07
@Niluge-KiWi
Copy link
Copy Markdown
Contributor Author

Thanks @liiight for the review, I updated my PR with all your suggestions, and also added more features:

  • multi tracker (they seem to randomly generate one in 4, I generate all 4 (easier for the test, maybe I should random the order and/or follow what upstream does: send only one, but it seems less robust, but then tests are harder to do..))
  • topic title as Display Name in magnet uri, hoping it would help with some clients when magnetization timeouts.

If you have time, could you please reply to my questions in my initial comment? Thanks !

@Niluge-KiWi
Copy link
Copy Markdown
Contributor Author

@githubbla It should work using their forum rss/atom feed: http://feed.rutracker.cc/atom/f/0.atom

If you also want music parsing I have a somewhat working version 0 there: https://github.com/Niluge-KiWi/Flexget/tree/dev-music
(usage: filter previous atom feed with an artist list)

@Niluge-KiWi Niluge-KiWi requested a review from liiight June 9, 2020 22:07
@Niluge-KiWi Niluge-KiWi force-pushed the dev-rutracker_urlrewrite branch from 85a32b8 to 5742fbb Compare June 11, 2020 20:36
@Niluge-KiWi
Copy link
Copy Markdown
Contributor Author

@liiight If you have time, could you please reply to my questions in my initial comment? Thanks !

@Niluge-KiWi Niluge-KiWi requested a review from liiight June 11, 2020 21:13
@liiight
Copy link
Copy Markdown
Member

liiight commented Jun 11, 2020

@Niluge-KiWi I'll try to respond ASAP

@Niluge-KiWi
Copy link
Copy Markdown
Contributor Author

Niluge-KiWi commented Jun 11, 2020 via email

Copy link
Copy Markdown
Member

@liiight liiight left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor tweaks requested, looks good

Uses api.t-ru.org to get topic data from topic id: get torrent hash,
use it to reconstruct a magnet URI:
- add all 4 rutracker trackers servers
- add topic title as Display Name

Test url_rewrite.
@Niluge-KiWi Niluge-KiWi force-pushed the dev-rutracker_urlrewrite branch from 5742fbb to 69e982a Compare June 14, 2020 21:06
@Niluge-KiWi
Copy link
Copy Markdown
Contributor Author

@liiight last round of requested changes done!

Side note: most of your (valid) remarks also apply to other files in sites/*.py (notably site_1337.py), it's thus confusing from an external contributor: I mimic what I see in the project, notably I tried to validate with flake8 initially (as documented there https://flexget.readthedocs.io/en/latest/develop/index.html#code-style) but got many errors running it on sites/*.py, so I concluded it's not enforced in the CI/mandatory. Maybe cleaning these and enforcing more things in flake8/pep8/other checker if needed could remove this manual back and forth for 4 months. I do understand it would take time. But once it's enforced by CI checks it won't diverge again: it's a one-time cost. I could help if needed (autopep8/yapf hopefully can lift most of the work).
It would give you more time to answer deeper questions about code architecture than coding style.

@liiight liiight merged commit 43934e9 into Flexget:develop Jun 15, 2020
@liiight
Copy link
Copy Markdown
Member

liiight commented Jun 15, 2020

@Niluge-KiWi thank you for your contribution!
You are absolutely right about the formatting issue. We do use pre-commit which runs black manually, I guess we should maybe just add it as a CI step and let it change and commit on its own. maybe...

@n00buk
Copy link
Copy Markdown

n00buk commented Jul 7, 2020

Unfortunately this completely breaks external RSS feeds with direct torrent links to RuTracker. Can the old rutracker_auth plugin be restored and work in parallel with this?

@BrutuZ
Copy link
Copy Markdown
Contributor

BrutuZ commented Jul 7, 2020

@n00buk In that case wouldn't it be fine to just disable the url rewriting for the task?

disable:
  - urlrewriting

@n00buk
Copy link
Copy Markdown

n00buk commented Jul 8, 2020

@BrutuZ, I might be doing it wrong, but it didn't work. Here is what i tried...

task-name:
  rss: {? key.rssurl ?}
  accept_all: yes
  rutracker_auth:
    username: \''{? key.rut_u ?}'\'
    password: \''{? key.rut_p ?}'\'
  set:
    path: '{? folder.ong_dl ?}'
  disable: urlrewriting
  transmission:
    host: localhost
    port: 9091
    ratio: -1
    username: '{? key.tr_u ?}'
    password: '{? key.tr_p ?}'

... and what I got ...

2020-07-08 22:05:47 INFO     transmission  task-name      Trying to connect to transmission...
2020-07-08 22:05:47 INFO     transmission  task-name      Successfully connected to transmission.
2020-07-08 22:05:47 VERBOSE  backlog       task-name      Added 3 entries from backlog
2020-07-08 22:05:47 VERBOSE  details       task-name      Produced 4 entries.
2020-07-08 22:05:47 VERBOSE  task          task-name      ACCEPTED: `torrent name 1` by accept_all plugin
2020-07-08 22:05:47 VERBOSE  task          task-name      ACCEPTED: `torrent name 2` by accept_all plugin
2020-07-08 22:05:47 VERBOSE  task          task-name      ACCEPTED: `torrent name 3` by accept_all plugin
2020-07-08 22:05:47 VERBOSE  task          task-name      ACCEPTED: `torrent name 4` by accept_all plugin
2020-07-08 22:05:47 CRITICAL plugin        task-name      Host unreachable.
2020-07-08 22:05:47 WARNING  task          task-name      Aborting task (plugin: rutracker_auth)

@Niluge-KiWi Niluge-KiWi deleted the dev-rutracker_urlrewrite branch September 29, 2020 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants