new sort_by_weight plugin by Andy2244 · Pull Request #1533 · Flexget/Flexget

Andy2244 · 2016-12-04T15:07:11Z

Motivation for changes:

I wanted the ability to better sort the discover results, so the best release could be picked-up. Basically i wanted to weight different stats/data fields, so the filter plugins can pick the best from the top.

Detailed changes:

new sort_by_weight plugin that takes multiple fields as input (at least 2)
stores the calculated weighted result in the sort_by_weight_sum and sorts so the highest weight entry is on top of the list
can use multiple config parameters to improve default rules

Config usage if relevant (new plugin or updated schema):

Simple

sort_by_weighted:
      - field: content_size
        weight: 80
      - field: newznab_grabs
        weight: 25

Advanced

        sort_by_weighted:
          - field: content_size
            weight: 80              # we want large files mainly = good quality
            delta_distance: 500     # anything within 500 MB gets the same weight
          - field: newznab_pubdate
            weight: 25              # we still like new releases
            delta_distance: 7 days      # anything within 7 days is similar
            upper_limit: 60         # confine results to 0-60 days, anything older 60 days gets 0 weight
            inverse: yes            # inverse weight results for date/age fields (oldest age gets lowest score)
          - field: newznab_grabs
            weight: 25              # we like releases that others already downloaded, aka safeguard against crap
            upper_limit: 100        # anything over 100 grabs is fine and gets maximum weight

Example of my own 720p based movie newznab settings:

    sort_by_weight:
      - field: content_size
        weight: 80
        delta_distance: 500
        upper_limit: 10000
      - field: newznab_hydraindexerscore
        weight: 25
        delta_distance: 1
      - field: newznab_pubdate
        weight: 30
        delta_distance: 30 days
        upper_limit: 500
        inverse: yes
      - field: newznab_grabs
        weight: 25
        upper_limit: 100

fixed dict exception

cvium · 2016-12-04T15:10:35Z

I think splitting limits_min_max into two min_value and max_value is better for the users. Alternatively, it could be called range, but instead of a list it's just a special string like 0-100. I prefer the former though.

The current format is weird and hackish imo.

cvium · 2016-12-04T15:16:54Z

flexget/plugins/modify/sort_by_weighted.py

+    field:          Name of the sort field
+    weight:         The sort weight used, values between 10-200 are good starts
+    weight_default: The default weight used if a sort 'field' could not be found or had a invalid entry (default is: 0)
+    inverse:        Use inverse weighting for the field, example: Date/Age fields


I don't understand this

What exactly inverse or weight_default?

The line that I'm commenting on. Line 38.

Inverse weighting means that the lowest result slot gets the highest score and the highest gets the lowest score. So for Age/Date fields the entries with (0 days) get the maximum weight and older entries start getting lower and lower weights.

cvium · 2016-12-04T15:17:44Z

flexget/plugins/modify/sort_by_weighted.py

+
+log = logging.getLogger('sort_by_weighted')
+
+SUPPORTED_TYPES = (


What's the point of this exactly?

The types?
Mainly to safeguard from using unsupported fields. Sorting by string for example would need something like "similarity" between them for expected results.

cvium · 2016-12-04T15:18:19Z

flexget/plugins/modify/sort_by_weighted.py

+    def on_task_filter(self, task, config):
+        # [field] = [weight, weight_default, delta, inverse, [min,max]]
+        settings = {}
+        for centry in config:


centry is a weird variable name imo

aye, will be changed

…_by_weighted

…gets more confined

Andy2244 · 2016-12-05T15:38:06Z

I'm happy with it at this point, so consider it for a merge.

cvium · 2016-12-05T15:47:31Z

flexget/plugins/modify/sort_by_weighted.py

+        'minItems': 2
+    }
+
+    #    def on_task_start(self, task, config):


Comments that serve no purpose should be removed

cvium · 2016-12-05T21:37:33Z

I feel the schema is overly complex, but I'm not sure how to remedy that.

Andy2244 · 2016-12-05T22:12:47Z

I don't expect this plugin to-be widely used and remember you only need this at a bare minimum, which looks less scary and still delivers usable results.

sort_by_weighted:
      - field: content_size
        weight: 80
      - field: newznab_grabs
        weight: 25

remove unused comment

paranoidi · 2016-12-06T13:25:27Z

Why is it 'sort_by_weighted' instead of 'sort_by_weight'? Could this all be combined into sort plugin. Not a huge fan of multiple plugins doing same things ..

Andy2244 · 2016-12-06T13:44:05Z

The name was just what came first to mind and coveys the idea of a 'weighted' score sort. As for combining the sorts, this would make the config schema more complex, but could be done. I simply did not feel confident enough to fiddle with a existing plugin and existing configs.
Also keep in mind that sort_by work on all_entries, while weighted uses accepted/undecided by design. I really had a hard time cramping yet another config parameter in the schema to set the states used for the sort.

paranoidi · 2016-12-06T13:49:15Z

flexget/plugins/modify/sort_by_weighted.py

+                value = value.days
+            elif isinstance(value, bool):
+                value = int(value)
+            if len(settings[key]) == 6:


Magic value

paranoidi · 2016-12-06T13:49:45Z

flexget/plugins/modify/sort_by_weighted.py

+                        if value is None:
+                            continue
+                        if key not in max_values:
+                            if len(settings[key]) == 6:


Magic value

paranoidi · 2016-12-06T13:50:06Z

flexget/plugins/modify/sort_by_weighted.py

+                            else:
+                                max_value = value
+                        else:
+                            if len(settings[key]) == 6:


Magic value

paranoidi · 2016-12-06T13:50:33Z

flexget/plugins/modify/sort_by_weighted.py

+                            continue
+                        if key not in max_values:
+                            if len(settings[key]) == 6:
+                                max_value = min(value, settings[key][5])


Magic value?

paranoidi · 2016-12-06T13:53:28Z

Assuming backwards compatibility it would be fine to combine this into sort.

Andy2244 · 2016-12-06T14:00:12Z

Can refactor the local settings array to a dict, to remove the magic values. I can try combining the two, but might need help with the schema.

paranoidi · 2016-12-06T19:25:53Z

Just took a fresh look at sort_byvpkugin, not so sure about merging anymore. It seems to be very field specific ..

Andy2244 · 2016-12-07T20:55:17Z

Yes i also feel it works quite different, so might confuse usage. I'm trying to refactor the algorithm so it can work with any type that has a lt/gt comparator, mainly to support the Quality type.

max value fixes Quality type support proper timedelta handling

fixed crash on weight calculation fixed lower defaults updated examples

Andy2244 · 2016-12-09T13:53:44Z

oki, did refactor the code to technically support all types, but the weight results for types like string are somewhat undefined. The mayor types like int/float/bool/Quality/datetime/timedelta work as expected.
I still don't like merging it with the sort_by plugin, since the config gets more complex and its more clear that this is a special type of sort.

cvium · 2016-12-09T13:54:28Z

flexget/plugins/modify/sort_by_weight.py

            return
        config = self.prepare_config(config)
-        log.info('sorting ´undecided´,´accepted´ entries by weight!')
+        log.info('sorting undecided, accepted entries by weight')


Seems like unnecessary logging. You would expect it to run if you've enabled it...

I wanted to make clear that the weighting is only calculated for undecided, accepted entries, which is different to how sort_by works. It still sorts all entries at the end, but the other states get the default weight of 0.

Andy2244 · 2016-12-14T08:50:44Z

Just wondering do i have to add a test for every new plugin i write, to-be accepted for a merge? So far none of my plugins have been merged yet or is there a minimum time that has to lapse without new changes?
What exactly are the requirements to get something merged? I see some pull request are like 6 months old, so i'm a little confused.

paranoidi · 2016-12-14T15:59:52Z

@Andy2244 Process is slow sometimes , but your contributions are greatly appreciated! Join IRC / Gitter for more hands on involvement :)

Unit tests are greatly appreciated as we are going to maintain this plugin long term. Only other issue that I can spot now is the too long docstring which should not prevent merging :)

Can you add some unit test and we'll get this one merged right away! It's not required but I would feel much better if it had some.

cleanup exceptions, logging

Andy2244 · 2016-12-17T15:35:32Z

Implemented suggested changes.

PS: I know its bad practice, but i want to finish the Anidb lookup plugin first, before i look at the testcode for each my plugins.

paranoidi · 2016-12-19T11:55:37Z

@Andy2244 Understandable, a lot to take in such short time. I'm merging this in. Please update the wiki page to include this plugin! :)

Andy2244 added 3 commits December 4, 2016 13:36

weighted sort implementation

8d8b8bf

reversed sort order

ca51735

fixed dict exception

better examples docu and fixed type

eb6607e

cvium reviewed Dec 4, 2016

View reviewed changes

Andy2244 added 2 commits December 4, 2016 16:57

fixed typo and change limits to separate config values

9336eda

Merge remote-tracking branch 'refs/remotes/Flexget/develop' into sort…

c1721da

…_by_weighted

Andy2244 changed the title ~~new sort_by_weighted plugin~~ [WIP] new sort_by_weighted plugin Dec 5, 2016

change to only work with accepted, undecided entries so weight range …

e1cc13d

…gets more confined

Andy2244 changed the title ~~[WIP] new sort_by_weighted plugin~~ new sort_by_weighted plugin Dec 5, 2016

cvium reviewed Dec 5, 2016

View reviewed changes

change priority to 127

e312398

remove unused comment

paranoidi reviewed Dec 6, 2016

View reviewed changes

Andy2244 changed the title ~~new sort_by_weighted plugin~~ [WIP] new sort_by_weighted plugin Dec 6, 2016

Andy2244 added 2 commits December 8, 2016 20:08

refactor to support more types (Quality/Dates)

e46e991

code cleanup

a905b22

max value fixes Quality type support proper timedelta handling

Andy2244 changed the title ~~[WIP] new sort_by_weighted plugin~~ [WIP] new sort_by_weight plugin Dec 8, 2016

Andy2244 added 2 commits December 9, 2016 14:35

fixed datetime handling

83463be

fixed crash on weight calculation fixed lower defaults updated examples

fix typo

0ab1e88

fix Non-ASCII character

f1c83ef

cvium reviewed Dec 9, 2016

View reviewed changes

change log statement to verbose and made more clear whats happening

9bbfc6c

Andy2244 changed the title ~~[WIP] new sort_by_weight plugin~~ new sort_by_weight plugin Dec 9, 2016

remove past.types

38b0ced

cleanup exceptions, logging

paranoidi merged commit a6cf081 into Flexget:develop Dec 19, 2016

Andy2244 deleted the sort_by_weighted branch January 6, 2017 12:55


		log = logging.getLogger('sort_by_weighted')

		SUPPORTED_TYPES = (

Conversation

Andy2244 commented Dec 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for changes:

Detailed changes:

Config usage if relevant (new plugin or updated schema):

Uh oh!

cvium commented Dec 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy2244 commented Dec 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cvium commented Dec 5, 2016

Uh oh!

Andy2244 commented Dec 5, 2016

Uh oh!

paranoidi commented Dec 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andy2244 commented Dec 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paranoidi commented Dec 6, 2016

Uh oh!

Andy2244 commented Dec 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paranoidi commented Dec 6, 2016

Uh oh!

Andy2244 commented Dec 7, 2016

Uh oh!

Andy2244 commented Dec 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy2244 Dec 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy2244 commented Dec 14, 2016

Uh oh!

paranoidi commented Dec 14, 2016

Uh oh!

Andy2244 commented Dec 17, 2016

Uh oh!

paranoidi commented Dec 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Andy2244 commented Dec 4, 2016 •

edited

Loading

cvium commented Dec 4, 2016 •

edited

Loading

paranoidi commented Dec 6, 2016 •

edited

Loading

Andy2244 commented Dec 6, 2016 •

edited

Loading

Andy2244 commented Dec 6, 2016 •

edited

Loading

Andy2244 Dec 9, 2016 •

edited

Loading