Skip to content

Add type annotations (top-level files)#2751

Merged
gazpachoking merged 2 commits intoFlexget:developfrom
vgerak:develop-typings
Oct 19, 2020
Merged

Add type annotations (top-level files)#2751
gazpachoking merged 2 commits intoFlexget:developfrom
vgerak:develop-typings

Conversation

@vgerak
Copy link
Copy Markdown
Contributor

@vgerak vgerak commented Oct 15, 2020

Motivation for changes:

Add/enrich type annotations

Detailed changes:

  • Add type annotation to update-changelog.py and files directly under flexget module
  • Add mypy_cache/ and dmypy.json to gitignore
# before
❯ dmypy run -- --follow-imports=skip flexget/*.py  | tail -1
Found 174 errors in 14 files (checked 16 source files)
# after
❯ dmypy run -- --follow-imports=skip flexget/*.py  | tail -1
Found 83 errors in 14 files (checked 16 source files)

I can keep adding commits to this PR

Addressed issues:

  • No such issue exists, we can add one if you wish

Implemented feature requests:

Config usage if relevant (new plugin or updated schema):

Log and/or tests output (preferably both):

To Do:

  • Add type annotations for the rest of the modules

@vgerak
Copy link
Copy Markdown
Contributor Author

vgerak commented Oct 15, 2020

Only annotated the vars that I could deduce with some confidence, so it's not complete, but should be correct.

3 thoughts moving forward:

  1. I was thinking to open different PRs for each module, rather than one (pretty big) PR that would keep being updated. This way it would be easier to review/comment and get integrated into the codebase. I leave the decision to the maintainers.
  2. Should I convert the logging strings to f-strings instead of str.format()? Since lazy logger evaluation (%) is not used, should be safe to do so.
  3. I get many warnings for logger.verbose() calls, I see that loguru.logger does not have that attribute. I assume it's correct (since it would be crashing everywhere otherwise), but I don't see how it's working. Please point me to the file that enables this behaviour, so I can take a look (curiosity).

Copy link
Copy Markdown
Contributor

@ianstalk ianstalk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

Copy link
Copy Markdown
Member

@gazpachoking gazpachoking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only scanned through the first half so far, but looks good to me. I mostly had questions on your changes in the use of Optional. Figured I'd just ask if there is a reason you changed them before scanning the rest.

@gazpachoking
Copy link
Copy Markdown
Member

3 thoughts moving forward:

1. I was thinking to open different PRs for each module, rather than one (pretty big) PR that would keep being updated. This way it would be easier to review/comment and get integrated into the codebase. I leave the decision to the maintainers.

Agreed, this would be nice, much easier to check over and merge in stuff that's good in smaller chunks.

2. Should I convert the logging strings to f-strings instead of `str.format()`? Since lazy logger evaluation (`%`) is not used, should be safe to do so.

We are using loguru, which does do lazy evaluation using the format formatting. I'm all for f strings in most locations, and I don't know how much that lazy evaluation saves us, but might as well leave them in that format for lazy evaluation.

3. I get many warnings for `logger.verbose()` calls, I see that loguru.logger does not have that attribute. I assume it's correct (since it would be crashing everywhere otherwise), but I don't see how it's working. Please point me to the file that enables this behaviour, so I can take a look (curiosity).

Loguru is a bit weird with how you add levels, and since we don't (can't) subclass the logger class a lot of tools give warnings on this. 😕 Here is where it's defined. Here are the loguru docs on the matter. If you can figure out a nicer way that PyCharm/mypy won't complain that would be great.

Thanks for all your work on this!

@vgerak
Copy link
Copy Markdown
Contributor Author

vgerak commented Oct 16, 2020

We are using loguru, which does do lazy evaluation using the format formatting. I'm all for f strings in most locations, and I don't know how much that lazy evaluation saves us, but might as well leave them in that format for lazy evaluation.

I couldn't find anything in the loguru docs regarding this. I know that stdlib logging is "lazy" (and safer) when using the "%s" formatting, as indicated by the relevant pylint warnings when trying to use .format() or f-strings.

I couldn't find anything indicating that loguru treats .format() lazily, only when marking a single entry that should be lazily evaluated if sink is at the appropriate level. I'm not against following the current implementation, just wondering if it is indeed lazy.

@gazpachoking
Copy link
Copy Markdown
Member

We are using loguru, which does do lazy evaluation using the format formatting. I'm all for f strings in most locations, and I don't know how much that lazy evaluation saves us, but might as well leave them in that format for lazy evaluation.

I couldn't find anything in the loguru docs regarding this. I know that stdlib logging is "lazy" (and safer) when using the "%s" formatting, as indicated by the relevant pylint warnings when trying to use .format() or f-strings.

I couldn't find anything indicating that loguru treats .format() lazily, only when marking a single entry that should be lazily evaluated if sink is at the appropriate level. I'm not against following the current implementation, just wondering if it is indeed lazy.

Hmm. I thought I had read this, but now I'm not so sure

@gazpachoking
Copy link
Copy Markdown
Member

Thinking about the lazy log stuff further:

  1. We are always logging at debug level in the background, so that we can create the crash logs when needed. This means only trace level stuff would avoid the evaluation.
  2. I don't know how much time it actually saves anyway.
  3. As you pointed out, loguru doesn't even advertise the format call is lazy. (it might be, I haven't dug into the code to find out)

Given that, I'm thinking fstrings are fine. I don't think we should necessarily go replace every logging call right away, but I don't think there is a problem with it. If we actually have expensive logging stuff we want to avoid always calling, maybe we put it on the trace level, with the explicit loguru way of making it lazy. @liiight @paranoidi Any thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants