Skip to content

Conversation

@TendTo
Copy link

@TendTo TendTo commented Feb 18, 2024

closes #4010

Trying to fix the annoying absence of type hinting for every function wrapped in the _log decorator.

# TODO: After https://youtrack.jetbrains.com/issue/PY-50952 is fixed, we can revisit this and
# consider adding Paramspec from typing_extensions to properly fix this. Currently a workaround

This solution should also be compatible with older versions of Python.

I have opted to put all the imports inside the TYPE_CHEKING branch. This requires the use of strings in the type annotations.
It is easy to go with the opposite approach.

Changes should be transparent to the end-user and not affect them in any way, except for the added support of static typing.

@Bibo-Joshi
Copy link
Member

Hi. Thanks for the PR!

I see that you explicitly use typing_extensions, which is not a dependency of PTB so far. We're generally trying rather strictly to keep the number of dependencies small so unless there is an overwhelming benefit in using it, I'd like to skip it.

Your changes also lead to a number of problems reported by mypy (see the pre-commit check). I haven't yet checked if they are genuine problems that surface due to improved type hinting or if some of them are due to problems in your changes.

Have you checked also the other options mentioned in microsoft/pyright#6669 (comment), i.e. using a typevar or no annotation at all, and how those behave with VSCode & PyCharm?

@TendTo
Copy link
Author

TendTo commented Feb 18, 2024

I see that you explicitly use typing_extensions, which is not a dependency of PTB so far. We're generally trying rather strictly to keep the number of dependencies small so unless there is an overwhelming benefit in using it, I'd like to skip it.

Have you checked also the other options mentioned in microsoft/pyright#6669 (comment), i.e. using a typevar or no annotation at all, and how those behave with VSCode & PyCharm?

I agree with you, and it seems that removing most of the type hinting annotation and leaving the static analysis tool do its job works.
EDIT: well, not exactly. As stated in the answer you linked, mypy is unable to infer the return type if not properly annotated. It is a matter of personal preferences, but I feel like proper type hinting is worth the extra package. Maybe it could be installed only when using versions of python < 3.10?
This said, there is a problem with the method itself: being inside the class, it is expected to have self as the first parameter.
Making it static creates problems with the documentation. Removing it from the class entirely may be the best option.

Your changes also lead to a number of problems reported by mypy (see the pre-commit check). I haven't yet checked if they are genuine problems that surface due to improved type hinting or if some of them are due to problems in your changes.

All of them seem to be genuine, and were quite easy to fix. The changes required are just in a few spots, so you can make sure everything is correct just by looking at the commits.

@TendTo
Copy link
Author

TendTo commented Feb 18, 2024

Is there a specific reason why input_chat_id in this line

input_chat_id = object()

is an object instead of the expected string or int?

Copy link
Member

@Bibo-Joshi Bibo-Joshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your analysis! I had a look at the follow-up changes in the other places and left some comments about them.

is an object instead of the expected string or int?

oh, that's just for testing. This makes it easier to check that we received the actualy object that we passed without having to compare a bunch of strings :)


Now to the acutal topic :D The typing of the log decorator seems to be a realy problem. The definiton place alone apparently is a problem: Note that it was originally placed outside the class bot #2686 moved it into the class (IIRC that was actually to improve the typing).

Looking at the function itself, what we acually want it to do is to do some logging when

  • we start calling a specific TG endpoint
  • we got the result
  • we we finish calling the TG endpoint

Instead of doing this via a decorator, we could that instead in a centralized place in _do_post.

Pros:

  • no need to add the @_log decorator everywhere (DRY)
  • no typing problems

Cons:

  • only endpoint is available, not the function name (i.e. sendMessage instead of send_message)
  • result available only as json data, not as TelegramObject

Both are acceptable for debug logging IMO.

I propose that we go forward in that direction instead.

return await self.get_bot().set_chat_administrator_custom_title(
chat_id=self.id,
user_id=user_id,
user_id=int(user_id),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of casting to int, we should adapt the type hint of the argument to match the one of Bot.set_admin_title. I see that in test_chat.py, the corresponding test test_set_administrator_custom_title and actually a few other tests are missing the assertions for check_shortcut_signature and check_shortcut_call. I expect that those would have already caught that …

We should add them. If not done within the scope of this PR. we should open an issue about that.

)
return await self.get_bot().send_game(
chat_id=chat_id,
chat_id=int(chat_id),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would vote for a # type: ignore[arg-type] instead, here. My reasoning:

  • chat_id is annotated here as str | int as return value of the internal _parse_quote_arguments, which is supposed to be used by all shortcut methods, even those that accept strings for chat_id. So forcing _parse_quote_arguments to return an integer doesn't make sense
  • However, we generally do not enforce type checks before passing values to Telegram. If a user would pass a string here, Telegram will through a descriptive error message if it can't parse it. That should be enough

Comment on lines 753 to 755
reply_markup=cast(
Optional["InlineKeyboardMarkup"], self._replace_keyboard(reply_markup)
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can instead improve the typing of _replace_keyboard here via a typevar, i.e.

_replace_keyboard(self, reply_markup: Optional[KT]) -> Optional[KT]:

_LOGGER.debug("Dropping pending updates from Telegram server")
await self.bot.set_webhook(
url=webhook_url,
url=webhook_url if webhook_url is not None else "",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

passing an empty string wouldn't make sense. Since this callback function is used only within an if webhook_url cause (see line 813), a type: ignore would be enough IMO.

@Poolitzer
Copy link
Member

@Bibo-Joshi yeah I am fine with the massively reduced complexity in favour of better type hinting. If people need more info about the called method they should just debug step into or write their own log lines in user level code

@harshil21
Copy link
Member

Instead of doing this via a decorator, we could that instead in a centralized place in _do_post

also agree with dropping the decorator.

@harshil21 harshil21 added enhancement 📋 pending-reply work status: pending-reply labels Feb 25, 2024
@Bibo-Joshi
Copy link
Member

@TendTo would you be interested in working on the alternative approach?

@TendTo
Copy link
Author

TendTo commented Feb 25, 2024

@TendTo would you be interested in working on the alternative approach?

I thought this would have been a matter of a few targeted changes, but I now see it is a bit more complex than that. I still want to give it a try, but since I am a bit busy, if someone can solve the issue faster, they are free to do so.

@TendTo
Copy link
Author

TendTo commented Feb 26, 2024

So, I tried following your advice of centralizing the logging in the method _do_post.
There are two ways I would use for logging the caller function: either logging the endpoint:

    async def _do_post(self, endpoint: str, ....):
        self._LOGGER.debug("Calling: %s", endpoint)
        ...
        result = await request.post(
            url=f"{self._base_url}/{endpoint}",
            request_data=request_data,
            read_timeout=read_timeout,
            write_timeout=write_timeout,
            connect_timeout=connect_timeout,
            pool_timeout=pool_timeout,
        )
        self._LOGGER.debug("Result from: %s", endpoint)
        self._LOGGER.debug(result)
        return result

or using the traceback provided by python:

    async def _do_post(self, endpoint: str, ....):
         # is this -4 always correct?
        self._LOGGER.debug("Calling: %s", traceback.extract_stack(limit=4)[-4].name)
        ...
        result = await request.post(
            url=f"{self._base_url}/{endpoint}",
            request_data=request_data,
            read_timeout=read_timeout,
            write_timeout=write_timeout,
            connect_timeout=connect_timeout,
            pool_timeout=pool_timeout,
        )
        self._LOGGER.debug("Result from: %s", traceback.extract_stack(limit=4)[-4].name)
        self._LOGGER.debug(result)
        return result

Based on the approach you prefer, there are a lot of tests that may need to be updated. E.g.

assert caplog.records[0].getMessage().startswith("Entering: get_me")

@harshil21
Copy link
Member

Definitely simply go with logging the endpoint. Trying to extract the traceback stack is a bit too complicated for such little benefit.

@harshil21 harshil21 changed the title fix: add type-hinting to _log decorator Remove @_log decorator by logging in Bot._do_post Feb 26, 2024
@harshil21 harshil21 added 📋 pending-reply work status: pending-reply and removed 📋 pending-reply work status: pending-reply labels Feb 26, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Apr 1, 2024
@Bibo-Joshi Bibo-Joshi added 🔌 enhancement pr description: enhancement and removed enhancement labels Nov 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

🔌 enhancement pr description: enhancement 📋 pending-reply work status: pending-reply

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Type Hinting] _log function is not properly annotated

4 participants