Skip to content

Enhance assertion output for assert_all_called#224

Merged
lundberg merged 1 commit intolundberg:masterfrom
sileht:master
Nov 29, 2022
Merged

Enhance assertion output for assert_all_called#224
lundberg merged 1 commit intolundberg:masterfrom
sileht:master

Conversation

@sileht
Copy link
Copy Markdown
Contributor

@sileht sileht commented Nov 28, 2022

When AllCalledAssertionError is raised its hard to guess which route got
called and which routes didn't get called.

This change aims to provide more details by listing not called routes
and by leveraging the repr of route object.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 28, 2022

Codecov Report

Base: 100.00% // Head: 100.00% // No change to project coverage 👍

Coverage data is based on head (b9df437) compared to base (6f8d05f).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #224   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           20        20           
  Lines         2757      2755    -2     
  Branches       417       417           
=========================================
- Hits          2757      2755    -2     
Impacted Files Coverage Δ
respx/models.py 100.00% <ø> (ø)
respx/router.py 100.00% <100.00%> (ø)
tests/test_mock.py 100.00% <100.00%> (ø)
tests/test_transports.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Copy Markdown
Owner

@lundberg lundberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a most welcome enhancement @sileht !

respx/router.py Outdated
raise AllCalledAssertionError("RESPX: some routes were not called!")
not_called_routes = {str(route) for route in self.routes if not route.called}
if not_called_routes:
formatted_not_called_routes = "* " + "\n* ".join(not_called_routes)
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can make use of regular failed assertion output, instead of manual rendering 🤔

I'm thinking that this would give us the desired output:

assert not_called_routes == set(), "RESPX: some routes were not called!"

@sileht
Copy link
Copy Markdown
Contributor Author

sileht commented Nov 28, 2022

Yeah, this looks even nicer. Here is a pytest example:

    ...
    def assert_all_called(self) -> None:
        not_called_routes = [route for route in self.routes if not route.called]
>       assert not_called_routes == [], "RESPX: some routes were not called!"
E       AssertionError: RESPX: some routes were not called!
E       assert [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>] == []
E         Left contains 6 more items, first extra item: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>
E         Full diff:
E           [
E         -  ,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>,
E         +  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>,
E           ]

I have to use a list instead of a set as the Route object is not hashable.

@sileht
Copy link
Copy Markdown
Contributor Author

sileht commented Nov 28, 2022

Another example, if we also list called routes with:

       called_routes = [route for route in self.routes if route.called]
       assert called_routes == list(self.routes), "RESPX: some routes were not called!"
    def assert_all_called(self) -> None:
        called_routes = [route for route in self.routes if route.called]
>       assert called_routes == list(self.routes), "RESPX: some routes were not called!"
E       AssertionError: RESPX: some routes were not called!
E       assert [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>] == [<Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>, <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>]
E         At index 3 diff: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>> != <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>
E         Right contains 6 more items, first extra item: <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>
E         Full diff:
E           [
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/app/installations/12345/access_tokens'> AND <Method eq 'POST'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/user/12345/installation'> AND <Method eq 'GET'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1'> AND <Method eq 'GET'>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/pulls/1/files'> AND <Method eq 'GET'>>,
E            <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify/config.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.github/mergify.yml'> AND NOT <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Method eq 'GET'> AND <Path eq '/repos/testing/name/contents/.mergify.yml'> AND <Params contains QueryParams('ref=base-sha')>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/commits/old-sha-one/check-runs'> AND <Method eq 'GET'>>,
E         -  <Route <Scheme eq 'https'> AND <Host eq 'api.github.com'> AND <Path eq '/repos/testing/name/check-runs'> AND <Method eq 'POST'>>,
E           ]

.tox/py310/lib/python3.10/site-packages/respx/router.py:106: AssertionError

Tell me which one you prefer and I will update the pull request.

@lundberg
Copy link
Copy Markdown
Owner

I have to use a list instead of a set as the Route object is not hashable.

👍

Tell me which one you prefer and I will update the pull request.

I'd say the first implementation aligns better with the assertion, i.e. all called == no remaining non-called

A third alternative that might give roughly the same output would be ...

assert len(non_called_routes) == 0

@sileht
Copy link
Copy Markdown
Contributor Author

sileht commented Nov 29, 2022

So it's ready 😀

@lundberg
Copy link
Copy Markdown
Owner

So it's ready 😀

Almost .. We should probably drop the AllCalledAssertionError class and adjust tests using that one.

@lundberg
Copy link
Copy Markdown
Owner

Once AllCalledAssertionError is dropped, all tests that are using it should be constrained to match part of the assertion message.

When AllCalledAssertionError is raised its hard to guess which route got
called and which routes didn't get called.

This change aims to provide more details by listing not called routes
and by leveraging the __repr__ of route object.
@lundberg lundberg changed the title chore: add more informations for AllCalledAssertionError Enhance assertion output for assert_all_called Nov 29, 2022
@lundberg lundberg merged commit 59e629e into lundberg:master Nov 29, 2022
@lundberg
Copy link
Copy Markdown
Owner

Thanks @sileht

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants