Skip to content

Fix apigateway authorizer path suffix#7557

Merged
calvernaz merged 1 commit intomasterfrom
fix-apigateway-authorizer-path-suffix
Feb 22, 2023
Merged

Fix apigateway authorizer path suffix#7557
calvernaz merged 1 commit intomasterfrom
fix-apigateway-authorizer-path-suffix

Conversation

@calvernaz
Copy link
Contributor

@calvernaz calvernaz commented Jan 26, 2023

This PR accomplishes two things:

  1. making sure literal booleans are correctly parsed on our VTL templates. Why ast.literal_eval and not json.loads? Imagine the following case:
# valid rendered template (well, given airspeed implementation)
tpl = '{"key": True}' 

# fails because the string above is not valid json, True is not a valid json literal
json.loads(tpl)
# works and returns the following dict {"key": True}
ast.literal_eval(tpl)
  1. routes with trailing slashes are correctly handled inside APIGW router, it uses the strict_slashes where it's needed.

There is a comprehensive test upstream for the templating and routing issue.

@calvernaz calvernaz temporarily deployed to localstack-ext-tests January 26, 2023 00:29 — with GitHub Actions Inactive
@calvernaz calvernaz marked this pull request as draft January 26, 2023 00:29
@github-actions
Copy link

github-actions bot commented Jan 26, 2023

LocalStack integration with Pro

       3 files  ±0         3 suites  ±0   1h 31m 16s ⏱️ +28s
1 738 tests ±0  1 384 ✔️ ±0  354 💤 ±0  0 ±0 
2 456 runs  ±0  1 760 ✔️ ±0  696 💤 ±0  0 ±0 

Results for commit e16d8a1. ± Comparison against base commit 44c6718.

♻️ This comment has been updated with latest results.

@joe4dev joe4dev temporarily deployed to localstack-ext-tests January 30, 2023 19:58 — with GitHub Actions Inactive
@calvernaz calvernaz force-pushed the fix-apigateway-authorizer-path-suffix branch from 381a8f2 to 70ceb0e Compare February 8, 2023 22:47
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 8, 2023 22:47 — with GitHub Actions Inactive
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 8, 2023 23:01 — with GitHub Actions Inactive
@calvernaz calvernaz force-pushed the fix-apigateway-authorizer-path-suffix branch from 9b4c3ee to e4c678a Compare February 16, 2023 09:59
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 16, 2023 09:59 — with GitHub Actions Inactive
@calvernaz calvernaz force-pushed the fix-apigateway-authorizer-path-suffix branch from e4c678a to db9e225 Compare February 17, 2023 14:54
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 17, 2023 14:54 — with GitHub Actions Inactive
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 17, 2023 21:24 — with GitHub Actions Inactive
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 18, 2023 09:06 — with GitHub Actions Inactive
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 18, 2023 14:03 — with GitHub Actions Inactive
# we render the template with the context data and the response content
variables = self.build_variables_mapping(api_context)
response._content = self.render_vtl(template, variables=variables)
response._content = json.dumps(ast.literal_eval(response._content))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the important bit, we only support json templates at the moment, and ast.literal_eval make sure the string is a valid deserialised type, meaning a valid JSON object will turn out to be a valid dictionary, and true for other types. Without this evaluation, we would have (issue found) a rendered template containing invalid json because the boolean literal was a python literal (e.g '{ "key": True}') and not a JSON boolean literal (e.g '{ "key": true}').

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out this key change here 👍

👀 This breaks in cases where a JSON template is requested but the template produces invalid JSON. It would be very helpful to have a parity test covering such a scenario.
I'm ok with doing this in a follow-up PR but it would be good to add a TODO or comment describing this limitation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It might be a bit cleaner to introduce a variable for the rendered vlt template instead of updating response._content inline.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parity test is a good idea, but definitely on a follow-up PR. On AWS I'm sure this will end up as a 500 if the content type expected is application/json. In our case a 500 is guaranteed, the content of the exception might not be the same tho.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that one could argue that Velocity templates could be considered language-independent. However, given Apache Velocity stating "Velocity is a Java-based template engine", it might still be useful to aim for parity with the Java-based reference implementation.

There, only true and false are accepted boolean values (see pointer in docs) and using True or False raises an exception (tested with https://github.com/inbravo/apache-velocity-examples). This behavior is independent of the output type (tested using text output, not json).

Hence, given the following template and variables:

template = """
{
#set( $myTrue = true )
"booleanKeyTrue": $booleanKeyTrue,
"booleanKeyFalse": $booleanKeyFalse,
"booleanLiteralTrue": true,
"booleanLiteralFalse": false,
#if( $booleanKeyTrue )
"conditionalOutcome": "PASSED"
#else
"conditionalOutcome": "FAILED"
#end
}
"""
variables = {
    "booleanKeyTrue": "true",
    "booleanKeyFalse": "false",
}

The suggested workaround with dumping JSON (json.dumps(ast.literal_eval(response._content))) produces the correct output for the content type JSON:

import json

from localstack.utils.aws.templating import VtlTemplate
rendered_template = VtlTemplate().render_vtl(template, variables)
print(rendered_template)
print(json.loads(rendered_template))
{
"booleanKeyTrue": true,
"booleanKeyFalse": false,
"booleanLiteralTrue": true,
"booleanLiteralFalse": false,
"conditionalOutcome": "PASSED"
}

Nevertheless one could still argue that the underlying airspeed templating output violates the (Java-based) Velocity protocol:

import airspeed
import json

t = airspeed.Template(template)
booleanKeyTrue = True
booleanKeyFalse = False
print(t.merge(locals()))
{
"booleanKeyTrue": True,
"booleanKeyFalse": False,
"booleanLiteralTrue": true,
"booleanLiteralFalse": false,
"conditionalOutcome": "PASSED"
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a sensible assumption to expect a 500 but better write a parity test in a follow-up as I have experienced many surprises ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use velocity templates to render HTML, the implementation of velocity templates is Java, that's what that means. What we want is to render json in same cases, and that once deserialised is language specific, anything else is a string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking of, there is a unit test failing with valid json but fails to evaluate using ast.literal_evalwith an indentation error.

E         File "<unknown>", line 3
E           {
E       IndentationError: unexpected indent

I'll have to double check on this one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, it doesn't support leading and trailing white-spaces.

@calvernaz calvernaz force-pushed the fix-apigateway-authorizer-path-suffix branch from da5d277 to 826c898 Compare February 18, 2023 15:10
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 18, 2023 15:10 — with GitHub Actions Inactive
@calvernaz calvernaz marked this pull request as ready for review February 18, 2023 15:11
@calvernaz calvernaz requested a review from joe4dev February 18, 2023 22:27
Copy link
Member

@joe4dev joe4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest fixing the formatting/linting issues (get build green) and then handle the further suggestions in a follow-up (see comments in ext PR)

# we render the template with the context data and the response content
variables = self.build_variables_mapping(api_context)
response._content = self.render_vtl(template, variables=variables)
response._content = json.dumps(ast.literal_eval(response._content))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out this key change here 👍

👀 This breaks in cases where a JSON template is requested but the template produces invalid JSON. It would be very helpful to have a parity test covering such a scenario.
I'm ok with doing this in a follow-up PR but it would be good to add a TODO or comment describing this limitation.

assert "false" in result
assert result == '{"booleanKeyTrue": true, "booleanKeyFalse": false}'
# test is valid json
json.loads(result)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good json parsing addition 👍
(there is also a as_json option in render_vlt)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove that option in the render_vtl template; the template should render whatever it is and make decisions outside the templating what is expected based on the content-type. There are a couple of usages, thus not being removed yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

# we render the template with the context data and the response content
variables = self.build_variables_mapping(api_context)
response._content = self.render_vtl(template, variables=variables)
response._content = json.dumps(ast.literal_eval(response._content))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It might be a bit cleaner to introduce a variable for the rendered vlt template instead of updating response._content inline.

@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 21, 2023 07:50 — with GitHub Actions Inactive
@calvernaz calvernaz requested a review from joe4dev February 21, 2023 09:59
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 21, 2023 10:33 — with GitHub Actions Inactive
@joe4dev
Copy link
Member

joe4dev commented Feb 21, 2023

changes look good 👍
Thank you for finishing this tricky one 🙏

The failing Kinesis test appears unrelated

@joe4dev
Copy link
Member

joe4dev commented Feb 21, 2023

Remember to fix the ext pipeline before merging them together: https://github.com/localstack/localstack-ext/actions/runs/4230870403/jobs/7348708566#step:17:2570
FAILED tests/integration/apigateway/test_rest_apis.py::TestRestAPIs::test_lambda_token_authorizer_path_suffixes

@calvernaz calvernaz force-pushed the fix-apigateway-authorizer-path-suffix branch from b1e9916 to e16d8a1 Compare February 21, 2023 20:12
@calvernaz calvernaz temporarily deployed to localstack-ext-tests February 21, 2023 20:12 — with GitHub Actions Inactive
@coveralls
Copy link

Coverage Status

Coverage: 84.979% (-0.02%) from 84.995% when pulling e16d8a1 on fix-apigateway-authorizer-path-suffix into 44c6718 on master.

@coveralls
Copy link

Coverage Status

Coverage: 84.979% (-0.02%) from 84.995% when pulling e16d8a1 on fix-apigateway-authorizer-path-suffix into 44c6718 on master.

@calvernaz calvernaz merged commit c7d0336 into master Feb 22, 2023
@calvernaz calvernaz deleted the fix-apigateway-authorizer-path-suffix branch February 22, 2023 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants