Skip to content

Add redis cluster support#177

Closed
erdody wants to merge 41 commits intoenvoyproxy:masterfrom
medallia:redisClusterPoC
Closed

Add redis cluster support#177
erdody wants to merge 41 commits intoenvoyproxy:masterfrom
medallia:redisClusterPoC

Conversation

@erdody
Copy link
Copy Markdown
Contributor

@erdody erdody commented Sep 15, 2020

No description provided.

@mattklein123 mattklein123 self-assigned this Sep 15, 2020
@mattklein123
Copy link
Copy Markdown
Member

At a high level this LGTM, though will need docs in the README. @ysawa0 do you mind taking a look?

@erdody
Copy link
Copy Markdown
Contributor Author

erdody commented Sep 16, 2020

Thanks @mattklein123, I thought draft PRs were not visible, but it's great you could take a look anyway.
We were planning to add documentation and an integration test before submitting the PR.
Please let us know if you prefer a different way to configure the different redis types.

@mattklein123
Copy link
Copy Markdown
Member

Nope this LGTM. docs/integration tests sounds great! Thank you!

stevesloka and others added 25 commits September 23, 2020 17:14
Signed-off-by: Steve Sloka <slokas@vmware.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Steve Sloka <slokas@vmware.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Fixes envoyproxy#138

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: David Black <david.black@autodesk.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
…oyproxy#143)

Even though the Makefile wants to encourage using mockgen@1.4.1, it
seems like the mocks have been generated using a pre-1.0 version of
mockgen. Using "go run github.com/golang/mock/mockgen" as a go:generate
command instead of just "mockgen" avoids the need to pre-install into
the developer's $PATH and uses the go.mod-specified version

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
My interest is the UDP protocol support which appeared in gotstats 0.3.10

There's a breaking change as of https://github.com/lyft/gostats/releases/tag/v0.3.0
which is that gostats no longer publishes stats as expvars.

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Tong Cai <caitong93@gmail.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
A newly-added test in envoyproxy#137 checks the exact text of an error message which
seems to vary when the network is tcp4 vs tcp6. This change relaxes the
assertion to look for "connection refused" in a panic without making
assumptions about what an IP address looks like.

Example failure:

--- FAIL: TestNewClientImpl (0.00s)
    --- FAIL: TestNewClientImpl/connection_refused (0.00s)
        cache_impl_test.go:442:
                Error Trace:    cache_impl_test.go:442
                Error:          func (assert.PanicTestFunc)(0x1724110) should panic with error message: "dial tcp 127.0.0.1:12345: connect: connection refused"
                                        Panic value:    "dial tcp [::1]:12345: connect: connection refused"
                                        Panic stack:    goroutine 27 [running]:

The testify assert package doesn't seem to support inexact matching on error messages, so the code gets a bit uglier than before.

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
…roxy#142)

This is a pure refactoring with no behavior changes. It's a step toward being able
to add memcache as a backend (see envoyproxy#140).

This PR moves RateLimitCache from the redis package to a new "limiter" package, along with
code for time/jitter, local cache stats,  and constructing cache keys. All that can be reused
with memcache.

After this PR, the redis package is imported in exactly two places:
- in service_cmd/runner/runner.go to call redis.NewRateLimiterCacheImplFromSettings()
- in service/ratelimit.go in ShouldRateLimit to identify if a recovered panic is a redis.RedisError. If so, a stat is incremented and the panic() propagation is ended and in favor of returning the error as a the function result.

The PR also includes changes by goimports to test/service/ratelimit_test.go
so that the difference between package name vs file path name is explicit
instead of implicit.

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
…oxy#148)

Previously an HTTP POST to /json would only return an HTTP status code,
not all the other details supported by grpc ratelimit responses.

With this change an HTTP POST to /json receives the full proto3 response
encoded as json by jsonpb.

It seems unlikely that anyone would be parsing the text "over limit" from
the HTTP body instead of just reading the 429 response code, but for anyone
doing that this would be a breaking change.

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
…s in runtime config folder directly instead of the runtime root dir. (envoyproxy#151)

Signed-off-by: Yuki Sawa <yukisawa@gmail.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
…nvoyproxy#153)

Signed-off-by: Petr Pchelko <ppchelko@wikimedia.org>
Signed-off-by: Diego Erdody <diego@medallia.com>
- Regenerate mocks based on new default protocol
- Manually transform v2 messages to v3 messages - some of the fields
were renamed thus json Marshal/Unmarshal does not work anymore
- Added tests that verify conversion v2<->v3 works for headers fields
- Update tests to use proto.Equal - simple assert.Equals might not
work correctly for protobuf messages.

Signed-off-by: Petr Pchelko <ppchelko@wikimedia.org>
Signed-off-by: Diego Erdody <diego@medallia.com>
This diff creates Dockerfile.integration for running integration tests with clearly-defined dependencies. Previously the dependencies of the integration tests were defined within the github actions config.

The new "make docker_tests" target should work for any developer with Docker installed.

Previously there was no single command that would run integration tests across platforms, which makes development and onboarding harder. Even copying the command from github actions wouldn't have worked before, since that command quietly assumed that redis was already running on port 6379.

Signed-off-by: David Weitzman <dweitzman@pinterest.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Fixes envoyproxy#154

Signed-off-by: Petr Pchelko <ppchelko@wikimedia.org>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Tong Cai <caitong93@gmail.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Yuki Sawa <yukisawa@gmail.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
envoyproxy#170)

Signed-off-by: Yuki Sawa <yukisawa@gmail.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Centralized log collection system works better with logs in json format.
E.g. DataDog strongly encourage setting up your logging library to produce your logs in JSON format to avoid the need for custom parsing rules.
So, the next small fix is all we need to get json logs.

Signed-off-by: Sergey Belyaev <sbelyaev@setronica.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
kriti-shaw and others added 16 commits September 23, 2020 17:14
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
Signed-off-by: Diego Erdody <diego@medallia.com>
@erdody
Copy link
Copy Markdown
Contributor Author

erdody commented Sep 24, 2020

Forgot to squash and merge the integration test. Closing in favor of a cleaner version #179

@erdody erdody closed this Sep 24, 2020
@erdody erdody deleted the redisClusterPoC branch September 24, 2020 02:31
@ysawa0
Copy link
Copy Markdown
Member

ysawa0 commented Sep 24, 2020

At a high level this LGTM, though will need docs in the README. @ysawa0 do you mind taking a look?

@mattklein123 Hi, sorry missed this. I can take a look!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants