[WiP] mysql proxy: separate downstream client and upstream server authentication. by qinggniq · Pull Request #15485 · envoyproxy/envoy

qinggniq · 2021-03-15T06:00:27Z

Signed-off-by: qinggniq livewithblank@gmail.com

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md

This pr implements the separate downstream client and upstream server authentication in mysql proxy.
How it works?

maintain a mysql client pool per thread, establish a valid mysql client is expensive, it need one tcp handshake and at least 1.5 round trip server greet -> client auth -> server response, if server need client switch auth plugin, it need additional 1 round trip. So I decide to maintain a mysql client pool for each cluster, once mysql client is released, it will be put back into pool.
when downstream connection is coming, mysql proxy acts as a server, sends greet message to client, wait for client auth message, then verify the auth message by downstream_auth_username, downstream_auth_password. Once client pass the authentication, base on the db in auth message, proxy will find corresponding cluster, and get a client from mysql client pool.
when proxy is acquiring a client, the mysql client pool first check whether there are any idle clients, if not it will use underlaying Tcp::ConnectionPool::Instance to acquire a connection, when success, it will mmic as a mysql client, deal with greet message, send auth message, when received ok message which means it pass the connection phase, the pool will put this connection into active clients list.

What will it affect?

Now mysql proxy needs to maintain connections of upstream like tcp proxy or redis proxy, so it become a terminal filter. In current mysql proxy documentation, it mentions that it's Dynamic Metadata could combine with rbac filter to control accesses to individual tables in a database.

but now mysql proxy becomes a terminal filter, so this feature is no longer support, but when mysql proxy maintain the connections of upstream, it becomes possible to support more features like mirror, fault, read load balance.

Commit Message: separate downstream client and upstream server authentication.
Additional Description:
Risk Level: medium
Testing: TBD
Docs Changes: TBD
Release Notes: support separate downstream client and upstream server authentication.
Platform Specific Features: N/A
[Optional Runtime guard:]
[Optional Fixes #Issue] #14697
[Optional Deprecated:]
[Optional API Considerations:]

Signed-off-by: qinggniq <livewithblank@gmail.com>

repokitteh-read-only · 2021-03-15T06:00:39Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
API shepherd assignee is @htuch
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #15485 was opened by qinggniq.

see: more, trace.

Signed-off-by: qinggniq <livewithblank@gmail.com>

…hentication Signed-off-by: qinggniq <livewithblank@gmail.com>

htuch · 2021-03-17T01:24:36Z

@qinggniq I don't understand:

this feature will make the mysql filter to be terminal filter, thus the previous feature collect query stat and use rbac filter to deny specific queries is no longer support.

for two reasons:

RBAC comes before any terminal filter, so why wouldn't this keep working?
From an API compatibility perspective, I don't think we can do this as you are effectively making an API breaking change. It's also not very clearly documented in the PR.

htuch · 2021-03-17T01:25:18Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+    google.protobuf.UInt32Value max_idle_connections = 2;
+
+    // Connections when envoy setup, default is 0.
+    google.protobuf.UInt32Value start_connections = 3;


What do these changes have to do with auth?

qinggniq · 2021-03-17T03:22:42Z

@htuch Hello, I filled up the pr message, just ping me if the message has any confused points.

Signed-off-by: qinggniq <livewithblank@gmail.com>

htuch · 2021-03-18T02:10:05Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+}
+
+message MySQLProtocolOptions {
+  config.core.v3.DataSource auth_password = 1 [(udpa.annotations.sensitive) = true];


Suggest a shared message for username/password.

Do you mean remove udpa.annotations.sensitive = true? I just found at redis username/password is secret at

envoy/api/envoy/extensions/filters/network/redis_proxy/v3/redis_proxy.proto

Lines 308 to 318 in a240824

message RedisProtocolOptions {

option (udpa.annotations.versioning).previous_message_type =

"envoy.config.filter.network.redis_proxy.v2.RedisProtocolOptions";

// Upstream server password as defined by the `requirepass` directive

// <https://redis.io/topics/config>`_ in the server's configuration file.

config.core.v3.DataSource auth_password = 1 [(udpa.annotations.sensitive) = true];

// Upstream server username as defined by the `user` directive

// <https://redis.io/topics/acl>`_ in the server's configuration file.

config.core.v3.DataSource auth_username = 2 [(udpa.annotations.sensitive) = true];

, I don't know is there any differences between them.

I'm saying grouping username/password with senstitivity options into a single message is good and makes sense als for downstream above; the same username/password message can be reused for both upstream/downstream.

htuch

I still think we need to maintain backwards compatibility. Otherwise, when we ship the next Envoy release, control planes will have their configuration rejected by Envoy.

This is a v3 stable filter (at API level), so we can't just say that certain uses of it are no longer supported. You need to allow the filter to function in the use cases it previously did limit the new capabilities for when it is a terminal filter.

htuch · 2021-03-18T02:10:45Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+
+  repeated Route routes = 3;
+
+  ConnectionPoolSettings setting = 4 [(validate.rules).message = {required: true}];


Can you split out the connection pool aspect of this PR from the split in the auth aspect? This is a very large PR and will be hard to review unless that is done IMHO.

htuch · 2021-03-18T02:13:34Z

@rshriram @venilnoronha for high-level API and design input as well.

qinggniq · 2021-03-19T03:57:17Z

@htuch I think there are two approaches to make it backward compatible:

Determine whether this filter is terminal filter by setting a is_terminal flag at protobuf file, we can use this to turn off the feature by default, but I'm not sure it's there are any way to implement it.
Make another mysql proxy and name it to mysql_proxy_plus, reserve current mysql proxy, and replace mysql_proxy with mysql_proxy_plus completely at v4.

htuch · 2021-03-19T21:29:02Z

@qinggniq I don't think (2) is great given duplication, so you will need to add an API field to distinguish from existing behaviors like (1).

qinggniq · 2021-03-22T02:57:18Z

@htuch Yes, I agree that if (1) and (2) could both work, (1) will be better. I'm trying to find out a way to determine the filter whether is a terminal filter at runtime #15565. And I think the method (2) could be able not to make much duplication, the code base of mysql proxy contains many common files which can be used by two filters like codec, session, decoder, utils, the difference between mysql_proxy and mysql_proxy_plus is config、filter. I think when (1) could work, we still need to write a new MySQLFilter for new feature, because now the base classes of MySQLFilter is deferent from before, they now are "different filters".

Signed-off-by: qinggniq <livewithblank@gmail.com>

…ysql-tool Signed-off-by: qinggniq <livewithblank@gmail.com>

Signed-off-by: qinggniq <livewithblank@gmail.com>

alyssawilk

(adding a drive by comment or two after looking this over from the runtime issue)

alyssawilk · 2021-03-22T14:18:30Z

source/extensions/filters/network/mysql_proxy/conn_pool_impl.h

+    ConnectionPoolSettings config_;
+    std::list<ThreadLocalActiveClientPtr> pending_clients_;
+    std::list<ThreadLocalActiveClientPtr> active_clients_;
+    std::list<ThreadLocalActiveClientPtr> busy_clients_;


connection reuse will be awesome! That said, this looks a lot like the legacy TCP connection pool and I don't think we want to reinvent this again, especially as the base pool supports connection reuse.

Can we please inherit from souce/common/conn_pool/conn_pool_base.* and refactor/extend where needed to avoid all the duplicate pending/active/busy work and reuse stats etc?

Yes, that's better.

alyssawilk · 2021-03-22T14:19:10Z

api/envoy/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto

  // Configuration for tunneling TCP over other transports or application layers.
-  // Tunneling is supported over both HTTP/1.1 and HTTP/2. Upstream protocol is
-  // determined by the cluster configuration.
+  // Currently, only HTTP/2 is supported. When other options exist, HTTP/2 will


Yes, I'll fix those unrelated changes.

htuch · 2021-03-23T01:04:17Z

@venilnoronha @rshriram would one of you be able to provide some MySQL CODEOWNERS input as well into the API discussion? Thanks.

Signed-off-by: qinggniq <livewithblank@gmail.com>

Signed-off-by: qinggniq <livewithblank@gmail.com> format Signed-off-by: qinggniq <livewithblank@gmail.com>

Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq

@htuch I have changed api a little, could you take a look?

qinggniq · 2021-03-31T11:13:54Z

.bazelrc


 # We already have absl in the build, define absl=1 to tell googletest to use absl for backtrace.
 build --define absl=1
+build --//bazel:http3=False


I will remove it in the final.

qinggniq · 2021-03-31T11:17:26Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+
+  repeated Route routes = 3;


Here I will add a field like bool new_feature_enabled to control after I replace the signature of isTerminalFilter. #15565 (comment)

Signed-off-by: qinggniq <livewithblank@gmail.com> fix: filter config Signed-off-by: qinggniq <livewithblank@gmail.com> add some comments Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com>

htuch · 2021-04-02T22:36:36Z

@lizan this was introduced by Tetrate I think; do you folks still consider MySQL something you are supporting?

htuch

@qinggniq I left some API comments, as long as we resolve this in a way that does not radically break existing users I'm fine with that.

Re: code review, I'm trying to find someone who can take a look. The existing CODEOWNERS appear MIA and this is a large non-trivial change that needs someone with MySQL experience.

We may be able to simply treat this as an external contribution and move it under the new contrib/ directory structure that @mattklein123 is planning on putting in place. I've had a look and your code appears to be localized to just the extension, so it would naturally fit that mold.

htuch · 2021-04-05T00:06:18Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+}
+
+message MySQLProtocolOptions {
+  config.core.v3.DataSource auth_password = 1 [(udpa.annotations.sensitive) = true];


I'm saying grouping username/password with senstitivity options into a single message is good and makes sense als for downstream above; the same username/password message can be reused for both upstream/downstream.

htuch · 2021-04-05T00:08:11Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+
+  reserved 3;
+
+  reserved "is_termianl";


Where does this come from?

Signed-off-by: qinggniq <livewithblank@gmail.com> fix spell Signed-off-by: qinggniq <livewithblank@gmail.com>

…-tool Signed-off-by: qinggniq <livewithblank@gmail.com>

Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq · 2021-04-06T08:33:02Z

We may be able to simply treat this as an external contribution and move it under the new contrib/ directory structure that @mattklein123 is planning on putting in place

@htuch Thanks for review, I am a little confused of above message.

Do you mean to move the source/extensions/filters/network/mysql_proxy to new directory contrib/extensions/filters/network/mysql_proxy?
When do you plan to do this?
What's the difference between treat this as an external contribution and improve exists extension?

htuch · 2021-04-06T23:25:57Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

  // If the access log field is empty, access logs will not be written.
  string access_log = 2;
+
+  // [#not-implemented-hide:] enable feature of separate authentication of downstream clients and upstream server. When feature_enable is true, the filter is a terminal filter, it's config should put at the last of filter chain, when feature_enable is false, the filter is not a terminal filter, it's config can not put at the last of filter chain. Default is false.


Needs to be wrapped and have a better name than "feature_enable"; maybe enable_protocol_management or something like that to indicate you're now basically the equivalent of HCM.

htuch · 2021-04-06T23:26:11Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+  //      - database: "database_b"
+  //        cluster: "cluster_b"
+  //
+  // When using the above routes, the following connect dsn would be sent to:


https://en.wikipedia.org/wiki/Data_source_name

htuch · 2021-04-06T23:26:53Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+  // * If username is empty but password is not, then proxy will only auth password.
+  // * If password is empty but username is not, then proxy will only auth username/
+  // * If the username is not match, then ER_ACCESS_DENIED_ERROR will return;
+  // * If connect db name is not found in envoy.extensions.filters.network.mysql_proxy.v3.MySQLProxy.routes, then ER_ER_BAD_DB_ERROR will return.


What does the last bullet point have to do with auth?

htuch · 2021-04-06T23:27:08Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+
+  // Downstream client username/password.
+  // * If username and password is empty, then authentication will always pass.
+  // * If username is empty but password is not, then proxy will only auth password.


Suggested change

// * If username is empty but password is not, then proxy will only auth password.

// * If username is empty but password is not, then proxy will only match the auth password.

Also, what's the point of wildcard username/passwords? Why not have exact match semantics, expecting an empty string when there is an empty username specified?

Yeah, it's weird not using exact match semantics for username/password. fixed.

htuch · 2021-04-06T23:27:28Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+  // Downstream client username/password.
+  // * If username and password is empty, then authentication will always pass.
+  // * If username is empty but password is not, then proxy will only auth password.
+  // * If password is empty but username is not, then proxy will only auth username/


Suggested change

// * If password is empty but username is not, then proxy will only auth username/

// * If password is empty but username is not, then proxy will only match the auth username.

htuch · 2021-04-06T23:29:15Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto


+// Auth information of MySQL.
+message AuthInfo {
+  config.core.v3.DataSource auth_username = 1;


Probably just put the sensitive annotations on the fields in here directly to factor them out.

Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com>

lizan

In general I think the direction is fine, but this PR is too large. Is it possible to split out a couple things out of this PR such: a) adding routing/conn pool support and b) authentication support?

lizan · 2021-04-07T05:12:04Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+  config.core.v3.DataSource auth_username = 1 [(udpa.annotations.sensitive) = true];
+
+  config.core.v3.DataSource auth_password = 2 [(udpa.annotations.sensitive) = true];


can be just username/password (without auth_ prefix).

lizan · 2021-04-07T05:14:06Z

api/envoy/extensions/filters/network/mysql_proxy/v3/mysql_proxy.proto

+  // [#not-implemented-hide:] enable manage MySQL protocol at proxy. Default is false.
+  // * When enable_manage_protocol is true, MySQL filter is a terminal filter, it's config should put at the last of filter chain, in the same time, MySQLProtocolOptions should be set at corresponding config of cluster for connecting to upstream database.
+  // * When enable_manage_protocol is false, MySQL filter is not a terminal filter, it's config can not put at the last of filter chain.
+  bool enable_manage_protocol = 3;


Isn't this implied by whether routes has element or not? Existing behavior delegate route select to tcp_proxy so in that case route shouldn't have anything.

Yes, when enable_manage_protocol is false, route and auth_info should not set value.

qinggniq · 2021-04-07T06:02:09Z

@lizan ok, I'll try to do it.

Signed-off-by: qinggniq <livewithblank@gmail.com>

This pull request is the part of #15485, help reduce the work of review. - refactor the BUILD struct of mysql proxy - replace auth related fields of message from `string` to `vector`, avoid `\0` character leading message to be partial. Commit Message: refactor BUILD struct of mysql proxy Additional Description: Risk Level: Low Testing: N/A Docs Changes: N/A Release Notes: N/A Signed-off-by: qinggniq <livewithblank@gmail.com>

This pull request is the part of envoyproxy#15485, help reduce the work of review. - refactor the BUILD struct of mysql proxy - replace auth related fields of message from `string` to `vector`, avoid `\0` character leading message to be partial. Commit Message: refactor BUILD struct of mysql proxy Additional Description: Risk Level: Low Testing: N/A Docs Changes: N/A Release Notes: N/A Signed-off-by: qinggniq <livewithblank@gmail.com>

github-actions · 2021-05-07T08:01:10Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

github-actions · 2021-05-14T08:07:51Z

This pull request has been automatically closed because it has not had activity in the last 37 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

impore envoy build doc

61ead0e

Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq requested a review from mattklein123 as a code owner March 15, 2021 06:00

repokitteh-read-only bot added the api label Mar 15, 2021

repokitteh-read-only bot assigned htuch Mar 15, 2021

qinggniq marked this pull request as draft March 15, 2021 06:00

qinggniq added 2 commits March 16, 2021 17:10

Merge branch 'main' of https://github.com/envoyproxy/envoy into main

72edd0c

Signed-off-by: qinggniq <livewithblank@gmail.com>

feat: mysql proxy: separate downstream client and upstream server aut…

6de80e2

…hentication Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq force-pushed the mysql-tool branch from 8d137bf to 6de80e2 Compare March 16, 2021 10:38

htuch reviewed Mar 17, 2021

View reviewed changes

comment some code

7016206

Signed-off-by: qinggniq <livewithblank@gmail.com>

htuch reviewed Mar 18, 2021

View reviewed changes

htuch suggested changes Mar 18, 2021

View reviewed changes

htuch assigned venilnoronha and rshriram Mar 18, 2021

qinggniq mentioned this pull request Mar 19, 2021

test: reinit connection_data_ when called poolReady at Tcp::Connection::MockInstance #15532

Merged

qinggniq mentioned this pull request Mar 22, 2021

Question: Is there are any way to determine whether is a terminal filter at runtime? #15565

Closed

qinggniq added 5 commits March 22, 2021 14:23

add test

80c970e

Signed-off-by: qinggniq <livewithblank@gmail.com>

Merge branch 'main' into mysql-tool

6d53ef2

delete decoder

eb79eb7

Signed-off-by: qinggniq <livewithblank@gmail.com>

Merge branch 'mysql-tool' of https://github.com/qinggniq/envoy into m…

dfdd229

…ysql-tool Signed-off-by: qinggniq <livewithblank@gmail.com>

gen udp proxy

817b58a

Signed-off-by: qinggniq <livewithblank@gmail.com>

alyssawilk reviewed Mar 22, 2021

View reviewed changes

qinggniq added 7 commits March 30, 2021 16:16

refctor connection pool

8bafe27

Signed-off-by: qinggniq <livewithblank@gmail.com>

refctor connection pool

6cb9b9b

Signed-off-by: qinggniq <livewithblank@gmail.com>

format

21c7df9

Signed-off-by: qinggniq <livewithblank@gmail.com>

change build option

defd51d

Signed-off-by: qinggniq <livewithblank@gmail.com>

merge from upstream

872226a

Signed-off-by: qinggniq <livewithblank@gmail.com>

pass Smoke Testing

d7059cb

Signed-off-by: qinggniq <livewithblank@gmail.com> format Signed-off-by: qinggniq <livewithblank@gmail.com>

fix bad merge

c3765cd

Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq commented Mar 31, 2021

View reviewed changes

add test

71b7dec

Signed-off-by: qinggniq <livewithblank@gmail.com> fix: filter config Signed-off-by: qinggniq <livewithblank@gmail.com> add some comments Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq requested a review from htuch April 2, 2021 11:01

htuch reviewed Apr 5, 2021

View reviewed changes

qinggniq added 3 commits April 6, 2021 13:37

make username password reuseable

91dfe5a

Signed-off-by: qinggniq <livewithblank@gmail.com> fix spell Signed-off-by: qinggniq <livewithblank@gmail.com>

Merge branch 'main' of https://github.com/envoyproxy/envoy into mysql…

644e377

…-tool Signed-off-by: qinggniq <livewithblank@gmail.com>

merge from master, add some comments

d0c38c2

Signed-off-by: qinggniq <livewithblank@gmail.com>

htuch reviewed Apr 6, 2021

View reviewed changes

add comments

7006899

Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com> fix typo Signed-off-by: qinggniq <livewithblank@gmail.com>

lizan suggested changes Apr 7, 2021

View reviewed changes

lizan reviewed Apr 7, 2021

View reviewed changes

fix api

de5460b

Signed-off-by: qinggniq <livewithblank@gmail.com>

qinggniq mentioned this pull request Apr 8, 2021

mysql proxy: refactor BUILD struct #15885

Merged

qinggniq mentioned this pull request Apr 20, 2021

mysql proxy: route and downstream/upstream separate auth implementation #16075

Closed

github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label May 7, 2021

github-actions bot closed this May 14, 2021

	message RedisProtocolOptions {
	option (udpa.annotations.versioning).previous_message_type =
	"envoy.config.filter.network.redis_proxy.v2.RedisProtocolOptions";

	// Upstream server password as defined by the `requirepass` directive
	// <https://redis.io/topics/config>`_ in the server's configuration file.
	config.core.v3.DataSource auth_password = 1 [(udpa.annotations.sensitive) = true];

	// Upstream server username as defined by the `user` directive
	// <https://redis.io/topics/acl>`_ in the server's configuration file.
	config.core.v3.DataSource auth_username = 2 [(udpa.annotations.sensitive) = true];


		repeated Route routes = 3;

		ConnectionPoolSettings setting = 4 [(validate.rules).message = {required: true}];

	// * If username is empty but password is not, then proxy will only auth password.
	// * If username is empty but password is not, then proxy will only match the auth password.

	// * If password is empty but username is not, then proxy will only auth username/
	// * If password is empty but username is not, then proxy will only match the auth username.

Conversation

qinggniq commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repokitteh-read-only bot commented Mar 15, 2021

Uh oh!

htuch commented Mar 17, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qinggniq commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch commented Mar 18, 2021

Uh oh!

qinggniq commented Mar 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

htuch commented Mar 19, 2021

Uh oh!

qinggniq commented Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alyssawilk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch commented Mar 23, 2021

Uh oh!

qinggniq left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch commented Apr 2, 2021

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qinggniq commented Apr 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

qinggniq commented Mar 15, 2021 •

edited

Loading

qinggniq commented Mar 17, 2021 •

edited

Loading

qinggniq commented Mar 19, 2021 •

edited

Loading

qinggniq commented Mar 22, 2021 •

edited

Loading