Metrics for Reindexing by ankikuma · Pull Request #111845 · elastic/elasticsearch

ankikuma · 2024-08-13T17:02:52Z

This PR adds metrics for the Reindexing plugin, to measure the end-to-end time taken by a reindex request.

This PR aims to resolve ES-8851

Refreshing main to latest

henningandersen

A couple initial comments, I'd like to see the missing class before I continue.

henningandersen · 2024-08-13T17:06:23Z

modules/reindex/src/main/java/org/elasticsearch/reindex/ReindexPlugin.java

+        final Collection<Object> components = new ArrayList<>();
+        components.add(new ReindexSslConfig(services.environment().settings(), services.environment(), services.resourceWatcherService()));
+        components.add(new ReindexMetrics(services.telemetryProvider().getMeterRegistry()));
+        return (components);


This can be simplified to:

return List.of(new ReindexSslConfig(services.environment().settings(), services.environment(), services.resourceWatcherService()), new ReindexMetrics(services.telemetryProvider().getMeterRegistry()));

henningandersen · 2024-08-13T17:08:42Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

        ScriptService scriptService,
-        ReindexSslConfig reindexSslConfig
+        ReindexSslConfig reindexSslConfig,
+        ReindexMetrics reindexMetrics


I think you missed adding this class when commiting the changes.

Ah Yes! Sorry about that.

ankikuma · 2024-08-13T17:21:54Z

Hi Team, this is an initial implementation to measure the total time taken by a reindex request and externalize it as a histogram via APM. Please take a close look since I am new to both Reindexing module and the metering infrastructure. Any and all feedback is welcome.
This initial version does not have any tests, just a basic implementation to collect the reindex metrics. If it looks good to the team, I plan to write tests and also extend the same idea to UpdateByQueryAction and DeleteByQueryAction.

ankikuma · 2024-08-13T17:37:49Z

I am looking into the test failures.

henningandersen

Overall looks good. Next step seems to be to add a (or amend an existing) test to verify that we get some took times out.

henningandersen · 2024-08-13T18:44:06Z

...enrich/src/main/java/org/elasticsearch/xpack/enrich/action/TransportEnrichReindexAction.java

            transportService,
-            new ReindexSslConfig(settings, environment, watcherService)
+            new ReindexSslConfig(settings, environment, watcherService),
+            null


Will this throw a NullPointerException when this special reindex task ends? I think we may have to pass an object here.

Hi Henning, do you expect the NullPointerException to be thrown inside Reindexer->execute() ? Here, I did handle the case where we pass in a null ReindexMetrics object to the Reindexer class constructor. But perhaps you are thinking of some other scenario ?
Right now I pass in null ReindexMetrics object to TransportReindexAction when called from TransportEnrichReindexAction, but that also means we will not capture metrics for reindexing operations done by the EnrichPlugin. So I also wanted your opinion on whether we want that behavior or not.
The thing is that a metric can only be registered once on a node. So I think we can only instantiate one ReindexMetrics object for the node. If we want to record reindexing metrics from the EnrichPlugin as well, I believe we will have to instantiate the ReindexMetrics object in NodeConstruction here instead of as a component of ReindexPlugin like I do right now.

I think it should work if you accept it as a parameter in TransportEnrichReindexAction. You expose it from the ReindexPlugin as a component and those are automatically injected whenever needed.

I'd prefer to do that over the null check. If there is a problem, perhaps you can let me know how it fails?

Hi Henning, if I simply try to pass ReindexMetrics as a parameter to TransportEnrichReindexAction constructor, I run into the following failure. Could it have anything to do with the order in which the ReindexPlugin and the EnrichPlugin are created ? I have made the change and uploaded it here in case it is easier to inspect the failure here.

Task :docs:yamlRestTest FAILED

=== Log output of node node{:docs:yamlRestTest-0} ===

» ↓ errors and warnings from /Users/ankitakumar/elastic/elasticsearch/docs/build/testclusters/yamlRestTest-0/logs/es.out ↓
» [2024-08-14T16:09:26.012689Z] [BUILD] Starting Elasticsearch process
» Enter password for the elasticsearch keystore : Aug 14, 2024 12:09:32 PM sun.util.locale.provider.LocaleProviderAdapter
» WARNING: COMPAT locale provider will be removed in a future release
» [2024-08-14T12:10:03,451][ERROR][o.e.b.Elasticsearch ] [node-0] fatal exception while booting Elasticsearch org.elasticsearch.injection.guice.CreationException: Guice creation errors:
»
» 1) Could not find a suitable constructor in org.elasticsearch.reindex.ReindexMetrics. Classes must have either one (and only one) constructor annotated with @org.elasticsearch.injection.guice.Inject or a zero-argument constructor that is not private.
» at org.elasticsearch.reindex.ReindexMetrics.class(Unknown Source)
» while locating org.elasticsearch.reindex.ReindexMetrics
» for parameter 11 at org.elasticsearch.xpack.enrich.action.TransportEnrichReindexAction.(Unknown Source)
» at unknown
»
» 1 error
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.injection.guice.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:312)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.injection.guice.InjectorBuilder.initializeStatically(InjectorBuilder.java:115)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.injection.guice.InjectorBuilder.build(InjectorBuilder.java:73)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.injection.guice.Guice.createInjector(Guice.java:69)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.injection.guice.ModulesBuilder.createInjector(ModulesBuilder.java:36)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.construct(NodeConstruction.java:1163)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:270)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.node.Node.(Node.java:193)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:239)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:239)
» at org.elasticsearch.server@8.16.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
»
» ERROR: Elasticsearch did not exit normally - check the logs at /Users/ankitakumar/elastic/elasticsearch/docs/build/testclusters/yamlRestTest-0/logs/yamlRestTest.log
»
» ERROR: Elasticsearch died while starting up, with exit code 1
» ↓ last 40 non error or warning messages from /Users/ankitakumar/elastic/elasticsearch/docs/build/testclusters/yamlRestTest-0/logs/es.out ↓
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded module [x-pack-eql]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-ukrainian]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-kuromoji]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [discovery-gce]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [mapper-murmur3]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-phonetic]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-icu]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [discovery-ec2]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [mapper-annotated-text]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-smartcn]
» [2024-08-14T12:09:43,219][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [mapper-size]
» [2024-08-14T12:09:43,220][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [discovery-azure-classic]
» [2024-08-14T12:09:43,220][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [store-smb]
» [2024-08-14T12:09:43,220][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-nori]
» [2024-08-14T12:09:43,220][INFO ][o.e.p.PluginsService ] [node-0] loaded plugin [analysis-stempel]
» [2024-08-14T12:09:43,512][INFO ][o.e.c.u.FeatureFlag ] [node-0] The current build is a snapshot, feature flag [failure_store] is enabled
» [2024-08-14T12:09:44,529][INFO ][o.e.c.u.FeatureFlag ] [node-0] The current build is a snapshot, feature flag [ccs_telemetry] is enabled
» [2024-08-14T12:09:44,836][INFO ][o.e.e.NodeEnvironment ] [node-0] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [851.2gb], net total_space [926.3gb], types [apfs]
» [2024-08-14T12:09:44,836][INFO ][o.e.e.NodeEnvironment ] [node-0] heap size [512mb], compressed ordinary object pointers [true]
» [2024-08-14T12:09:44,858][INFO ][o.e.n.Node ] [node-0] node name [node-0], node ID [41A1Dw_hTsGdR6iragiuYQ], cluster name [yamlRestTest], roles [remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen, ingest, data_cold, data]
» [2024-08-14T12:09:48,502][INFO ][o.e.c.u.FeatureFlag ] [node-0] The current build is a snapshot, feature flag [connector_secrets] is enabled
» [2024-08-14T12:09:49,400][INFO ][o.e.i.r.RecoverySettings ] [node-0] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]

If we don't care to record metrics from the TransportEnrichReindexAction, we could also call the TransportReindexAction constructor with a NOOP like so:
new ReindexMetrics(TelemetryProvider.NOOP.getMeterRegistry())

Is that better than passing a null ?

Discussed this with Ryan here https://elastic.slack.com/archives/C8UUBNASY/p1724102081167389

ankikuma · 2024-08-14T12:46:35Z

Thanks for taking a look Henning. Will work on a test today.

henningandersen

Couple more comments.

henningandersen · 2024-08-14T13:30:42Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

-                    listener
+                    ActionListener.runAfter(listener, () -> {
+                        long elapsedTime = TimeUnit.NANOSECONDS.toSeconds(System.nanoTime() - startTime);
+                        if (reindexMetrics != null) reindexMetrics.recordTookTime(elapsedTime);


We normally prefer the curly-brace form, even for one-line if statements:

Suggested change

if (reindexMetrics != null) reindexMetrics.recordTookTime(elapsedTime);

if (reindexMetrics != null) {

reindexMetrics.recordTookTime(elapsedTime);

}

henningandersen · 2024-08-14T13:34:55Z

...enrich/src/main/java/org/elasticsearch/xpack/enrich/action/TransportEnrichReindexAction.java

            transportService,
-            new ReindexSslConfig(settings, environment, watcherService)
+            new ReindexSslConfig(settings, environment, watcherService),
+            null


I think it should work if you accept it as a parameter in TransportEnrichReindexAction. You expose it from the ReindexPlugin as a component and those are automatically injected whenever needed.

I'd prefer to do that over the null check. If there is a problem, perhaps you can let me know how it fails?

Refresh to latest

ankikuma · 2024-08-20T19:23:19Z

Adding @Tim-Brooks as a reviewer since Henning and Francisco are out this week.
I am still working on the UpdateByQuery and DeleteByQueryMetrics. I wanted to get some feedback on the test I wrote. For some reason it is not deterministic. For example, after the first reindexing request in ReindexBasicTests#testFiltering, I will sometimes see measurements.size() to be 0 while I expect there to be 1 measurement. But this does not happen in every run. Some runs will pass. Any ideas why that might be happening.

fcofdez

Looks good, I left a few comments

fcofdez · 2024-08-21T07:36:09Z

modules/reindex/src/main/java/org/elasticsearch/reindex/ReindexPlugin.java

+            new UpdateByQueryMetrics(services.telemetryProvider().getMeterRegistry())
+        );
+
+        /*


nit: we should get rid of this commented code

fcofdez · 2024-08-21T07:37:35Z

modules/reindex/src/main/java/org/elasticsearch/reindex/ReindexMetrics.java

+    private final LongHistogram reindexTimeSecsHistogram;
+
+    public ReindexMetrics(MeterRegistry meterRegistry) {
+        this(meterRegistry.registerLongHistogram(TOOK_TIME_HISTOGRAM, "Time to reindex by search", "seconds"));


are we sure that we want to use seconds granularity? we might end up reporting 0 seconds for fast reindex operations

The max span of APM histograms is 100,000. That was the reason for choosing seconds granularity as per the discussion here: https://elastic.slack.com/archives/C05ENM6MEDD/p1722522943486389?thread_ts=1722522639.141469&cid=C05ENM6MEDD
Perhaps we can capture in centiseconds ? We probably don't expect operations to take more than 15 min ?

Ah right, I didn't remember that limitation. Let's keep it in seconds then 👍

modules/reindex/src/main/java/org/elasticsearch/reindex/TransportDeleteByQueryAction.java

fcofdez · 2024-08-21T07:39:31Z

modules/reindex/src/main/java/org/elasticsearch/reindex/TransportUpdateByQueryAction.java

        ScriptService scriptService,
-        ClusterService clusterService
+        ClusterService clusterService,
+        UpdateByQueryMetrics updateByQueryMetrics


maybe we should annotate this variable as @Nullable too?

Yes, will do.

fcofdez · 2024-08-21T07:40:11Z

modules/reindex/src/main/java/org/elasticsearch/reindex/UpdateByQueryMetrics.java

+import org.elasticsearch.telemetry.metric.MeterRegistry;
+
+public class UpdateByQueryMetrics {
+    public static final String TOOK_TIME_HISTOGRAM = "es.update_by_query.took_time.histogram";


maybe we should name this metric es.update_by_query.duration.histogram? instead of using took_time?

fcofdez · 2024-08-21T07:45:25Z

modules/reindex/src/test/java/org/elasticsearch/reindex/ReindexBasicTests.java

        );
        assertHitCount(prepareSearch("source").setSize(0), 4);

+        TestTelemetryPlugin testTelemetryPlugin = getTestTelemetryPlugin();


I think that we should write a new test case for the newly introduced metrics instead of updating an existing test

ok sure. I will write a new test. Where should the test be located. Should it go inside the reindex package or somewhere else ? @henningandersen mentioned that ReindexBasicTests was not in the right folder.

yes, it looks like it's an integration test. Integration tests should go into the internalClusterTest folder.

I think we can move the ReindexBasicTests into internalClusterTest and write a new test case, wdyt?

I think we can move the ReindexBasicTests into internalClusterTest and write a new test case, wdyt?

I've just noticed that you wrote a new test, we can do what I suggested here in a follow-up PR.

fcofdez · 2024-08-21T07:46:42Z

modules/reindex/src/test/java/org/elasticsearch/reindex/ReindexBasicTests.java

+        testTelemetryPlugin.resetMeter();
        // Copy all the docs
        ReindexRequestBuilder copy = reindex().source("source").destination("dest").refresh(true);
+        testTelemetryPlugin.collect();


we're calling collect() before the reindex is executed, we should call that after

Thanks! I wasn't quite sure how collect works.

Refresh to latest

fcofdez

Looks good, left a few more comments about the test

fcofdez · 2024-08-26T13:37:23Z

modules/reindex/src/test/java/org/elasticsearch/reindex/ReindexPluginMetricsTests.java

+import static org.hamcrest.Matchers.equalTo;
+
+@ESIntegTestCase.ClusterScope(numDataNodes = 0, numClientNodes = 0, scope = ESIntegTestCase.Scope.TEST)
+public class ReindexPluginMetricsTests extends ESIntegTestCase {


can we move this test into internalClusterTest?

Sure, which folder inside internalClusterTest should I move it to ?
Perhaps server/src/internalClusterTest/java/org/elasticsearch/plugins ?

Or should I create a new folder inside server/src/internalClusterTest/java/org/elasticsearch/ called reindex ?

You should move it into modules/reindex/src/internalClusterTest/java/org/elasticsearch/index/reindex/

Ah ok inside modules

fcofdez · 2024-08-26T13:39:26Z

modules/reindex/src/test/java/org/elasticsearch/reindex/ReindexPluginMetricsTests.java

+        assertThat(copy.get(), matcher().created(2));
+
+        // wait for all threads to complete so that we get deterministic results
+        waitUntil(() -> (tps[0] = tp.stats()).stats().stream().allMatch(s -> s.active() == 0));


usually what we do when an assertion might need to wait until something completes in the background is use assertBusy which asserts a few times until either the assertion is true or fails once the timeout is reached.

Ah ok, so I don't need to do the explicit wait if I use assertBusy () ?

No, you don't need to do the explicit wait, assertBusy waits implicitly.

fcofdez

Looks good, just a few minor comments about the tests before approving 👍

fcofdez · 2024-08-27T07:41:52Z

...dex/src/internalClusterTest/java/org/elasticsearch/index/reindex/ReindexPluginMetricsIT.java

+        );
+        assertHitCount(prepareSearch("source").setSize(0), 4);
+
+        final String dataNodeName = internalCluster().getRandomNodeName();


since this test uses a single node started with internalCluster().startNode(); can we get the name from that node instead of using getRandomNode that way it's clearer that we'll be fetching the metrics from that node.

Yes! Let me make the change.

fcofdez · 2024-08-27T07:42:45Z

...dex/src/internalClusterTest/java/org/elasticsearch/index/reindex/ReindexPluginMetricsIT.java

+        copy = reindex().source("source").destination("none").filter(termQuery("foo", "no_match")).refresh(true);
+        assertThat(copy.get(), matcher().created(0));
+        assertHitCount(prepareSearch("none").setSize(0), 0);
+        // wait for all threads to complete so that we get deterministic results


can we remove this unused code?

Yes! Sorry I wanted to keep it around until I made sure the assertBusy() was working :-). I did the ci build multiple times to make sure results are deterministic.

fcofdez · 2024-08-27T07:48:16Z

...dex/src/internalClusterTest/java/org/elasticsearch/index/reindex/ReindexPluginMetricsIT.java

+        // Limit with maxDocs
+        copy = reindex().source("source").destination("dest_size_one").maxDocs(1).refresh(true);
+        assertThat(copy.get(), matcher().created(1));
+        assertHitCount(prepareSearch("dest_size_one").setSize(0), 1);


Since this tests are meant to test that we do collect the metrics on the different reindex operations I wonder if we should assert that reindex and related operations work as expected, maybe we can simplify this a bit and remove these assertions? wdyt?

The reason I kept those asserts is because I thought they were very basic asserts to check if the operation went through as expected. And I thought it would sort of make sense to verify that in conjunction with the metrics.
Do we want to remove them so the test would run faster or because it really doesn't matter if the operation succeeds or not - I guess metrics should get collected even for failed operations, right ?

Do we want to remove them so the test would run faster or because it really doesn't matter if the operation succeeds or not - I guess metrics should get collected even for failed operations, right ?

Yes, that's my thinking, since we're just interested in testing the metric collection in these tests.

Do we want to remove them so the test would run faster or because it really doesn't matter if the operation succeeds or not - I guess metrics should get collected even for failed operations, right ?

Yes, that's my thinking, since we're just interested in testing the metric collection in these tests.

Sure let me remove it. I can also remove the refresh(true) option in that case.

fcofdez

LGTM 👍

Refresh to latest

This PR adds metrics for the Reindexing plugin, to measure the end-to-end time taken by a reindex request, update-by-query request and delete-by-query request.

ankikuma added 3 commits August 13, 2024 11:27

Implement metrics for reindex

170926b

Merge remote-tracking branch 'upstream/main' into feature/ES8851

4ca9059

Refreshing main to latest

Make fanout changes to TransportEnrichReindexAction.java

4296219

ankikuma added WIP :Distributed/Reindex Issues relating to reindex that are not caused by issues further down labels Aug 13, 2024

ankikuma requested review from fcofdez and henningandersen August 13, 2024 17:02

elasticsearchmachine added the v8.16.0 label Aug 13, 2024

henningandersen reviewed Aug 13, 2024

View reviewed changes

addComment

d3758e8

Create only one instance of ReindexMetrics

016cfdb

henningandersen reviewed Aug 13, 2024

View reviewed changes

Create only one instance of ReindexMetrics

e6e19aa

henningandersen reviewed Aug 14, 2024

View reviewed changes

ankikuma added 3 commits August 14, 2024 13:17

Pass ReindexMetrics to TransportEnrichReindexAction

c3fb524

Add tests to ReindexBasicTests

51c18e4

Merge remote-tracking branch 'upstream/main' into feature/ES8851

aed7873

Refresh to latest

ankikuma marked this pull request as ready for review August 20, 2024 13:33

ankikuma added 2 commits August 20, 2024 13:02

Fix test in ReindexBasicTests

32f15a1

Fix test in ReindexBasicTests

93b7e68

ankikuma requested a review from Tim-Brooks August 20, 2024 19:17

fcofdez reviewed Aug 21, 2024

View reviewed changes

ankikuma added 2 commits August 22, 2024 13:15

comment

f889673

Merge remote-tracking branch 'upstream/main' into feature/ES8851

7eaf741

Refresh to latest

ankikuma added 4 commits August 22, 2024 13:44

Addressed review comments

cf55113

Added new tests

8a406ec

Modified tests

9e1c421

Try to make Test deterministic

911034d

fcofdez reviewed Aug 26, 2024

View reviewed changes

Address review comments and add new tests

4a0fd63

ankikuma removed the request for review from Tim-Brooks August 26, 2024 15:57

Add integ tests for DeleteByQuery and UpdateByQuery

236c0dc

fcofdez reviewed Aug 27, 2024

View reviewed changes

Update tests according to review comments

f2bda07

fcofdez approved these changes Aug 28, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into feature/ES8851

6f4674d

Refresh to latest

ankikuma merged commit 32374db into elastic:main Aug 28, 2024

samxbr mentioned this pull request Nov 18, 2025

Reindex metrics can be inaccurate with slicing enabled #138258

Closed

Conversation

ankikuma commented Aug 13, 2024

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma commented Aug 13, 2024

Uh oh!

ankikuma commented Aug 13, 2024

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Aug 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma commented Aug 14, 2024

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma commented Aug 20, 2024

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankikuma Aug 14, 2024 •

edited

Loading

ankikuma Aug 15, 2024 •

edited

Loading

ankikuma Aug 21, 2024 •

edited

Loading