Skip to content

Dynamically load tasks; sync target improvements; StopIteration fix#963

Merged
rgalanakis merged 10 commits intomainfrom
dynamic-tasks
Jun 8, 2025
Merged

Dynamically load tasks; sync target improvements; StopIteration fix#963
rgalanakis merged 10 commits intomainfrom
dynamic-tasks

Conversation

@rgalanakis
Copy link
Contributor

@rgalanakis rgalanakis commented Jun 7, 2025

Improve sync target stat tracking

Track full call duration, not just remote calls,
to find slow queries.

Better cursor page name.

Use better summary names and calculations.


Support disabling sync targets


Expose str2inthash method and converter

So that replicators can convert a column into a hash value
for partitioning, add a converter.

This is useful when a 'parent' (like a calendar) needs to
store the hash used for children (events),
so clients can easily query.


Dynamically load tasks

Allow other modules to define tasks.


Replace httpx and down to fix StopIteration bugs

The StopIteration problem with down and httpx turned
into a giant can of worms that I wasn't able to fix.
There was no way to tell the difference between redirects and 304s,
since we'd get a StopIteration for both.

To fix this, I decided to rip out Down, since it wasn't providing
useful abstraction over anything.
I also needed to rip out HTTPX, since its streaming implementation
was busted and wasn't easy to patch or replace.

Instead, we use the more mature, and in my opinion clearer,
HTTP gem. We wrap it more thoroughly with chunked_download.

This required some small changes to Webhookdb::Http to make
it more flexible for the HTTP responses (not just HTTParty).

It included a much more annoying change to handle the content type
and charset detection fixes we made on top of HTTPX and Down,
to handle the bad HTTP servers we deal with. These are small and
subtle things, so it's good they were tested!

Unfortunately the Webhook HTTP lib wrapper makes too many
assumptions, and we can't use it to unit test these charset changes.
So we use thin to start a webserver we can use
for better http testing.

Overall I think this code is much simpler and will be easier
to fix and modify. Look at all the exception types we had to
handle, for example- we'll need to add more, almost for sure,
but it's much more clear.

rgalanakis added 10 commits June 5, 2025 10:34
Allow other modules to define tasks.
So that replicators can convert a column into a hash value
for partitioning, add a converter.

This is useful when a 'parent' (like a calendar) needs to
store the hash used for children (events),
so clients can easily query.
Track full call duration, not just remote calls,
to find slow queries.

Better cursor page name.

Use better summary names and calculations.
No idea why this just started.
The StopIteration problem with `down` and `httpx` turned
into a giant can of worms that I wasn't able to fix.
There was no way to tell the difference between redirects and 304s,
since we'd get a StopIteration for both.

To fix this, I decided to rip out Down, since it wasn't providing
useful abstraction over anything.
I also needed to rip out HTTPX, since its streaming implementation
was busted and wasn't easy to patch or replace.

Instead, we use the more mature, and in my opinion clearer,
`HTTP` gem. We wrap it more thoroughly with `chunked_download`.

This required some small changes to `Webhookdb::Http` to make
it more flexible for the `HTTP` responses (not just HTTParty).

It included a much more annoying change to handle the content type
and charset detection fixes we made on top of HTTPX and Down,
to handle the bad HTTP servers we deal with. These are small and
subtle things, so it's good they were tested!

Unfortunately the Webhook HTTP lib wrapper makes too many
assumptions, and we can't use it to unit test these charset changes.
So we use `thin` to start a webserver we can use
for better http testing.

Overall I think this code is much simpler and will be easier
to fix and modify. Look at all the exception types we had to
handle, for example- we'll need to add more, almost for sure,
but it's much more clear.
@codecov
Copy link

codecov bot commented Jun 8, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.07%. Comparing base (6092b84) to head (a27c87a).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #963      +/-   ##
==========================================
+ Coverage   97.06%   97.07%   +0.01%     
==========================================
  Files         487      487              
  Lines       30867    30994     +127     
==========================================
+ Hits        29960    30088     +128     
+ Misses        907      906       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rgalanakis rgalanakis merged commit b316a33 into main Jun 8, 2025
4 checks passed
@rgalanakis rgalanakis deleted the dynamic-tasks branch June 8, 2025 01:48
@rgalanakis rgalanakis changed the title Dynamically load tasks; sync target improvements Dynamically load tasks; sync target improvements; StopIteration fix Jun 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant