Skip to content

Conversation

@noble-varghese
Copy link
Contributor

@noble-varghese noble-varghese commented Oct 8, 2024

Adding the functionality to retry the log sending on getting a request timeout (408) from the HTTP sink. Since the default behavior is to drop the events during such events, this would affect the log propagation to the sinks.

solves: #10870

@bits-bot
Copy link

bits-bot commented Oct 8, 2024

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Oct 8, 2024
@noble-varghese noble-varghese changed the title enhancement(sinks): Retrying the HTTP sink in case of 404s and request timeouts enhancement(http sink): Retrying the HTTP sink in case of 404s and request timeouts Oct 8, 2024
@noble-varghese noble-varghese marked this pull request as ready for review October 8, 2024 20:51
@noble-varghese noble-varghese requested a review from a team as a code owner October 8, 2024 20:51
@pront
Copy link
Member

pront commented Oct 8, 2024

Hi @noble-varghese, thank you for the contribution. I think retrying for these codes is reasonable, I will take a closer look tomorrow.

Note: Need to update the docs as well. See https://vector.dev/docs/reference/configuration/sinks/http/#retry-policy.

@noble-varghese noble-varghese requested review from a team as code owners October 9, 2024 03:30
@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Oct 9, 2024
@noble-varghese
Copy link
Contributor Author

@pront Thanks for pointing it out ! I have updated the docs page and added unit tests on the http utils as well :)

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @noble-varghese, I noticed the E2E test http_to_http failed. This change actually is a breaking one. Since it is hard to evaluate the risk, it is better to err on the side of caution. After some discussion with the team, I created #21469 to layout a solution for this.

@pront
Copy link
Member

pront commented Oct 9, 2024

To unblock this PR, perhaps we can add 408 to the defaults? Still a breaking change, but seems desirable and low risk. cc @jszwedko

@pront
Copy link
Member

pront commented Oct 30, 2024

I tried to push directly to the fork but I don't have write permissions.

@noble-varghese
Copy link
Contributor Author

I tried to push directly to the fork but I don't have write permissions.

I will push those changes. Thanks !

@noble-varghese
Copy link
Contributor Author

@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
@jszwedko
Copy link
Member

@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?

You could try cargo test --no-default-features --features sinks-http. This and other tips are mentioned here: https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md#tips-and-tricks

@pront
Copy link
Member

pront commented Oct 31, 2024

@pront Just curious to know, how do you run these tests on the local. It was taking me about 30-40mins for it to run. Is there a faster way ?

You could try cargo test --no-default-features --features sinks-http. This and other tips are mentioned here: https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md#tips-and-tricks

Thanks Jesse, very neat.

Also, I always run the following before pushing to the remote branch:

  • cargo install -f --path vdev #cd to the repo root (only need this once)
  • cargo vdev fmt
  • cargo vdev check rust --clippy

@pront pront enabled auto-merge November 5, 2024 16:14
@pront pront added this pull request to the merge queue Nov 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 6, 2024
@pront pront added this pull request to the merge queue Nov 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 6, 2024
@pront pront added this pull request to the merge queue Nov 6, 2024
Merged via the queue into vectordotdev:master with commit e3cd143 Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants