fix(postgresql): Make pg error clear with table name#5733
fix(postgresql): Make pg error clear with table name#5733yevgenypats wants to merge 2 commits intomainfrom
Conversation
There was a problem hiding this comment.
Thanks for the PR @yevgenypats can you explain how getting the error from the pgError instance doesn't work? We already log it in
Also do we want to stop the sync on a batch failure? Before we kept syncing so at least you got some data. Stopping and reporting the offending table does makes sense as users can chose to skip the table and re-sync.
| // maybe in the future we can provide a flag to switch batching mechanism. | ||
| func (c *Client) findErrorInBatch(ctx context.Context, items []*batchItem) error { | ||
| for _, item := range items { | ||
| _, err := c.conn.Exec(ctx, item.sql, item.arguments...) |
There was a problem hiding this comment.
Are queries guaranteed to behave the same the second time we execute them? I'm concerned this can generated unrelated errors
| } | ||
| atomic.AddUint64(&c.metrics.Errors, 1) | ||
| c.logger.Error().Err(pgErr).Str("table", pgErr.TableName).Msg("failed to execute batch with pgerror") | ||
| return fmt.Errorf("failed to execute batch and was unable to pinpoint table: %w", batchErr) |
There was a problem hiding this comment.
Doesn't this mean everything was successfully inserted? Maybe log it and not return an error
|
@erezrokah Good catch! Seems you are correct. apparently there are two types of errors. One that is happening on Postgres side and for that we should usually have the In the meanwhile i'll close this one and open PR to update pg to Follow-up PR here: #5757 |
This is a follow-up to #5733 (see discussion there). and needed so we can incorporate this nice bit: jackc/pgx#1441 The jackc/pgx#1441 is not a pre-requirement for this to go through. even better we should prob ship this first and see that `v5` works for us and for our users and if not we can do the same fix for `v4` (we will just have to fork it as I don't think `v4` accepts anymore contributions)
Sometimes we can have bugs in our type systems and/or destination which we need to find and fix as fast as we can. In PostgreSQL case we are using batch which makes it impossible to know which table and/or query caused it understand what is the bug. This finds the violating query/table before exiting and prints it so users can report us the issue.