Conversation
|
Hi, thanks for the PR, are there any performance improvements in benchmarks? |
|
I like the direction this is going, but I am concerned about the change in the type DB interface {
Query(ctx context.Context, sql string, optionsAndArgs ...interface{}) (pgx.Rows, error)
QueryRow(ctx context.Context, sql string, optionsAndArgs ...interface{}) pgx.Row
CopyFrom(ctx context.Context, tableName pgx.Identifier, columnNames []string, rowSrc pgx.CopyFromSource) (int64, error)
}An application or library using interface Would it be sufficient to use the default query exec mode without the ability to override? AFAIK, overriding the mode per query is extremely rare. I only included it on If it really needs to be controlled per call, perhaps a new optional interface could be created that |
|
@jackc I really need to override it to avoid the the prepare statement and the cache 😅 We are using pgx in timescale/promscale to ingest metrics and traces, we insert the data using COPY. Basically we have a table per metric, and our COPY operations are always of the same 3 columns. We want to get rid of the extra roundtrip per copy. In our use case the cache would be a lot of unnecessary memory since we are always sending (time, int64 and float64), which the Type map resolves correctly.
I'll try to give ☝️ a shot. |
2940679 to
9ecf9da
Compare
|
I'm going to try our use case with allowing setting the Exec mode only via I reverted the signature changes to This is the benchmark comparison. One thing, this is running against a DB listening on localhost, meaning that this does not account for the extra latency of having to go to connect to an external instance: |
9ecf9da to
7655e8b
Compare
7655e8b to
049170a
Compare
|
|
||
| func TestConnCopyWithKnownOIDQueryExecModes(t *testing.T) { | ||
|
|
||
| for _, mode := range pgxtest.KnownOIDQueryExecModes { |
There was a problem hiding this comment.
This tests is only for the pgxtest.KnownOIDQueryExecModes because we can't resolved between varchar and text, or between date and timestampz, with the exec modes that rely on the pgtype.Map
fd2d502 to
ed05619
Compare
CopyFrom had to create a prepared statement to get the OIDs of the data types that were going to be copied into the table. Every COPY operation required an extra round trips to retrieve the type information. There was no way to customize this behavior. By leveraging the QueryExecMode feature, like in `Conn.Query`, users can specify if they want to cache the prepared statements, execute them on every request (like the old behavior), or bypass the prepared statement relying on the pgtype.Map to get the type information. The `QueryExecMode` behave exactly like in `Conn.Query` in the way the data type OIDs are fetched, meaning that: - `QueryExecModeCacheStatement`: caches the statement. - `QueryExecModeCacheDescribe`: caches the statement and assumes they do not change. - `QueryExecModeDescribeExec`: gets the statement description on every execution. This is like to the old behavior of `CopyFrom`. - `QueryExecModeExec` and `QueryExecModeSimpleProtocol`: maintain the same behavior as before, which is the same as `QueryExecModeDescribeExec`. It will keep getting the statement description on every execution The `QueryExecMode` can only be set via `ConnConfig.DefaultQueryExecMode`, unlike `Conn.Query` there's no support for specifying the `QueryExecMode` via optional arguments in the function signature.
ed05619 to
f32632b
Compare
| case QueryExecModeExec, QueryExecModeSimpleProtocol: | ||
| // These modes don't support the binary format. Before the inclusion of the | ||
| // QueryExecModes, Conn.Prepare was called on every COPY operation to get | ||
| // the OIDs. These prepared statements were not cached. | ||
| // | ||
| // Since that's the same behavior provided by QueryExecModeDescribeExec, | ||
| // we'll default to that mode. | ||
| ct.mode = QueryExecModeDescribeExec | ||
| fallthrough |
There was a problem hiding this comment.
@jackc I updated the PR, I'd say this is the most contentious part. Is this ok? or do you want me to just call Prepare directly here?
|
LGTM! Thanks! |
Use DefaultQueryExecMode in CopyFrom
CopyFrom had to create a prepared statement to get the OIDs of the data
types that were going to be copied into the table. Every COPY operation
required an extra round trips to retrieve the type information. There
was no way to customize this behavior.
By leveraging the QueryExecMode feature, like in
Conn.Query, users canspecify if they want to cache the prepared statements, execute
them on every request (like the old behavior), or bypass the prepared
statement relying on the pgtype.Map to get the type information.
The
QueryExecModebehave exactly like inConn.Queryin the way thedata type OIDs are fetched, meaning that:
QueryExecModeCacheStatement: caches the statement.QueryExecModeCacheDescribe: caches the statement and assumes they donot change.
QueryExecModeDescribeExec: gets the statement description on everyexecution. This is like to the old behavior of
CopyFrom.QueryExecModeExecandQueryExecModeSimpleProtocol: maintain thesame behavior as before, which is the same as
QueryExecModeDescribeExec.It will keep getting the statement description on every execution
The
QueryExecModecan only be set viaConnConfig.DefaultQueryExecMode, unlikeConn.Querythere's nosupport for specifying the
QueryExecModevia optional argumentsin the function signature.