-
Notifications
You must be signed in to change notification settings - Fork 4.1k
sql: COPY seemingly spends a lot of time in the optimizer #52071
Copy link
Copy link
Closed
Labels
A-sql-optimizerSQL logical planning and optimizations.SQL logical planning and optimizations.C-investigationFurther steps needed to qualify. C-label will change.Further steps needed to qualify. C-label will change.T-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Description
Trying out the osm2pgsql tool, which utilizes copies. Thus far I have found it spends most of its time in the optimizer trying to copy over ~10M rows (~4 of these in parallel). Here is a CPU profile over 10s during the COPY period:
Repro steps:
- Checkout osm2pgsql-dev/osm2pgsql@master...otan-cockroach:crdb_branch, build using
cd build && cmake .. && make && cd .. - Run cockroach single node built with sql: allow escape of any character for COPY #52068 included.
- Download the dataset: https://download.geofabrik.de/europe/iceland-latest.osm.pbf
- Run the tool
./build/osm2pgsql -U root -d defaultdb -H localhost -P 26257 --number-processes 1 -C 20480 ~/iceland-latest.osm.pbf - Profile during this step:
Processing: Node(6965k 6965.0k/s) Way(356k 14.24k/s) Relation(5290 661.2/s)
PostgreSQL is able to do this in ~7s, we take ~49s. (maybe fodder for https://accidentallyquadratic.tumblr.com/)
Jira issue: CRDB-3984
Epic CRDB-14477
Epic CRDB-14916
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-sql-optimizerSQL logical planning and optimizations.SQL logical planning and optimizations.C-investigationFurther steps needed to qualify. C-label will change.Further steps needed to qualify. C-label will change.T-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Type
Projects
Status
Done
