Query cancellation with CTRL-C for Python client#3749
Merged
Mytherin merged 2 commits intoduckdb:masterfrom Jun 8, 2022
Merged
Query cancellation with CTRL-C for Python client#3749Mytherin merged 2 commits intoduckdb:masterfrom
Mytherin merged 2 commits intoduckdb:masterfrom
Conversation
Collaborator
|
Ups, I had worked on the same thing yday, but you beat me to the PR :). The GIL thing is indeed a pity, maybe we should run some numbers? |
Member
Author
|
Great idea, please do that :P |
pdet
approved these changes
Jun 8, 2022
Collaborator
pdet
left a comment
There was a problem hiding this comment.
I've did try out different data sizes and number of threads, and it seems there is no considerable impact.
import duckdb
from datetime import datetime
import threading
import queue as Queue
class DuckDBThreaded:
def __init__(self,duckdb_insert_thread_count,thread_function):
self.duckdb_insert_thread_count = duckdb_insert_thread_count
self.threads = []
self.thread_function = thread_function
def multithread_test(self,size,if_all_true=True):
duckdb_conn = duckdb.connect(check_same_thread=False)
queue = Queue.Queue()
return_value = False
for i in range(0,self.duckdb_insert_thread_count):
self.threads.append(threading.Thread(target=self.thread_function, args=(duckdb_conn,queue,size),name='duckdb_thread_'+str(i)))
for i in range(0,len(self.threads)):
self.threads[i].start()
if not if_all_true:
if queue.get():
return_value = True
else:
if i == 0 and queue.get():
return_value = True
elif queue.get() and return_value:
return_value = True
for i in range(0,len(self.threads)):
self.threads[i].join()
assert (return_value)
def fetchone_query(duckdb_conn, queue, size):
try:
a = duckdb_conn.execute("select sum(range) from range ("+str(size)+")").fetchone()
queue.put(True)
except:
queue.put(False)
def run_test(num_threads, size):
cur = datetime.now()
duck_threads = DuckDBThreaded(num_threads,fetchone_query)
duck_threads.multithread_test(size)
print (str(num_threads) + "T " + str(size) + " elements")
print (datetime.now() - cur)
run_test (10, 100000000)
run_test (20, 100000000)
run_test (100, 100000)
Collaborator
|
Great! Thanks :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to #3747, adding query cancellation to Python client. Should fix #3742. One caveat that we maybe have to check out still is the need for the GIL when checking for interrupts. Hopefully that does not lead to a performance degradation.