(this issue was first taken to a Google employee; they recommended this bug report be submitted here too)

Expected Behavior

QueryJob.result() always returns when a submitted query completes

Issue

At random and unexplainable frequencies, the QueryJob.result() runs indefinitely even after the submitted job_id is shown as completed in BigQuery's Job History.

Anecdotical observation which motivated this outreach:

• On 2024-04-29 17:16:20 UTC, job_id "A" was submitted to BigQuery.
• On 2024-05-06, we identified the hung python process, observed through its logs that it was waiting for job.result() to complete.
• In the BigQuery console, the Job History for"A" shows that the query was submitted on 2024-04-29 at 17:16:20 UTC and completed ~2 minutes later on 2024-04-29 at 17:18:26 UTC.
• The python process executing job.result() required manual termination over 6 days later on 2024-05-06.

We are implementing process-level timeouts to prevent this specific issue from running indefinitely, but this is a bandaid solution to a bug in the google-cloud-bigquery python package.

Environment details

• OS type and version: Debian Bookworm
• Python version: 3.11.4
• pip version: 24.0
• google-cloud-bigquery version: 3.21.0

Steps to reproduce

We are unable to deterministically reproduce this issue.

Code example

Simplified code example depicting the objects and methods used to

import uuid
from google.cloud import bigquery as bq

project_id = "dummy-project"
client = bq.Client(project=project_id)

job_id = str(uuid.uuid4())
sql = "select 1;"  # fake query

job_config = bq.QueryJobConfig(priority=bq.enums.QueryPriority.BATCH)
job = client.query(
    sql,
    job_config=job_config,
    api_method=bq.enums.QueryApiMethod.INSERT,  # this necessary to specify the job_id
    job_id=job_id
)
print(f"Submitting query: {job_id}")
result = job.result()  # run job and wait

# do other things with the result once completed
# sometimes this code is never reached, nor any error raised