Drop `allow_none` for finding repository by mvdbeek · Pull Request #1358 · galaxyproject/planemo

mvdbeek · 2023-03-08T12:10:48Z

This was only used in the RealizedRepository class where it is being caught anyway. Should make debugging mismatched in name/owner of .shed.yml much easier. Probably fixes #1353 ?

This was only used in the RealizedRepository class where it is being caught anyway. Should make debugging mismatched in name/owner of .shed.yml much easier.

planemo/shed/__init__.py

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

nsoranzo · 2023-03-08T14:23:33Z

Failing tests seem unrelated.

mvdbeek · 2023-03-08T14:26:35Z

Yeah, probably a change in the dev branch that's not playing nice with sqlite:

galaxy.workflow.run DEBUG 2023-03-08 13:25:16,334 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Workflow step 3 of invocation 1 invoked (1.195 ms)
galaxy.managers.collections DEBUG 2023-03-08 13:25:16,381 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Created collection with 1 elements
galaxy.tools.actions.model_operations INFO 2023-03-08 13:25:16,383 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Calling produce_outputs, tool is <galaxy.tools.ApplyRulesTool object at 0x7f025f54e650>
galaxy.tools.execute DEBUG 2023-03-08 13:25:16,383 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Tool __APPLY_RULES__ created job None (14.443 ms)
galaxy.web_stack.handlers INFO 2023-03-08 13:25:16,413 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] (Job[id=3,tool_id=__APPLY_RULES__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.tools.execute DEBUG 2023-03-08 13:25:16,415 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Created 1 job(s) for tool __APPLY_RULES__ request (78.566 ms)
galaxy.workflow.run DEBUG 2023-03-08 13:25:16,420 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Workflow step 2 of invocation 1 invoked (86.011 ms)
galaxy.tools.actions INFO 2023-03-08 13:25:16,436 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Handled output named out_file1 for tool cat_list (1.032 ms)

galaxy.tools ERROR 2023-03-08 13:25:21,447 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Exception caught while attempting to execute tool with id 'cat_list':
Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/__init__.py", line 2069, in handle_single_execution
    skip=skip,
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/__init__.py", line 2155, in execute
    self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/actions/__init__.py", line 638, in execute
    history.add_pending_items(set_output_hid=set_output_hid)
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2900, in add_pending_items
    object_session(self), self._pending_additions, set_hid=set_output_hid, quota=False, flush=False
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2992, in add_datasets
    self.add_dataset(dataset, parent_id=parent_id, genome_build=genome_build, set_hid=set_hid, quota=quota)
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2960, in add_dataset
    dataset.hid = self._next_hid()
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2928, in _next_hid
    conn.execute(update_stmt)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 335, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE history SET update_time=?, hid_counter=(history.hid_counter + ?) WHERE history.id = ?]
[parameters: ('2023-03-08 13:25:16.441239', 1, 1)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
galaxy.tools.execute WARNING 2023-03-08 13:25:21,452 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] There was a failure executing a job for tool [cat_list] - Error executing tool with id 'cat_list': (sqlite3.OperationalError) database is locked
[SQL: UPDATE history SET update_time=?, hid_counter=(history.hid_counter + ?) WHERE history.id = ?]
[parameters: ('2023-03-08 13:25:16.441239', 1, 1)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
galaxy.jobs.handler DEBUG 2023-03-08 13:25:22,113 [pN:main.1,p:20282,tN:JobHandlerQueue.monitor_thread] Grabbing Job failed (serialization failures are ok): (sqlite3.OperationalError) database is locked
[SQL: UPDATE job SET update_time=?, handler=? WHERE job.id IN (SELECT job.id 
FROM job 
WHERE job.handler IN (?) AND job.state = ? ORDER BY job.id)]
[parameters: ('2023-03-08 13:25:17.105546', 'main.1', '_default_', <states.NEW: 'new'>)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

kombu.mixins WARNING 2023-03-08 13:25:22,402 [pN:main.1,p:20282,tN:Thread-1] Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/mixins.py", line 171, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/mixins.py", line 193, in consume
    conn.drain_events(timeout=safety_interval)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/connection.py", line 316, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 971, in drain_events
    get(self._deliver, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 1009, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 736, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 400, in _poll
    return cycle.get(callback)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 403, in _get_and_deliver
    message = self._get(queue)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 167, in _get
    self.session.execute('BEGIN IMMEDIATE TRANSACTION')
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1714, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 335, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: BEGIN IMMEDIATE TRANSACTION]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

any idea on that @jdavcs ?

jdavcs · 2023-03-08T17:58:57Z

Yeah, probably a change in the dev branch that's not playing nice with sqlite:

It's very possible that it may be related/caused by recent changes to db transaction handling. I'll investigate.

jdavcs · 2023-06-08T12:37:27Z

The sqlite database locking error has been fixed in galaxyproject/galaxy#16208.

Drop allow_none for finding repository

80865bf

This was only used in the RealizedRepository class where it is being caught anyway. Should make debugging mismatched in name/owner of .shed.yml much easier.

nsoranzo approved these changes Mar 8, 2023

View reviewed changes

planemo/shed/__init__.py Outdated Show resolved Hide resolved

Use f-string for exception message

622aa4a

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

nsoranzo enabled auto-merge March 8, 2023 12:34

mvdbeek disabled auto-merge March 8, 2023 14:27

mvdbeek merged commit 0bf7c24 into galaxyproject:master Mar 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop `allow_none` for finding repository#1358

Drop `allow_none` for finding repository#1358
mvdbeek merged 2 commits intogalaxyproject:masterfrom
mvdbeek:fail_with_better_error

mvdbeek commented Mar 8, 2023

Uh oh!

Uh oh!

nsoranzo commented Mar 8, 2023

Uh oh!

mvdbeek commented Mar 8, 2023

Uh oh!

jdavcs commented Mar 8, 2023

Uh oh!

jdavcs commented Jun 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mvdbeek commented Mar 8, 2023

Uh oh!

Uh oh!

nsoranzo commented Mar 8, 2023

Uh oh!

mvdbeek commented Mar 8, 2023

Uh oh!

jdavcs commented Mar 8, 2023

Uh oh!

jdavcs commented Jun 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants