AsyncAdapter causes ActiveStorage attachment to hang in db:seed #34939
Description
TLDR trying to attach two or more files using ActiveStorage in a db:seed task hangs after the first attachment.
I suspect that ActiveJob::QueueAdapters::AsyncAdapter is not releasing the DB lock, which causes the Rake task to hang during the second call to attach.
Rails has config.active_job.queue_adapter = :async by default, so this issue happens in a fresh Rails 5.2.2 app. When setting config.active_job.queue_adapter = :inline, the task completes with no errors.
Might be related to #33500 .
Steps to reproduce
git clone https://github.com/lbrito1/active-storage-bug
cd active-storage-bug
bundle
rails db:migrate
rails db:seed # will hang here
Although I was able to reproduce this many times with the steps above on different machines, it becomes non-reproduceable after the first run, even after clearing tmp/* and storage/* and running db:drop (i.e. rails db:seed runs normally).
To reproduce more than once on the same machine, I had to resort to Docker:
docker run -it ruby
apt-get update && apt-get install -y nodejs
# git clone ...
Expected behavior
Rake exits.
Actual behavior
Rake hangs.
System configuration
Rails version: 5.2.2
Ruby version: 2.6.0
Solution
I think we can config.active_job.queue_adapter = :inline in development.rb. Non-development envs seem more troublesome to me, though: as far as I can tell they're probably also affected by this.
Traces
This is the trace I get if I interrupt the task after a while with ctrl+c:
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:221:in `wait_for'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:190:in `block in yield_shares'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:189:in `yield_shares'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:213:in `block in exec_query'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:212:in `exec_query'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:122:in `exec_insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:154:in `insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:185:in `_insert_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:732:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/counter_cache.rb:184:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/locking/optimistic.rb:70:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/attribute_methods/dirty.rb:140:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:346:in `block in _create_record'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:816:in `_run_create_callbacks'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:346:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/timestamp.rb:102:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:703:in `create_or_update'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:342:in `block in create_or_update'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:342:in `create_or_update'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:306:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/validations.rb:52:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:315:in `block in save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `block in transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:212:in `transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:315:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/suppressor.rb:48:in `save!'
/usr/local/bundle/gems/activestorage-5.2.2/app/models/active_storage/blob.rb:65:in `tap'
/usr/local/bundle/gems/activestorage-5.2.2/app/models/active_storage/blob.rb:65:in `create_after_upload!'
/usr/local/bundle/gems/activestorage-5.2.2/lib/active_storage/attached.rb:28:in `create_blob_from'
/usr/local/bundle/gems/activestorage-5.2.2/lib/active_storage/attached/one.rb:24:in `attach'
/active-storage-bug/db/seeds.rb:7:in `<main>'