Seed database with inline ActiveJob job adapter#34953
Seed database with inline ActiveJob job adapter#34953rafaelfranca merged 1 commit intorails:masterfrom
Conversation
There was a problem hiding this comment.
I think this doesn't do what you want. Rails.application.load_seed will return true.
1b04af9 to
cc8e86e
Compare
cc8e86e to
66cc0e7
Compare
Seed database with inline ActiveJob job adapter
|
This caused us a myriad of problems and I wish it were:
We have auto-publication code that pushes things out to Kafka or RabbitMQ when records are saved. We want to be able to use this in development because it's harmless (it just queues a job with It took spying on
That's a lot of surprising behavior and a lot of places to search to find it. If anything, setting |
|
As @rafaelfranca pointed out this has been fixed in 5-2-stable by only overriding Still feels like it might be confusing, so I'm hoping anyone surprised finds this as a reference later on. |
Seed database with inline ActiveJob job adapter rails/rails#34953)
Before rails#34953, when using the `:async` Active Job queue adapter, jobs enqueued in `db/seeds.rb`, such as Active Storage analysis jobs, would cause a hang (see rails#34939). Therefore, rails#34953 changed all jobs enqueued in `db/seeds.rb` to use the `:inline` queue adapter instead. (This behavior was later limited to only take effect when the `:async` adapter was configured, see rails#35905.) However, inline jobs in `db/seeds.rb` cleared `CurrentAttributes` values (see rails#37526). Therefore, rails#37568 changed the `:inline` adapter to wrap each job in its own thread, for isolation. However, wrapping a job in its own thread affects which database connection it uses. Thus inline jobs can no longer execute within the calling thread's database transaction, including seeing any uncommitted changes. Additionally, if the calling thread is not wrapped with the executor, the inline job thread (which is wrapped with the executor) can deadlock on the load interlock. And when testing (with `connection_pool.lock_thread = true`), the inline job thread can deadlock on one of the locks added by rails#28083. Therefore, this commit reverts the solutions of rails#34953 and rails#37568, and instead wraps evaluation of `db/seeds.rb` with the executor. This eliminates the original hang from rails#34939, which was also due to running multiple threads and not wrapping all of them with the executor. And, because nested calls to `executor.wrap` are ignored, any inline jobs in `db/seeds.rb` will not clear `CurrentAttributes` values. Alternative fix for rails#34939. Reverts rails#34953. Reverts rails#35905. Partially reverts rails#35896. Alternative fix for rails#37526. Reverts rails#37568. Fixes rails#40552.
Before rails#34953, when using the `:async` Active Job queue adapter, jobs enqueued in `db/seeds.rb`, such as Active Storage analysis jobs, would cause a hang (see rails#34939). Therefore, rails#34953 changed all jobs enqueued in `db/seeds.rb` to use the `:inline` queue adapter instead. (This behavior was later limited to only take effect when the `:async` adapter was configured, see rails#35905.) However, inline jobs in `db/seeds.rb` cleared `CurrentAttributes` values (see rails#37526). Therefore, rails#37568 changed the `:inline` adapter to wrap each job in its own thread, for isolation. However, wrapping a job in its own thread affects which database connection it uses. Thus inline jobs can no longer execute within the calling thread's database transaction, including seeing any uncommitted changes. Additionally, if the calling thread is not wrapped with the executor, the inline job thread (which is wrapped with the executor) can deadlock on the load interlock. And when testing (with `connection_pool.lock_thread = true`), the inline job thread can deadlock on one of the locks added by rails#28083. Therefore, this commit reverts the solutions of rails#34953 and rails#37568, and instead wraps evaluation of `db/seeds.rb` with the executor. This eliminates the original hang from rails#34939, which was also due to running multiple threads and not wrapping all of them with the executor. And, because nested calls to `executor.wrap` are ignored, any inline jobs in `db/seeds.rb` will not clear `CurrentAttributes` values. Alternative fix for rails#34939. Reverts rails#34953. Reverts rails#35905. Partially reverts rails#35896. Alternative fix for rails#37526. Reverts rails#37568. Fixes rails#40552.
Before rails#34953, when using the `:async` Active Job queue adapter, jobs enqueued in `db/seeds.rb`, such as Active Storage analysis jobs, would cause a hang (see rails#34939). Therefore, rails#34953 changed all jobs enqueued in `db/seeds.rb` to use the `:inline` queue adapter instead. (This behavior was later limited to only take effect when the `:async` adapter was configured, see rails#35905.) However, inline jobs in `db/seeds.rb` cleared `CurrentAttributes` values (see rails#37526). Therefore, rails#37568 changed the `:inline` adapter to wrap each job in its own thread, for isolation. However, wrapping a job in its own thread affects which database connection it uses. Thus inline jobs can no longer execute within the calling thread's database transaction, including seeing any uncommitted changes. Additionally, if the calling thread is not wrapped with the executor, the inline job thread (which is wrapped with the executor) can deadlock on the load interlock. And when testing (with `connection_pool.lock_thread = true`), the inline job thread can deadlock on one of the locks added by rails#28083. Therefore, this commit reverts the solutions of rails#34953 and rails#37568, and instead wraps evaluation of `db/seeds.rb` with the executor. This eliminates the original hang from rails#34939, which was also due to running multiple threads and not wrapping all of them with the executor. And, because nested calls to `executor.wrap` are ignored, any inline jobs in `db/seeds.rb` will not clear `CurrentAttributes` values. Alternative fix for rails#34939. Reverts rails#34953. Reverts rails#35905. Partially reverts rails#35896. Alternative fix for rails#37526. Reverts rails#37568. Fixes rails#40552. (cherry picked from commit 648da12)
Summary
Closes #34939.
It appears that
ActiveStorageenqueuesActiveStorage::AnalyzeJobs when attaching a file. This promptsActiveJobto enqueue jobs (via the async queue adapter by default in development) and can cause problems for one-off scripts such as database seeding.This may also be a symptom of a bigger problem with
ActiveStorage's queueing assumptions, or maybe the async job adapter. For now, I think temporarily setting the queue adapter to inline for all seeding in all environments is reasonable.