We experienced a strange issue where Raven + ActiveJob consumed all of our Redis memory by creating jobs with enormous payload size. This can happen when a background job is scheduled with a model in arguments, but this model doesn't exist when the job is executed AND when SentryJob inherits from ActiveJob.
Environment
- Rails 5.0.2
- sentry-raven 2.3.1
- Sidekiq 4.2.9
Steps to reproduce
# config/initializers/raven.rb
Raven.configure do |config|
config.dsn = 'http://public:secret@example.com/project-id'
config.async = lambda { |event|
SentryJob.perform_later(event.to_hash)
}
end
# app/jobs/sentry_job.rb
class SentryJob < ApplicationJob
queue_as :default
def perform(event)
Raven.send_event(event)
end
end
# app/jobs/test_job.rb
class TestJob < ApplicationJob
queue_as :default
def perform(user)
# Do something later
end
end
In rails console:
user = User.create!(name: "John Doe")
TestJob.perform_later(user)
user.destroy!
Start Sidekiq and watch the apocalypse:
Example application with steps to reproduce
https://github.com/mrhead/sentry-raven-active-job
Expected result
TestJob should fail with ActiveJob::DeserializationError and SentryJob should successfully report this failure.
Actual result
TestJob fails with ActiveJob::DeserializationError, then SentryJob is scheduled but it fails with ActiveJob::DeserializationError too. This failure event schedules another SentryJob with the payload size double of the original one. And this goes over and over again.
My guess is that SentryJob fails, because it contains a global ID url in the payload and ActiveJob is trying to load that object even if it wasn't directly provided as an argument.
Workaround
Do not use ActiveJob for SentryJob.
We experienced a strange issue where Raven +
ActiveJobconsumed all of our Redis memory by creating jobs with enormous payload size. This can happen when a background job is scheduled with a model in arguments, but this model doesn't exist when the job is executed AND whenSentryJobinherits fromActiveJob.Environment
Steps to reproduce
In
rails console:Start Sidekiq and watch the apocalypse:
Example application with steps to reproduce
https://github.com/mrhead/sentry-raven-active-job
Expected result
TestJobshould fail withActiveJob::DeserializationErrorandSentryJobshould successfully report this failure.Actual result
TestJobfails withActiveJob::DeserializationError, thenSentryJobis scheduled but it fails withActiveJob::DeserializationErrortoo. This failure event schedules anotherSentryJobwith the payload size double of the original one. And this goes over and over again.My guess is that
SentryJobfails, because it contains a global ID url in the payload andActiveJobis trying to load that object even if it wasn't directly provided as an argument.Workaround
Do not use
ActiveJobforSentryJob.