Skip to content

Raven Ruby + ActiveJob can kill BG processing #642

@mrhead

Description

@mrhead

We experienced a strange issue where Raven + ActiveJob consumed all of our Redis memory by creating jobs with enormous payload size. This can happen when a background job is scheduled with a model in arguments, but this model doesn't exist when the job is executed AND when SentryJob inherits from ActiveJob.

Environment

  • Rails 5.0.2
  • sentry-raven 2.3.1
  • Sidekiq 4.2.9

Steps to reproduce

# config/initializers/raven.rb
Raven.configure do |config|
  config.dsn = 'http://public:secret@example.com/project-id'

  config.async = lambda { |event|
    SentryJob.perform_later(event.to_hash)
  }
end
# app/jobs/sentry_job.rb
class SentryJob < ApplicationJob
  queue_as :default

  def perform(event)
    Raven.send_event(event)
  end
end
# app/jobs/test_job.rb
class TestJob < ApplicationJob
  queue_as :default

  def perform(user)
    # Do something later
  end
end

In rails console:

user = User.create!(name: "John Doe")
TestJob.perform_later(user)
user.destroy!

Start Sidekiq and watch the apocalypse:

bundle exec sidekiq

Example application with steps to reproduce

https://github.com/mrhead/sentry-raven-active-job

Expected result

TestJob should fail with ActiveJob::DeserializationError and SentryJob should successfully report this failure.

Actual result

TestJob fails with ActiveJob::DeserializationError, then SentryJob is scheduled but it fails with ActiveJob::DeserializationError too. This failure event schedules another SentryJob with the payload size double of the original one. And this goes over and over again.

My guess is that SentryJob fails, because it contains a global ID url in the payload and ActiveJob is trying to load that object even if it wasn't directly provided as an argument.

Workaround

Do not use ActiveJob for SentryJob.

Metadata

Metadata

Assignees

Labels

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions