Skip to content

Increase Memory usage in Logging #2641

@frederikspang

Description

@frederikspang

Issue Description

We're trying out logging with Sentry - Firstly as a partial log-sink for Rails logs. Rails is not supported officially yet, but our initial attempts has shown somewhat highly increased memory in preprod environments. Is this something you're aware of, or something that we/I should dig into?

Reproduction Steps

Add "mirrored" logging to for config.logger in Rails initializer:

dual_logger = Class.new(Logger) do
    LOGGER_TO_SENTRY = {
      Logger::Severity::DEBUG => :debug,
      Logger::Severity::INFO => :info,
      Logger::Severity::WARN => :warning,
      Logger::Severity::ERROR => :error,
      Logger::Severity::FATAL => :fatal,
      Logger::Severity::UNKNOWN => :fatal
    }
    def initialize(input)
      super(input)

      @stdout_logger = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(input))
      @stdout_logger.level = Logger::DEBUG
    end

    def add(severity, message = nil, progname = nil)
      message ||= yield if block_given?

      return true if message.nil? || message.empty?

      @stdout_logger.add(severity, message, progname)
      Sentry.logger.log(LOGGER_TO_SENTRY[severity], message, parameters: []) if severity >= Logger::Severity::INFO
    end
  end

  Rails.logger = dual_logger.new($stdout)

This is not exactly production-ready, or pretty; But solves the issue of logging to default STDOUT logger, as well as Sentry.logger.

And while this DOES work - Memory increases massively (~40% in our test-env on our application until OOM in Kubernetes).

I have not determined if it's leaking, or if it's just fragmenting a lot more, and leaving huge empty pages etc.

Expected Behavior

Memory can increase, but should not increase this much for logging.

Actual Behavior

Memory increases until OOM

Ruby Version

3.4.3

SDK Version

5.24.0

Integration and Its Version

ruby, rails, sidekiq

Sentry Config

  require 'p2b/sentry/active_record_instantiation_subscriber'

  Sentry.init do |config|
    config.dsn = 'https://onprem.sentry/...'
    config.enabled_environments = %w[production staging]
    config.skip_rake_integration = true
    config.send_default_pii = true
    config.send_modules = false

    config.traces_sample_rate = Rails.env.production? ? 0.4 : 1
    config.profiles_sample_rate = Rails.env.production? ? 0.2 : 1
    if Rails.env.staging?
      config.debug = true
      config.enable_logs = true
      config.sdk_logger.level = Logger::Severity::DEBUG
    else
      config.sdk_logger.level = Logger::Severity::WARN
    end

    config.enabled_patches += [:sidekiq_cron, :excon, :redis, :graphql]
    # Hold that thought. Would be nice to have this as nested spans. How though.
    config.rails.tracing_subscribers << P2b::Sentry::ActiveRecordInstantiationSubscriber
    config.before_send = lambda do |event, hint|
      if hint[:exception].is_a?(ActiveRecord::RecordInvalid)
        begin
          event.set_tags(e.record.errors.to_hash.to_h { ["errors.#{_1}", [*_2].join(';')] })
        rescue
          # This is ok. Don't error out if we can't set tags.
        end
      end

      event
    end
  end

Metadata

Metadata

Assignees

No fields configured for issues without a type.

Projects

Status

Waiting for: Product Owner

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions