class Sidekiq::Metrics::Histogram

Implements space-efficient but statistically useful histogram storage. A precise time histogram stores every time. Instead we break times into a set of known buckets and increment counts of the associated time bucket. Even if we call the histogram a million times, we'll still only store 26 buckets. NB: needs to be thread-safe or resiliant to races.

To store this data, we use Redis' BITFIELD command to store unsigned 16-bit counters per bucket per klass per minute. It's unlikely that most people will be executing more than 1000 job/sec for a full minute of a specific type.

Constants

BUCKET_INTERVALS

This number represents the maximum milliseconds for this bucket. 20 means all job executions up to 20ms, e.g. if a job takes 280ms, it'll increment bucket. Note we can track job executions up to about 5.5 minutes. After that, it's assumed you're probably not too concerned with its performance.

FETCH
LABELS

Attributes

buckets[R]

Public Class Methods

new(klass) click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 59
def initialize(klass)
  @klass = klass
  @buckets = Array.new(BUCKET_INTERVALS.size) { Counter.new }
end

Public Instance Methods

each() { |value| ... } click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 50
def each
  buckets.each { |counter| yield counter.value }
end
fetch(conn, now = Time.now) click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 72
def fetch(conn, now = Time.now)
  window = now.utc.strftime("%d-%H:%-M")
  key = "#{@klass}-#{window}"
  conn.bitfield(key, *FETCH)
end
label(idx) click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 54
def label(idx)
  LABELS[idx]
end
persist(conn, now = Time.now) click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 78
def persist(conn, now = Time.now)
  buckets, @buckets = @buckets, []
  window = now.utc.strftime("%d-%H:%-M")
  key = "#{@klass}-#{window}"
  cmd = [key, "OVERFLOW", "SAT"]
  buckets.each_with_index do |counter, idx|
    val = counter.value
    cmd << "INCRBY" << "u16" << "##{idx}" << val.to_s if val > 0
  end

  conn.bitfield(*cmd) if cmd.size > 3
  conn.expire(key, 86400)
  key
end
record_time(ms) click to toggle source
# File lib/sidekiq/metrics/shared.rb, line 64
def record_time(ms)
  index_to_use = BUCKET_INTERVALS.each_index do |idx|
    break idx if ms < BUCKET_INTERVALS[idx]
  end

  @buckets[index_to_use].increment
end