NAME
    Data::Histogram::Shared - shared-memory HdrHistogram for Linux

SYNOPSIS
        use Data::Histogram::Shared;

        # track values in [1, 3_600_000_000] with 3 significant figures, anonymous
        my $h = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3);

        $h->record(120);              # record the value 120 once
        $h->record(2500, 4);          # record the value 2500 four times

        $h->value_at_percentile(50);  # median (50th percentile)
        $h->percentile(99);           # 99th percentile (alias)
        $h->min; $h->max; $h->mean;   # min / max / arithmetic mean
        $h->total_count;              # number of recorded values

        # bulk record in a single lock acquisition
        $h->record_many([ 100, 250, 250, 900, 1500 ]);

        # merge another histogram of identical geometry (cellwise add)
        my $other = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3);
        $other->record_many([ 50, 75, 80 ]);
        $h->merge($other);

        # share across processes via a backing file
        my $shared = Data::Histogram::Shared->new("/tmp/latency.hdr", 1, 3_600_000_000, 3);

        # freeze and ship: query it read-only (lock-free) on other machines
        $shared->freeze;
        my $ro = Data::Histogram::Shared->new_readonly("/tmp/latency.hdr");
        $ro->value_at_percentile(99);

DESCRIPTION
    A High Dynamic Range histogram (HdrHistogram) in shared memory: a
    fixed-size structure for recording integer values over a wide range and
    querying percentiles, minimum, maximum and mean. It is suited to latency
    measurements that span many orders of magnitude (microseconds to seconds).

    You construct the histogram with the "lowest" and "highest" values it must
    track and the number of "sig_figs" (significant figures) of precision you
    require. Values are bucketed logarithmically -- one bucket per power of
    two of magnitude -- and linearly within each bucket. Values in the same
    sub-bucket are equivalent. Bucket widths are at most "1 / 10**sig_figs" of
    the value, except where the minimum width "2**floor(log2($lowest))"
    dominates. Memory is proportional to the dynamic range and the precision,
    not to the number of values recorded: you can record billions of samples
    into a fixed array of counters.

    The histogram stores integer values only. To record floating-point
    quantities, scale them to integers yourself before recording -- for
    example, record a latency in microseconds (or nanoseconds) rather than
    fractional seconds, and divide the reported percentiles back down.

    Because the counts array lives in a shared mapping, several processes
    share one histogram: any process that opens the same backing file,
    inherits the anonymous mapping across "fork", or reopens a passed memfd,
    sees the others' recordings and contributes its own. A write-preferring
    futex rwlock with dead-process recovery guards mutation, so many processes
    may "record" and query concurrently. Two histograms of identical geometry
    (same "lowest", "highest" and "sig_figs") can be combined with "merge"
    (cellwise add of their counts arrays), which yields a histogram whose
    distribution is the union of the two input streams. Linux-only. Requires
    64-bit Perl.

METHODS
  Constructors
        my $h = Data::Histogram::Shared->new($path, $lowest, $highest, $sig_figs, $mode);
        my $h = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3);   # defaults
        my $h = Data::Histogram::Shared->new_memfd($name, $lowest, $highest, $sig_figs);
        my $h = Data::Histogram::Shared->new_from_fd($fd);
        my $ro = Data::Histogram::Shared->new_readonly($path);   # frozen file, read-only

    $path is the backing file ("undef" or omitted for an anonymous mapping).
    $lowest sets the minimum resolution and must be ">= 1" (default 1).
    $highest is the highest value that can be tracked and must be ">= 2 *
    $lowest" (default "3_600_000_000", i.e. one hour in microseconds).
    $sig_figs is the number of significant figures of precision and must be in
    the range 1..5 (default 3). "new" and "new_memfd" croak if any argument is
    out of range.

    $mode sets the permission bits used when "new" creates the backing file
    (applied exactly via "fchmod", so umask does not narrow it); the default
    is 0600, owner-only. Pass e.g. 0660 to opt in to sharing the file with
    other users in the group. It applies only at creation: reopening an
    existing file does not change its permissions, and the argument is ignored
    for anonymous histograms ($path undef or omitted). "new_memfd" and
    "new_from_fd" do not take a mode.

    $lowest additionally must satisfy "floor(log2($lowest)) + ceil(log2(2 *
    10**$sig_figs)) - 1 <= 61" (a histogram-geometry limit); in practice this
    only rejects a very large $lowest -- above roughly "2**51" at the default
    "sig_figs = 3". Such a histogram croaks at construction.

    From these the histogram derives its bucket geometry: a unit magnitude of
    "floor(log2($lowest))", "2 * 10**sig_figs" sub-buckets per power of two
    (rounded up to a power of two), and as many buckets as are needed to cover
    $highest. When reopening an existing file or memfd, the stored geometry
    wins and the caller's $lowest/$highest/$sig_figs arguments are ignored.
    "new_memfd" creates a Linux memfd (transferable via its "memfd"
    descriptor); "new_from_fd" reopens one in another process. The descriptor
    you pass is duplicated ("F_DUPFD_CLOEXEC"), so it stays yours to close and
    closing it does not disturb the handle. "new_readonly" opens a frozen file
    read-only for lock-free querying (see "FROZEN (READ-ONLY) MODE").

    "new" and "new_memfd" create a new segment sparse: pages are allocated as
    they are first written, and once the filesystem is full, a "record" into a
    page not yet allocated dies with "SIGBUS". Set
    "DATA_HISTOGRAM_SHARED_SPARSE=0" to reserve the whole segment at creation,
    so a full filesystem makes them croak instead; on tmpfs and memfd that
    commits the segment's memory at once, and a memory cgroup too small for it
    gets an OOM kill rather than a croak.

  Recording values
        my $count = $h->record($value);          # record once;  returns new total_count
        my $count = $h->record($value, $n);      # record $n times; returns new total_count
        my $added = $h->record_many(\@values);    # record each once; returns how many
        $h->reset;                               # clear every count back to empty

    "record" adds $n (default 1) occurrences of the integer $value to the
    histogram and returns the new "total_count". $value must be in the range
    "0 .. $highest": a negative value croaks, and a value above $highest
    croaks ("value ... exceeds highest_trackable_value"). Values strictly
    below $lowest are recorded at the minimum bucket width. $n is an unsigned
    integer.

    "record_many" takes an array reference and records each element once under
    a single write lock, returning the number recorded. Every element is
    range-checked before the lock is taken, so an out-of-range element croaks
    without recording any of the batch (and without holding the lock).

    Percentiles return the highest equivalent value of the selected bucket.
    They can exceed the largest recorded value or the configured $highest when
    either lies inside a bucket.

  Querying
        my $v = $h->value_at_percentile($p);     # value at the $p-th percentile (0..100)
        my $v = $h->percentile($p);              # alias for value_at_percentile
        my $c = $h->count_at_value($value);      # count recorded in $value's bucket
        my $lo = $h->min;                        # lowest recorded value (0 if empty)
        my $hi = $h->max;                        # highest recorded value (0 if empty)
        my $mu = $h->mean;                       # arithmetic mean (0 if empty)
        my $n  = $h->total_count;                # number of recorded values
        my $n  = $h->count;                      # alias for total_count

    "value_at_percentile" returns the value below which $p percent of the
    recorded values lie: the highest equivalent value of the bucket at that
    percentile, so it never underestimates. $p is a percentile in the range
    0..100 (not a 0..1 quantile). "percentile" is a short alias. An empty
    histogram returns 0 for any percentile.

    "count_at_value" returns the number of values recorded in the bucket that
    $value falls into (so values equivalent to $value are counted together).
    $value is range-checked like "record": a negative value, or a value above
    $highest, croaks. "min" and "max" are the exact lowest and highest values
    ever recorded (each 0 when nothing has been recorded). "mean" is the
    arithmetic mean, computed from each bucket's median-equivalent value.
    "total_count" (aliased "count") is the number of recorded values -- the
    sum of all the per-bucket counts.

  Merging
        $h->merge($other);

    Folds $other's counts array into $h by cellwise addition, so $h then
    represents the combined distribution of both histograms; $h's
    "total_count", "min" and "max" are updated to span both. Both histograms
    must have identical geometry -- the same "lowest", "highest" and
    "sig_figs" ("merge" croaks on a mismatch). $other is read under its own
    lock into a private snapshot first, so merging is deadlock-free even if
    two processes merge each other concurrently; $other is not modified.
    Counts that would overflow a 64-bit cell saturate at the maximum value.

  Introspection and lifecycle
        $h->lowest; $h->highest; $h->sig_figs; $h->counts_len; $h->stats;
        $h->path; $h->memfd; $h->sync; $h->unlink;   # or Class->unlink($path)

    "lowest", "highest" and "sig_figs" return the configured geometry;
    "counts_len" is the number of 64-bit counter cells. "sync" flushes the
    mapping to its backing store (a no-op for anonymous and memfd histograms,
    which have none); "unlink" removes the backing file (also callable as
    "Class->unlink($path)"); "path" returns the backing path ("undef" for
    anonymous, memfd, or fd-reopened histograms) and "memfd" the backing
    descriptor -- the memfd of a "new_memfd" histogram or the dup'd fd of a
    "new_from_fd" histogram, and -1 for file-backed or anonymous histograms.

STATS
    stats() returns a hashref describing the histogram:

    *   "lowest" -- the lowest trackable value.

    *   "highest" -- the highest trackable value.

    *   "sig_figs" -- the significant figures of precision.

    *   "count" -- the number of recorded values ("total_count").

    *   "min" -- the lowest recorded value (0 if empty).

    *   "max" -- the highest recorded value (0 if empty).

    *   "mean" -- the arithmetic mean of the recorded values (0.0 if empty).

    *   "counts_len" -- the number of 64-bit counter cells.

    *   "bucket_count" -- the number of logarithmic buckets.

    *   "sub_bucket_count" -- the number of linear sub-buckets per bucket.

    *   "ops" -- running count of mutating operations ("record",
        "record_many", "merge", "reset").

    *   "mmap_size" -- bytes of the shared mapping.

    *   "frozen" -- 1 if the histogram has been sealed by "freeze"
        (immutable), else 0.

    *   "readonly" -- 1 if this handle is a read-only view (from
        "new_readonly", or the handle that called "freeze"), else 0.

SHARING ACROSS PROCESSES
    Processes sharing a mutable histogram must be in the same PID namespace;
    lock recovery uses local process IDs.

    The histogram lives in a shared mapping, shared the same three ways as the
    rest of the family: a backing file (every process calls "new($path, ...)"
    on the same path with matching geometry), an anonymous mapping inherited
    across "fork", or a memfd whose descriptor is passed to an unrelated
    process (over a UNIX socket via "SCM_RIGHTS", or via "/proc/$pid/fd/$n")
    and reopened with new_from_fd($fd). Because the mapping is shared, every
    process records into and queries the same counts array, so the
    distribution reflects the combined stream all of them have recorded.

        # producer and consumer share one histogram with no coordination
        my $h = Data::Histogram::Shared->new(undef, 1, 1_000_000, 3);   # before fork
        unless (fork) { $h->record(500) for 1 .. 10; exit }
        wait;
        print $h->value_at_percentile(50), "\n";   # the child's recordings

FROZEN (READ-ONLY) MODE
    A file-backed histogram can be frozen and then shipped to other machines,
    where consumers open it read-only and query it with no locking at all.

        # producer: build, freeze, ship the file
        my $h = Data::Histogram::Shared->new("/tmp/latency.hdr", 1, 3_600_000_000, 3);
        $h->record_many(\@samples);
        $h->freeze;                  # seal: now immutable, and $h itself is read-only
        # ... copy /tmp/latency.hdr to another host ...

        # consumer (any process, same architecture): read-only, lock-free
        my $ro = Data::Histogram::Shared->new_readonly("/tmp/latency.hdr");
        $ro->value_at_percentile(99) for @queries;

    "freeze" takes the write lock, marks the histogram permanently immutable
    (there is no unfreeze -- rebuild the file to change it), and flushes the
    seal to disk. A frozen histogram rejects every mutator ("record",
    "record_many", "merge", "reset") with a croak, and a read-write reopen
    ("new($path, ...)") of a sealed file is refused -- so a shipped artifact
    can never be silently mutated out from under its readers.

    new_readonly($path) maps the file "O_RDONLY" / "PROT_READ" and requires it
    to be frozen (it croaks on a file that was never "freeze"d). Because a
    sealed histogram's counts and geometry are immutable,
    "value_at_percentile", "percentile", "count_at_value", "min", "max",
    "mean", "total_count", "count" and "stats" read them directly, taking no
    reader lock -- the mapping is never written, so a read-only view works
    from a read-only file descriptor or a read-only filesystem, and any number
    of processes can share one "PROT_READ" mapping. "frozen" and "readonly"
    report the two states.

    Portability. The on-disk format is native binary (native-endian 64-bit
    words), so a frozen file may be copied only between machines of the same
    architecture; a wrong-endian file is rejected at open by the magic check.
    Copy the file to each consumer -- do not share one file over a network
    filesystem: the lock is a Linux futex (process-local to one kernel), and
    the "no live writer" contract assumes a static copy. Linux-only; 64-bit
    Perl.

SECURITY
    Backing files are created with mode 0600 (owner-only) by default, so only
    the creating user can open and attach them. To share a backing file across
    users, pass an explicit octal file mode such as 0660 as the last argument
    to "new"; the mode is applied when the file is created, and when a file
    left behind by an interrupted create is re-initialized (see "CRASH
    SAFETY"); a file already in use keeps its own permissions. The file is
    opened with "O_NOFOLLOW", so a symlink planted at the path is refused, and
    created with "O_EXCL"; the on-disk header is validated when the file is
    attached. Any process you grant write access to a shared mapping is
    trusted not to corrupt its contents while other processes are using it.

CRASH SAFETY
    Mutation is guarded by a futex-based write-preferring rwlock with
    PID-encoded ownership; if a holder dies, the next contender detects the
    dead owner and recovers. Recovery releases locks; it does not roll back an
    interrupted mutation. A killed "record", "merge" or "reset" can leave
    partially applied counts or extrema, skewing percentiles and the mean
    until the next "reset". Limitation: PID reuse is not detected (very
    unlikely in practice).

    Reader-slot exhaustion (slotless readers): dead-process recovery
    attributes a crashed lock holder's contribution through its reader-slot.
    The slot table holds 1024 entries (one per handle that takes a lock). If
    more than that many handles share one mapping at once, a reader that
    cannot claim a slot proceeds "slotless" -- it still takes the read lock
    but leaves no per-process record. If such a slotless reader is then killed
    while holding the read lock, its share of the lock cannot be attributed to
    a dead process, so writer recovery cannot reclaim it and writers may block
    until the mapping is recreated. Reaching this needs more than 1024 handles
    on one mapping plus a crash in the brief read-lock window; the
    dead-process slot reclaim keeps the table from filling with stale entries,
    so in practice it is very unlikely.

    An interrupted create is recovered too. A creator killed after the backing
    file is sized but before its header is committed leaves a full-size,
    all-zero file. "new" re-initializes such a file automatically, but only
    when it is exactly the size the requested geometry needs, is owned by your
    effective uid, and is still entirely zero -- a file holding data is never
    re-initialized. If the creator got as far as writing part of the header,
    the file cannot be told apart from a corrupt one and "new" croaks with
    "incomplete histogram file left by an interrupted create; remove it and
    retry". A file left behind by an interrupted create never held data, so
    removing it is safe -- but a file whose header was corrupted after the
    fact reaches the same croak, so confirm it is an abandoned create before
    deleting anything you care about.

SEE ALSO
    Data::CountMinSketch::Shared, Data::HyperLogLog::Shared,
    Data::BloomFilter::Shared, Data::Intern::Shared, Data::SortedSet::Shared,
    Data::SpatialHash::Shared, and the rest of the "Data::*::Shared" family.

AUTHOR
    vividsnow

LICENSE
    This is free software; you can redistribute it and/or modify it under the
    same terms as Perl itself.