com.clearspring.analytics.stream.cardinality

Class HyperLogLogPlus

java.lang.Object

com.clearspring.analytics.stream.cardinality.HyperLogLogPlus

All Implemented Interfaces:

ICardinality, Serializable

public class HyperLogLogPlus
extends Object
implements ICardinality, Serializable

Implementation of HyperLogLog++ described in

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf

Brief HyperLogLog++ Overview

Uses 64 bit hashing instead of 32 Has two representation modes: sparse and normal

'normal' is approximately the same as regular hyperloglog (still uses 64 bits)

'sparse' handles lower cardinality values with a highly accurate but poorly scaling strategy and leverages data compression to compete with 'normal' for as long as possible (sparse has the advantage on accuracy per unit of memory at low cardinality but quickly falls behind).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HyperLogLogPlus.Builder
protected static class	HyperLogLogPlus.HyperLogLogPlusMergeException

Constructor Summary

Constructors

Constructor and Description
HyperLogLogPlus(int p) This constructor disables the sparse set.
HyperLogLogPlus(int p, int sp) Basic constructor for creating a instance that supports sparse and normal representations.
HyperLogLogPlus(int p, int sp, List<byte[]> deltaByteSet) Constructor to support instances serialized with the legacy sparse encoding scheme.

Method Summary

Methods

Modifier and Type	Method and Description
void	addAll(HyperLogLogPlus other) Add all the elements of the other set to this set.
long	cardinality() Gather the cardinality estimate from this estimator.
byte[]	getBytes()
protected RegisterSet	getRegisterSet() exposed for testing
protected int[]	getSparseSet() exposed for testing
ICardinality	merge(ICardinality... estimators) Merge this HLL++ with a bunch of others! The power of minions! Most of the logic consists of case analysis about the state of this HLL++ and each one it wants to merge with.
protected void	mergeTempList() Script-esque function that handles preparing to and executing merging the sparse set and the temp list.
boolean	offer(Object o) Add data to estimator based on the mode it is in
boolean	offerHashed(int hashedInt) Offer the value as a hashed long value
boolean	offerHashed(long hashedLong) Offer the value as a hashed long value
int	sizeof()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperLogLogPlus

public HyperLogLogPlus(int p)

This constructor disables the sparse set. If the counter is likely to exceed the sparse set thresholds than using this constructor will help avoid the extra memory pressure created by maintaining the sparse set until that threshold is breached.

Parameters:

p - - the precision value for the normal set

HyperLogLogPlus

public HyperLogLogPlus(int p,
               int sp)

Basic constructor for creating a instance that supports sparse and normal representations. The values of p and sp define the precision of the Normal and Sparse set representations for the data structure. p must be a value between 4 and sp and sp must be less than 32.

Parameters:

p - - the precision value for the normal set

sp - - the precision value for the sparse set

HyperLogLogPlus

public HyperLogLogPlus(int p,
               int sp,
               List<byte[]> deltaByteSet)

Constructor to support instances serialized with the legacy sparse encoding scheme.

Parameters:

p - - the precision value for the normal set

sp - - the precision value for the sparse set

deltaByteSet - - a list of varint byte arrays encoded using a delta encoding scheme

Method Detail

offerHashed

public boolean offerHashed(long hashedLong)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedLong - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream

offerHashed

public boolean offerHashed(int hashedInt)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedInt - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream

offer

public boolean offer(Object o)

Add data to estimator based on the mode it is in

Specified by:

offer in interface ICardinality

Parameters:

o - stream element

Returns:

Will almost always return true for sparse mode because the additions are batched in

cardinality

public long cardinality()

Gather the cardinality estimate from this estimator.

Has two procedures based on current mode. 'Normal' mode works similar to HLL but has some new bias corrections. 'Sparse' mode is linear counting.

Specified by:

cardinality in interface ICardinality

Returns:

sizeof

public int sizeof()

Specified by:

sizeof in interface ICardinality

Returns:

size in bytes needed for serialization

getBytes

public byte[] getBytes()
                throws IOException

Specified by:

getBytes in interface ICardinality

Returns:

Throws:

IOException

mergeTempList

protected void mergeTempList()

Script-esque function that handles preparing to and executing merging the sparse set and the temp list.

Set up the delta encoding, sort the temp list, merge the lists, blow up the temp list.

Exposed for testing purposes

addAll

public void addAll(HyperLogLogPlus other)
            throws HyperLogLogPlus.HyperLogLogPlusMergeException

Add all the elements of the other set to this set.

If possible, the sparse mode is protected. A switch to the normal mode is triggered only if the resulting set exceed the threshold.

This operation does not imply a loss of precision.

Parameters:

other - A compatible Hyperloglog++ instance (same p and sp)

Throws:

CardinalityMergeException - if other is not compatible

HyperLogLogPlus.HyperLogLogPlusMergeException

merge

public ICardinality merge(ICardinality... estimators)
                   throws CardinalityMergeException

Merge this HLL++ with a bunch of others! The power of minions!

Most of the logic consists of case analysis about the state of this HLL++ and each one it wants to merge with. If either of them is 'normal' mode then the other converts to 'normal' as well. A touching sacrifice. 'Normal's combine just like regular HLL estimators do.

If they happen to be both sparse, then it checks if their combined size would be too large and if so, they get relegated to normal mode anyway. Otherwise, the mergeEstimators function is called, and a new sparse HLL++ is born.

Specified by:

merge in interface ICardinality

Parameters:

estimators - the estimators to merge with this one

Returns:

a new estimator with their combined knowledge

Throws:

CardinalityMergeException

getRegisterSet

protected RegisterSet getRegisterSet()

exposed for testing

getSparseSet

protected int[] getSparseSet()

exposed for testing