Module com.carrotsearch.hppc
Package com.carrotsearch.hppc

Class IntPgmIndex

java.lang.Object
com.carrotsearch.hppc.IntPgmIndex
All Implemented Interfaces:
Accountable
@Generated(date="2024-06-04T15:20:17+0200", value="KTypePgmIndex.java") public class IntPgmIndex extends Object implements Accountable
Space-efficient index that enables fast rank/range search operations on a sorted sequence of int.

Implementation of the PGM-Index described at https://pgm.di.unipi.it/, based on the paper 

   Paolo Ferragina and Giorgio Vinciguerra.
   The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds.
   PVLDB, 13(8): 1162-1175, 2020.
It provides rank and range search operations. indexOf() is faster than B+Tree, and the index is much more compact. contains() is between 4x to 7x slower than IntHashSet#contains(), but between 2.5x to 3x faster than Arrays.binarySearch(long[], long).

Its compactness (40KB for 200MB of keys) makes it efficient for very large collections, the index fitting easily in the L2 cache. The epsilon parameter should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage. In practice, epsilon 64 is a good sweet spot.

Internally the index uses an optimal piecewise linear mapping from keys to their position in the sorted order. This mapping is represented as a sequence of linear models (segments) which are themselves recursively indexed by other piecewise linear mappings.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    IntPgmIndex.IntBuilder
    Builds a IntPgmIndex on a provided sorted list of keys.
    protected static class 
    IntPgmIndex.RangeIterator
    Iterator over a range of elements in a sorted array.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    BEYOND_EPSILON_JUMP
    Initial value of the exponential jump when scanning out of the epsilon range.
    static final int
    DOUBLE_KEY_SIZE
    2x KEY_SIZE.
    static final IntPgmIndex
    EMPTY
    Empty immutable IntPgmIndex.
    final int
    epsilon
    The epsilon range used to build this index.
    static final int
    EPSILON
    Epsilon approximation range when searching the list of keys.
    static final int
    EPSILON_RECURSIVE
    Epsilon approximation range for the segments layers.
    final int
    epsilonRecursive
    The recursive epsilon range used to build this index.
    final int
    firstKey
    The lowest key in keys.
    static final int
    KEY_SIZE
    Size of a key, measured in Integer.BYTES because the key is stored in an int[].
    final IntArrayList
    keys
    The list of keys for which this index is built.
    final int
    lastKey
    The highest key in keys.
    final int[]
    levelOffsets
    The offsets in segmentData of the first segment of each segment level.
    static final int
    SEGMENT_DATA_SIZE
    Data size of a segment, measured in Integer.BYTES, because segments are stored in an int[].
    final int[]
    segmentData
    The index data.
    final int
    size
    The size of the key set.

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    contains(int key)
    Returns whether this key set contains the given key.
    <T extends IntProcedure>
    T
    forEachInRange(T procedure, int minKey, int maxKey)
    Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).
    int
    indexOf(int key)
    Searches the specified key, and returns its index in the element list.
    boolean
    isEmpty()
    Returns whether this key set is empty.
    long
    ramBytesAllocated()
    Estimates the allocated memory.
    long
    ramBytesUsed()
    Estimates the bytes that are actually used.
    int
    rangeCardinality(int minKey, int maxKey)
    Returns the number of keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).
    Iterator<IntCursor>
    rangeIterator(int minKey, int maxKey)
    Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).
    int
    rank(int x)
    Returns, for any value x, the number of keys in the sorted list which are smaller than x.
    int
    size()
    Returns the size of the key set.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • EMPTY

      public static final IntPgmIndex EMPTY
      Empty immutable IntPgmIndex.

    • EPSILON

      public static final int EPSILON
      Epsilon approximation range when searching the list of keys. Controls the size of the returned search range, strictly greater than 0. It should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage.

      With EPSILON=64 the benchmark with 200MB of keys shows that this PGM index requires only 2% additional memory on average (40KB). It depends on the distribution of the keys. This epsilon value is good even for 2MB of keys. With EPSILON=32: +5% speed, but 4x space (160KB).

      See Also:

    • EPSILON_RECURSIVE

      public static final int EPSILON_RECURSIVE
      Epsilon approximation range for the segments layers. Controls the size of the search range in the hierarchical segment lists, strictly greater than 0.
      See Also:

    • KEY_SIZE

      public static final int KEY_SIZE
      Size of a key, measured in Integer.BYTES because the key is stored in an int[].

    • DOUBLE_KEY_SIZE

      public static final int DOUBLE_KEY_SIZE
      2x KEY_SIZE.

    • SEGMENT_DATA_SIZE

      public static final int SEGMENT_DATA_SIZE
      Data size of a segment, measured in Integer.BYTES, because segments are stored in an int[].

    • BEYOND_EPSILON_JUMP

      public static final int BEYOND_EPSILON_JUMP
      Initial value of the exponential jump when scanning out of the epsilon range.
      See Also:

    • keys

      public final IntArrayList keys
      The list of keys for which this index is built. It is sorted and may contain duplicate elements.

    • size

      public final int size
      The size of the key set. That is, the number of distinct elements in keys.

    • firstKey

      public final int firstKey
      The lowest key in keys.

    • lastKey

      public final int lastKey
      The highest key in keys.

    • epsilon

      public final int epsilon
      The epsilon range used to build this index.

    • epsilonRecursive

      public final int epsilonRecursive
      The recursive epsilon range used to build this index.

    • levelOffsets

      public final int[] levelOffsets
      The offsets in segmentData of the first segment of each segment level.

    • segmentData

      public final int[] segmentData
      The index data. It contains all the segments for all the levels.

  • Method Details

    • size

      public int size()
      Returns the size of the key set. That is, the number of distinct elements in keys.

    • isEmpty

      public boolean isEmpty()
      Returns whether this key set is empty.

    • contains

      public boolean contains(int key)
      Returns whether this key set contains the given key.

    • indexOf

      public int indexOf(int key)
      Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
      Returns:
      The index of the searched key if it is present; otherwise, (-(<i>insertion point</i>) - 1). The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this guarantees that the return value will be >= 0 if and only if the key is found.

    • rank

      public int rank(int x)
      Returns, for any value x, the number of keys in the sorted list which are smaller than x. It is equal to indexOf(int) if x belongs to the list, or -indexOf(int)-1 otherwise.

      If multiple elements are equal to the specified key, there is no guarantee which one will be found.

      Returns:
      The index of the searched key if it is present; otherwise, the insertion point. The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys# size() if all the elements are less than the specified key. Note that this method always returns a value >= 0.

    • rangeCardinality

      public int rangeCardinality(int minKey, int maxKey)
      Returns the number of keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

    • rangeIterator

      public Iterator<IntCursor> rangeIterator(int minKey, int maxKey)
      Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

    • forEachInRange

      public <T extends IntProcedure> T forEachInRange(T procedure, int minKey, int maxKey)
      Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

    • ramBytesAllocated

      public long ramBytesAllocated()
      Estimates the allocated memory. It does not count the memory for the list of keys, only for the index itself.
      Specified by:
      ramBytesAllocated in interface Accountable
      Returns:
      Ram allocated in bytes

    • ramBytesUsed

      public long ramBytesUsed()
      Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the index itself.
      Specified by:
      ramBytesUsed in interface Accountable
      Returns:
      Ram used in bytes