org.carrot2.clustering.stc

Class STCClusteringAlgorithm

java.lang.Object

org.carrot2.core.ProcessingComponentBase

org.carrot2.clustering.stc.STCClusteringAlgorithm

All Implemented Interfaces:

IClusteringAlgorithm, IProcessingComponent

@Bindable(prefix="STCClusteringAlgorithm",
          inherit=CommonAttributes.class)
@Label(value="STC Clustering")
public final class STCClusteringAlgorithm
extends ProcessingComponentBase
implements IClusteringAlgorithm

Suffix Tree Clustering (STC) algorithm. Pretty much as described in: Oren Zamir, Oren Etzioni, Grouper: A Dynamic Clustering Interface to Web Search Results, 1999. Some liberties were taken wherever STC's description was not clear enough or where we thought some improvements could be made.

Field Summary

Fields

Modifier and Type	Field and Description
List<Cluster>	clusters Clusters created by the algorithm.
double	documentCountBoost Document count boost.
List<Document>	documents Documents to cluster.
int	ignoreWordIfInFewerDocs Minimum word-document recurrences.
double	ignoreWordIfInHigherDocsPercent Maximum word-document ratio.
int	maxBaseClusters Maximum base clusters count.
int	maxClusters Maximum final clusters.
int	maxDescPhraseLength Maximum words per label.
double	maxPhraseOverlap Maximum cluster phrase overlap.
int	maxPhrases Maximum phrases per label.
double	mergeThreshold Base cluster merge threshold.
double	minBaseClusterScore Minimum base cluster score.
int	minBaseClusterSize Minimum documents per base cluster.
double	mostGeneralPhraseCoverage Minimum general phrase coverage.
MultilingualClustering	multilingualClustering A helper for performing multilingual clustering.
int	optimalPhraseLength Optimal label length.
double	optimalPhraseLengthDev Phrase length tolerance.
IPreprocessingPipeline	preprocessingPipeline Common preprocessing tasks handler.
String	query Query that produced the documents.
double	scoreWeight Balance between cluster score and size during cluster sorting.
double	singleTermBoost Single term boost.

Constructor Summary

Constructors

Constructor and Description
STCClusteringAlgorithm()

Method Summary

Methods

Modifier and Type	Method and Description
void	afterProcessing() Memory cleanups.
void	process() Performs STC clustering of documents.

Methods inherited from class org.carrot2.core.ProcessingComponentBase

beforeProcessing, dispose, getContext, getSharedExecutor, init

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.carrot2.core.IProcessingComponent

beforeProcessing, dispose, init

Field Detail

query

@Processing
@Input
@Internal
@Attribute(key="query",
           inherit=true)
public String query

Query that produced the documents. The query will help the algorithm to create better clusters. Therefore, providing the query is optional but desirable.

documents

@Processing
@Input
@Required
@Internal
@Attribute(key="documents",
           inherit=true)
public List<Document> documents

Documents to cluster.

clusters

@Processing
@Output
@Internal
@Attribute(key="clusters",
           inherit=true)
public List<Cluster> clusters

Clusters created by the algorithm.

ignoreWordIfInFewerDocs

@Processing
@Input
@Attribute
@IntRange(min=2)
@Level(value=MEDIUM)
@Group(value="Word filtering")
public int ignoreWordIfInFewerDocs

Minimum word-document recurrences.

ignoreWordIfInHigherDocsPercent

@Processing
@Input
@Attribute
@Level(value=MEDIUM)
@Group(value="Word filtering")
public double ignoreWordIfInHigherDocsPercent

Maximum word-document ratio. A number between 0 and 1, if a word exists in more snippets than this ratio, it is ignored.

minBaseClusterScore

@Processing
@Input
@Attribute
@Level(value=ADVANCED)
@Group(value="Base clusters")
public double minBaseClusterScore

Minimum base cluster score.

maxBaseClusters

@Processing
@Input
@Attribute
@IntRange(min=2)
@Level(value=ADVANCED)
@Group(value="Base clusters")
public int maxBaseClusters

Maximum base clusters count. Trims the base cluster array after N-th position for the merging phase.

minBaseClusterSize

@Processing
@Input
@Attribute
@IntRange(min=2,
          max=20)
@Level(value=ADVANCED)
@Group(value="Base clusters")
public int minBaseClusterSize

Minimum documents per base cluster.

maxClusters

@Processing
@Input
@Attribute
@IntRange(min=1)
@Level(value=BASIC)
@Group(value="Merging and output")
public int maxClusters

Maximum final clusters.

mergeThreshold

@Processing
@Input
@Attribute
@Level(value=ADVANCED)
@Group(value="Merging and output")
public double mergeThreshold

Base cluster merge threshold.

maxPhraseOverlap

@Processing
@Input
@Attribute
@Level(value=ADVANCED)
@Group(value="Labels")
public double maxPhraseOverlap

Maximum cluster phrase overlap.

mostGeneralPhraseCoverage

@Processing
@Input
@Attribute
@Level(value=ADVANCED)
@Group(value="Labels")
public double mostGeneralPhraseCoverage

Minimum general phrase coverage. Minimum phrase coverage to appear in cluster description.

maxDescPhraseLength

@Processing
@Input
@Attribute
@IntRange(min=1)
@Level(value=BASIC)
@Group(value="Labels")
public int maxDescPhraseLength

Maximum words per label. Base clusters formed by phrases with more words than this ratio are trimmed.

maxPhrases

@Processing
@Input
@Attribute
@IntRange(min=1)
@Level(value=BASIC)
@Group(value="Labels")
public int maxPhrases

Maximum phrases per label. Maximum number of phrases from base clusters promoted to the cluster's label.

singleTermBoost

@Processing
@Input
@Attribute
@Level(value=MEDIUM)
@Group(value="Base clusters")
public double singleTermBoost

Single term boost. A factor in calculation of the base cluster score. If greater then zero, single-term base clusters are assigned this value regardless of the penalty function.

optimalPhraseLength

@Processing
@Input
@Attribute
@IntRange(min=1)
@Level(value=BASIC)
@Group(value="Base clusters")
public int optimalPhraseLength

Optimal label length. A factor in calculation of the base cluster score.

optimalPhraseLengthDev

@Processing
@Input
@Attribute
@Level(value=MEDIUM)
@Group(value="Base clusters")
public double optimalPhraseLengthDev

Phrase length tolerance. A factor in calculation of the base cluster score.

documentCountBoost

@Processing
@Input
@Attribute
@Level(value=MEDIUM)
@Group(value="Base clusters")
public double documentCountBoost

Document count boost. A factor in calculation of the base cluster score, boosting the score depending on the number of documents found in the base cluster.

preprocessingPipeline

@Input
@Attribute
@Internal
@Level(value=ADVANCED)
public IPreprocessingPipeline preprocessingPipeline

Common preprocessing tasks handler.

scoreWeight

@Input
@Processing
@Attribute
@Label(value="Size-Score sorting ratio")
@Level(value=MEDIUM)
@Group(value="Clusters")
public double scoreWeight

Balance between cluster score and size during cluster sorting. Value equal to 0.0 will sort clusters based only on cluster size. Value equal to 1.0 will sort clusters based only on cluster score.

multilingualClustering

public final MultilingualClustering multilingualClustering

A helper for performing multilingual clustering.

Constructor Detail

STCClusteringAlgorithm

public STCClusteringAlgorithm()

Method Detail

process

public void process()
             throws ProcessingException

Performs STC clustering of documents.

Specified by:

process in interface IProcessingComponent

Overrides:

process in class ProcessingComponentBase

Throws:

ProcessingException - when processing failed. If thrown, the IProcessingComponent.afterProcessing() method will be called and the component will be ready to accept further requests or to be disposed of. Finally, the exception will be rethrown from the controller method that caused the component to perform processing.

afterProcessing

public void afterProcessing()

Memory cleanups.

Specified by:

afterProcessing in interface IProcessingComponent

Overrides:

afterProcessing in class ProcessingComponentBase