org.carrot2.text.preprocessing.pipeline

Class BasicPreprocessingPipeline

java.lang.Object

org.carrot2.text.preprocessing.pipeline.BasicPreprocessingPipeline

All Implemented Interfaces:

IPreprocessingPipeline

Direct Known Subclasses:

CompletePreprocessingPipeline

@Bindable(prefix="PreprocessingPipeline")
public class BasicPreprocessingPipeline
extends Object
implements IPreprocessingPipeline

Performs basic preprocessing steps on the provided documents. The preprocessing consists of the following steps:

1. Tokenizer.tokenize(PreprocessingContext)
2. CaseNormalizer.normalize(PreprocessingContext)
3. LanguageModelStemmer.stem(PreprocessingContext)
4. StopListMarker.mark(PreprocessingContext)

Field Summary

Fields

Modifier and Type	Field and Description
CaseNormalizer	caseNormalizer Case normalizer used by the algorithm, contains bindable attributes.
LanguageModelStemmer	languageModelStemmer Stemmer used by the algorithm, contains bindable attributes.
ILexicalDataFactory	lexicalDataFactory Lexical data factory.
IStemmerFactory	stemmerFactory Stemmer factory.
StopListMarker	stopListMarker Stop list marker used by the algorithm, contains bindable attributes.
Tokenizer	tokenizer Tokenizer used by the algorithm, contains bindable attributes.
ITokenizerFactory	tokenizerFactory Tokenizer factory.

Constructor Summary

Constructors

Constructor and Description
BasicPreprocessingPipeline()

Method Summary

Methods

Modifier and Type	Method and Description
PreprocessingContext	preprocess(List<Document> documents, String query, LanguageCode language) Performs preprocessing on the provided list of documents.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenizer

public final Tokenizer tokenizer

Tokenizer used by the algorithm, contains bindable attributes.

caseNormalizer

public final CaseNormalizer caseNormalizer

Case normalizer used by the algorithm, contains bindable attributes.

languageModelStemmer

public final LanguageModelStemmer languageModelStemmer

Stemmer used by the algorithm, contains bindable attributes.

stopListMarker

public final StopListMarker stopListMarker

Stop list marker used by the algorithm, contains bindable attributes.

tokenizerFactory

@Input
@Processing
@Internal
@Attribute
@Level(value=ADVANCED)
@Group(value="Preprocessing")
public ITokenizerFactory tokenizerFactory

Tokenizer factory. Creates the tokenizers to be used by the clustering algorithm.

stemmerFactory

@Input
@Processing
@Internal
@Attribute
@Level(value=ADVANCED)
@Group(value="Preprocessing")
public IStemmerFactory stemmerFactory

Stemmer factory. Creates the stemmers to be used by the clustering algorithm.

lexicalDataFactory

@Input
@Processing
@Internal
@Attribute
@Level(value=ADVANCED)
@Group(value="Preprocessing")
public ILexicalDataFactory lexicalDataFactory

Lexical data factory. Creates the lexical data to be used by the clustering algorithm, including stop word and stop label dictionaries.

Constructor Detail

BasicPreprocessingPipeline

public BasicPreprocessingPipeline()

Method Detail

preprocess

public PreprocessingContext preprocess(List<Document> documents,
                              String query,
                              LanguageCode language)

Performs preprocessing on the provided list of documents. Results can be obtained from the returned PreprocessingContext.

Specified by:

preprocess in interface IPreprocessingPipeline