org.carrot2.text.preprocessing

Class CaseNormalizer

java.lang.Object

org.carrot2.text.preprocessing.CaseNormalizer

@Bindable(prefix="CaseNormalizer")
public final class CaseNormalizer
extends Object

Performs case normalization and calculates a number of frequency statistics for words. The aim of case normalization is to find the most frequently appearing variants of words in terms of case. For example, if in the input documents MacOS appears 20 times, Macos 5 times and macos 2 times, case normalizer will select MacOS to represent all variants and assign the aggregated term frequency of 27 to it.

This class saves the following results to the PreprocessingContext:

• PreprocessingContext.AllTokens.wordIndex
• PreprocessingContext.AllWords.image
• PreprocessingContext.AllWords.tf
• PreprocessingContext.AllWords.tfByDocument

This class requires that Tokenizer be invoked first.

Field Summary

Fields

Modifier and Type	Field and Description
int	dfThreshold Word Document Frequency threshold.

Constructor Summary

Constructors

Constructor and Description
CaseNormalizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	normalize(PreprocessingContext context) Performs normalization and saves the results to the context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dfThreshold

@Processing
@Input
@Attribute
@IntRange(min=1,
          max=100)
@Label(value="Word document frequency threshold")
@Level(value=ADVANCED)
@Group(value="Preprocessing")
public int dfThreshold

Word Document Frequency threshold. Words appearing in fewer than dfThreshold documents will be ignored.

Constructor Detail

CaseNormalizer

public CaseNormalizer()

Method Detail

normalize

public void normalize(PreprocessingContext context)

Performs normalization and saves the results to the context.