org.carrot2.text.analysis

Class TokenTypeUtils

java.lang.Object

org.carrot2.text.analysis.TokenTypeUtils

public final class TokenTypeUtils
extends Object

Utility methods for working with ITokenizer attributes.

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	isCommon(int flag) Return true if ITokenizer.TF_COMMON_WORD is set.
static boolean	isDocumentSeparator(int type) Returns true if the given type has ITokenizer.TF_SEPARATOR_DOCUMENT set.
static boolean	isFieldSeparator(int type) Returns true if the given type has ITokenizer.TF_SEPARATOR_FIELD set.
static boolean	isInQuery(int flag) Return true if ITokenizer.TF_QUERY_WORD is set.
static boolean	isTerminator(int type) Returns true if the given type has ITokenizer.TF_TERMINATOR set.
static int	maskType(int rawType) Mask the given raw token type and leave just the token type bits.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

maskType

public static int maskType(int rawType)

Mask the given raw token type and leave just the token type bits.

isDocumentSeparator

public static boolean isDocumentSeparator(int type)

Returns true if the given type has ITokenizer.TF_SEPARATOR_DOCUMENT set.

isFieldSeparator

public static boolean isFieldSeparator(int type)

Returns true if the given type has ITokenizer.TF_SEPARATOR_FIELD set.

isTerminator

public static boolean isTerminator(int type)

Returns true if the given type has ITokenizer.TF_TERMINATOR set.

isCommon

public static boolean isCommon(int flag)

Return true if ITokenizer.TF_COMMON_WORD is set.

isInQuery

public static boolean isInQuery(int flag)

Return true if ITokenizer.TF_QUERY_WORD is set.