Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.lucene.analysis.compound
Class CompoundWordTokenFilterBase

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.TokenFilter
                  org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase

All Implemented Interfaces:
  Closeable

Direct Known Subclasses:
  DictionaryCompoundWordTokenFilter, HyphenationCompoundWordTokenFilter

    public abstract class CompoundWordTokenFilterBase

extends TokenFilter

Base class for decomposition token filters.

You must specify the required `Version` compatibility when creating CompoundWordTokenFilterBase:

- As of 3.1, CompoundWordTokenFilterBase correctly handles Unicode 4.0 supplementary characters in strings and char arrays provided as compound word dictionaries.

If you pass in a `CharArraySet` as dictionary, it should be case-insensitive unless it contains only lowercased entries and you have `LowerCaseFilter` before this filter in your analysis chain. For optional performance (as this filter does lots of lookups to the dictionary, you should use the latter analysis chain/CharArraySet). Be aware: If you supply arbitrary `Sets` to the ctors or `String[]` dictionaries, they will be automatically transformed to case-insensitive!

Nested Class Summary
`protected class` | `CompoundWordTokenFilterBase.CompoundToken` Helper class to hold decompounded token information

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary
`static int` | `DEFAULT_MAX_SUBWORD_SIZE` The default for maximal length of subwords that get propagated to the output of this filter
`static int` | `DEFAULT_MIN_SUBWORD_SIZE` The default for minimal length of subwords that get propagated to the output of this filter
`static int` | `DEFAULT_MIN_WORD_SIZE` The default for minimal word length that gets decomposed
`protected CharArraySet` | `dictionary`
`protected int` | `maxSubwordSize`
`protected int` | `minSubwordSize`
`protected int` | `minWordSize`
`protected OffsetAttribute` | `offsetAtt`
`protected boolean` | `onlyLongestMatch`
`protected CharTermAttribute` | `termAtt`
`protected LinkedList<CompoundWordTokenFilterBase.CompoundToken>` | `tokens`

Fields inherited from class org.apache.lucene.analysis.TokenFilter
`input`

Constructor Summary
`protected` | `CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set)` instead
`protected` | `CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean)` instead
`protected` | `CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean)` instead
`protected` | `CompoundWordTokenFilterBase(TokenStream input, String[] dictionary)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[])` instead
`protected` | `CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, boolean onlyLongestMatch)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean)` instead
`protected` | `CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)` Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean)` instead
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary)`
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)`
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary)`
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary, boolean onlyLongestMatch)`
`protected` | `CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`

Method Summary
`protected abstract void` | `decompose()` Decomposes the current `termAtt` and places `CompoundWordTokenFilterBase.CompoundToken` instances in the `tokens` list.
`boolean` | `incrementToken()` Consumers (i.e., `IndexWriter`) use this method to advance the stream to the next token.
`static CharArraySet` | `makeDictionary(Version matchVersion, String[] dictionary)` Deprecated. Only available for backwards compatibility.
`void` | `reset()` Reset the filter as well as the input TokenStream.

Methods inherited from class org.apache.lucene.analysis.TokenFilter
`close, end`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### DEFAULT_MIN_WORD_SIZE

    public static final int DEFAULT_MIN_WORD_SIZE

  The default for minimal word length that gets decomposed

  See Also:
    Constant Field Values

### DEFAULT_MIN_SUBWORD_SIZE

    public static final int DEFAULT_MIN_SUBWORD_SIZE

  The default for minimal length of subwords that get propagated to the output of this filter

  See Also:
    Constant Field Values

### DEFAULT_MAX_SUBWORD_SIZE

    public static final int DEFAULT_MAX_SUBWORD_SIZE

  The default for maximal length of subwords that get propagated to the output of this filter

  See Also:
    Constant Field Values

### dictionary

    protected final CharArraySet dictionary

### tokens

    protected final LinkedList<CompoundWordTokenFilterBase.CompoundToken> tokens

### minWordSize

    protected final int minWordSize

### minSubwordSize

    protected final int minSubwordSize

### maxSubwordSize

    protected final int maxSubwordSize

### onlyLongestMatch

    protected final boolean onlyLongestMatch

### termAtt

    protected final CharTermAttribute termAtt

### offsetAtt

    protected final OffsetAttribute offsetAtt

Constructor Detail

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     String[] dictionary,
                                                     int minWordSize,
                                                     int minSubwordSize,
                                                     int maxSubwordSize,
                                                     boolean onlyLongestMatch)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean)` instead

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     String[] dictionary,
                                                     boolean onlyLongestMatch)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean)` instead

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     Set<?> dictionary,
                                                     boolean onlyLongestMatch)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean)` instead

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     String[] dictionary)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, String[])` instead

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     Set<?> dictionary)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set)` instead

### CompoundWordTokenFilterBase

    @Deprecated
    protected CompoundWordTokenFilterBase(TokenStream input,
                                                     Set<?> dictionary,
                                                     int minWordSize,
                                                     int minSubwordSize,
                                                     int maxSubwordSize,
                                                     boolean onlyLongestMatch)

  Deprecated. use `CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean)` instead

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          String[] dictionary,
                                          int minWordSize,
                                          int minSubwordSize,
                                          int maxSubwordSize,
                                          boolean onlyLongestMatch)

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          String[] dictionary,
                                          boolean onlyLongestMatch)

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          Set<?> dictionary,
                                          boolean onlyLongestMatch)

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          String[] dictionary)

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          Set<?> dictionary)

### CompoundWordTokenFilterBase

    protected CompoundWordTokenFilterBase(Version matchVersion,
                                          TokenStream input,
                                          Set<?> dictionary,
                                          int minWordSize,
                                          int minSubwordSize,
                                          int maxSubwordSize,
                                          boolean onlyLongestMatch)

Method Detail

### makeDictionary

    @Deprecated
    public static CharArraySet makeDictionary(Version matchVersion,
                                                         String[] dictionary)

  Deprecated. Only available for backwards compatibility.

### incrementToken

    public final boolean incrementToken()
                                 throws IOException

  Description copied from class: `TokenStream`
  Consumers (i.e., `IndexWriter`) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate `AttributeImpl`s with the attributes of the next token.

  The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use `AttributeSource.captureState()` to create a copy of the current attribute state.

  This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to `AttributeSource.addAttribute(Class)` and `AttributeSource.getAttribute(Class)`, references to all `AttributeImpl`s that this stream uses should be retrieved during instantiation.

  To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in `TokenStream.incrementToken()`.

  Specified by:
    `incrementToken` in class `TokenStream`
  Returns:
    false for end of stream; true otherwise
  Throws:
    `IOException`

### decompose

    protected abstract void decompose()

  Decomposes the current `termAtt` and places `CompoundWordTokenFilterBase.CompoundToken` instances in the `tokens` list. The original token may not be placed in the list, as it is automatically passed through this filter.

### reset

    public void reset()
               throws IOException

  Description copied from class: `TokenFilter`
  Reset the filter as well as the input TokenStream.

  Overrides:
    `reset` in class `TokenFilter`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD