org.apache.solr.analysis
Class RemoveDuplicatesTokenFilter
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter
org.apache.solr.analysis.BufferedTokenStream
org.apache.solr.analysis.RemoveDuplicatesTokenFilter
public class RemoveDuplicatesTokenFilter
- extends BufferedTokenStream
A TokenFilter which filters out Tokens at the same position and Term
text as the previous token in the stream.
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
RemoveDuplicatesTokenFilter
public RemoveDuplicatesTokenFilter(TokenStream input)
process
protected Token process(Token t)
throws IOException
- Description copied from class:
BufferedTokenStream
- Process a token. Subclasses may read more tokens from the input stream,
write more tokens to the output stream, or simply return the next token
to be output. Subclasses may return null if the token is to be dropped.
If a subclass writes tokens to the output stream and returns a
non-null Token, the returned Token is considered to be at the head of
the token output stream.
- Specified by:
process
in class BufferedTokenStream
- Throws:
IOException
Copyright © 2010 Apache Software Foundation. All Rights Reserved.