## org.apache.lucene.analysis.wikipedia
Class WikipediaTokenizer

    java.lang.Object
      org.apache.lucene.util.AttributeSource
          org.apache.lucene.analysis.TokenStream
              org.apache.lucene.analysis.Tokenizer
                  org.apache.lucene.analysis.wikipedia.WikipediaTokenizer

All Implemented Interfaces:
  Closeable

    public final class WikipediaTokenizer

extends Tokenizer

Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
`AttributeSource.AttributeFactory, AttributeSource.State`

Field Summary
`static int` `ACRONYM_ID`
`static int` `ALPHANUM_ID`
`static int` `APOSTROPHE_ID`
`static String` `BOLD`
`static int` `BOLD_ID`
`static String` `BOLD_ITALICS`
`static int` `BOLD_ITALICS_ID`
`static int` `BOTH` Output the both the untokenized token and the splits
`static String` `CATEGORY`
`static int` `CATEGORY_ID`
`static String` `CITATION`
`static int` `CITATION_ID`
`static int` `CJ_ID`
`static int` `COMPANY_ID`
`static int` `EMAIL_ID`
`static String` `EXTERNAL_LINK`
`static int` `EXTERNAL_LINK_ID`
`static String` `EXTERNAL_LINK_URL`
`static int` `EXTERNAL_LINK_URL_ID`
`static String` `HEADING`
`static int` `HEADING_ID`
`static int` `HOST_ID`
`static String` `INTERNAL_LINK`
`static int` `INTERNAL_LINK_ID`
`static String` `ITALICS`
`static int` `ITALICS_ID`
`static int` `NUM_ID`
`static String` `SUB_HEADING`
`static int` `SUB_HEADING_ID`
`static String[]` `TOKEN_TYPES` String token types that correspond to token type int constants
`static int` `TOKENS_ONLY` Only output tokens
`static int` `UNTOKENIZED_ONLY` Only output untokenized tokens, which are tokens that would normally be split into several tokens
`static int` `UNTOKENIZED_TOKEN_FLAG` This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`WikipediaTokenizer(AttributeSource.AttributeFactory factory, Reader input, int tokenOutput, Set<String> untokenizedTypes)` Creates a new instance of the `WikipediaTokenizer`.
`WikipediaTokenizer(AttributeSource source, Reader input, int tokenOutput, Set<String> untokenizedTypes)` Creates a new instance of the `WikipediaTokenizer`.
`WikipediaTokenizer(Reader input)` Creates a new instance of the `WikipediaTokenizer`.
`WikipediaTokenizer(Reader input, int tokenOutput, Set<String> untokenizedTypes)` Creates a new instance of the `WikipediaTokenizer`.

Method Summary
`void` `end()`
`boolean` `incrementToken()`
`void` `reset()`

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close, correctOffset, setReader`

Methods inherited from class org.apache.lucene.util.AttributeSource
`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### INTERNAL_LINK

    public static final String INTERNAL_LINK

See Also:
  Constant Field Values

### EXTERNAL_LINK

    public static final String EXTERNAL_LINK

See Also:
  Constant Field Values

### EXTERNAL_LINK_URL

    public static final String EXTERNAL_LINK_URL

See Also:
  Constant Field Values

### CITATION

    public static final String CITATION

See Also:
  Constant Field Values

### CATEGORY

    public static final String CATEGORY

See Also:
  Constant Field Values

### BOLD

    public static final String BOLD

See Also:
  Constant Field Values

### ITALICS

    public static final String ITALICS

See Also:
  Constant Field Values

### BOLD_ITALICS

    public static final String BOLD_ITALICS

See Also:
  Constant Field Values

### HEADING

    public static final String HEADING

See Also:
  Constant Field Values

### SUB_HEADING

    public static final String SUB_HEADING

See Also:
  Constant Field Values

### ALPHANUM_ID

    public static final int ALPHANUM_ID

See Also:
  Constant Field Values

### APOSTROPHE_ID

    public static final int APOSTROPHE_ID

See Also:
  Constant Field Values

### ACRONYM_ID

    public static final int ACRONYM_ID

See Also:
  Constant Field Values

### COMPANY_ID

    public static final int COMPANY_ID

See Also:
  Constant Field Values

### EMAIL_ID

    public static final int EMAIL_ID

See Also:
  Constant Field Values

### HOST_ID

    public static final int HOST_ID

See Also:
  Constant Field Values

### NUM_ID

    public static final int NUM_ID

See Also:
  Constant Field Values

### CJ_ID

    public static final int CJ_ID

See Also:
  Constant Field Values

### INTERNAL_LINK_ID

    public static final int INTERNAL_LINK_ID

See Also:
  Constant Field Values

### EXTERNAL_LINK_ID

    public static final int EXTERNAL_LINK_ID

See Also:
  Constant Field Values

### CITATION_ID

    public static final int CITATION_ID

See Also:
  Constant Field Values

### CATEGORY_ID

    public static final int CATEGORY_ID

See Also:
  Constant Field Values

### BOLD_ID

    public static final int BOLD_ID

See Also:
  Constant Field Values

### ITALICS_ID

    public static final int ITALICS_ID

See Also:
  Constant Field Values

### BOLD_ITALICS_ID

    public static final int BOLD_ITALICS_ID

See Also:
  Constant Field Values

### HEADING_ID

    public static final int HEADING_ID

See Also:
  Constant Field Values

### SUB_HEADING_ID

    public static final int SUB_HEADING_ID

See Also:
  Constant Field Values

### EXTERNAL_LINK_URL_ID

    public static final int EXTERNAL_LINK_URL_ID

See Also:
  Constant Field Values

### TOKEN_TYPES

    public static final String[] TOKEN_TYPES

  String token types that correspond to token type int constants

### TOKENS_ONLY

    public static final int TOKENS_ONLY

  Only output tokens

  See Also:
    Constant Field Values

### UNTOKENIZED_ONLY

    public static final int UNTOKENIZED_ONLY

  Only output untokenized tokens, which are tokens that would normally be split into several tokens

  See Also:
    Constant Field Values

### BOTH

    public static final int BOTH

  Output the both the untokenized token and the splits

  See Also:
    Constant Field Values

### UNTOKENIZED_TOKEN_FLAG

    public static final int UNTOKENIZED_TOKEN_FLAG

  This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.

  See Also:
    Constant Field Values

Constructor Detail

### WikipediaTokenizer

    public WikipediaTokenizer(Reader input)

  Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a newly created JFlex scanner.

  Parameters:
    `input` - The Input Reader

### WikipediaTokenizer

    public WikipediaTokenizer(Reader input,
                              int tokenOutput,
                              Set<String> untokenizedTypes)

  Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a the newly created JFlex scanner.

  Parameters:
    `input` - The input
    `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

### WikipediaTokenizer

    public WikipediaTokenizer(AttributeSource.AttributeFactory factory,
                              Reader input,
                              int tokenOutput,
                              Set<String> untokenizedTypes)

  Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a the newly created JFlex scanner. Uses the given `AttributeSource.AttributeFactory`.

  Parameters:
    `input` - The input
    `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

### WikipediaTokenizer

    public WikipediaTokenizer(AttributeSource source,
                              Reader input,
                              int tokenOutput,
                              Set<String> untokenizedTypes)

  Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a the newly created JFlex scanner. Uses the given `AttributeSource`.

  Parameters:
    `input` - The input
    `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

Method Detail

### incrementToken

    public final boolean incrementToken()
                                 throws IOException

  Specified by:
    `incrementToken` in class `TokenStream`
  Throws:
    `IOException`

### reset

    public void reset()
               throws IOException

  Overrides:
    `reset` in class `TokenStream`
  Throws:
    `IOException`

### end

    public void end()

  Overrides:
    `end` in class `TokenStream`

Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.