org.apache.lucene.analysis.standard

## Interface StandardTokenizerInterface

- All Known Implementing Classes:
    StandardTokenizerImpl, StandardTokenizerImpl31, StandardTokenizerImpl34, UAX29URLEmailTokenizerImpl, UAX29URLEmailTokenizerImpl31, UAX29URLEmailTokenizerImpl34, UAX29URLEmailTokenizerImpl36

      public interface StandardTokenizerInterface

  Internal interface for supporting versioned grammars.
  NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `YYEOF` This character denotes the end of file

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `int` | `getNextToken()` Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
    `void` | `getText(CharTermAttribute t)` Copies the matched text into the CharTermAttribute
    `int` | `yychar()` Returns the current position.
    `int` | `yylength()` Returns the length of the matched text region.
    `void` | `yyreset(Reader reader)` Resets the scanner to read from a new input stream.

- - ### Field Detail

    - #### YYEOF

          static final int YYEOF

      This character denotes the end of file
      See Also:
        Constant Field Values

  - ### Method Detail

    - #### getText

          void getText(CharTermAttribute t)

      Copies the matched text into the CharTermAttribute

    - #### yychar

          int yychar()

      Returns the current position.

    - #### yyreset

          void yyreset(Reader reader)

      Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to `ZZ_INITIAL`.
      Parameters:
        `reader` - the new input stream

    - #### yylength

          int yylength()

      Returns the length of the matched text region.

    - #### getNextToken

          int getNextToken()
                           throws IOException

      Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
      Returns:
        the next token, `YYEOF` on end of stream
      Throws:
        `IOException` - if any I/O-Error occurs

Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.