net.sourceforge.pmd.cpd

## Class AbstractTokenizer

- java.lang.Object
- - net.sourceforge.pmd.cpd.AbstractTokenizer

All Implemented Interfaces:
  Tokenizer

Direct Known Subclasses:
  EcmascriptTokenizer, FortranTokenizer, RubyTokenizer

    public abstract class AbstractTokenizer
    extends Object
    implements Tokenizer

### Field Summary

Fields
`protected List<String>` `ignorableCharacter`
`protected List<String>` `ignorableStmt`
`protected char` `oneLineCommentChar`
`protected boolean` `spanMultipleLinesString`
`protected List<String>` `stringToken`

### Constructor Summary

Constructors
Constructor and Description
`AbstractTokenizer()`

### Method Summary

Methods
`void` `tokenize(SourceCode tokens, Tokens tokenEntries)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### stringToken

    protected List<String> stringToken

#### ignorableCharacter

    protected List<String> ignorableCharacter

#### ignorableStmt

    protected List<String> ignorableStmt

#### oneLineCommentChar

    protected char oneLineCommentChar

#### spanMultipleLinesString

    protected boolean spanMultipleLinesString

### Constructor Detail

#### AbstractTokenizer

    public AbstractTokenizer()

### Method Detail

#### tokenize

    public void tokenize(SourceCode tokens,
                Tokens tokenEntries)

Specified by:
  `tokenize` in interface `Tokenizer`