Standard
|
The StandardTokenizer type exposes the following members.
Properties
| Name | Description | |
|---|---|---|
|
Factory | Returns the used AttributeFactory. (Inherited from AttributeSource) |
|
|
HasAttributes | Returns true, iff this AttributeSource has any attributes (Inherited from AttributeSource) |
|
|
MaxTokenLength | Set the max allowed token length. Any token longer than this is skipped. |
See Also