StandardTokenizer Properties |
The StandardTokenizer type exposes the following members.
Properties| Name | Description | |
|---|---|---|
 | Factory | Returns the used AttributeFactory. (Inherited from AttributeSource.) |
| HasAttributes | Returns true, iff this AttributeSource has any attributes (Inherited from AttributeSource.) |
| MaxTokenLength | Set the max allowed token length. Any token longer
than this is skipped.
|
See Also