Standard
|
The StandardTokenizer type exposes the following members.
| Name | Description | |
|---|---|---|
|
|
ACRONYM | |
|
|
ACRONYM_DEP |
Obsolete.
|
|
|
ALPHANUM | |
|
|
APOSTROPHE | |
|
|
CJ | |
|
|
COMPANY | |
|
|
||
|
|
HOST | |
|
|
NUM | |
|
|
TOKEN_TYPES | String token types that correspond to token type int constants |