StringTokenizeris a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.
If you look at String.split() and compare it to StringTokenizer, the relevant difference is that String.split() uses a regular expression, whereas StringTokenizer just uses verbatim split characters. So if I wanted to tokenize a string with more complex logic than single characters (e.g. split on \r\n), I can't use StringTokenizer but I can use String.split().