StringTokenizer
[ Home ] [ Up ] [ StringTokenizer ] [ StreamTokenizer ]

 

 

The StringTokenizer class

public class StringTokenizer 
	implements Enumeration

The StringTokenizer class allows an application to break a string into tokens.

The set of delimiters (the characters that separate tokens) may be specified either at StringTokenizer creation time or on a per-token basis.

An instance of StringTokenizer behaves in one of two ways, depending on whether it was created with the returnTokens flag having the value true or false:

  • If the flag is false, delimiter characters serve to separate tokens. A token is a maximal sequence of consecutive characters that are not delimiters.
  • If the flag is true, delimiter characters are also considered to be tokens. A token is either one delimiter character, or a maximal sequence of consecutive characters that are not delimiters.

Here's a simple example:

package inputOutput;

import java.util.StringTokenizer;

public class TokenizeString
{
    public static void main(String[] args)
    {
        StringTokenizer st = new StringTokenizer("Java is great!");
        while (st.hasMoreTokens()) 
        {
            System.out.println(st.nextToken());
        }
    }
}

which prints out:

Java
is
great!

Note that the default delimiter characters are:

  • space (' ')
  • tab ('\t')
  • newline ('\n')
  • carriage return ('\r')

Here's another, modified, version, where we specify the delimiters explicitly:

package inputOutput;

import java.util.StringTokenizer;

public class TokenizeString
{
    public static void main(String[] args)
    {
        String delimiters = ".;!?;:";
        String paragraph =
        "The spirit of Java is alive in the world! " +
        "How can you be ignorant of it? " +
        "Parse, compile, and assimilate; " +
        "perhaps the world will become your oyster! " +
        "An example: 'Write once, run anywhere.'";
        StringTokenizer st = 
                  new StringTokenizer(paragraph, delimiters);
        while (st.hasMoreTokens()) 
        {
            System.out.println(st.nextToken());
        }
    }
}

It outputs the following:

The spirit of Java is alive in the world
 How can you be ignorant of it
 Parse, compile, and assimilate
 perhaps the world will become your oyster
 An example
 'Write once, run anywhere
'

Note that, in the above example, we are considering tokens to be sentences (well, kind of -- actually, we're just using the punctuation as delimiters). In other words, tokens do not have to be words, or single characters only. A token may be any sequence of characters.

If we want the delimiter characters also to be returned as tokens, we can make a slight further modification:

package inputOutput;

import java.util.StringTokenizer;

public class TokenizeString
{
    public static void main(String[] args)
    {
        String delimiters = ".;!?;:";
        String paragraph =
        "The spirit of Java is alive in the world! " +
        "How can you be ignorant of it? " +
        "Parse, compile, and assimilate; " +
        "perhaps the world will become your oyster! " +
        "An example: 'Write once, run anywhere.'";
        StringTokenizer st = 
                  new StringTokenizer(paragraph, delimiters, true);
        while (st.hasMoreTokens()) 
        {
            System.out.println(st.nextToken());
        }
    }
}

which outputs:

The spirit of Java is alive in the world
!
 How can you be ignorant of it
?
 Parse, compile, and assimilate
;
 perhaps the world will become your oyster
!
 An example
:
 'Write once, run anywhere
.
'

Perhaps we'd like to handle things inside single quotes a little better:

package inputOutput;

import java.util.StringTokenizer;

public class TokenizeString
{
    public static void main(String[] args)
    {
        String delimiters = ".;!?;:'"; // Added single quote 
        String paragraph =
        "The spirit of Java is alive in the world! " +
        "How can you be ignorant of it? " +
        "Parse, compile, and assimilate; " +
        "perhaps the world will become your oyster! " +
        "An example: 'Write once, run anywhere.'";
        StringTokenizer st = 
                  new StringTokenizer(paragraph, delimiters, true);
        while (st.hasMoreTokens()) 
        {
            String token = st.nextToken();
            if (token.equals("'"))
            {
                if (st.hasMoreTokens())
                    token += st.nextToken("'"); // Use different delimiter set.
                if (st.hasMoreTokens())
                    token += st.nextToken();
            }
            System.out.println(token);
        }
    }
}

which outputs:

The spirit of Java is alive in the world
!
 How can you be ignorant of it
?
 Parse, compile, and assimilate
;
 perhaps the world will become your oyster
!
 An example
:

'Write once, run anywhere.'
 
The page was last updated February 19, 2008