{"id":169,"date":"2021-01-19T21:31:52","date_gmt":"2021-01-19T21:31:52","guid":{"rendered":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/?page_id=169"},"modified":"2021-01-19T21:43:22","modified_gmt":"2021-01-19T21:43:22","slug":"tokenizers","status":"publish","type":"page","link":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/course-topics\/i-o-files-streams\/tokenizers\/","title":{"rendered":"Tokenizers"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a6d4acaadcb6\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a6d4acaadcb6\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/course-topics\/i-o-files-streams\/tokenizers\/#The_StringTokenizer_Class\" >The StringTokenizer Class<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/course-topics\/i-o-files-streams\/tokenizers\/#The_StreamTokenizer_Class\" >The StreamTokenizer Class<\/a><\/li><\/ul><\/nav><\/div>\n\n<p class=\"wp-block-paragraph\">Just as you would like to format output data (using the <code><strong>Format<\/strong><\/code> family, or the <code><strong>printf<\/strong><\/code> family), you may also have a need to read in&nbsp;<em><strong>formatted input data<\/strong><\/em>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There are two classes that are useful for this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>(Package java.util)<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Object<strong>\n    StringTokenizer<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>(Package java.io)<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Object<strong>\n    StreamTokenizer<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">While they don&#8217;t provide full parsing capabilities they are still extremely useful.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><strong>Note: <\/strong>If you really would like to do some more serious parsing in Java, see <a rel=\"noreferrer noopener\" href=\"http:\/\/java-source.net\/open-source\/parser-generators\" target=\"_blank\">http:\/\/java-source.net\/open-source\/parser-generators<\/a> for how to find more capable parsers, such as ANTLR, JavaCC, and CUP, etc.<\/p><\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_StringTokenizer_Class\"><\/span>The StringTokenizer Class<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>public class StringTokenizer \n\timplements Enumeration<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The&nbsp;<strong>StringTokenizer<\/strong>&nbsp;class allows an application to break a string into tokens.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The set of delimiters (the characters that separate tokens) may be specified either at&nbsp;<strong>StringTokenizer<\/strong>&nbsp;creation time or on a per-token basis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An instance of&nbsp;<strong>StringTokenizer<\/strong>&nbsp;behaves in one of two ways, depending on whether it was created with the&nbsp;<strong>returnTokens<\/strong>&nbsp;flag having the value&nbsp;<strong>true<\/strong>&nbsp;or&nbsp;<strong>false<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>If the flag is false, delimiter characters serve to separate tokens. A token is a maximal sequence of consecutive characters that are not delimiters.<\/li><li>If the flag is true, delimiter characters are also considered to be tokens. A token is either one delimiter character, or a maximal sequence of consecutive characters that are not delimiters.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s a simple example:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; auto-links: false; title: ; quick-code: false; notranslate\" title=\"\">\npackage inputOutput;\n\nimport java.util.StringTokenizer;\n\npublic class TokenizeString\n{\n    public static void main(String&#x5B;] args)\n    {\n        StringTokenizer st = new StringTokenizer(&quot;Java is great!&quot;);\n        while (st.hasMoreTokens()) \n        {\n            System.out.println(st.nextToken());\n        }\n    }\n}\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">which prints out:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>Java\nis\ngreat!<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note that the&nbsp;<em>default delimiter characters<\/em>&nbsp;are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>space (&#8216; &#8216;)<\/li><li>tab (&#8216;\\t&#8217;)<\/li><li>newline (&#8216;\\n&#8217;)<\/li><li>carriage return (&#8216;\\r&#8217;)<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s another, modified, version, where we specify the delimiters explicitly:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; auto-links: false; title: ; quick-code: false; notranslate\" title=\"\">\npackage inputOutput;\n\nimport java.util.StringTokenizer;\n\npublic class TokenizeString\n{\n    public static void main(String&#x5B;] args)\n    {\n        String delimiters = &quot;.;!?;:&quot;;\n        String paragraph =\n        &quot;The spirit of Java is alive in the world! &quot; +\n        &quot;How can you be ignorant of it? &quot; +\n        &quot;Parse, compile, and assimilate; &quot; +\n        &quot;perhaps the world will become your oyster! &quot; +\n        &quot;An example: &#039;Write once, run anywhere.&#039;&quot;;\n        StringTokenizer st = \n                  new StringTokenizer(paragraph, delimiters);\n        while (st.hasMoreTokens()) \n        {\n            System.out.println(st.nextToken());\n        }\n    }\n}\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">It outputs the following:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>The spirit of Java is alive in the world\n How can you be ignorant of it\n Parse, compile, and assimilate\n perhaps the world will become your oyster\n An example\n 'Write once, run anywhere\n'<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note that, in the above example, we are considering tokens to be sentences (well, kind of &#8212; actually, we&#8217;re just using the punctuation as delimiters). In other words, tokens do not have to be words, or single characters only. A token may be any sequence of characters.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If we want the delimiter characters also to be returned as tokens, we can make a slight further modification:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; auto-links: false; title: ; quick-code: false; notranslate\" title=\"\">\npackage inputOutput;\n\nimport java.util.StringTokenizer;\n\npublic class TokenizeString\n{\n    public static void main(String&#x5B;] args)\n    {\n        String delimiters = &quot;.;!?;:&quot;;\n        String paragraph =\n        &quot;The spirit of Java is alive in the world! &quot; +\n        &quot;How can you be ignorant of it? &quot; +\n        &quot;Parse, compile, and assimilate; &quot; +\n        &quot;perhaps the world will become your oyster! &quot; +\n        &quot;An example: &#039;Write once, run anywhere.&#039;&quot;;\n        StringTokenizer st = \n                  new StringTokenizer(paragraph, delimiters, true);\n        while (st.hasMoreTokens()) \n        {\n            System.out.println(st.nextToken());\n        }\n    }\n}\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">which outputs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>The spirit of Java is alive in the world\n!\n How can you be ignorant of it\n?\n Parse, compile, and assimilate\n;\n perhaps the world will become your oyster\n!\n An example\n:\n 'Write once, run anywhere\n.\n'<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Perhaps we&#8217;d like to handle things inside single quotes a little better:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; auto-links: false; title: ; quick-code: false; notranslate\" title=\"\">\npackage inputOutput;\n\nimport java.util.StringTokenizer;\n\npublic class TokenizeString\n{\n    public static void main(String&#x5B;] args)\n    {\n        String delimiters = &quot;.;!?;:&#039;&quot;; \/\/ Added single quote \n        String paragraph =\n        &quot;The spirit of Java is alive in the world! &quot; +\n        &quot;How can you be ignorant of it? &quot; +\n        &quot;Parse, compile, and assimilate; &quot; +\n        &quot;perhaps the world will become your oyster! &quot; +\n        &quot;An example: &#039;Write once, run anywhere.&#039;&quot;;\n        StringTokenizer st = \n                  new StringTokenizer(paragraph, delimiters, true);\n        while (st.hasMoreTokens()) \n        {\n            String token = st.nextToken();\n            if (token.equals(&quot;&#039;&quot;))\n            {\n                if (st.hasMoreTokens())\n                    token += st.nextToken(&quot;&#039;&quot;); \/\/ Use different delimiter set.\n                if (st.hasMoreTokens())\n                    token += st.nextToken();\n            }\n            System.out.println(token);\n        }\n    }\n}\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">which outputs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>The spirit of Java is alive in the world\n!\n How can you be ignorant of it\n?\n Parse, compile, and assimilate\n;\n perhaps the world will become your oyster\n!\n An example\n:\n\n'Write once, run anywhere.'<\/strong><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_StreamTokenizer_Class\"><\/span>The StreamTokenizer Class<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>public class StreamTokenizer<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The&nbsp;<strong>StreamTokenizer<\/strong>&nbsp;class takes an input stream and parses it into &#8220;tokens&#8221;, allowing the tokens to be read one at a time. The parsing process is controlled by a table and a number of flags that can be set to various states. The stream tokenizer can recognize identifiers, numbers, quoted strings, and various comment styles.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Each byte read from the input stream is regarded as a character in the range &#8216;\\u0000&#8217; through &#8216;\\u00FF&#8217;. The character value is used to look up five possible attributes of the character:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>white space<\/li><li>alphabetic<\/li><li>numeric<\/li><li>string quote<\/li><li>comment character.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Each character can have zero or more of these attributes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In addition, an instance has four flags. These flags indicate:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Whether line terminators are to be returned as tokens or treated as white space that merely separates tokens.<\/li><li>Whether C-style comments are to be recognized and skipped.<\/li><li>Whether C++-style comments are to be recognized and skipped.<\/li><li>Whether the characters of identifiers are converted to lowercase.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A typical application first constructs an instance of this class, sets up the syntax tables, and then repeatedly loops calling the&nbsp;nextToken&nbsp;method in each iteration of the loop until it returns the value&nbsp;TT_EOF.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s an example, which parses Java files:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; auto-links: false; title: ; quick-code: false; notranslate\" title=\"\">\npackage inputOutput;\n\nimport java.io.BufferedReader;\nimport java.io.FileReader;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.io.StreamTokenizer;\n\npublic class TokenizeStream\n{\n    \/**\n    *   Main entry point.\n    *   This expects a single argument: \n    *     the name of the file to be tokenized.\n    *\/\n    public static void main(String&#x5B;] args)\n    {\n        String fileName = args&#x5B;0];\n        System.out.println(&quot;Parsing file: &quot; + fileName);\n        try\n        {\n            BufferedReader reader = \n                    new BufferedReader(\n                             new FileReader(fileName) );\n            StreamTokenizer st = new StreamTokenizer(reader);\n            st.slashSlashComments(true);\n            st.slashStarComments(true);\n            int token = StreamTokenizer.TT_NUMBER; \/\/ Just not TT_EOF\n            while (token != StreamTokenizer.TT_EOF) \n            {\n                token = st.nextToken();\n                switch (token)\n                {\n                    case StreamTokenizer.TT_NUMBER:\n                        System.out.println(&quot;Number: &quot; + st.nval);\n                        break;\n                        \n                    case StreamTokenizer.TT_WORD:\n                        System.out.println(&quot;Word: &quot; + st.sval);\n                        break;\n                        \n                    case StreamTokenizer.TT_EOL:\n                        System.out.println(&quot;End of line&quot;);\n                        break;\n                        \n                    case StreamTokenizer.TT_EOF:\n                        System.out.println(&quot;End of Stream&quot;);\n                        break;\n                        \n                    default:\n                        System.out.println(&quot;Character: &quot; + \n                                           (char)(token));\n                        break;\n                }\n            }\n        }\n        catch (FileNotFoundException e)\n        {\n            e.printStackTrace();\n        }\n        catch (IOException e)\n        {\n            e.printStackTrace();\n        }\n    }\n}\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">which, when told to parse itself, outputs (I&#8217;ve removed lots of lines in the middle, or it would get far too long):<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong>Parsing file: TokenizeStream.java\nWord: package\nWord: inputOutput\nCharacter: ;\nWord: import\nWord: java.io.BufferedReader\nCharacter: ;\nWord: import\nWord: java.io.FileReader\nCharacter: ;\nWord: import\nWord: java.io.FileNotFoundException\nCharacter: ;\nWord: import\nWord: java.io.IOException\nCharacter: ;\nWord: import\nWord: java.io.StreamTokenizer\nCharacter: ;\nWord: public\nWord: class\nWord: TokenizeStream\nCharacter: {\n\t...\nWord: catch\nCharacter: (\nWord: FileNotFoundException\nWord: e\nCharacter: )\nCharacter: {\nWord: e.printStackTrace\nCharacter: (\nCharacter: )\nCharacter: ;\nCharacter: }\nWord: catch\nCharacter: (\nWord: IOException\nWord: e\nCharacter: )\nCharacter: {\nWord: e.printStackTrace\nCharacter: (\nCharacter: )\nCharacter: ;\nCharacter: }\nCharacter: }\nCharacter: }\nEnd of Stream<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">You can use the\u00a0<code><strong>StreamTokenizer<\/strong><\/code>\u00a0class relatively simply, as above. However, many applications demand more sophisticated usage, where you specify the attributes of word characters, etc. Details are left to the imaginative reader&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Just as you would like to format output data (using the Format family, or the printf family), you may also have a need to read in&nbsp;formatted input data. There are two classes that are useful for this: (Package java.util) Object StringTokenizer (Package java.io) Object StreamTokenizer While they don&#8217;t provide full parsing capabilities they are still [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":67,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_eb_attr":"","ocean_front_end_style_editor":"no","ocean_post_layout":"left-sidebar","ocean_both_sidebars_style":"","ocean_both_sidebars_content_width":0,"ocean_both_sidebars_sidebars_width":0,"ocean_sidebar":"ocs-course-topics-sidebar","ocean_second_sidebar":"0","ocean_disable_margins":"enable","ocean_add_body_class":"","ocean_shortcode_before_top_bar":"","ocean_shortcode_after_top_bar":"","ocean_shortcode_before_header":"","ocean_shortcode_after_header":"","ocean_has_shortcode":"","ocean_shortcode_after_title":"","ocean_shortcode_before_footer_widgets":"","ocean_shortcode_after_footer_widgets":"","ocean_shortcode_before_footer_bottom":"","ocean_shortcode_after_footer_bottom":"","ocean_display_top_bar":"default","ocean_display_header":"default","ocean_header_style":"","ocean_center_header_left_menu":"0","ocean_custom_header_template":"0","ocean_custom_logo":0,"ocean_custom_retina_logo":0,"ocean_custom_logo_max_width":0,"ocean_custom_logo_tablet_max_width":0,"ocean_custom_logo_mobile_max_width":0,"ocean_custom_logo_max_height":0,"ocean_custom_logo_tablet_max_height":0,"ocean_custom_logo_mobile_max_height":0,"ocean_header_custom_menu":"0","ocean_menu_typo_font_family":"0","ocean_menu_typo_font_subset":"","ocean_menu_typo_font_size":0,"ocean_menu_typo_font_size_tablet":0,"ocean_menu_typo_font_size_mobile":0,"ocean_menu_typo_font_size_unit":"px","ocean_menu_typo_font_weight":"","ocean_menu_typo_font_weight_tablet":"","ocean_menu_typo_font_weight_mobile":"","ocean_menu_typo_transform":"","ocean_menu_typo_transform_tablet":"","ocean_menu_typo_transform_mobile":"","ocean_menu_typo_line_height":0,"ocean_menu_typo_line_height_tablet":0,"ocean_menu_typo_line_height_mobile":0,"ocean_menu_typo_line_height_unit":"","ocean_menu_typo_spacing":0,"ocean_menu_typo_spacing_tablet":0,"ocean_menu_typo_spacing_mobile":0,"ocean_menu_typo_spacing_unit":"","ocean_menu_link_color":"","ocean_menu_link_color_hover":"","ocean_menu_link_color_active":"","ocean_menu_link_background":"","ocean_menu_link_hover_background":"","ocean_menu_link_active_background":"","ocean_menu_social_links_bg":"","ocean_menu_social_hover_links_bg":"","ocean_menu_social_links_color":"","ocean_menu_social_hover_links_color":"","ocean_disable_title":"default","ocean_disable_heading":"default","ocean_post_title":"","ocean_post_subheading":"","ocean_post_title_style":"","ocean_post_title_background_color":"","ocean_post_title_background":0,"ocean_post_title_bg_image_position":"","ocean_post_title_bg_image_attachment":"","ocean_post_title_bg_image_repeat":"","ocean_post_title_bg_image_size":"","ocean_post_title_height":0,"ocean_post_title_bg_overlay":0.5,"ocean_post_title_bg_overlay_color":"","ocean_disable_breadcrumbs":"default","ocean_breadcrumbs_color":"","ocean_breadcrumbs_separator_color":"","ocean_breadcrumbs_links_color":"","ocean_breadcrumbs_links_hover_color":"","ocean_display_footer_widgets":"default","ocean_display_footer_bottom":"default","ocean_custom_footer_template":"0","footnotes":""},"class_list":["post-169","page","type-page","status-publish","hentry","entry"],"_links":{"self":[{"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/pages\/169","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/comments?post=169"}],"version-history":[{"count":3,"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/pages\/169\/revisions"}],"predecessor-version":[{"id":175,"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/pages\/169\/revisions\/175"}],"up":[{"embeddable":true,"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/pages\/67"}],"wp:attachment":[{"href":"https:\/\/bhiggs.x10hosting.com\/HighOctaneJava\/wp-json\/wp\/v2\/media?parent=169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}