Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!eternal-september.org!feeder.eternal-september.org!mx05.eternal-september.org!.POSTED!not-for-mail
From: Jeff Higgins <jeff@invalid.invalid>
Newsgroups: comp.lang.java.programmer
Subject: Re: How to scan Java source texts?
Date: Wed, 12 Jun 2013 04:04:27 -0400
Organization: A noiseless patient Spider
Lines: 41
Message-ID: <kp99n0$1hr$1@dont-email.me>
References: <Java-Scanner-20130611180636@ram.dialup.fu-berlin.de> <kp81ku$kst$1@dont-email.me> <tokens-20130611230207@ram.dialup.fu-berlin.de> <kp8kpn$fk1$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Wed, 12 Jun 2013 07:58:24 +0000 (UTC)
Injection-Info: mx05.eternal-september.org; posting-host="6b9e92482af47766954d85fb77d372f5"; logging-data="1595"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/p8kYccj/p88p6hFRo6V6o6j9SBb2gvXY="
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130529 Icedove/17.0.5
In-Reply-To: <kp8kpn$fk1$1@dont-email.me>
Cancel-Lock: sha1:KsHBjOcy6ty7JTwdU9IRjCEUTRU=
Xref: csiph.com comp.lang.java.programmer:24374

On 06/11/2013 10:07 PM, Jeff Higgins wrote:
> On 06/11/2013 05:02 PM, Stefan Ram wrote:
>> Jeff Higgins <jeff@invalid.invalid> writes:
>>> On 06/11/2013 12:26 PM, Stefan Ram wrote:
>>>> I'd like to scan Java source texts, printing one token per line.
>>> Do you mean these tokens:
>>> <http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>
>>
>>    Yes.
>>
>>>> I thought it might be possible with the compiler API, and
>>>> have read that it can return an AST, but I do not know how
>>>> to just obtain the tokens from the source code AST.
>>> An AST is built from the tokens above.
>>
>>    Yes. That's why the compiler still might have a copy of
>>    the tokens lying around somewhere or might have a method
>>    to get the next token. I just can't find such a method.
>>
> I suspect, but don't know, that these tokens may have lost some
> of the information associated with their being 'InputElements'
> by the time the AST is constructed. It shouldn't be too hard
> to find a Java lexer that will output as you request.
> I'll look around when I have a little more time.
>

 From OpenJDK:

package com.sun.tools.javac.parser;

/** The lexical analyzer maps an input stream consisting of
  *  ASCII characters and Unicode escapes into a token sequence.
  *
  *  <p><b>This is NOT part of any supported API.
  *  If you write code that depends on this, you do so at your own risk.
  *  This code and its internal interfaces are subject to change or
  *  deletion without notice.</b>
  */
public class Scanner implements Lexer {