Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #24374

Re: How to scan Java source texts?

Show all headers | View raw

On 06/11/2013 10:07 PM, Jeff Higgins wrote:
> On 06/11/2013 05:02 PM, Stefan Ram wrote:
>> Jeff Higgins <jeff@invalid.invalid> writes:
>>> On 06/11/2013 12:26 PM, Stefan Ram wrote:
>>>> I'd like to scan Java source texts, printing one token per line.
>>> Do you mean these tokens:
>>> <http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>
>>
>>    Yes.
>>
>>>> I thought it might be possible with the compiler API, and
>>>> have read that it can return an AST, but I do not know how
>>>> to just obtain the tokens from the source code AST.
>>> An AST is built from the tokens above.
>>
>>    Yes. That's why the compiler still might have a copy of
>>    the tokens lying around somewhere or might have a method
>>    to get the next token. I just can't find such a method.
>>
> I suspect, but don't know, that these tokens may have lost some
> of the information associated with their being 'InputElements'
> by the time the AST is constructed. It shouldn't be too hard
> to find a Java lexer that will output as you request.
> I'll look around when I have a little more time.
>

 From OpenJDK:

package com.sun.tools.javac.parser;

/** The lexical analyzer maps an input stream consisting of
  *  ASCII characters and Unicode escapes into a token sequence.
  *
  *  <p><b>This is NOT part of any supported API.
  *  If you write code that depends on this, you do so at your own risk.
  *  This code and its internal interfaces are subject to change or
  *  deletion without notice.</b>
  */
public class Scanner implements Lexer {

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 16:40 -0400
  Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 22:07 -0400
    Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:04 -0400
      Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:15 -0400
        Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:22 -0400
          Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:55 -0400
          Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 14:38 -0400
            Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 15:38 -0400

csiph-web