Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #24364 > unrolled thread

Re: How to scan Java source texts?

Back to article view | Back to comp.lang.java.programmer

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 16:40 -0400
    Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 22:07 -0400
      Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:04 -0400
        Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:15 -0400
          Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:22 -0400
            Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:55 -0400
            Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 14:38 -0400
              Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 15:38 -0400

#24364 — Re: How to scan Java source texts?

On 06/11/2013 12:26 PM, Stefan Ram wrote:
>    I'd like to scan Java source texts, printing one token per line.

Do you mean these tokens:
<http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>

>    I thought it might be possible with the compiler API, and
>    have read that it can return an AST, but I do not know how
>    to just obtain the tokens from the source code AST.

An AST is built from the tokens above.

[snip]

[toc] | [next] | [standalone]

#24370

On 06/11/2013 05:02 PM, Stefan Ram wrote:
> Jeff Higgins <jeff@invalid.invalid> writes:
>> On 06/11/2013 12:26 PM, Stefan Ram wrote:
>>> I'd like to scan Java source texts, printing one token per line.
>> Do you mean these tokens:
>> <http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>
>
>    Yes.
>
>>> I thought it might be possible with the compiler API, and
>>> have read that it can return an AST, but I do not know how
>>> to just obtain the tokens from the source code AST.
>> An AST is built from the tokens above.
>
>    Yes. That's why the compiler still might have a copy of
>    the tokens lying around somewhere or might have a method
>    to get the next token. I just can't find such a method.
>
I suspect, but don't know, that these tokens may have lost some
of the information associated with their being 'InputElements'
by the time the AST is constructed. It shouldn't be too hard
to find a Java lexer that will output as you request.
I'll look around when I have a little more time.

[toc] | [prev] | [next] | [standalone]

#24374

On 06/11/2013 10:07 PM, Jeff Higgins wrote:
> On 06/11/2013 05:02 PM, Stefan Ram wrote:
>> Jeff Higgins <jeff@invalid.invalid> writes:
>>> On 06/11/2013 12:26 PM, Stefan Ram wrote:
>>>> I'd like to scan Java source texts, printing one token per line.
>>> Do you mean these tokens:
>>> <http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>
>>
>>    Yes.
>>
>>>> I thought it might be possible with the compiler API, and
>>>> have read that it can return an AST, but I do not know how
>>>> to just obtain the tokens from the source code AST.
>>> An AST is built from the tokens above.
>>
>>    Yes. That's why the compiler still might have a copy of
>>    the tokens lying around somewhere or might have a method
>>    to get the next token. I just can't find such a method.
>>
> I suspect, but don't know, that these tokens may have lost some
> of the information associated with their being 'InputElements'
> by the time the AST is constructed. It shouldn't be too hard
> to find a Java lexer that will output as you request.
> I'll look around when I have a little more time.
>

 From OpenJDK:

package com.sun.tools.javac.parser;

/** The lexical analyzer maps an input stream consisting of
  *  ASCII characters and Unicode escapes into a token sequence.
  *
  *  <p><b>This is NOT part of any supported API.
  *  If you write code that depends on this, you do so at your own risk.
  *  This code and its internal interfaces are subject to change or
  *  deletion without notice.</b>
  */
public class Scanner implements Lexer {

[toc] | [prev] | [next] | [standalone]

#24375

On 06/12/2013 04:04 AM, Jeff Higgins wrote:
>
>  From OpenJDK:
>
<http://openjdk.java.net/groups/compiler>

[toc] | [prev] | [next] | [standalone]

#24384

On 06/12/2013 04:15 AM, Jeff Higgins wrote:
> On 06/12/2013 04:04 AM, Jeff Higgins wrote:
>>
>>  From OpenJDK:
>>
> <http://openjdk.java.net/groups/compiler>
It turns out to be surprisingly easy to build javac.
It shouldn't be too hard to add a commandline switch -tokens
and the requisite code to output tokens one per line as
they appear using the Lexer interface.
I don't see a way to do what you want using the existing API.

[toc] | [prev] | [next] | [standalone]

#24387

On 06/12/2013 12:44 PM, Stefan Ram wrote:
> Jeff Higgins <jeff@invalid.invalid> writes:
>> I don't see a way to do what you want using the existing API.
>
>    Thanks for your remarks!, which helped me
>    to find out that it can be done, once one
>    is willing to use the »com.sun....«-classes,
>    such as »Scanner«. »tools.jar« needs to be in
>    the classpath for this.
>
>    Now, there indeed is the risk that these classes
>    will change in future JDK versions. But still
>    I estimate them to be more stable than some
>    third-party libraries. For example, for the same
>    purpose I used a third-party program before that
>    now has not been adapted to Java >= 1.5, so that I
>    now needed to find some means to accomplish this
>    for Java >= 1.5.
>
Well, right there under my nose! :-O :-))
It was fun building my own compiler though!
Maybe a new language to go with it Jeffa!

[toc] | [prev] | [next] | [standalone]

#24390

On 06/12/2013 12:44 PM, Stefan Ram wrote:
> Jeff Higgins <jeff@invalid.invalid> writes:
>> I don't see a way to do what you want using the existing API.
>
>    Thanks for your remarks!, which helped me
>    to find out that it can be done, once one
>    is willing to use the »com.sun....«-classes,
>    such as »Scanner«. »tools.jar« needs to be in
>    the classpath for this.

The only problem I see now is gaining access to the protected
Scanner constructor outside of the com.sun.tools.javac.parser package.

Maybe I'll try extending my newly built javac
to include a -tokenize extension as above.


>    Now, there indeed is the risk that these classes
>    will change in future JDK versions. But still
>    I estimate them to be more stable than some
>    third-party libraries. For example, for the same
>    purpose I used a third-party program before that
>    now has not been adapted to Java >= 1.5, so that I
>    now needed to find some means to accomplish this
>    for Java >= 1.5.
>

[toc] | [prev] | [next] | [standalone]

#24393

On 06/12/2013 02:45 PM, Stefan Ram wrote:
> Jeff Higgins <jeff@invalid.invalid> writes:
>> The only problem I see now is gaining access to the protected
>> Scanner constructor outside of the com.sun.tools.javac.parser package.
>
>    You need to use a factory method of a ScannerFactory, and to
>    get that, you need to use yet another factory-like method of
>    ScannerFactory, which needs a Context, but this time you can
>    use Context's default constructor.
>
ScannerFactory.instance(new Context()).newScanner(args[0], true);
Wonderful! Thanks.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web