Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.java.programmer > #24374
| Path | csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!eternal-september.org!feeder.eternal-september.org!mx05.eternal-september.org!.POSTED!not-for-mail |
|---|---|
| From | Jeff Higgins <jeff@invalid.invalid> |
| Newsgroups | comp.lang.java.programmer |
| Subject | Re: How to scan Java source texts? |
| Date | Wed, 12 Jun 2013 04:04:27 -0400 |
| Organization | A noiseless patient Spider |
| Lines | 41 |
| Message-ID | <kp99n0$1hr$1@dont-email.me> (permalink) |
| References | <Java-Scanner-20130611180636@ram.dialup.fu-berlin.de> <kp81ku$kst$1@dont-email.me> <tokens-20130611230207@ram.dialup.fu-berlin.de> <kp8kpn$fk1$1@dont-email.me> |
| Mime-Version | 1.0 |
| Content-Type | text/plain; charset=ISO-8859-1; format=flowed |
| Content-Transfer-Encoding | 7bit |
| Injection-Date | Wed, 12 Jun 2013 07:58:24 +0000 (UTC) |
| Injection-Info | mx05.eternal-september.org; posting-host="6b9e92482af47766954d85fb77d372f5"; logging-data="1595"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/p8kYccj/p88p6hFRo6V6o6j9SBb2gvXY=" |
| User-Agent | Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130529 Icedove/17.0.5 |
| In-Reply-To | <kp8kpn$fk1$1@dont-email.me> |
| Cancel-Lock | sha1:KsHBjOcy6ty7JTwdU9IRjCEUTRU= |
| Xref | csiph.com comp.lang.java.programmer:24374 |
Show key headers only | View raw
On 06/11/2013 10:07 PM, Jeff Higgins wrote:
> On 06/11/2013 05:02 PM, Stefan Ram wrote:
>> Jeff Higgins <jeff@invalid.invalid> writes:
>>> On 06/11/2013 12:26 PM, Stefan Ram wrote:
>>>> I'd like to scan Java source texts, printing one token per line.
>>> Do you mean these tokens:
>>> <http://docs.oracle.com/javase/specs/jls/se7/html/jls-3.html#jls-3.5>
>>
>> Yes.
>>
>>>> I thought it might be possible with the compiler API, and
>>>> have read that it can return an AST, but I do not know how
>>>> to just obtain the tokens from the source code AST.
>>> An AST is built from the tokens above.
>>
>> Yes. That's why the compiler still might have a copy of
>> the tokens lying around somewhere or might have a method
>> to get the next token. I just can't find such a method.
>>
> I suspect, but don't know, that these tokens may have lost some
> of the information associated with their being 'InputElements'
> by the time the AST is constructed. It shouldn't be too hard
> to find a Java lexer that will output as you request.
> I'll look around when I have a little more time.
>
From OpenJDK:
package com.sun.tools.javac.parser;
/** The lexical analyzer maps an input stream consisting of
* ASCII characters and Unicode escapes into a token sequence.
*
* <p><b>This is NOT part of any supported API.
* If you write code that depends on this, you do so at your own risk.
* This code and its internal interfaces are subject to change or
* deletion without notice.</b>
*/
public class Scanner implements Lexer {
Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 16:40 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-11 22:07 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:04 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 04:15 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:22 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 12:55 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 14:38 -0400
Re: How to scan Java source texts? Jeff Higgins <jeff@invalid.invalid> - 2013-06-12 15:38 -0400
csiph-web