Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #91736 > unrolled thread

Parser needed.

Back to article view | Back to comp.lang.python

Contents

  Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 02:29 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 02:55 +0200
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:02 +0200
        Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:10 +0200
          Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:16 +0200
            Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:27 +0200
              Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:29 +0200
                Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:31 +0200
                  Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:34 +0200
                    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:02 +0200
                      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:22 +0200
                        Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:27 +0200
                          Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:39 +0200
                  Re: Parser needed. Joel Goldstick <joel.goldstick@gmail.com> - 2015-06-01 21:41 -0400
                    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:04 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 02:56 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:01 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:03 +0200
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:04 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:07 +0200
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:07 +0200
    Re: Parser needed. Michael Torrie <torriem@gmail.com> - 2015-06-01 19:12 -0600
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:19 +0200
        Re: Parser needed. Michael Torrie <torriem@gmail.com> - 2015-06-01 19:31 -0600
          Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 03:35 +0200
            Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:11 +0200
              Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 04:20 +0200
        Re: Parser needed. Grant Edwards <invalid@invalid.invalid> - 2015-06-02 13:48 +0000
    Re: Parser needed. Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-06-02 04:42 +0100
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:23 +0200
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:29 +0200
        Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:39 +0200
          Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:45 +0200
            Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:51 +0200
              Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 06:53 +0200
            Re: Parser needed. MRAB <python@mrabarnett.plus.com> - 2015-06-02 17:43 +0100
              Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-02 19:09 +0200
            Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 18:01 +0200
    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:02 +0200
      Re: Parser needed. Steven D'Aprano <steve@pearwood.info> - 2015-06-05 03:14 +1000
        Re: Parser needed. Michael Torrie <torriem@gmail.com> - 2015-06-04 11:25 -0600
      Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:23 +0200
        Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:34 +0200
          Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:38 +0200
            Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:44 +0200
              Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 19:50 +0200
                Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-04 20:00 +0200
                  Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-09 04:30 +0200
                    Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-09 14:20 +0200
                      Re: Parser needed. Michael Torrie <torriem@gmail.com> - 2015-06-09 17:33 -0600
                        Re: Parser needed. Rustom Mody <rustompmody@gmail.com> - 2015-06-09 19:08 -0700
                        Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-11 13:15 +0200
                          Re: Parser needed. Joel Goldstick <joel.goldstick@gmail.com> - 2015-06-11 08:35 -0400
                          Re: Parser needed. Larry Martell <larry.martell@gmail.com> - 2015-06-11 08:38 -0400
                            Re: Parser needed. Rustom Mody <rustompmody@gmail.com> - 2015-06-11 08:35 -0700
                    Re: Parser needed. Tony the Tiger <tony@tiger.invalid> - 2015-06-14 16:01 +0000
                  Re: Parser needed. "Skybuck Flying" <skybuck2000@hotmail.com> - 2015-06-09 04:31 +0200

Page 1 of 3  [1] 2 3  Next page →

#91736 — Parser needed.

Hello,

I need some kind of parser and some kind of way to access the data contained 
in a file like the one below:

(text file):

http://www.skybuck.org/Games/StartrekOnline/Parser/SpaceFleetAlertEnemyExample.demo

I am interested in learning the entity numbers:

"EntityRef <number>"

and their positions:

"
Pos <x>, <z>, <y>
"

I am also interested in the creation of these entities, especially their 
type:

"entityTypeEnum <type>"

So basically the parser should produce a few lists:

EntityNumber = []
EntityType = []
EntityPositionX = []
EntityPositionY = []
EntityPositionZ = []

Any extra information about entities is welcome.

The parser should be able to parse a textfile of somewhere between 20.000 
lines to 50.000 lines in about 1 to 2 seconds.

My environment is SikuliX 1.1

Bye,
  Skybuck. 

[toc] | [next] | [standalone]

#91740

I will try and help for as far as possible, maybe I will end up writing it 
myself in the process.

The first problem seems to be, how to read a textfile into python all at 
once... and then perhaps process them per line.

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91743

This will probably help:

http://stackoverflow.com/questions/14676265/how-to-read-text-file-into-a-list-or-array-with-python

text_file = open("filename.dat", "r")
lines = text_file.readlines()
print lines
print len(lines)
text_file.close() Usually I like to consult officals docs though...Bye, 
Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91750

Readlines is indeed documented as reading EOF:

http://www.tutorialspoint.com/python/file_methods.htm

Bye,
  skybuck.

"Skybuck Flying"  wrote in message 
news:37cda$556d009d$5419aafe$38529@news.ziggo.nl...

This will probably help:

http://stackoverflow.com/questions/14676265/how-to-read-text-file-into-a-list-or-array-with-python

text_file = open("filename.dat", "r")
lines = text_file.readlines()
print lines
print len(lines)
text_file.close() Usually I like to consult officals docs though...Bye,
Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91754

So far this is what I got... I like to name things for what they are so 
FileObject I like better than something abstract/weird like "text file", my 
code:

BotDemoFolder = "C:\\Games\\Startrek Online\\Startrek Online\\Cryptic 
Studios\\Star Trek Online\\Live\\demos"
BotDemoFile = "SpaceFleetAlert.demo"

def Main():
DemoFilePath = BotDemoFolder + "\\" + BotDemoFile

FileObject = open( DemoFilePath, "r")

DemoLines = FileObject.readlines()

print DemoLines
print len(Demolines)

FileObject.close()
return

Main()

So far it took multiple seconds to print the demo lines...

Hopefully that's just a "printing" issue to the console and not a processing 
limitation of python.

I know python/sikulix can do about 500.000 if statements per second.

I shall have to add some timing code, to rule out sikulix startup time...

Hmmm..

Bye,
  Skybuck.

[toc] | [prev] | [next] | [standalone]

#91757

Ok, so far so good, a little start has been made.

Text file is read into lines... I am not so sure if this is a good idea...

Maybe it's easier if the entire file is one gigant array of characters 
instead of fragmented lines.

However I don't know yet exactly how to read as one gigant array of 
characters.

I don't really like this line approach but maybe it's nice:

But I am not faced with a new problem: "How to process individually lines".

So this solution has created more problems than it solves :)

BotDemoFolder = "C:\\Games\\Startrek Online\\Startrek Online\\Cryptic 
Studios\\Star Trek Online\\Live\\demos"
BotDemoFile = "SpaceFleetAlert.demo"

import time

def ParseDemoLines( ParaLines ):
print "Parsing " + str( len(ParaLines) ) + " lines."

for LineIndex in range(0, len(ParaLines)):
  if "{" in ParaLines[LineIndex]: # how to process a line.. hmmm...
   print "yup"
return

def Main():
DemoFilePath = BotDemoFolder + "\\" + BotDemoFile

FileObject = open( DemoFilePath, "r")

DemoLines = FileObject.readlines()

ParseDemoLines( DemoLines )

# print DemoLines

FileObject.close()
return

print "program started"

Tick1 = time.time()
Main()
Tick2 = time.time()

Seconds = Tick2 - Tick1

print "Time in seconds: " + str(Seconds)

print "program finished"

Bye,
  Skybuck.

[toc] | [prev] | [next] | [standalone]

#91758

Since the file is probably ascii... not sure... I might come away with:

"
file.read([size])
Reads at most size bytes from the file (less if the read hits EOF before 
obtaining size bytes).

"

The doc does not mention is size is optionally... I will try and leave it 
out, see what happens, otherwise a big number will have to be given or so... 
or perhaps retrieve file size and add it.



Bye,

  Skybuck.

[toc] | [prev] | [next] | [standalone]

#91760

Yes this will work:

DemoChars = FileObject.read()

I think this is a cleaner solution. EOL can be ignored and focusses on { } 
and stuff like that... when extracting information EOL could be used as 
well.

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91762

Nice char based code:

BotDemoFolder = "C:\\Games\\Startrek Online\\Startrek Online\\Cryptic 
Studios\\Star Trek Online\\Live\\demos"
BotDemoFile = "SpaceFleetAlert.demo"

import time

def ParseDemoLines( ParaLines ):
print "Parsing " + str( len(ParaLines) ) + " lines."

for LineIndex in range(0, len(ParaLines)):
  if "{" in ParaLines[LineIndex]: # how to process a line.. hmmm...
   print "yup"
return

def ParseDemoChars( ParaChars ):
print "Parsing " + str( len(ParaChars) ) + " chars."

for CharIndex in range(0, len(ParaChars)):
  if ParaChars[CharIndex] == "{":
   print "yup"
return

def Main():
DemoFilePath = BotDemoFolder + "\\" + BotDemoFile

FileObject = open( DemoFilePath, "r")

# DemoLines = FileObject.readlines()
# ParseDemoLines( DemoLines )

DemoChars = FileObject.read()
ParseDemoChars( DemoChars )

FileObject.close()
return

print "program started"

Tick1 = time.time()
Main()
Tick2 = time.time()

Seconds = Tick2 - Tick1

print "Time in seconds: " + str(Seconds)

print "program finished"

[toc] | [prev] | [next] | [standalone]

#91766

I am trying to rush it a bit but things are starting to go to hell.

SikuliX needs to be restarted continously to deal with bugs.

Plus Eclipse breakpoints cannot be set.

So both IDEs totally suck for any complex code development.

Plus I can already tell this is going to be insanely slow in python for some 
reason.

I am starting to have serious doubts for python as a parser platform:

This simple piece of code already takes 10 seconds for just 120 characters ? 
HOLYSHIT ?!

Bad sign.

BotDemoFolder = "C:\\Games\\Startrek Online\\Startrek Online\\Cryptic 
Studios\\Star Trek Online\\Live\\demos"
BotDemoFile = "SpaceFleetAlertEnemyExample.demo"


import time

def ParseDemoLines( ParaLines ):
print "Parsing " + str( len(ParaLines) ) + " lines."

for LineIndex in range(0, len(ParaLines)):
  if "{" in ParaLines[LineIndex]: # how to process a line.. hmmm...
   print "yup"
return

def ParseStructureOpen( ParaChars, CharIndex ):
# find eol before it and re-read characters until eol this is the structure 
name
while (CharIndex >= 0) and (CharIndex < len(ParaChars)) and 
(ParaChars[CharIndex] <> "\n"):
  print "CharIndex: " + str(CharIndex)
  CharIndex = CharIndex - 1


# re-read characters for structure name
StructureName = ""
while (CharIndex < len(ParaChars)) and (ParaChars[CharIndex] <> "\n"):
  print "CharIndex: " + str(CharIndex)
  StructureName = StructureName + ParaChars[CharIndex]
  CharIndex = CharIndex + 1

print StructureName
return

def ParseStructureClose( ParaChars, CharIndex ):
pass
return

def ParseDemoChars( ParaChars ):
print "Parsing " + str( len(ParaChars) ) + " chars."
for CharIndex in range(0, len(ParaChars)):
  print "Parsing char index: " + str(CharIndex)
  if ParaChars[CharIndex] == "{":
   ParseStructureOpen( ParaChars, CharIndex )
  elif ParaChars[CharIndex] == "}":
   ParseStructureClose( ParaChars, CharIndex )
return

def Main():
DemoFilePath = BotDemoFolder + "\\" + BotDemoFile

FileObject = open( DemoFilePath, "r")

# DemoLines = FileObject.readlines()
# ParseDemoLines( DemoLines )

DemoChars = FileObject.read()
ParseDemoChars( DemoChars )

FileObject.close()
return

print "program started"

Tick1 = time.time()
Main()
Tick2 = time.time()

Seconds = Tick2 - Tick1

print "Time in seconds: " + str(Seconds)

print "program finished"


Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91772

Test program shows, python/sikulix can handle about 3m chars per second so I 
must continue with a parser attempt.

Apperently there is some bottleneck/slowdown in posted code...

I may have to slow down a bit take more time... and perhaps use global vars 
or so... instead of parameters.

Maybe that will help, not sure.

Kinda strange...  how test program is super fast in other thread and this 
one is slow.

Maybe it's the printing that's slow... Hmm...

I shall test print speed instead to figure this out.

Bye,
  Skybuck.

[toc] | [prev] | [next] | [standalone]

#91773

Apperently print speed has some exponential time issue in SikuliX kinda 
weird.

But at least processing is high... however, debug console/print is only 
thing I got to diagnose problems...

Could get time-wise nasty... hmmm...

Perhaps I should make a small little parser in Delphi and port it to Python 
instead...

But it a bit risky if not using python only code hmm...

# test print speed.

# 1000 prints costs 3 seconds
# 10.000 prints costed 60 seconds.

# this is weird indeed, seems to be some kind of exponential growth.

import time

MaxPrints = 10000
def PrintMeARiver():
for vIndex in range(0, MaxPrints):
  print str(vIndex)
return

def Main():
PrintMeARiver()
return

print "program started"
Tick1 = time.time()
Main()
Tick2 = time.time()
Seconds = Tick2 - Tick1
print "Seconds: " + str(Seconds)
print "program finished"


Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91774

Perhaps the list.append operation is the cause of the slowdown, or perhaps 
my algo is flawed.

No idea yet with slowdown of parser code so far.

Instead of trying to debug everything going to add some time outputs to the 
whole thing to get to bottom of this ;)

Ok another sikuli failure... I just won't run the script... this is 
seriously getting on my tits...and I am seriously getting on your tits.

Time to nuke sikuli for now... and either give up on this idea...

Or try it in a more serious dev environment cause this sucks balls.

To bad this was a failure, bye bye.

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91765

On Mon, Jun 1, 2015 at 9:31 PM, Skybuck Flying <skybuck2000@hotmail.com> wrote:
> Yes this will work:
>
> DemoChars = FileObject.read()
>
> I think this is a cleaner solution. EOL can be ignored and focusses on { }
> and stuff like that... when extracting information EOL could be used as
> well.
>
>
> Bye,
>  Skybuck.
> --
> https://mail.python.org/mailman/listinfo/python-list

This is kind of a q and a mailing list.  I'm baffled by all of your
posts, that seem to be a conversation with yourself.  Do you have a
question you need help with?

-- 
Joel Goldstick
http://joelgoldstick.com

[toc] | [prev] | [next] | [standalone]

#91767

"Joel Goldstick"  wrote in message 
news:mailman.34.1433209315.13271.python-list@python.org...

On Mon, Jun 1, 2015 at 9:31 PM, Skybuck Flying <skybuck2000@hotmail.com> 
wrote:
> Yes this will work:
>
> DemoChars = FileObject.read()
>
> I think this is a cleaner solution. EOL can be ignored and focusses on { }
> and stuff like that... when extracting information EOL could be used as
> well.
>
>
> Bye,
>  Skybuck.
> --
> https://mail.python.org/mailman/listinfo/python-list

"
This is kind of a q and a mailing list.  I'm baffled by all of your
posts, that seem to be a conversation with yourself.  Do you have a
question you need help with?
"

All of my postings contain things I run into.

Missing documentation, python limitations, and other problems.

Algo design considerations etc.

What I need help with is a parser for the file mentioned in the link.

So far it looks bad for python.

I may give the parser frameworks a try to see if they are speedier then 120 
chars per second ?! :P :)

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91741

Some further information about the demo file:

It seems to be split up into "creation sections"

and "update" sections.

The update sections contain the positions.

The update sections also contain a reference number to the created entities.

(There is one empty creation section but that's because I cut it out... so 
it can be ignored for now).

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91742

This can help with parsing the C like bracketted data:

http://stackoverflow.com/questions/1651487/python-parsing-bracketed-blocks

algo:

"
For each string in the array:
    Find the first '{'. If there is none, leave that string alone.
    Init a counter to 0.
    For each character in the string:
        If you see a '{', increment the counter.
        If you see a '}', decrement the counter.
        If the counter reaches 0, break.
    Here, if your counter is not 0, you have invalid input (unbalanced 
brackets)
    If it is, then take the string from the first '{' up to the '}' that put 
the
     counter at 0, and that is a new element in your array."Not sure what 
this is, but more points: Or this pyparsing version:>>> from pyparsing 
import nestedExpr
>>> txt = "{ { a } { b } { { { c } } } }"
>>>
>>> nestedExpr('{','}').parseString(txt).asList()
[[['a'], ['b'], [[['c']]]]]
>>> Not sure if I want a data structure like that... not sure what data 
>>> structure it is anyway... seems to be a list in a list in a list... 
>>> hmm.... Bye,  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91745

From examining the demo file it seems each bracketted data sections if 
pre-header/pre-fixed with it's name.

So it's somewhat similar to a C structure:

structure
{

}

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91746

Except for the first bracket... it has no structure name.

Perhaps the demo filename could be used as structure name or just be left 
empty.

Bye,
  Skybuck.


"Skybuck Flying"  wrote in message 
news:aef84$556d00e7$5419aafe$39141@news.ziggo.nl...

From examining the demo file it seems each bracketted data sections if
pre-header/pre-fixed with it's name.

So it's somewhat similar to a C structure:

structure
{

}

Bye,
  Skybuck. 

[toc] | [prev] | [next] | [standalone]

#91747

I really like this python help file, it has served me well so far:

http://www.tutorialspoint.com/python/python_functions.htm

It's unusually better than the rest on the net ;)

It matches nicely Sikuli's python's capabilities.

Bye,
  Skybuck :)

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web