Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #44439 > unrolled thread

File Read issue by using module binascii

Back to article view | Back to comp.lang.python

Contents

  File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-26 20:57 -0700
    Re: File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-26 21:22 -0700
      Re: File Read issue by using module binascii Fábio Santos <fabiosantosart@gmail.com> - 2013-04-27 10:56 +0100
        Re: File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-27 03:42 -0700
    Re: File Read issue by using module binascii Peter Otten <__peter__@web.de> - 2013-04-27 12:57 +0200
      Re: File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-27 04:23 -0700
        Re: File Read issue by using module binascii Fábio Santos <fabiosantosart@gmail.com> - 2013-04-27 12:40 +0100
        Re: File Read issue by using module binascii Peter Otten <__peter__@web.de> - 2013-04-27 14:01 +0200
    Re: File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-27 05:46 -0700
    Re: File Read issue by using module binascii Tim Roberts <timr@probo.com> - 2013-04-27 21:34 -0700
      Re: File Read issue by using module binascii Peter Otten <__peter__@web.de> - 2013-04-28 09:42 +0200
      Re: File Read issue by using module binascii jt@toerring.de (Jens Thoms Toerring) - 2013-04-28 12:04 +0000
        Re: File Read issue by using module binascii Jimmie He <jimmie.he@gmail.com> - 2013-04-28 06:32 -0700

#44439 — File Read issue by using module binascii

When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
Thank you in advance!

Python Code as below!!

import binascii

def read_bmp():
    f = open('example.bmp','rb')
    rawdata = f.read()                       #f.read(1000) is ok
    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
    bsstr = bin (int(hexstr,16))[2:]
    f.close()
    print('bin: ',bsstr,type(bsstr))
    return

[toc] | [next] | [standalone]

#44440

when I commet the line of "print('bin: ',bsstr,type(bsstr)) ",it can be run,so maybe the problem is the memory allocate of so long strings......Am I right?

在 2013年4月27日星期六UTC+8上午11时57分45秒，Jimmie He写道：
> When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
> 
> Thank you in advance!
> 
> 
> 
> Python Code as below!!
> 
> 
> 
> import binascii
> 
> 
> 
> def read_bmp():
> 
>     f = open('example.bmp','rb')
> 
>     rawdata = f.read()                       #f.read(1000) is ok
> 
>     hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
> 
>     bsstr = bin (int(hexstr,16))[2:]
> 
>     f.close()
> 
>     print('bin: ',bsstr,type(bsstr))
> 
>     return

[toc] | [prev] | [next] | [standalone]

#44442

[Multipart message — attachments visible in raw view] — view raw

It may be that you are printing too much data at once. 100k is a bit too
much to have in memory but it should run anyway. But your console may be
having trouble. Try looping over small chunks of the file and printing them
one at a time. Use a while loop. I do know that in windows the console is
not very efficient at printing so when I print too much data the console
itself starts taking up a lot of processor time.
On 27 Apr 2013 05:28, "Jimmie He" <jimmie.he@gmail.com> wrote:

> when I commet the line of "print('bin: ',bsstr,type(bsstr)) ",it can be
> run,so maybe the problem is the memory allocate of so long strings......Am
> I right?
>
> 在 2013年4月27日星期六UTC+8上午11时57分45秒，Jimmie He写道：
> > When I run the readbmp on an example.bmp(about 100k),the Shell is become
> to "No respose",when I change f.read() to f.read(1000),it is ok,could
> someone tell me the excat reason for this?
> >
> > Thank you in advance!
> >
> >
> >
> > Python Code as below!!
> >
> >
> >
> > import binascii
> >
> >
> >
> > def read_bmp():
> >
> >     f = open('example.bmp','rb')
> >
> >     rawdata = f.read()                       #f.read(1000) is ok
> >
> >     hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
> >
> >     bsstr = bin (int(hexstr,16))[2:]
> >
> >     f.close()
> >
> >     print('bin: ',bsstr,type(bsstr))
> >
> >     return
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#44443

What you said should make sense and I've already correct my code by your advice,thanks for your response!


在 2013年4月27日星期六UTC+8下午5时56分08秒，Fábio Santos写道：
> It may be that you are printing too much data at once. 100k is a bit too much to have in memory but it should run anyway. But your console may be having trouble. Try looping over small chunks of the file and printing them one at a time. Use a while loop. I do know that in windows the console is not very efficient at printing so when I print too much data the console itself starts taking up a lot of processor time.
> 
> 
> On 27 Apr 2013 05:28, "Jimmie He" <jimm...@gmail.com> wrote:
> 
> when I commet the line of "print('bin: ',bsstr,type(bsstr)) ",it can be run,so maybe the problem is the memory allocate of so long strings......Am I right?
> 
> 
> 
> 在 2013年4月27日星期六UTC+8上午11时57分45秒，Jimmie He写道：
> 
> > When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
> 
> >
> 
> > Thank you in advance!
> 
> >
> 
> >
> 
> >
> 
> > Python Code as below!!
> 
> >
> 
> >
> 
> >
> 
> > import binascii
> 
> >
> 
> >
> 
> >
> 
> > def read_bmp():
> 
> >
> 
> >     f = open('example.bmp','rb')
> 
> >
> 
> >     rawdata = f.read()                       #f.read(1000) is ok
> 
> >
> 
> >     hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
> 
> >
> 
> >     bsstr = bin (int(hexstr,16))[2:]
> 
> >
> 
> >     f.close()
> 
> >
> 
> >     print('bin: ',bsstr,type(bsstr))
> 
> >
> 
> >     return
> 
> --
> 
> http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#44444

Jimmie He wrote:

> When I run the readbmp on an example.bmp(about 100k),the Shell is become
> to "No respose",when I change f.read() to f.read(1000),it is ok,could
> someone tell me the excat reason for this? Thank you in advance!
> 
> Python Code as below!!
> 
> import binascii
> 
> def read_bmp():
>     f = open('example.bmp','rb')
>     rawdata = f.read()                       #f.read(1000) is ok
>     hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
>     bsstr = bin (int(hexstr,16))[2:]
>     f.close()
>     print('bin: ',bsstr,type(bsstr))
>     return

What shell are you using? The one provided by Idle?

[toc] | [prev] | [next] | [standalone]

#44445

> What shell are you using? The one provided by Idle?

Yes. I use IDLE,the python version is 3.3.1.What else could I use??

[toc] | [prev] | [next] | [standalone]

#44446

[Multipart message — attachments visible in raw view] — view raw

You could use your operating system's shell. Even if it is windows, it
should be a lot better and faster, and thus not block so easily.
On 27 Apr 2013 12:28, "Jimmie He" <jimmie.he@gmail.com> wrote:

>
> > What shell are you using? The one provided by Idle?
>
> Yes. I use IDLE,the python version is 3.3.1.What else could I use??
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#44447

Jimmie He wrote:

>> What shell are you using? The one provided by Idle?
> 
> Yes. I use IDLE,the python version is 3.3.1.What else could I use??

The shell provided by the operating system is usually much faster. When I 
modify your code to

import binascii

def read_bmp():
    f = open('example.bmp','rb')
    rawdata = f.read()                       #f.read(1000) is ok
    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
    bsstr = bin (int(hexstr,16))[2:]
    f.close()
    print('bin: ',bsstr,type(bsstr))
    return

if __name__ == "__main__":
    read_bmp()

and generate a dummy example.bmp with 2**20 (about 1 million) bytes it takes 
about 2 seconds to terminate -- on hardware that is quite old. If I redirect 
the output it is even faster:

$ time python3 bmp_to_bin.py > /dev/null

real    0m0.766s
user    0m0.300s
sys     0m0.180s

I am a Linux user, but expect similar numbers on Windows (in the DOS box or 
one of its successors).

I have considered filing a bug* to ask for a tweak in idle that improves its 
responsiveness, but first wanted you to confirm that this was indeed the 
problem.

(*) on http://bugs.python.org, if you want to do it yourself

[toc] | [prev] | [next] | [standalone]

#44449

Peter,
  Thanks for your test in details. Because I'm an newbie in Python and I can not conform whether it is an bug. Here I attach my code include my emample.bmp as link below.
https://hotfile.com/dl/204795514/62fed41/BMPTool.zip.html

[toc] | [prev] | [next] | [standalone]

#44462

Jimmie He <jimmie.he@gmail.com> wrote:

>When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
>Thank you in advance!
>
>Python Code as below!!
>
>import binascii
>
>def read_bmp():
>    f = open('example.bmp','rb')
>    rawdata = f.read()                       #f.read(1000) is ok
>    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
>    bsstr = bin (int(hexstr,16))[2:]

I suspect the root of the problem here is that you don't understand what
this is actually doing.  You should run this code in the command-line
interpreter, one line at a time, and print the results.

The "read" instruction produces a string with 100k bytes.  The b2a_hex then
produces a string with 200k bytes.  Then, int(hexstr,16) takes that 200,000
byte hex string and converts it to an integer, roughly equal to 10 to the
240,000 power, a number with some 240,000 decimal digits.  You then convert
that integer to a binary string.  That string will contain 800,000 bytes.
You then drop the first two characters and print the other 799,998 bytes,
each of which will be either '0' or '1'.

I am absolutely, positively convinced that's not what you wanted to do.
What point is there in printing out the binary equavalent of a bitmap?

Even if you did, it would be much quicker for you to do the conversion one
byte at a time, completely skipping the conversion to hex and then the
creation of a massive multi-precision number.  Example:

    f = open('example.bmp','rb')
    rawdata = f.read()
    bsstr = []
    for b in rawdata:
        bsstr.append( bin(ord(b)) )
    bsstr = ''.join(bsstr)

or even:
    f = open('example.bmp','rb')
    bsstr = ''.join( bin(ord(b))[2:] for b in f.read() )
-- 
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[toc] | [prev] | [next] | [standalone]

#44464

Tim Roberts wrote:

> Jimmie He <jimmie.he@gmail.com> wrote:
> 
>>When I run the readbmp on an example.bmp(about 100k),the Shell is become
>>to "No respose",when I change f.read() to f.read(1000),it is ok,could
>>someone tell me the excat reason for this? Thank you in advance!
>>
>>Python Code as below!!
>>
>>import binascii
>>
>>def read_bmp():
>>    f = open('example.bmp','rb')
>>    rawdata = f.read()                       #f.read(1000) is ok
>>    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
>>    bsstr = bin (int(hexstr,16))[2:]
> 
> I suspect the root of the problem here is that you don't understand what
> this is actually doing.  You should run this code in the command-line
> interpreter, one line at a time, and print the results.
> 
> The "read" instruction produces a string with 100k bytes.  The b2a_hex
> then
> produces a string with 200k bytes.  Then, int(hexstr,16) takes that
> 200,000 byte hex string and converts it to an integer, roughly equal to 10
> to the
> 240,000 power, a number with some 240,000 decimal digits.  You then
> convert
> that integer to a binary string.  That string will contain 800,000 bytes.
> You then drop the first two characters and print the other 799,998 bytes,
> each of which will be either '0' or '1'.
> 
> I am absolutely, positively convinced that's not what you wanted to do.
> What point is there in printing out the binary equavalent of a bitmap?
> 
> Even if you did, it would be much quicker for you to do the conversion one
> byte at a time, completely skipping the conversion to hex and then the
> creation of a massive multi-precision number.  Example:

Hm, if you fix the long integer arithmetic "problem" you should also attack 
the unbounded memory consumption problem in general ;)

>     f = open('example.bmp','rb')
>     rawdata = f.read()
>     bsstr = []
>     for b in rawdata:
>         bsstr.append( bin(ord(b)) )
>     bsstr = ''.join(bsstr)
> 
> or even:
>     f = open('example.bmp','rb')
>     bsstr = ''.join( bin(ord(b))[2:] for b in f.read() )

Yes, the original is horrible newbie code ;) but that's what you tend to 
write while learning to program -- and python can handle it alright. On the 
other hand, Idle becomes unresponsive when I do

>>> print("a"*10**6)

in its shell. I'm still investigating, but the problem seems to be that it's 
a single line.

>>> print(("a"*100+"\n") * 10**4)

takes under 7 secs. Not as good as konsole (KDE's terminal emulation) which 
finishes in 0.5 secs, but acceptable.

[toc] | [prev] | [next] | [standalone]

#44468

Tim Roberts <timr@probo.com> wrote:
> Jimmie He <jimmie.he@gmail.com> wrote:

> >When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
> >Thank you in advance!
> >
> >Python Code as below!!
> >
> >import binascii
> >
> >def read_bmp():
> >    f = open('example.bmp','rb')
> >    rawdata = f.read()                       #f.read(1000) is ok
> >    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
> >    bsstr = bin (int(hexstr,16))[2:]

> I suspect the root of the problem here is that you don't understand what
> this is actually doing.  You should run this code in the command-line
> interpreter, one line at a time, and print the results.

> The "read" instruction produces a string with 100k bytes.  The b2a_hex then
> produces a string with 200k bytes.  Then, int(hexstr,16) takes that 200,000
> byte hex string and converts it to an integer, roughly equal to 10 to the
> 240,000 power, a number with some 240,000 decimal digits.  You then convert
> that integer to a binary string.  That string will contain 800,000 bytes.
> You then drop the first two characters and print the other 799,998 bytes,
> each of which will be either '0' or '1'.

> I am absolutely, positively convinced that's not what you wanted to do.
> What point is there in printing out the binary equavalent of a bitmap?

> Even if you did, it would be much quicker for you to do the conversion one
> byte at a time, completely skipping the conversion to hex and then the
> creation of a massive multi-precision number.  Example:

>     f = open('example.bmp','rb')
>     rawdata = f.read()
>     bsstr = []
>     for b in rawdata:
>         bsstr.append( bin(ord(b)) )
>     bsstr = ''.join(bsstr)

> or even:
>     f = open('example.bmp','rb')
>     bsstr = ''.join( bin(ord(b))[2:] for b in f.read() )

Exactly my idea at first. But then I started to time it (using
the timeit module) by comparing the following functions:

  # Original version
  
  def c1( rawdata ) :
      h = binascii.b2a_hex( rawdata )
      z = bin( int( h, 16 ) )[ 2 : ]
      return '0' * ( 8 * len( r ) - len( z ) ) + z

  # Convert each byte directly

  def c2( rawdata ) :
      return ''.join( bin( ord( x ) )[ 2 : ].rjust( 8, '0' ) for x in r )

  # Convert each byte using a list for table look-up

  def c3( rawdata ) :
      h = [ bin( i )[ 2 : ].rjust( 8, '0' ) for i in range( 256 ) ]
      return ''.join( h[ ord( x ) ] for x in rawdata )

  # Convert each byte using a dictionary for table look-up (avoids
  # lots of ord() calls)

  def c4( rawdata ) :
      h = { chr( i ) : bin( i )[ 2 : ].rjust( 8, '0' ) for i in range( 256 ) }
      return ''.join( h[ x ] for x in rawdata )

As you can see I even in c3() and c4() tried to speed things up
further by using a table look-up instead if calling bin() etc.
on each byte. But the results was that c2() is nearly 15 times
slower than c1(), c3() about 3 times and c4() still more than 2
times slower! So the method the OP uses seems to be quite a bit
more efficient than one might be tempted to assume.

I would guess that the reason is that c1() does just a small
number of calls of functions that probably aren't implemented
in Python but in C and thus can be a lot faster then anything
you could achieve with Python, while the other functions use a
for loop in Python, which seems to account for a good part of
the CPU time used. To test for that I split the 'rawdata' string
into a list of character (i.e. single letter strings) and re-
assembled it using join() and a for loop:

    r = list( rawdata( )
    z = ''.join( x for x in r )

The second line alone took about 1.7 times longer than the
whole, seemingly convoluted c1() function!

What I take away from this is that a lot of the assumption one
is prone to make when coming from e.g. a C/C++ background can
be quite misleading when extrapolating to Python (or other in-
terpreted languages)...
                          Best regards, Jens
-- 
  \   Jens Thoms Toerring  ___      jt@toerring.de
   \__________________________      http://toerring.de

[toc] | [prev] | [next] | [standalone]

#44469

On Sunday, April 28, 2013 8:04:04 PM UTC+8, Jens Thoms Toerring wrote:
> Tim Roberts <timr@probo.com> wrote:
> 
> > Jimmie He <jimmie.he@gmail.com> wrote:
> 
> 
> 
> > >When I run the readbmp on an example.bmp(about 100k),the Shell is become to "No respose",when I change f.read() to f.read(1000),it is ok,could someone tell me the excat reason for this?
> 
> > >Thank you in advance!
> 
> > >
> 
> > >Python Code as below!!
> 
> > >
> 
> > >import binascii
> 
> > >
> 
> > >def read_bmp():
> 
> > >    f = open('example.bmp','rb')
> 
> > >    rawdata = f.read()                       #f.read(1000) is ok
> 
> > >    hexstr = binascii.b2a_hex(rawdata)       #Get an HEX number
> 
> > >    bsstr = bin (int(hexstr,16))[2:]
> 
> 
> 
> > I suspect the root of the problem here is that you don't understand what
> 
> > this is actually doing.  You should run this code in the command-line
> 
> > interpreter, one line at a time, and print the results.
> 
> 
> 
> > The "read" instruction produces a string with 100k bytes.  The b2a_hex then
> 
> > produces a string with 200k bytes.  Then, int(hexstr,16) takes that 200,000
> 
> > byte hex string and converts it to an integer, roughly equal to 10 to the
> 
> > 240,000 power, a number with some 240,000 decimal digits.  You then convert
> 
> > that integer to a binary string.  That string will contain 800,000 bytes.
> 
> > You then drop the first two characters and print the other 799,998 bytes,
> 
> > each of which will be either '0' or '1'.
> 
> 
> 
> > I am absolutely, positively convinced that's not what you wanted to do.
> 
> > What point is there in printing out the binary equavalent of a bitmap?
> 
> 
> 
> > Even if you did, it would be much quicker for you to do the conversion one
> 
> > byte at a time, completely skipping the conversion to hex and then the
> 
> > creation of a massive multi-precision number.  Example:
> 
> 
> 
> >     f = open('example.bmp','rb')
> 
> >     rawdata = f.read()
> 
> >     bsstr = []
> 
> >     for b in rawdata:
> 
> >         bsstr.append( bin(ord(b)) )
> 
> >     bsstr = ''.join(bsstr)
> 
> 
> 
> > or even:
> 
> >     f = open('example.bmp','rb')
> 
> >     bsstr = ''.join( bin(ord(b))[2:] for b in f.read() )
> 
> 
> 
> Exactly my idea at first. But then I started to time it (using
> 
> the timeit module) by comparing the following functions:
> 
> 
> 
>   # Original version
> 
>   
> 
>   def c1( rawdata ) :
> 
>       h = binascii.b2a_hex( rawdata )
> 
>       z = bin( int( h, 16 ) )[ 2 : ]
> 
>       return '0' * ( 8 * len( r ) - len( z ) ) + z
> 
> 
> 
>   # Convert each byte directly
> 
> 
> 
>   def c2( rawdata ) :
> 
>       return ''.join( bin( ord( x ) )[ 2 : ].rjust( 8, '0' ) for x in r )
> 
> 
> 
>   # Convert each byte using a list for table look-up
> 
> 
> 
>   def c3( rawdata ) :
> 
>       h = [ bin( i )[ 2 : ].rjust( 8, '0' ) for i in range( 256 ) ]
> 
>       return ''.join( h[ ord( x ) ] for x in rawdata )
> 
> 
> 
>   # Convert each byte using a dictionary for table look-up (avoids
> 
>   # lots of ord() calls)
> 
> 
> 
>   def c4( rawdata ) :
> 
>       h = { chr( i ) : bin( i )[ 2 : ].rjust( 8, '0' ) for i in range( 256 ) }
> 
>       return ''.join( h[ x ] for x in rawdata )
> 
> 
> 
> As you can see I even in c3() and c4() tried to speed things up
> 
> further by using a table look-up instead if calling bin() etc.
> 
> on each byte. But the results was that c2() is nearly 15 times
> 
> slower than c1(), c3() about 3 times and c4() still more than 2
> 
> times slower! So the method the OP uses seems to be quite a bit
> 
> more efficient than one might be tempted to assume.
> 
> 
> 
> I would guess that the reason is that c1() does just a small
> 
> number of calls of functions that probably aren't implemented
> 
> in Python but in C and thus can be a lot faster then anything
> 
> you could achieve with Python, while the other functions use a
> 
> for loop in Python, which seems to account for a good part of
> 
> the CPU time used. To test for that I split the 'rawdata' string
> 
> into a list of character (i.e. single letter strings) and re-
> 
> assembled it using join() and a for loop:
> 
> 
> 
>     r = list( rawdata( )
> 
>     z = ''.join( x for x in r )
> 
> 
> 
> The second line alone took about 1.7 times longer than the
> 
> whole, seemingly convoluted c1() function!
> 
> 
> 
> What I take away from this is that a lot of the assumption one
> 
> is prone to make when coming from e.g. a C/C++ background can
> 
> be quite misleading when extrapolating to Python (or other in-
> 
> terpreted languages)...
> 
>                           Best regards, Jens
> 
> -- 
> 
>   \   Jens Thoms Toerring  ___      jt@toerring.de
> 
>    \__________________________      http://toerring.de
Hi,Jens &Peter &Tim,
   Thank you very much for your wonderful analysis for my newbie question.
   I admit that I throw this question to much early because I just want some guru to inspire me;-) If it really confuse you,excuse my noise:-)
   What I intend to do is to make an BMP Font Maker(Covert the BMP to an data array,what I did wrong is print it directly to screen and had not understand it at all firstly.
   C1()~C4() which Jens provided deeply indicate that we should think about the effiency because it is an interpreted language.
   Anyway thanks for all your kindly help :-)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web