Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #3580 > unrolled thread

Pickling over a socket

Back to article view | Back to comp.lang.python

Contents

  Pickling over a socket Roger Alexander <rtalexander@mac.com> - 2011-04-19 11:53 -0700
    Re: Pickling over a socket Chris Rebert <clp2@rebertia.com> - 2011-04-19 12:21 -0700
    Re: Pickling over a socket Chris Angelico <rosuav@gmail.com> - 2011-04-20 05:29 +1000
    Re: Pickling over a socket Dan Stromberg <drsalists@gmail.com> - 2011-04-19 12:30 -0700
    Re: Pickling over a socket Chris Angelico <rosuav@gmail.com> - 2011-04-20 05:37 +1000
      Re: Pickling over a socket Roger Alexander <rtalexander@mac.com> - 2011-04-19 15:27 -0700
        Re: Pickling over a socket Jean-Paul Calderone <calderone.jeanpaul@gmail.com> - 2011-04-19 19:28 -0700
          Re: Pickling over a socket Bastian Ballmann <balle@chaostal.de> - 2011-04-20 08:44 +0200
          Re: Pickling over a socket Chris Angelico <rosuav@gmail.com> - 2011-04-20 16:59 +1000
          Re: Pickling over a socket Bastian Ballmann <balle@chaostal.de> - 2011-04-20 09:34 +0200
            Re: Pickling over a socket Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-04-20 10:25 +0200
              [OT] Re: Pickling over a socket Bastian Ballmann <balle@chaostal.de> - 2011-04-20 10:59 +0200
          Re: Pickling over a socket Chris Angelico <rosuav@gmail.com> - 2011-04-20 19:26 +1000
          Re: Pickling over a socket Bastian Ballmann <balle@chaostal.de> - 2011-04-20 11:41 +0200

#3580 — Pickling over a socket

Hi,

I'm trying to understand how to pickle Python objects over a TCP
socket.

In the example below (based on code from Foundations of Python Network
Programming), a client creates a dictionary (lines 34-38) and uses
pickle.dump at line 42 to write the pickled object using file handle
make from a socket. The server de-pickles with pickle.load  (line 24),
again using a file handle made from a socket.

When I run the program, the following output is produced:

    Listening at ('127.0.0.1', 1060)
    Accepted connection from ('127.0.0.1', 49938)
    Traceback (most recent call last):
    File "pickles.py", line 24, in <module>
        d = pickle.load( s_fh )
    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
        return Unpickler(file).load()
    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
        key = read(1)
    File "/usr/local/lib/python2.7/socket.py", line 380, in read
        data = self._sock.recv(left)
    socket.error: [Errno 107] Transport endpoint is not connected

I'm at a loss, can anyone provide any guidance?

Thanks,

Roger Alexander

 1  import pickle
 2  import socket, sys
 3
 4  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 5
 6  HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
 7  PORT = 1060
 8
 9  if sys.argv[1:] == ['server']:
10
11      s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
12      s.bind((HOST, PORT))
13      s.listen(1)
14
15      print 'Listening at', s.getsockname()
16
17      sc, sockname = s.accept()
18
19      print 'Accepted connection from', sockname
20
21      sc.shutdown(socket.SHUT_WR)
22      sf = s.makefile( "rb" )
23
24      d = pickle.load(sf)
25
26      sc.close()
27      s.close()
28
29  elif sys.argv[1:] == ['client']:
30
31      s.connect((HOST, PORT))
32      s.shutdown(socket.SHUT_RD)
33
34      d = dict()
35
36      d[ 'Name' ]     = 'Jake Thompson.'
37      d[ 'Age' ]      = 25
38      d[ 'Location' ] = 'Washington, D.C.'
39
40      sf = s.makefile( "wb" )
41
42      pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
43
44      s.close()
45
46  else:
47      print >>sys.stderr, 'usage: streamer.py server|client [host]'

[toc] | [next] | [standalone]

#3583

On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <rtalexander@mac.com> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-38) and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load  (line 24),
> again using a file handle made from a socket.
>
> When I run the program, the following output is produced:
>
>    Listening at ('127.0.0.1', 1060)
>    Accepted connection from ('127.0.0.1', 49938)
>    Traceback (most recent call last):
>    File "pickles.py", line 24, in <module>
>        d = pickle.load( s_fh )
>    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
>        return Unpickler(file).load()
>    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
>        key = read(1)
>    File "/usr/local/lib/python2.7/socket.py", line 380, in read
>        data = self._sock.recv(left)
>    socket.error: [Errno 107] Transport endpoint is not connected
>
> I'm at a loss, can anyone provide any guidance?
>
> Thanks,
>
> Roger Alexander
>
>  1  import pickle
>  2  import socket, sys
>  3
>  4  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>  5
>  6  HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
>  7  PORT = 1060
>  8
>  9  if sys.argv[1:] == ['server']:
> 10
> 11      s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> 12      s.bind((HOST, PORT))
> 13      s.listen(1)
> 14
> 15      print 'Listening at', s.getsockname()
> 16
> 17      sc, sockname = s.accept()
> 18
> 19      print 'Accepted connection from', sockname
> 20
> 21      sc.shutdown(socket.SHUT_WR)

[Haven't done any network programming, so please excuse the naivete of
this suggestion.]

Have you tried removing line #21 and/or #32?

http://docs.python.org/library/socket.html#socket.socket.shutdown :
"socket.shutdown(how) - Shut down one or both halves of the
connection. [...] Depending on the platform, shutting down one half of
the connection can also close the opposite half"

Cheers,
Chris
--
http://blog.rebertia.com

> 22      sf = s.makefile( "rb" )
> 23
> 24      d = pickle.load(sf)
> 25
> 26      sc.close()
> 27      s.close()
> 28
> 29  elif sys.argv[1:] == ['client']:
> 30
> 31      s.connect((HOST, PORT))
> 32      s.shutdown(socket.SHUT_RD)
<snip>
> 42      pickle.dump( d, sf, pickle.HIGHEST_PROTOCOL )
> 43
> 44      s.close()

[toc] | [prev] | [next] | [standalone]

#3584

On Wed, Apr 20, 2011 at 4:53 AM, Roger Alexander <rtalexander@mac.com> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-38) and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load  (line 24),
> again using a file handle made from a socket.

Whenever there's a problem, create simplicity. I would recommend not
using the file-from-socket method, and simply using pickle.dumps() and
pickle.loads() to pickle to/from strings; those strings can then be
sent/received over the socket using standard recv/send functions.

Also, Chris Rebert's idea is a good one, and worth trying.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#3585

On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <rtalexander@mac.com> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-38) and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load  (line 24),
> again using a file handle made from a socket.
>
> When I run the program, the following output is produced:
>
>    Listening at ('127.0.0.1', 1060)
>    Accepted connection from ('127.0.0.1', 49938)
>    Traceback (most recent call last):
>    File "pickles.py", line 24, in <module>
>        d = pickle.load( s_fh )
>    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
>        return Unpickler(file).load()
>    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
>        key = read(1)
>    File "/usr/local/lib/python2.7/socket.py", line 380, in read
>        data = self._sock.recv(left)
>    socket.error: [Errno 107] Transport endpoint is not connected
>
> I'm at a loss, can anyone provide any guidance?
>
> Thanks,
>
> Roger Alexander

I played around with it until something worked, and ended up with the
below.  The most significant change was probably using sc.makefile
instead of s.makefile in the server, but I seemed to need some data
framing too despite the pickling.  It's possible you won't need that
if you just flush your file in the client; I don't much pickling
experience - in particular, I don't know if you can concatenate
pickled objects and load them serially from a file-like object without
any (additional) framing.

I like to use bufsock for this sort of stuff, but I'm probably unique
in that.  ^_^   http://stromberg.dnsalias.org/~strombrg/bufsock.html

#!/usr/bin/python

import time
import pickle
import socket, sys
import pprint

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
PORT = 1060

if sys.argv[1:] == ['server']:

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((HOST, PORT))
    s.listen(1)

    print 'Listening at', s.getsockname()

    sc, sockname = s.accept()

    print 'Accepted connection from', sockname

    sf = sc.makefile( "rb" )

    length_list = []
    while True:
        char = sf.read(1)
        if char == '\n':
            break
        else:
            length_list.append(int(char))
    length = 0
    for digit in length_list:
        length = length * 10 + digit
    data = sf.read(length)

    d = pickle.loads(data)

    pprint.pprint(d)

    sc.shutdown(socket.SHUT_RDWR)
    sc.close()
    s.close()

elif sys.argv[1:] == ['client']:

    s.connect((HOST, PORT))
    # s.shutdown(socket.SHUT_RD)

    d = dict()

    d[ 'Name' ]     = 'Jake Thompson.'
    d[ 'Age' ]      = 25
    d[ 'Location' ] = 'Washington, D.C.'

    sf = s.makefile( "wb" )

    string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
    sf.write('%d\n' % len(string))
    sf.write(string)
    sf.flush()

    #time.sleep(10)
    sf.close()
    s.shutdown(socket.SHUT_RDWR)
    # s.close()

else:
    print >>sys.stderr, 'usage: streamer.py server|client [host]'

[toc] | [prev] | [next] | [standalone]

#3586

On Wed, Apr 20, 2011 at 5:30 AM, Dan Stromberg <drsalists@gmail.com> wrote:
> I played around with it until something worked, and ended up with the
> below.  The most significant change was probably using sc.makefile
> instead of s.makefile in the server...

Oh! I didn't notice that in the OP. Yep, that would do it!

ChrisA

[toc] | [prev] | [next] | [standalone]

#3598

Thanks everybody, got it working.

 I appreciate the help!

Roger.

[toc] | [prev] | [next] | [standalone]

#3630

On Apr 19, 6:27 pm, Roger Alexander <rtalexan...@mac.com> wrote:
> Thanks everybody, got it working.
>
>  I appreciate the help!
>
> Roger.

It's too bad none of the other respondents pointed out to you that you
_shouldn't do this_!  Pickle is not suitable for use over the network
like this.  Your server accepts arbitrary code from clients and
executes it.  It is completely insecure.  Do not use pickle and
sockets together.  Notice the large red box at the top of <http://
docs.python.org/library/pickle.html>.

Jean-Paul

[toc] | [prev] | [next] | [standalone]

#3657

[Multipart message — attachments visible in raw view] — view raw

Am Tue, 19 Apr 2011 19:28:50 -0700 (PDT)
schrieb Jean-Paul Calderone <calderone.jeanpaul@gmail.com>:

> It is completely insecure.  Do not use pickle and
> sockets together.

Yes pickle is like eval, but that doesnt mean that one should never
ever use it over a socket connection. 
What about ssl sockets where client and server authenticate each other?
Or you encrypt the pickle dump with symmetric encryption and only load
it if you can decrypt it? There are ways to ensure that the data you
get can be handled as trusted.
Greets

Basti

[toc] | [prev] | [next] | [standalone]

#3660

On Wed, Apr 20, 2011 at 4:44 PM, Bastian Ballmann <balle@chaostal.de> wrote:
> Yes pickle is like eval, but that doesnt mean that one should never
> ever use it over a socket connection.
> What about ssl sockets where client and server authenticate each other?
> Or you encrypt the pickle dump with symmetric encryption and only load
> it if you can decrypt it? There are ways to ensure that the data you
> get can be handled as trusted.

No, I disagree. And I'll cite Caesary as evidence of why.

Caesary is a multiplayer game that uses Flash as its client. (I'm told
the back end is Java, which would explain why it starts lagging
horribly when everyone's online at once.) It has some measure of
authentication of the client, but it's not difficult to spoof;
obviously you could go more elaborate and harder to spoof, but that
still doesn't solve the problem. Even public/private key systems won't
work here; someone could get hold of your client and its private key,
and poof.

Caesary uses an Adobe Message Format system, whereby complex objects
get serialized and transmitted in both directions. It's fundamentally
the same as pickling. When I started poking around with things, it
took me very little time to start transmitting my own requests to the
server; my requests were benign (asking it for information), but other
people figured out the same thing and were rather less ethical.

That's why I tend to use and create much simpler protocols for network
transmission. Also, I like to use a MUD client to test my servers,
ergo textual protocols similar to SMTP. Sure, it may be a tad more
verbose than some, but it's usually easy to parse and verify.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#3662

[Multipart message — attachments visible in raw view] — view raw

Am Wed, 20 Apr 2011 16:59:19 +1000
schrieb Chris Angelico <rosuav@gmail.com>:
 
> Even public/private key systems won't
> work here; someone could get hold of your client and its private key,
> and poof.

Oh yeah but than all kinds of trusted computing wont work. Sure
one can see it on the net these days looking at the rsa or commodo or
ps3 hack and the like.

No system is totally secure. You can _always_ poke around if a program
uses user input. For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters. Now would you say one shouldnt use sql
databases cause of that? ;)

My point is using ssl authentication / encryption together with another
symmetric encryption builds up two layers, which I would say is secure
enough to handle the data as trusted.

Greets

Basti

[toc] | [prev] | [next] | [standalone]

#3667

Am 20.04.2011 09:34, schrieb Bastian Ballmann:

> No system is totally secure. You can _always_ poke around if a program
> uses user input.

It depends on what the program does with the input. If it treats it 
appropriately, nothing can happen.


> For example one can totally own a complete computer by
> nothing more than a single sql injection attack even if the programmer
> implemented some filters.

What do yu want with filters here? Not filtering is appropriate against 
SQL injection, but escaping.

If Little Bobby Tables is really called "Robert'); DROP TABLE STUDENTS; 
--", it is wrong to reject this string - instead, all dangerous 
characters inside it must be quoted (in this case: ') and then it does 
not harm at all.


 > Now would you say one shouldnt use sql
> databases cause of that? ;)

No, just beware of what can happen and use the dbs and its functions 
appropriately.


Thomas

[toc] | [prev] | [next] | [standalone]

#3670 — [OT] Re: Pickling over a socket

[Multipart message — attachments visible in raw view] — view raw

Am Wed, 20 Apr 2011 10:25:14 +0200
schrieb Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>:

> It depends on what the program does with the input. If it treats it 
> appropriately, nothing can happen.

Yes, but the question seems to be what is appropriately.
 

> What do yu want with filters here? Not filtering is appropriate
> against SQL injection, but escaping.

Escaping in strings, filtering with numbers etc.

 
> If Little Bobby Tables is really called "Robert'); DROP TABLE
> STUDENTS; --", it is wrong to reject this string - instead, all
> dangerous characters inside it must be quoted (in this case: ') and
> then it does not harm at all.

Well you forgot to escape ; and \ but this seems to slide into OT ;)
Greets

Basti

[toc] | [prev] | [next] | [standalone]

#3671

On Wed, Apr 20, 2011 at 7:17 PM, Bastian Ballmann <balle@chaostal.de> wrote:
> Well you forgot to escape ; and \ but this seems to slide into OT ;)

The semicolon doesn't need to be escaped in a quoted string, and the
backslash does only if it's the escape character. The
string-safetifier function that I used with DB2 was called "dblapos"
because it simply doubled every apostrophe - nothing else needed. On
the other hand, mysql_real_escape_string will escape quite a few
characters, for convenience in reading dumps.

> Am Wed, 20 Apr 2011 18:43:01 +1000
> schrieb Chris Angelico <rosuav@gmail.com>:
>
>> So, like Jean-Paul said, you simply do not trust anything that comes
>> from the network. Ever.
>
> If you generalize it in this way you should never trust any user input
> regardless if it comes from the net or from local or the environment
> etc.

Yes, but the other half of the issue is that you have to treat
anything that comes over the network as "user input", even if you
think it's from your own program that you control.

>> Urrrm. You can "own" a "complete computer" with SQL injection? Then
>> someone has some seriously weak protection.
>
> Yes and the database is poorly protected, but this happens way too
> often.

That's just *sad*.

>> SQL injection is easier to
>> protect against than buffer overruns, and with a lot of function
>> libraries
>
> I totally disagree. Buffer overflow is just a matter of size checking,
> but sql injection is a matter of syntax. It's more than just throwing
> the input into a magic auto-escape function.

Buffer overruns can happen in all sorts of places; SQL injection can
only happen where you talk to the database. And it IS just a matter of
using a magic auto-escape function, if your library is set up right -
unless, of course, you allow your users to submit SQL themselves (eg a
WHERE clause). That's almost impossible to sanitize, which is why I
would never EVER allow such a thing unless it's actually a trusted
environment (eg PHPMyAdmin - anyone who has access to PMA has access
to the database anyway).

> We both agree that one should never trust user input blindly and we also
> seem to conform to one can use user input in a appropriate way that's
> not the case, but if i read your mail i think you want to tell me one
> should never ever use the internet or only write programs without user
> input at all.

Not at all; just never *trust* user input. Where thou typest foo,
someone someday will type...

ChrisA

[toc] | [prev] | [next] | [standalone]

#3672

[Multipart message — attachments visible in raw view] — view raw

Am Wed, 20 Apr 2011 19:26:44 +1000
schrieb Chris Angelico <rosuav@gmail.com>:

> Yes, but the other half of the issue is that you have to treat
> anything that comes over the network as "user input", even if you
> think it's from your own program that you control.

Sure.

 
> Buffer overruns can happen in all sorts of places; SQL injection can
> only happen where you talk to the database. And it IS just a matter of
> using a magic auto-escape function, if your library is set up right -

No. Not all data is strings.


> Not at all; just never *trust* user input. Where thou typest foo,
> someone someday will type...

I never *trust* the user *blindly* as you do with your
magic-escape-function so where do we disagree?
Greets

Basti

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web