Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #69966

Re: threading

From Marko Rauhamaa <marko@pacujo.net>
Newsgroups comp.lang.python
Subject Re: threading
Date 2014-04-09 19:48 +0300
Organization A noiseless patient Spider
Message-ID <87ob0ala99.fsf@elektro.pacujo.net> (permalink)
References (4 earlier) <mailman.8986.1396914468.18130.python-list@python.org> <877g70wg8p.fsf@elektro.pacujo.net> <li3hki$grv$1@ger.gmane.org> <CAPTjJmqwhb8O8vq84mMTv+-Rkc3Ff1AQDXe5cs8Y5gY02kHyNg@mail.gmail.com> <mailman.9078.1397053837.18130.python-list@python.org>

Show all headers | View raw

"Frank Millman" <frank@chagford.com>:

> Does reading from disk count as blocking? Strictly speaking I would
> have thought 'yes'.

You have touched upon a very interesting topic there.

I can't speak for Windows, but linux doesn't really let you control the
blocking of disk access. In fact, linux doesn't really let you control
the blocking of RAM access, either. RAM and the disk are considered two
sides of the same coin. It is very difficult to guarantee that a process
has all of its memory "cached" in RAM short of not having a physical
disk mounted.

There is what's known as AIO, and it's supposedly supported in linux,
but I have never seen anybody use it, and I don't know how well tested
it is. Also, I don't know how well it integrates with regular asyncio.

On the other hand, you don't know if disk access ever blocks. Quite
often you will find that the whole active part of the file system is
kept in RAM by linux.

My rule of thumb, two processes per CPU, should alleviate disk blocking
issues. When that isn't enough, you may be forced to write a small file
server that translates disk access to socket/pipe access.

Sockets and pipes are different beasts because, unlike files, they are
allocated memory buffers in the kernel memory. Also, they are accessed
strictly sequentially while files can be sought back and forth.

I do think it would be a nice addition to linux if they added a, say,
AF_FILE socket type that provided a buffered socket abstraction for
physical files.


Marko

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 13:05 +1000
  Re: threading Roy Smith <roy@panix.com> - 2014-04-06 23:48 -0400
    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 13:56 +1000
      Re: threading Roy Smith <roy@panix.com> - 2014-04-07 08:26 -0400
        Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 22:34 +1000
          Re: threading Roy Smith <roy@panix.com> - 2014-04-07 09:22 -0400
            Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-07 14:41 +0100
            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 16:49 +0300
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:27 +1000
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 17:51 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 01:12 +1000
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:24 +1000
      Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-08 18:09 -0700
        Re: threading "Neil D. Cerutti" <neilc@norwich.edu> - 2014-04-09 09:50 -0400
          Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-09 08:51 -0700
            Re: threading MRAB <python@mrabarnett.plus.com> - 2014-04-09 18:47 +0100
              Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:35 +1200
                Re: threading Roy Smith <roy@panix.com> - 2014-04-09 19:53 -0400
                Re: threading Andrew Berg <robotsondrugs@gmail.com> - 2014-04-09 19:02 -0500
                Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 02:43 +0000
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:08 +1000
                Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-10 09:23 +0100
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:11 +1000
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 04:00 +1000
            Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 03:44 +0000
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:54 +1000
    Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 15:22 +1000
    Re: threading Ethan Furman <ethan@stoneleaf.us> - 2014-04-08 11:09 -0700
    Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 21:41 +0200
      Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-08 20:30 +0000
        Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 00:32 +0200
          Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-08 19:17 -0700
  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:10 +0300
    Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-06 22:39 -0700
      Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:46 +0300
      Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-07 19:47 -0400
        Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 08:19 +0300
          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 10:47 +0000
            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 15:10 +0300
              Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:37 +0000
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 20:17 +0300
            Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:19 -0400
              Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:44 +0000
                Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:38 -0700
                Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-09 14:42 +0100
          Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 15:23 +0200
            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 16:55 +0300
              Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:46 +0200
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 20:31 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 03:52 +1000
                Re: threading Mark H Harris <harrismh777@gmail.com> - 2014-04-10 08:29 -0500
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 19:20 +0000
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 23:47 +1000
            Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:44 -0400
          Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:30 +0200
            Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:52 -0400
              Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:19 +1200
            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 19:48 +0300
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 00:44 +1000
          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 15:29 +0000
          Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-09 12:14 -0400
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 02:25 +1000
          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 16:32 +0000
          Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-09 19:44 -0400
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 11:05 +1000
          Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 11:17 +0200
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:40 +1000
          Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 13:10 +0200
            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 14:43 +0300
              Re: threading Roy Smith <roy@panix.com> - 2014-04-10 08:56 -0400
              Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-10 15:24 +0000
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:20 +0300
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 01:32 +1000
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:25 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 03:08 +1000
                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 11:14 -0700
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 22:44 +0300
                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 13:21 -0700
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 23:44 +0300
                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 22:15 -0700
                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 23:50 -0700
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 18:36 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-12 01:53 +1000
                Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-11 16:58 +0100
                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-11 11:54 -0700
                Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 22:27 +0300
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 01:51 +0200
                Re: threading Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-04-11 05:35 +0000
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 09:26 +0000
                Re: threading Roy Smith <roy@panix.com> - 2014-04-11 08:36 -0400
                Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-11 16:18 +0000
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 02:21 +0200
                Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-10 20:23 -0400
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 21:19 +1000
      Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 02:06 +0000
        Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 11:07 +0000
          Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:13 -0400
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 23:23 +1000
              Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 14:15 +0000
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:06 +0000
            Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:40 +0000
              Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:46 -0700
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 02:46 +1000
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 17:17 +0000
          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:19 +0000

csiph-web