Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #197522

async I/O via threads is extremly slow (Was: Does Python Need Virtual Threads?)

Show key headers only | View raw

Hi,

async I/O in Python is extremly disappointing
and an annoying bottleneck.

The problem is async I/O via threads is currently
extremly slow. I use a custom async I/O file property
predicate. It doesn't need to be async for file

system access. But by some historical circumstances
I made it async since the same file property routine
might also do a http HEAD request. But what I was

testing and comparing was a simple file system access
inside a wrapped thread, that is async awaited.
Such a thread is called for a couple of directory

entries to check a directory tree whether updates
are need. Here some measurement doing this simple
involving some little async I/O:

node.js: 10 ms (usual Promises and stuff)
JDK 24: 50 ms (using Threads, not yet VirtualThreads)
pypy: 2000 ms

So currently PyPy is 200x times slower than node.js
when it comes to async I/O. No files were read or
written in the test case, only "mtime" was read,

via this Python line:

stats = await asyncio.to_thread(os.stat, url)

Bye

Mild Shock schrieb:
> 
> Concerning virtual threads the only problem
> with Java I have is, that JDK 17 doesn't have them.
> And some linux distributions are stuck with JDK 17.
> 
> Otherwise its not an idea that belongs solely
> to Java, I think golang pioniered them with their
> goroutines. I am planning to use them more heavily
> 
> when they become more widely available, and I don't
> see any principle objection that Python wouldn't
> have them as well. It would make async I/O based
> 
> on async waithing for a thread maybe more lightweight.
> But this would be only important if you have a high
> number of tasks.
> 
> Lawrence D'Oliveiro schrieb:
>> Short answer: no.
>>
>> <https://discuss.python.org/t/add-virtual-threads-to-python/91403>
>>
>> Firstly, anybody appealing to Java as an example of how to design a
>> programming language should immediately be sending your bullshit detector
>> into the yellow zone.
>>
>> Secondly, the link to a critique of JavaScript that dates from 2015, from
>> before the language acquired its async/await constructs, should be 
>> another
>> warning sign.
>>
>> Looking at that Java spec, a “virtual thread” is just another name for
>> “stackful coroutine”. Because that’s what you get when you take away
>> implicit thread preemption and substitute explicit preemption instead.
>>
>> The continuation concept is useful in its own right. Why not concentrate
>> on implementing that as a new primitive instead?
>>
> 

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-14 04:11 +0000
  Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-14 04:29 -0700
    Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-14 23:10 +0000
      Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-14 18:25 -0700
        Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-15 02:13 +0000
          Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 13:24 -0700
            Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-15 20:59 +0000
              Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 14:33 -0700
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-16 01:14 +0000
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 21:02 -0700
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-17 02:12 +0000
  Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Mild Shock <janburse@fastmail.fm> - 2025-06-14 23:23 +0200
    async I/O via threads is extremly slow (Was: Does Python Need Virtual Threads?) Mild Shock <janburse@fastmail.fm> - 2025-06-23 13:29 +0200
      What does stats = await asyncio.to_thread(os.stat, url) do? (Was async I/O via threads is extremly slow) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:32 +0200
        What does the Async Detour usually cost (Was: What does stats = await asyncio.to_thread(os.stat, url) do?) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:42 +0200
          Which Python System is affected? (Was: What does the Async Detour usually cost) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:48 +0200
            Schachner, Joseph was the Big Moron [September 2021 16:30] (Was: Which Python System is affected?) Mild Shock <janburse@fastmail.fm> - 2025-06-24 08:53 +0200

csiph-web