Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #197523

What does stats = await asyncio.to_thread(os.stat, url) do? (Was async I/O via threads is extremly slow)

Show all headers | View raw

So what does:

stats = await asyncio.to_thread(os.stat, url)

Whell it calls in a sparate new secondary thread:

os.stat(url)

It happends that url is only a file path, and
the file path points to an existing file. So the
secondary thread computs the stats, and terminates,

and the async framework hands the stats back to
the main thread that did the await, and the main
thread stops his waiting and continues to run

cooperatively with the other tasks in the current
event loop. The test case measures the wall time.
The results are:

 > node.js: 10 ms (usual Promises and stuff)
 > JDK 24: 50 ms (using Threads, not yet VirtualThreads)
 > pypy: 2000 ms

I am only using one main task, sequentially on
such await calles, with a couple of file, not
more than 50 files.

I could compare with removing the async detour,
to qualify the async I/O detour overhead.

Mild Shock schrieb:
> Hi,
> 
> async I/O in Python is extremly disappointing
> and an annoying bottleneck.
> 
> The problem is async I/O via threads is currently
> extremly slow. I use a custom async I/O file property
> predicate. It doesn't need to be async for file
> 
> system access. But by some historical circumstances
> I made it async since the same file property routine
> might also do a http HEAD request. But what I was
> 
> testing and comparing was a simple file system access
> inside a wrapped thread, that is async awaited.
> Such a thread is called for a couple of directory
> 
> entries to check a directory tree whether updates
> are need. Here some measurement doing this simple
> involving some little async I/O:
> 
> node.js: 10 ms (usual Promises and stuff)
> JDK 24: 50 ms (using Threads, not yet VirtualThreads)
> pypy: 2000 ms
> 
> So currently PyPy is 200x times slower than node.js
> when it comes to async I/O. No files were read or
> written in the test case, only "mtime" was read,
> 
> via this Python line:
> 
> stats = await asyncio.to_thread(os.stat, url)
> 
> Bye
> 
> Mild Shock schrieb:
>>
>> Concerning virtual threads the only problem
>> with Java I have is, that JDK 17 doesn't have them.
>> And some linux distributions are stuck with JDK 17.
>>
>> Otherwise its not an idea that belongs solely
>> to Java, I think golang pioniered them with their
>> goroutines. I am planning to use them more heavily
>>
>> when they become more widely available, and I don't
>> see any principle objection that Python wouldn't
>> have them as well. It would make async I/O based
>>
>> on async waithing for a thread maybe more lightweight.
>> But this would be only important if you have a high
>> number of tasks.
>>
>> Lawrence D'Oliveiro schrieb:
>>> Short answer: no.
>>>
>>> <https://discuss.python.org/t/add-virtual-threads-to-python/91403>
>>>
>>> Firstly, anybody appealing to Java as an example of how to design a
>>> programming language should immediately be sending your bullshit 
>>> detector
>>> into the yellow zone.
>>>
>>> Secondly, the link to a critique of JavaScript that dates from 2015, 
>>> from
>>> before the language acquired its async/await constructs, should be 
>>> another
>>> warning sign.
>>>
>>> Looking at that Java spec, a “virtual thread” is just another name for
>>> “stackful coroutine”. Because that’s what you get when you take away
>>> implicit thread preemption and substitute explicit preemption instead.
>>>
>>> The continuation concept is useful in its own right. Why not concentrate
>>> on implementing that as a new primitive instead?
>>>
>>
> 

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-14 04:11 +0000
  Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-14 04:29 -0700
    Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-14 23:10 +0000
      Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-14 18:25 -0700
        Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-15 02:13 +0000
          Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 13:24 -0700
            Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-15 20:59 +0000
              Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 14:33 -0700
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-16 01:14 +0000
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Paul Rubin <no.email@nospam.invalid> - 2025-06-15 21:02 -0700
                Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-06-17 02:12 +0000
  Re: Does Python Need Virtual Threads? (Posting On Python-List Prohibited) Mild Shock <janburse@fastmail.fm> - 2025-06-14 23:23 +0200
    async I/O via threads is extremly slow (Was: Does Python Need Virtual Threads?) Mild Shock <janburse@fastmail.fm> - 2025-06-23 13:29 +0200
      What does stats = await asyncio.to_thread(os.stat, url) do? (Was async I/O via threads is extremly slow) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:32 +0200
        What does the Async Detour usually cost (Was: What does stats = await asyncio.to_thread(os.stat, url) do?) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:42 +0200
          Which Python System is affected? (Was: What does the Async Detour usually cost) Mild Shock <janburse@fastmail.fm> - 2025-06-24 00:48 +0200
            Schachner, Joseph was the Big Moron [September 2021 16:30] (Was: Which Python System is affected?) Mild Shock <janburse@fastmail.fm> - 2025-06-24 08:53 +0200

csiph-web