Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: MRAB <python@mrabarnett.plus.com>
Newsgroups: comp.lang.python
Subject: Re: PyGILState_Ensure() deadlocks, why?
Date: Sun, 7 Jul 2024 23:44:01 +0100
Lines: 144
Message-ID: <mailman.16.1720392430.2981.python-list@python.org>
References: <y-6AMIAbWkvLII_tfcbRs5NtbKgQRos3XlZ83qbGCwuq7RSQJBGniGJEn4Nzwr_iVQfJOdvLVzOycTR0V-F-pDKt2rNPj3cHmFVY06vNQnI=@protonmail.ch> <DGC1c0YSrMcV0n6rNNeplHySXzBWUfTUwJbBVPgOocO1evufk5MKe8QWWEzO6Zj5-_0VrJJbK3OpB5-vrvMdDRKITiZrvq0RMY69IND05u4=@protonmail.ch> <fd0e2e55-34ef-4149-84c8-a2a26596b6b0@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Cancel-Lock: sha1:0MvFx5OIYKXJAlC+e46ou5S3bU8= sha256:Rxz8uJP5GmK3ntWx/RHRHZMtLkydquiNiEVpUeGpv3Y=
User-Agent: Mozilla Thunderbird
Content-Language: en-GB
In-Reply-To: <DGC1c0YSrMcV0n6rNNeplHySXzBWUfTUwJbBVPgOocO1evufk5MKe8QWWEzO6Zj5-_0VrJJbK3OpB5-vrvMdDRKITiZrvq0RMY69IND05u4=@protonmail.ch>
Precedence: list
Xref: csiph.com comp.lang.python:196401

On 2024-07-07 20:40, Tomas Ukkonen via Python-list wrote:
> Hi
> 
> There was a bug in the example code. I fixed it and it STILL deadlocks (my larger software project deadlocks when I call python from C++).
> 
> Updated code:
> 
> /* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
>   *
>   * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
>   * ./a.out
>   *
>   * uname:
>   * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
>   */
> 
> #include <Python.h>
> #include <thread>
> #include <vector>
> #include <iostream>
> 
> 
> void thread_function() {
>      // Ensure this thread has the GIL
>      PyGILState_STATE gstate = PyGILState_Ensure();
> 
>      // Execute some Python code
>      PyRun_SimpleString("print('Hello from std::thread!')");
> 
>      // Release the GIL
>      PyGILState_Release(gstate);
> }
> 
> int main() {
>      // Initialize the Python Interpreter
>      Py_Initialize();
> 
>      // Create a vector of threads
>      std::vector<std::thread> threads;
> 
>      // Launch threads
>      for (int i = 0; i < 5; ++i) {
>          threads.push_back(std::thread(thread_function));
>      }
> 
>      // Join threads
>      for (auto& t : threads) {
>          t.join();
>      }
> 
>      // Finalize the Python Interpreter
>      Py_Finalize();
> 
>      return 0;
> }
> sunnuntaina 7. heinäkuuta 2024 klo 10:24 ip, Tomas Ukkonen <tomas.ukkonen@protonmail.ch> kirjoitti:
> 
>> Hello
>> Is this python c api bug? The following C++ code (g++) deadlocks on Ubuntu Linux.
>> 
> 
>> /* * This code deadlocks on python3-dev 3.12.3 (ubuntu 24.04 lts)
>>  *
>>  * g++ python_thread_test.cpp `python3-config --cflags --libs --embed`
>>  * ./a.out
>>  *
>>  * uname:
>>  * Linux softice 6.8.0-36-generic SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
>>  */
>> 
> 
>> #include <Python.h>
>> #include <thread>
>> #include <vector>
>> #include <iostream>
>> 
> 
>> void perform_long_operation() {
>>     // Simulate a long-running task
>>     std::this_thread::sleep_for(std::chrono::seconds(5));
>> }
>> 
> 
>> void thread_function() {
>>     // Ensure this thread has the GIL
>>     PyGILState_STATE gstate = PyGILState_Ensure();
>> 
> 
>>     // Execute some Python code
>>     PyRun_SimpleString("print('Hello from std::thread!')");
>> 
> 
>>     // Release the GIL for long operation
>>     Py_BEGIN_ALLOW_THREADS
>>     perform_long_operation();
>>     Py_END_ALLOW_THREADS
>> 
> 
>>     // Re-acquire the GIL and execute more Python code
>>     gstate = PyGILState_Ensure();
>>     PyRun_SimpleString("print('Thread operation completed!')");
>> 
> 
>>     // Release the GIL
>>     PyGILState_Release(gstate);
>> }
>> 
> 
>> int main() {
>>     // Initialize the Python Interpreter
>>     Py_Initialize();
>> 
At this point, there's only one thread (the main thread) and it owns the 
GIL.
> 
>>     // Create a vector of threads
>>     std::vector<std::thread> threads;
>> 
> 
>>     // Launch threads
>>     for (int i = 0; i < 5; ++i) {
>>         threads.push_back(std::thread(thread_function));
>>     }
>> 
The threads will each try to acquire and release the GIL, but it's still 
owned by the main thread.
> 
>>     // Join threads
>>     for (auto& t : threads) {
>>         t.join();
>>     }
The main thread is waiting for the sub-threads to finish, and the 
threads waiting for the GIL, but the main thread still owns the GIL, so 
they'll be waiting forever. Deadlock.
>> 
> 
>>     // Finalize the Python Interpreter
>>     Py_Finalize();
>> 
> 
>>     return 0;
>> }
>>