Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1275156 > unrolled thread

Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0')

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-19 13:10 +0100
    Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Cordell Bloor <cgmb@slerp.xyz> - 2025-12-19 19:30 +0100
      Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-19 21:20 +0100
        Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-19 21:20 +0100
          Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Cordell Bloor <cgmb@slerp.xyz> - 2025-12-19 23:50 +0100
            Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-20 00:50 +0100
              Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-20 01:00 +0100
                Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0') Petter Reinholdtsen <pere@hungry.com> - 2025-12-20 01:10 +0100

#1275156 — Bug#1123651: llama-server: Crash with SIGABRT on assertion (`system_regions_fine_.size() > 0')

Package: llama.cpp-tools
Version: 5882+dfsg-3

Dear Maintainer,

The llama-server program now crashes hwere it used to succeed.  This is
the message reported when trying to start it:

  % llama-server -ngl 256  -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../models-llama/DeepSeek-R1-Distill-Qwen-14B-Q8_0.gguf
  load_backend: loaded BLAS backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-blas.so
  ROCm calling rocblas_initialize as a workaround for a rocBLAS bug
  llama-server: ./src/core/runtime/runtime.cpp:198: void rocr::core::Runtime::RegisterAgent(rocr::core::Agent*, bool): Assertion `system_regions_fine_.size() > 0' failed.
  Abort (SIGABRT)          llama-server -ngl 256 -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../models-llama/DeepSeek-R1-Distill-Qwen-14B-Q8_0.gguf
  %

I have not tested other models.

-- System Information:
Debian Release: forky/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 6.17.11+deb14-amd64 (SMP w/32 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=nb_NO.UTF-8, LC_CTYPE=nb_NO.UTF-8 (charmap=UTF-8), LANGUAGE=nb_NO:nb:no_NO:no:nn_NO:nn:da:sv:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages llama.cpp-tools depends on:
ii  libc6        2.42-5
ii  libcurl4t64  8.17.0-3
ii  libgcc-s1    15.2.0-11
ii  libggml0     0.0~git20250712.d62df60-5
ii  libllama0    5882+dfsg-3
ii  libstdc++6   15.2.0-11

llama.cpp-tools recommends no packages.

llama.cpp-tools suggests no packages.

-- no debconf information

[toc] | [next] | [standalone]

#1275192

[Multipart message — attachments visible in raw view] — view raw

Hi Petter,

Could you provide the versions of libamdhip64, libhsakmt1, libhsa-runtime64, librocblas, and libhipblas?

What CPU and GPU are you using? The output of rocminfo can provide that information.

My guess would be that this may be caused by updates to libhsakmt1 or libhsa-runtime64-1, but it's plausible it could be related to the Linux kernel driver.

Sincerely,
Cory Bloor

[toc] | [prev] | [next] | [standalone]

#1275201

Hi Cordell,

[Cordell Bloor]
> Could you provide the versions of libamdhip64, libhsakmt1,
> libhsa-runtime64, librocblas, and libhipblas?
>
> What CPU and GPU are you using? The output of rocminfo can provide
> that information.

Sure.

% rocminfo; dpkg -l libamdhip64 libhsakmt1 libhsa-runtime64 librocblas libhipblas
ROCk module is loaded
rocminfo: ./src/core/runtime/runtime.cpp:198: void rocr::core::Runtime::RegisterAgent(rocr::core::Agent*, bool): Assertion `system_regions_fine_.size() > 0' failed.
Aborted                    rocminfo
dpkg-query: no packages found matching libamdhip64
dpkg-query: no packages found matching libhsa-runtime64
dpkg-query: no packages found matching librocblas
dpkg-query: no packages found matching libhipblas
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version      Architecture Description
+++-================-============-============-=================================
ii  libhsakmt1:amd64 6.4.3+dfsg-4 amd64        Thunk library for AMD KFD (shlib)
%

Trying to install the packages only claimed no such packages exist.

As rocminfo failed, here is the lspci info:

% lspci|grep VGA
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX/7900 GRE/7900M] (rev c8)
%

/proc/pcuinfo report "AMD Ryzen 9 7950X 16-Core Processor" as the CPU.

-- 
Happy hacking
Petter Reinholdtsen

[toc] | [prev] | [next] | [standalone]

#1275202

I tried a more flexible dpkg call, and got this:

% rocminfo; dpkg -l libamdhip64* libhsakmt1* libhsa-runtime64* librocblas* libhipblas*
ROCk module is loaded
rocminfo: ./src/core/runtime/runtime.cpp:198: void rocr::core::Runtime::RegisterAgent(rocr::core::Agent*, bool): Assertion `system_regions_fine_.size() > 0' failed.
Aborted                    rocminfo
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                     Version      Architecture Description
+++-========================-============-============-=================================================================
ii  libamdhip64-5:amd64      5.7.1-7      amd64        Heterogeneous Interface for Portability - AMD GPUs implementation
ii  libamdhip64-dev          5.7.1-7      amd64        Header files for the AMD implementation of HIP
un  libamdhip64-doc          <none>       <none>       (no description available)
ii  libhipblas-dev           5.5.1-4      amd64        portable interface for GPU BLAS - headers
un  libhipblas-doc           <none>       <none>       (no description available)
ii  libhipblas0              5.5.1-4      amd64        portable interface for GPU BLAS - library
ii  libhsa-runtime64-1:amd64 6.1.2-3      amd64        HSA Runtime API and runtime for ROCm
ii  libhsakmt1:amd64         6.4.3+dfsg-4 amd64        Thunk library for AMD KFD (shlib)
ii  librocblas-dev           5.5.1+dfsg-7 amd64        ROCm library for basic linear algebra - headers
un  librocblas-doc           <none>       <none>       (no description available)
ii  librocblas0              5.5.1+dfsg-7 amd64        ROCm library for basic linear algebra - library
%

-- 
Happy hacking
Petter Reinholdtsen

[toc] | [prev] | [next] | [standalone]

#1275228

[Multipart message — attachments visible in raw view] — view raw

Hi Petter,

I suspect that the problem is libhsakmt1 6.4.3+dfsg-4. You could try installing libhsakmt1 6.1.2-3 from stable to see if that fixes the problem.

The upstream version of libhsakmt1 should match that of libhsa-runtime64-1. You cannot upgrade the libhsa-runtime64-1, so you must downgrade libhsakmt1.

Of course, it is still possible that it's really a driver problem. You could try booting into a slightly older kernel if downgrading libhsakmt1 does not work.

Sincerely,
Cory Bloor

On December 19, 2025 12:33:27 p.m. MST, Petter Reinholdtsen <pere@hungry.com> wrote:
>
>I tried a more flexible dpkg call, and got this:
>
>% rocminfo; dpkg -l libamdhip64* libhsakmt1* libhsa-runtime64* librocblas* libhipblas*
>ROCk module is loaded
>rocminfo: ./src/core/runtime/runtime.cpp:198: void rocr::core::Runtime::RegisterAgent(rocr::core::Agent*, bool): Assertion `system_regions_fine_.size() > 0' failed.
>Aborted                    rocminfo
>Desired=Unknown/Install/Remove/Purge/Hold
>| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>||/ Name                     Version      Architecture Description
>+++-========================-============-============-=================================================================
>ii  libamdhip64-5:amd64      5.7.1-7      amd64        Heterogeneous Interface for Portability - AMD GPUs implementation
>ii  libamdhip64-dev          5.7.1-7      amd64        Header files for the AMD implementation of HIP
>un  libamdhip64-doc          <none>       <none>       (no description available)
>ii  libhipblas-dev           5.5.1-4      amd64        portable interface for GPU BLAS - headers
>un  libhipblas-doc           <none>       <none>       (no description available)
>ii  libhipblas0              5.5.1-4      amd64        portable interface for GPU BLAS - library
>ii  libhsa-runtime64-1:amd64 6.1.2-3      amd64        HSA Runtime API and runtime for ROCm
>ii  libhsakmt1:amd64         6.4.3+dfsg-4 amd64        Thunk library for AMD KFD (shlib)
>ii  librocblas-dev           5.5.1+dfsg-7 amd64        ROCm library for basic linear algebra - headers
>un  librocblas-doc           <none>       <none>       (no description available)
>ii  librocblas0              5.5.1+dfsg-7 amd64        ROCm library for basic linear algebra - library
>%
>
>-- 
>Happy hacking
>Petter Reinholdtsen
>

[toc] | [prev] | [next] | [standalone]

#1275244

[Cordell Bloor]
> I suspect that the problem is libhsakmt1 6.4.3+dfsg-4. You could try
> installing libhsakmt1 6.1.2-3 from stable to see if that fixes the
> problem.

Downgrading libhsakmt1 got rocminfo working.  Next, I tried to apt
dist-upgrade and apt install libhsa-runtime-dev to force a consistent
version 6.4.3+dfsg-4 of libhsa-runtime-dev, libhsa-runtime64-1 and
libhsakmt1.  This seem to solve the problem for both rocminfo and
llama-server.

Once this was done, the llama-server program started as it should.

I guess some dependencies, conflicts or breaks should be added to make
sure upgrades avoid incomplatible packages.

-- 
Happy hacking
Petter Reinholdtsen

[toc] | [prev] | [next] | [standalone]

#1275246

Petter Reinholdtsen <pere@hungry.com> writes:

> [Cordell Bloor]
>
> Once this was done, the llama-server program started as it should.

Not quite.  It started and no longer crashed, but not as it should.  I
starts with this:

load_backend: loaded BLAS backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-blas.so
load_backend: loaded CPU backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-cpu-icelake.so
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions
build: 5882 (Debian) with cc (Debian 14.3.0-5) 14.3.0 for x86_64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32


And there is no sign of any GPU being used. :(

I tried to work around this by installing libggml0-backend-hip, but
alas, it do not work:

% LC_ALL=C sudo apt install libggml0-backend-hip
Solving dependencies... Error!  
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

Unsatisfied dependencies:
 libggml0-backend-hip : Depends: libggml0 (>= 0.9.4) but 0.0~git20250712.d62df60-5 is to be installed
                        Depends: libhipblas2 (>= 6.4.3) but it is not installable
                        Depends: librocblas4 (>= 5.3.3) but it is not installable
Error: Unable to satisfy dependencies. Reached two conflicting decisions:
   1. libggml0:amd64=0.9.4-4+hip~exp2 is not selected for install
   2. libggml0:amd64=0.9.4-4+hip~exp2 is selected as an upgrade because:
      1. libggml0-backend-hip:amd64=0.9.4-4+hip~exp2 is selected for install
      2. libggml0-backend-hip:amd64 Depends libggml0 (>= 0.9.4)
%

I tried disabling experimental as a source in sources.list, but then I
only got this:

Package libggml0-backend-hip is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Error: Package 'libggml0-backend-hip' has no installation candidate

-- 
Happy hacking
Petter Reinholdtsen

[toc] | [prev] | [next] | [standalone]

#1275247

I have completely misunderstood.  I thought I got llama.cpp from the
Debian archive, but apparently it is missing in testing, so the packages
I got was a mixture of incompatible versions.  When I tried to remove
all ggml and llama.cpp related packages and reinstall them, I discovered
that there are no packages available from Trixie.

And trying to build the source from salsa failed too, as the HIP edition
of ggml is currently disabled. :)
-- 
Happy hacking
Petter Reinholdtsen

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web