Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #58206 > unrolled thread

Bug#865549: cargo still segfaults / rustc still FTBFS with the newest fixed kernel

Back to article view | Back to linux.debian.kernel

Contents

  Bug#865549: cargo still segfaults / rustc still FTBFS with the newest fixed kernel Ximin Luo <infinity0@debian.org> - 2017-06-29 17:20 +0200
    Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel Ximin Luo <infinity0@debian.org> - 2017-06-29 17:30 +0200
    Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel Ximin Luo <infinity0@debian.org> - 2017-06-29 19:50 +0200
      Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel Ben Hutchings <ben@decadent.org.uk> - 2017-06-30 00:40 +0200
        Bug#865549: [Pkg-rust-maintainers] Bug#865549: cargo still segfaults / rustc still FTBFS with the newest fixed kernel Ximin Luo <infinity0@debian.org> - 2017-06-30 11:10 +0200

#58206 — Bug#865549: cargo still segfaults / rustc still FTBFS with the newest fixed kernel

Control: unmerge 866185
Control: unmerge 865549
Control: forcemerge 865549 866185
Control: notfixed linux/4.11.6-1

Unfortunately I spoke too soon, the patch does not work. I had been testing with `cargo -V` and the segfault on this is now fixed by the patch, but `cargo build` still segfaults. More details in https://github.com/rust-lang/cargo/issues/4197

Disabling ASLR or setting `ulimit -s 24576` works around the segfault.

I am still fairly certain the issue is a kernel problem, because before June 20 I had built these things 5 times successfully on Debian buildds and more times on porterboxes:

https://buildd.debian.org/status/logs.php?pkg=rustc see versions 1.17.0+dfsg2-1 to 1.17.0+dfsg2-5

Some help would be appreciated as I have no idea how to debug these types of low-level situations. I'll try to step through the program at least and send the output of that here.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[toc] | [next] | [standalone]

#58207 — Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel

Control: notfixed -1 4.11.6-1
Control: affects -1 - jsvc security.debian.org libreoffice

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[toc] | [prev] | [next] | [standalone]

#58211 — Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel

Ximin Luo:
> [..]
> 
> The segfault occurs on various commands at different frequencies and after 
> differing amounts of time (but all less than a few seconds). The stack traces 
> are all different too. [..]

With some help from arielby from #rust-internals, we noticed that the stack getting allocated was always 192KB even though `ulimit -s` says 8192 (i.e. 8MB), and when the program tries to grow beyond this, is when the segfaults occur. Hope that's useful.

(sid_ppc64el-dchroot)infinity0@plummer:~/test/rust-1.18.0-powerpc64le-unknown-linux-gnu$ gdb -q -d ~/glibc-2.24/debian/ -d ~/glibc-2.24/malloc/ -d ~/glibc-2.24/stdio-common/ -ex 'set disable-randomization off' -ex 'run install regex' cargo/bin/cargo 
Reading symbols from cargo/bin/cargo...done.
Starting program: /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo install regex
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000028ada598 in regex_syntax::parser::Parser::parse_expr::h69b18aabf3de27d8 ()
(gdb) info inferiors
  Num  Description       Executable        
* 1    process 30272     /home/infinity0/test/rust-1.18.0-powerpc64le-unknown-linux-gnu/cargo/bin/cargo 
(gdb) shell command grep stack /proc/30272/maps
3fffcf960000-3fffcf990000 rw-p 00000000 00:00 0                          [stack]

(gdb) shell command grep stack /proc/29614/maps
3ffff4360000-3ffff4390000 rw-p 00000000 00:00 0                          [stack]

(gdb) shell command grep stack /proc/29586/maps
3fffe2800000-3fffe2830000 rw-p 00000000 00:00 0                          [stack]

Also, I earlier stated that this could be worked around by disabling ASLR (which gdb does implicitly, so we had to re-enable it). This was true on a deb8u1 kernel, but seems no longer to be true on the newer deb8u2 kernels - i.e. now when disabling ASLR I still see the segfaults. :(

Running using the Debian rustc and cargo package, with the same command line that the rustc build would run:

(sid_ppc64el-dchroot)infinity0@plummer:~/rustc$ gdb -q -ex 'run build --manifest-path /home/infinity0/rustc/src/bootstrap/Cargo.toml --frozen' cargo
Reading symbols from cargo...Reading symbols from /usr/lib/debug/.build-id/e9/f3607a78a9b60bc2112c3810f72b42b9a5e1f9.debug...done.
done.
Starting program: /usr/bin/cargo build --manifest-path /home/infinity0/rustc/src/bootstrap/Cargo.toml --frozen
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000020376b78 in regex::exec::ExecBuilder::build::h46818bd83e889eca ()
(gdb) info inferiors
  Num  Description       Executable        
* 1    process 32472     /usr/bin/cargo    
(gdb) shell command grep stack /proc/32472/maps
3ffffffd0000-400000000000 rw-p 00000000 00:00 0                          [stack]

Setting `ulimit -s 24576` still works, though.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[toc] | [prev] | [next] | [standalone]

#58221 — Bug#865549: [Pkg-rust-maintainers] cargo still segfaults / rustc still FTBFS with the newest fixed kernel

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2017-06-29 at 17:45 +0000, Ximin Luo wrote:
> Ximin Luo:
> > [..]
> > 
> > The segfault occurs on various commands at different frequencies and after 
> > differing amounts of time (but all less than a few seconds). The stack traces 
> > are all different too. [..]
> 
> With some help from arielby from #rust-internals, we noticed that the
> stack getting allocated was always 192KB even though `ulimit -s` says
> 8192 (i.e. 8MB),

That's normal.

> and when the program tries to grow beyond this, is when the segfaults
> occur. Hope that's useful.

That's obviously not.  A page fault below the stack mapping and not too
far below the stack pointer should cause the mapping to expand, up to
the limit, without the process having to do anything about it.

The shell ulimit command, and the underlying setrlimit(2) system call,
set the limit for expansion of the stack mapping.  Expansion is also
restricted by the requirement of a gap between the stack and the next
mapping below, but so long as the size limit is not 'unlimited' this
*should* make no difference.  The gap used to be 1 page and is now
configurable but defaults to 256 pages.

The stack gap used to be included in the range shown in /proc/*/maps
and was counted toward the stack limit.  The first version of the fix
did not change that, so it effectively reduced all existing limits by
255 pages.  The second version of the fix excludes the stack gap from
both.

This expanding behaviour normally only applies to the initial thread's
stack mapping, which is allocated automatically by execve(2).  For new
threads created using clone(2) the caller usually sets up a fixed size
stack mapping.

Hopefully that explanation can help you and the Rust developers
identify what's upsetting Rust, and I can then try to get the kernel
behaviour further refined to avoid doing so.

[...]
> Also, I earlier stated that this could be worked around by disabling
> ASLR (which gdb does implicitly, so we had to re-enable it). This was
> true on a deb8u1 kernel, but seems no longer to be true on the newer
> deb8u2 kernels - i.e. now when disabling ASLR I still see the
> segfaults. :(
> 
> Running using the Debian rustc and cargo package, with the same
> command line that the rustc build would run:
> 
> > (sid_ppc64el-dchroot)infinity0@plummer:~/rustc$ gdb -q -ex 'run build --manifest-path /home/infinity0/rustc/src/bootstrap/Cargo.toml --frozen' cargo
> Reading symbols from cargo...Reading symbols from /usr/lib/debug/.build-id/e9/f3607a78a9b60bc2112c3810f72b42b9a5e1f9.debug...done.
> done.
> Starting program: /usr/bin/cargo build --manifest-path /home/infinity0/rustc/src/bootstrap/Cargo.toml --frozen
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000020376b78 in regex::exec::ExecBuilder::build::h46818bd83e889eca ()
> (gdb) info inferiors
>   Num  Description       Executable        
> * 1    process 32472     /usr/bin/cargo    
> (gdb) shell command grep stack /proc/32472/maps
> 3ffffffd0000-400000000000 rw-p 00000000 00:00 0                          [stack]
> 
> Setting `ulimit -s 24576` still works, though.

Do you know how much stack space this function needs?  Can you get the
page fault address?  (I don't remember how to do that in gdb.)

Ben.

-- 
Ben Hutchings
When in doubt, use brute force. - Ken Thompson

[toc] | [prev] | [next] | [standalone]

#58222 — Bug#865549: [Pkg-rust-maintainers] Bug#865549: cargo still segfaults / rustc still FTBFS with the newest fixed kernel

Ben Hutchings:
> On Thu, 2017-06-29 at 17:45 +0000, Ximin Luo wrote:
>> Ximin Luo:
>>> [..]
>>>
>>> The segfault occurs on various commands at different frequencies and after 
>>> differing amounts of time (but all less than a few seconds). The stack traces 
>>> are all different too. [..]
>>
>> With some help from arielby from #rust-internals, we noticed that the
>> stack getting allocated was always 192KB even though `ulimit -s` says
>> 8192 (i.e. 8MB),
> 
> That's normal.
> 
>> and when the program tries to grow beyond this, is when the segfaults
>> occur. Hope that's useful.
> 
> That's obviously not.  A page fault below the stack mapping and not too
> far below the stack pointer should cause the mapping to expand, up to
> the limit, without the process having to do anything about it.
> 
> The shell ulimit command, and the underlying setrlimit(2) system call,
> set the limit for expansion of the stack mapping.  Expansion is also
> restricted by the requirement of a gap between the stack and the next
> mapping below, but so long as the size limit is not 'unlimited' this
> *should* make no difference.  The gap used to be 1 page and is now
> configurable but defaults to 256 pages.
> 
> The stack gap used to be included in the range shown in /proc/*/maps
> and was counted toward the stack limit.  The first version of the fix
> did not change that, so it effectively reduced all existing limits by
> 255 pages.  The second version of the fix excludes the stack gap from
> both.
> 
> [..]

By the way, PAGESIZE on these ppc64el machines is 64KB not 4KB, so a 256-page stack guard would be 16MB which is way higher than the default `ulimit -s` of 8MB.

Although you said that the stack guard (in theory) should not be part of the stack limit, the fact that a 24MB limit made the segfaults go away yet a 16MB did not, suggests there might be a link. What do you think?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web