Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1499201 > unrolled thread
| Started by | Laura Abbott <labbott@redhat.com> |
|---|---|
| First post | 2016-10-11 22:50 +0200 |
| Last post | 2016-10-12 20:20 +0200 |
| Articles | 4 — 2 participants |
Back to article view | Back to linux.kernel
Intermittent perf build failures Laura Abbott <labbott@redhat.com> - 2016-10-11 22:50 +0200
Re: Intermittent perf build failures Jiri Olsa <jolsa@redhat.com> - 2016-10-11 23:00 +0200
Re: Intermittent perf build failures Jiri Olsa <jolsa@redhat.com> - 2016-10-12 12:20 +0200
Re: Intermittent perf build failures Laura Abbott <labbott@redhat.com> - 2016-10-12 20:20 +0200
| From | Laura Abbott <labbott@redhat.com> |
|---|---|
| Date | 2016-10-11 22:50 +0200 |
| Subject | Intermittent perf build failures |
| Message-ID | <src9Q-2XG-11@gated-at.bofh.it> |
Hi,
While building today's Fedora rawhide kernel, there was a failure
building perf with -j4 [1]:
gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
ld -r -o plugin_function-in.o plugin_function.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_jbd2.so plugin_jbd2-in.o
gcc -Wp,-MD,./.plugin_cfg80211.o.d,-MT,plugin_cfg80211.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_cfg80211.o plugin_cfg80211.c
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_hrtimer.so plugin_hrtimer-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kmem.so plugin_kmem-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kvm.so plugin_kvm-in.o
ld -r -o plugin_xen-in.o plugin_xen.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_mac80211.so plugin_mac80211-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_sched_switch.so plugin_sched_switch-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_function.so plugin_function-in.o
ld -r -o plugin_cfg80211-in.o plugin_cfg80211.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_xen.so plugin_xen-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_cfg80211.so plugin_cfg80211-in.o
gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
ld -r -o plugin_scsi-in.o plugin_scsi.o
plugin_scsi.o: file not recognized: File truncated
make[3]: *** [/builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/build/Makefile.build:140: plugin_scsi-in.o] Error 1
make[2]: *** [Makefile:189: plugin_scsi-in.o] Error 2
make[1]: *** [Makefile.perf:570: libtraceevent_plugins] Error 2
make: *** [Makefile:109: install-traceevent-plugins] Error 2
The gcc call to compile plugin_scsi.o seems to be invoked twice which would
account for the ld failing. Unfortunately, this failure seems to be
intermittent and I've only been able to reproduce it once outside of
the Fedora build environment so bisection doesn't seem feasible.
The build invocation for this was changed in
0c3b7e42616f ("tools build: Add support for host programs format") so
I'm suspicious of that commit. I'm reverting that commit for today's
build but I'm not sure how much that's going to indicate given the
intermittent nature of the failure.
Any ideas?
Thanks,
Laura
[1] https://kojipkgs.fedoraproject.org//work/tasks/415/16050415/build.log
[toc] | [next] | [standalone]
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2016-10-11 23:00 +0200 |
| Message-ID | <srcjw-30Y-5@gated-at.bofh.it> |
| In reply to | #1499201 |
On Tue, Oct 11, 2016 at 01:43:36PM -0700, Laura Abbott wrote:
> Hi,
>
> While building today's Fedora rawhide kernel, there was a failure
> building perf with -j4 [1]:
>
>
> gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
> ld -r -o plugin_function-in.o plugin_function.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_jbd2.so plugin_jbd2-in.o
> gcc -Wp,-MD,./.plugin_cfg80211.o.d,-MT,plugin_cfg80211.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_cfg80211.o plugin_cfg80211.c
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_hrtimer.so plugin_hrtimer-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kmem.so plugin_kmem-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kvm.so plugin_kvm-in.o
> ld -r -o plugin_xen-in.o plugin_xen.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_mac80211.so plugin_mac80211-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_sched_switch.so plugin_sched_switch-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_function.so plugin_function-in.o
> ld -r -o plugin_cfg80211-in.o plugin_cfg80211.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_xen.so plugin_xen-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_cfg80211.so plugin_cfg80211-in.o
> gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
> ld -r -o plugin_scsi-in.o plugin_scsi.o
> plugin_scsi.o: file not recognized: File truncated
> make[3]: *** [/builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/build/Makefile.build:140: plugin_scsi-in.o] Error 1
> make[2]: *** [Makefile:189: plugin_scsi-in.o] Error 2
> make[1]: *** [Makefile.perf:570: libtraceevent_plugins] Error 2
> make: *** [Makefile:109: install-traceevent-plugins] Error 2
>
> The gcc call to compile plugin_scsi.o seems to be invoked twice which would
> account for the ld failing. Unfortunately, this failure seems to be
> intermittent and I've only been able to reproduce it once outside of
> the Fedora build environment so bisection doesn't seem feasible.
hi,
yep, looks like build race.. but -j4 should not be that bad ;-)
some questions:
- how often do you see this?
- is it always plugin_scsi.o failing?
- hows the build invoked?
- could you please attach full build log?
thanks,
jirka
>
> The build invocation for this was changed in
> 0c3b7e42616f ("tools build: Add support for host programs format") so
> I'm suspicious of that commit. I'm reverting that commit for today's
> build but I'm not sure how much that's going to indicate given the
> intermittent nature of the failure.
>
> Any ideas?
>
> Thanks,
> Laura
>
>
> [1] https://kojipkgs.fedoraproject.org//work/tasks/415/16050415/build.log
[toc] | [prev] | [next] | [standalone]
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2016-10-12 12:20 +0200 |
| Message-ID | <sroNH-33M-11@gated-at.bofh.it> |
| In reply to | #1499202 |
On Tue, Oct 11, 2016 at 02:18:49PM -0700, Laura Abbott wrote: > On 10/11/2016 01:59 PM, Jiri Olsa wrote: > > On Tue, Oct 11, 2016 at 01:43:36PM -0700, Laura Abbott wrote: > > > Hi, > > > > > > While building today's Fedora rawhide kernel, there was a failure > > > building perf with -j4 [1]: ok, the -j 4 is the problem running "make -j 4 install-bin install-traceevent-plugins" BUILD: Doing 'make -j4' parallel build BUILD: Doing 'make -j4' parallel build will run paralel make instances for install-bin and install-traceevent-plugins which will eventually touch same files and crash.. the main perf Makefile is actualy detecting number of cpus and runs Makefile.perf with -j X option so there's no need to specify it on top level.. you can always customize it via JOBS=X make variable so if you don't specify the -j X option it will run the 'Makefile.perf install-bin install-traceevent-plugins' with -j X set and it should execute sequentialy and fix your problem thanks, jirka
[toc] | [prev] | [next] | [standalone]
| From | Laura Abbott <labbott@redhat.com> |
|---|---|
| Date | 2016-10-12 20:20 +0200 |
| Message-ID | <srwid-838-5@gated-at.bofh.it> |
| In reply to | #1499462 |
On 10/12/2016 03:12 AM, Jiri Olsa wrote: > On Tue, Oct 11, 2016 at 02:18:49PM -0700, Laura Abbott wrote: >> On 10/11/2016 01:59 PM, Jiri Olsa wrote: >>> On Tue, Oct 11, 2016 at 01:43:36PM -0700, Laura Abbott wrote: >>>> Hi, >>>> >>>> While building today's Fedora rawhide kernel, there was a failure >>>> building perf with -j4 [1]: > > ok, the -j 4 is the problem > > running "make -j 4 install-bin install-traceevent-plugins" > > BUILD: Doing 'make -j4' parallel build > BUILD: Doing 'make -j4' parallel build > > will run paralel make instances for install-bin and install-traceevent-plugins > which will eventually touch same files and crash.. > > the main perf Makefile is actualy detecting number of cpus > and runs Makefile.perf with -j X option so there's no need > to specify it on top level.. you can always customize it via > JOBS=X make variable > > so if you don't specify the -j X option it will run the > 'Makefile.perf install-bin install-traceevent-plugins' with > -j X set and it should execute sequentialy and fix your problem > Thanks for pointing that out. The build command has been there a while and apparently has bit rotted. I've removed the extra -j X flags from the command. > thanks, > jirka > Thanks, Laura
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web