Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1025182 > unrolled thread

Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Matthew Vernon <matthew@debian.org> - 2020-09-15 13:40 +0200
    Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Timo Sirainen <timo@sirainen.com> - 2020-09-17 23:30 +0200
      Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Noah Meyerhans <noahm@debian.org> - 2020-09-20 19:30 +0200
    Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Dimitry Andric <dimitry@andric.com> - 2020-09-21 15:50 +0200
      Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Noah Meyerhans <noahm@debian.org> - 2020-09-22 00:50 +0200
        Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Noah Meyerhans <noahm@debian.org> - 2020-09-22 20:10 +0200
      Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Timo Sirainen <timo@sirainen.com> - 2020-09-23 12:40 +0200
        Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Noah Meyerhans <noahm@debian.org> - 2020-09-23 17:50 +0200
          Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder Noah Meyerhans <noahm@debian.org> - 2020-09-23 22:30 +0200

#1025182 — Bug#970386: dovecot-imapd: assertion failure in message_part_finish when searching large folder

Package: dovecot-imapd
Version: 1:2.2.27-3+deb9u6
Severity: important

Hi,

One of my IMAP users reports failures when trying to do full-text
searches of a large (3G) mailbox; subject-only searches are OK.

The backtrace in syslog is:

Sep 15 11:51:37 aragorn dovecot: imap(atreic): Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)
Sep 15 11:51:37 aragorn dovecot: imap(atreic): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x96442) [0x7f7a45a5d442] -> /usr/lib/dovecot/libdovecot.so.0(+0x9653d) [0x7f7a45a5d53d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7a459f2b41] -> /usr/lib/dovecot/libdovecot.so.0(+0x7de96) [0x7f7a45a44e96] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0xdc) [0x7f7a45a4645c] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa0) [0x7f7a45a47d30] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xb289f) [0x7f7a45d9c89f] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) [0x7f7a45d24685] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xb24b0) [0x7f7a45d9c4b0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xb4513) [0x7f7a45d9e513] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x114) [0x7f7a45d9ec24] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x22) [0x7f7a45d2b382] -> dov
 ecot/imap(+0x2241f) [0x561b87f2a41f] -> dovecot/imap(command_exec+0xa5) [0x561b87f23735] -> dovecot/imap(+0x21522) [0x561b87f29522] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x106) [0x7f7a45a71e06] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xc3) [0x7f7a45a73593] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f7a45a7203c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f7a45a721e8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7a459f9083] -> dovecot/imap(main+0x328) [0x561b87f14e68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f7a456482e1] -> dovecot/imap(_start+0x2a) [0x561b87f14fea]
Sep 15 11:51:37 aragorn dovecot: imap(atreic): Fatal: master: service(imap): child 9125 killed with signal 6 (core dumps disabled)

Regards,

Matthew

-- Package-specific info:

dovecot configuration
---------------------
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-12-amd64 x86_64 Debian 9.13 
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = 
}
passdb {
  driver = pam
}
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
protocols = " imap"
ssl_cert = </var/lib/dehydrated/certs/aragorn.weathertop.principate.org.uk/fullchain.pem
ssl_key =  # hidden, use -P to show it
userdb {
  driver = passwd
}

-- System Information:
Debian Release: 9.13
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-12-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages dovecot-imapd depends on:
ii  dovecot-core  1:2.2.27-3+deb9u6
ii  libbz2-1.0    1.0.6-8.1
ii  libc6         2.24-11+deb9u4
ii  liblz4-1      0.0~r131-2+b1
ii  liblzma5      5.2.2-1.2+b1
ii  ucf           3.0036
ii  zlib1g        1:1.2.8.dfsg-5

dovecot-imapd recommends no packages.

Versions of packages dovecot-imapd suggests:
pn  ufw  <none>

Versions of packages dovecot-imapd is related to:
ii  dovecot-core [dovecot-common]  1:2.2.27-3+deb9u6
pn  dovecot-dbg                    <none>
pn  dovecot-dev                    <none>
ii  dovecot-gssapi                 1:2.2.27-3+deb9u6
ii  dovecot-imapd                  1:2.2.27-3+deb9u6
ii  dovecot-ldap                   1:2.2.27-3+deb9u6
pn  dovecot-lmtpd                  <none>
pn  dovecot-managesieved           <none>
ii  dovecot-mysql                  1:2.2.27-3+deb9u6
ii  dovecot-pgsql                  1:2.2.27-3+deb9u6
pn  dovecot-pop3d                  <none>
ii  dovecot-sieve                  1:2.2.27-3+deb9u6
ii  dovecot-sqlite                 1:2.2.27-3+deb9u6

-- no debconf information

[toc] | [next] | [standalone]

#1025489

[Multipart message — attachments visible in raw view] — view raw

On 15. Sep 2020, at 14.00, Matthew Vernon <matthew@debian.org> wrote:
> One of my IMAP users reports failures when trying to do full-text
> searches of a large (3G) mailbox; subject-only searches are OK.
> 
> The backtrace in syslog is:
> 
> Sep 15 11:51:37 aragorn dovecot: imap(atreic): Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)

The original backported patch for v2.2 was accidentally wrong. Also I'm not sure if Debian backport had the "--" suffix boundary fix either? Attached anyway patches for both fixes.

[toc] | [prev] | [next] | [standalone]

#1025780

On Fri, Sep 18, 2020 at 12:05:09AM +0300, Timo Sirainen wrote:
> > One of my IMAP users reports failures when trying to do full-text
> > searches of a large (3G) mailbox; subject-only searches are OK.
> > 
> > The backtrace in syslog is:
> > 
> > Sep 15 11:51:37 aragorn dovecot: imap(atreic): Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)
> 
> The original backported patch for v2.2 was accidentally wrong. Also I'm not sure if Debian backport had the "--" suffix boundary fix either? Attached anyway patches for both fixes.
> 

Thanks for looking at this, Timo. The 2.2 packages in Debian 9 (stretch
LTS) should have the latest patches.  See [1] and [2].

1. https://salsa.debian.org/debian/dovecot/-/blob/stable/stretch/debian/patches/CVE-2020-12100-14.patch#L57
2. https://salsa.debian.org/debian/dovecot/-/blob/stable/stretch/debian/patches/CVE-2020-12100-16.patch

Matthew, the stack trace doesn't include the most relevant symbols, so
the code path isn't entirely clear.  It's likely that the problem isn't
specifically related to a large mailbox, as originally suggested, but
rather a specific message in that mailbox.  It might be interesting if
we could get a copy of that message, if it can be identified and the
contents aren't sensitive. Feel free to provide it to me privately if
posting it to the BTS isn't desirable.  It'd help track this down, and
I'd be interested in testing the buster and sid dovecot packages against
it.

Thanks
noah

[toc] | [prev] | [next] | [standalone]

#1025889

I have also seen similar panic messages, when a user was searching for
some plain text in their inbox, e.g.:

Sep 21 13:48:12 hostname dovecot: imap-login: Login: user=<username>, method=LOGIN, rip=12.34.56.78, lip=23.45.67.89, mpid=29488, TLS, session=<zs7bbNGvzOI0fYtc>
Sep 21 14:04:00 hostname dovecot: imap(username)<29488><zs7bbNGvzOI0fYtc>: Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)
Sep 21 14:04:00 hostname dovecot: imap(username)<29488><zs7bbNGvzOI0fYtc>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xdb62b) [0x7fa8e060262b] -> /usr/lib/dovecot/libdovecot.so.0(+0xdb6c1) [0x7fa8e06026c1] -> /usr/lib/dovecot/libdovecot.so.0(+0x4a149) [0x7fa8e0571149] -> /usr/lib/dovecot/libdovecot.so.0(+0x474ac) [0x7fa8e056e4ac] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x104) [0x7fa8e05ea844] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa8) [0x7fa8e05ecdf8] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcf89e) [0x7fa8e078589e] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) [0x7fa8e0707445] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd0774) [0x7fa8e0786774] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd1a68) [0x7fa8e0787a68] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x10d) [0x7fa8e078818d] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x28) [0x7fa8e0710e58] -> dovecot/imap [username 12.34.56.78 UID SEARCH](+0x2691f) [0x55e4a815691f] -> dovecot/imap [username 12.34.56.78 UID SEARCH](command_exec+0x70) [0x55e4a814fd80] -> dovecot/imap [username 12.34.56.78 UID SEARCH](+0x25ed2) [0x55e4a8155ed2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x111) [0x7fa8e06188f1] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xd0) [0x7fa8e061a070] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7fa8e0618b7c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x7fa8e0618ce0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fa8e05990d3] -> dovecot/imap [username 12.34.56.78 UID SEARCH](main+0x325) [0x55e4a8140bf5] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7fa8e037909b] -> dovecot/imap [username 12.34.56.78 UID SEARCH](_start+0x2a) [0x55e4a8140d8a]
Sep 21 14:04:00 hostname dovecot: imap(username)<29488><zs7bbNGvzOI0fYtc>: Fatal: master: service(imap): child 29488 killed with signal 6 (core dumped)

Backtrace of one particular crash:

#0  __GI_raise (sig=sig@entry=6)
   at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffa11aac535 in __GI_abort () at abort.c:79
#2  0x00007ffa11ca6481 in default_fatal_finish (status=0,
   type=LOG_TYPE_PANIC) at failures.c:460
#3  fatal_handler_real (ctx=<optimized out>, format=<optimized out>,
   args=<optimized out>) at failures.c:472
#4  0x00007ffa11d376c1 in i_internal_fatal_handler (
   ctx=<optimized out>, format=<optimized out>, args=<optimized out>)
   at failures.c:849
#5  0x00007ffa11ca6149 in i_panic (
   format=format@entry=0x7ffa11d710d8 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:524
#6  0x00007ffa11ca34ac in message_part_finish (ctx=<optimized out>)
   at message-parser.c:174
#7  0x00007ffa11d1f844 in message_parser_parse_next_block (
   ctx=0x563a3b04c000, block_r=block_r@entry=0x7fffa0bb7420)
   at message-parser.c:836
#8  0x00007ffa11d21df8 in message_search_msg_real (
   error_r=0x7fffa0bb7480, parts=<optimized out>,
   input=0x563a3b0557c0, ctx=0x563a3b055e20) at message-search.c:219
#9  message_search_msg (ctx=ctx@entry=0x563a3b055e20,
   input=0x563a3b0557c0, parts=<optimized out>,
   error_r=error_r@entry=0x7fffa0bb7480) at message-search.c:244
#10 0x00007ffa11eba89e in search_body (arg=0x563a3b07edd0,
   ctx=0x7fffa0bb7510) at index-search.c:697
#11 0x00007ffa11e3c445 in mail_search_args_foreach (
   args=args@entry=0x563a3b07edd0,
   callback=callback@entry=0x7ffa11eba810 <search_body>,
   context=context@entry=0x7fffa0bb7510) at mail-search.c:448
#12 0x00007ffa11ebb774 in search_arg_match_text (ctx=0x563a3b050f80,
   args=0x563a3b07edd0) at index-search.c:847
#13 search_match_once (ctx=0x563a3b050f80) at index-search.c:1400
#14 0x00007ffa11ebca68 in search_match_next (ctx=0x563a3b050f80)
   at index-search.c:1507
#15 search_more_with_mail (mail=<optimized out>, ctx=0x563a3b050f80)
   at index-search.c:1624
#16 search_more_with_prefetching (mail_r=<optimized out>,
   ctx=<optimized out>) at index-search.c:1706
#17 search_more (ctx=0x563a3b050f80, mail_r=0x7fffa0bb7670)
   at index-search.c:1780
#18 0x00007ffa11ebd18d in index_storage_search_next_nonblock (
   _ctx=0x563a3b050f80, mail_r=0x7fffa0bb7700,
   tryagain_r=0x7fffa0bb76ff) at index-search.c:1808
#19 0x00007ffa11e45e58 in mailbox_search_next_nonblock (
   ctx=0x563a3b050f80, mail_r=mail_r@entry=0x7fffa0bb7700,
   tryagain_r=tryagain_r@entry=0x7fffa0bb76ff) at mail-storage.c:2140
#20 0x0000563a3abf991f in cmd_search_more (cmd=0x563a3b0013f8)
   at imap-search.c:440
#21 0x0000563a3abf2d80 in command_exec (cmd=0x563a3b0013f8)
   at imap-commands.c:201
#22 0x0000563a3abf8ed2 in cmd_search_more_callback (
   cmd=<optimized out>) at imap-search.c:518
#23 0x00007ffa11d4d8f1 in io_loop_handle_timeouts_real (
   ioloop=0x563a3afdef40) at ioloop.c:665
#24 io_loop_handle_timeouts (ioloop=ioloop@entry=0x563a3afdef40)
   at ioloop.c:679
#25 0x00007ffa11d4f070 in io_loop_handler_run_internal (
   ioloop=ioloop@entry=0x563a3afdef40) at ioloop-epoll.c:194
#26 0x00007ffa11d4db7c in io_loop_handler_run (ioloop=0x563a3afdef40)
   at ioloop.c:750
#27 0x00007ffa11d4dce0 in io_loop_run (ioloop=0x563a3afdef40)
   at ioloop.c:723
#28 0x00007ffa11cce0d3 in master_service_run (service=0x563a3afdedd0,
   callback=callback@entry=0x563a3abfe8b0 <client_connected>)
   at master-service.c:781
#29 0x0000563a3abe3bf5 in main (argc=<optimized out>,
   argv=<optimized out>) at main.c:520

I managed to track down at least three mails that seem to cause this
panic, which appears to be caused by bad MIME headers. (Spammers are
rather fond of these. :)

Note, I cannot attach these mails to this message, as otherwise the
Debian bug system refuses them, since they appear to contain malware,
which was probably the spammer's intent. So I uploaded them here:

https://www.andric.com/debian/bug970386-1.eml.xz
https://www.andric.com/debian/bug970386-2.eml.xz
https://www.andric.com/debian/bug970386-3.eml.xz

Please take care when unpacking and handling these; the only interesting
parts about them are the bad MIME headers!

For instance, bad-mail-1.eml has an "X-Amavis-Alert: BAD HEADER SECTION,
MIME error: error: part did not end with expected boundary" warning,
bad-mail-2.eml has an "X-Amavis-Alert: BAD HEADER SECTION, MIME error:
error: unexpected end of header" warning, and bad-mail-3.eml even has
multiple of these warnings.

I also tried downgrading to 2.3.4.1-5+deb10u2 packages, and that made
the panic messages go away. It now seems obvious that they are caused by
the fixes for CVE-2020-12100 ("Receiving mail with deeply nested MIME
parts leads to resource exhaustion as Dovecot attempts to parse it"):

https://tracker.debian.org/news/1168771/accepted-dovecot-12341-5deb10u3-source-into-proposed-updates-stable-new-proposed-updates/

-Dimitry

[toc] | [prev] | [next] | [standalone]

#1025966

Control: tags -1 + confirmed

On Mon, Sep 21, 2020 at 03:43:31PM +0200, Dimitry Andric wrote:
> I managed to track down at least three mails that seem to cause this
> panic, which appears to be caused by bad MIME headers. (Spammers are
> rather fond of these. :)
> 
> Note, I cannot attach these mails to this message, as otherwise the
> Debian bug system refuses them, since they appear to contain malware,
> which was probably the spammer's intent. So I uploaded them here:
> 
> https://www.andric.com/debian/bug970386-1.eml.xz
> https://www.andric.com/debian/bug970386-2.eml.xz
> https://www.andric.com/debian/bug970386-3.eml.xz

Thanks for these.  I have used them to successfully reproduce a crash
when performing a server-side search using 1:2.2.27-3+deb9u6

noah

[toc] | [prev] | [next] | [standalone]

#1026049

On Mon, Sep 21, 2020 at 03:43:32PM -0700, Noah Meyerhans wrote:
> > Note, I cannot attach these mails to this message, as otherwise the
> > Debian bug system refuses them, since they appear to contain malware,
> > which was probably the spammer's intent. So I uploaded them here:
> > 
> > https://www.andric.com/debian/bug970386-1.eml.xz
> > https://www.andric.com/debian/bug970386-2.eml.xz
> > https://www.andric.com/debian/bug970386-3.eml.xz
> 
> Thanks for these.  I have used them to successfully reproduce a crash
> when performing a server-side search using 1:2.2.27-3+deb9u6

I have also reproduced the crash with the buster version
(1:2.3.4.1-5+deb10u4 from stable-proposed-updates), but I have not been
able to crash with 1:2.3.11.3+dfsg1-2 from sid/bullseye.

noah

[toc] | [prev] | [next] | [standalone]

#1026124

[Multipart message — attachments visible in raw view] — view raw

On 21. Sep 2020, at 16.43, Dimitry Andric <dimitry@andric.com> wrote:
> 
> Sep 21 14:04:00 hostname dovecot: imap(username)<29488><zs7bbNGvzOI0fYtc>: Panic: file message-parser.c: line 174 (message_part_finish): assertion failed: (ctx->nested_parts_count > 0)

Oh, right, this is fixed by: https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b024fb <https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b024fb>

[toc] | [prev] | [next] | [standalone]

#1026173

On Wed, Sep 23, 2020 at 01:34:26PM +0300, Timo Sirainen wrote:
>      Sep 21 14:04:00 hostname dovecot:
>      imap(username)<29488><zs7bbNGvzOI0fYtc>: Panic: file message-parser.c:
>      line 174 (message_part_finish): assertion failed:
>      (ctx->nested_parts_count > 0)
> 
>    Oh, right, this is fixed
>    by: [2]https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b024fb

Well this looks promising.  Thanks!  Will look into cherry-picking this change.

noah

[toc] | [prev] | [next] | [standalone]

#1026204

Control: tags -1 + upstream fixed-upstream buster stretch

On Wed, Sep 23, 2020 at 08:41:19AM -0700, Noah Meyerhans wrote:
> >      Sep 21 14:04:00 hostname dovecot:
> >      imap(username)<29488><zs7bbNGvzOI0fYtc>: Panic: file message-parser.c:
> >      line 174 (message_part_finish): assertion failed:
> >      (ctx->nested_parts_count > 0)
> > 
> >    Oh, right, this is fixed
> >    by: [2]https://github.com/dovecot/core/commit/a668d767a710ca18ab6e7177d8e8be22a6b024fb
> 
> Well this looks promising.  Thanks!  Will look into cherry-picking this change.

Confirmed that this fixes the crash in both stretch and buster.  I'll
prepare an update targeting buster for the 10.7 release.  The stretch
release will go through the LTS process.

noah

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web