Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #152211

Re: How many wide characters may mbstowcs store?

From Tim Rentsch <tr.17687@z991.linuxsc.com>
Newsgroups comp.lang.c
Subject Re: How many wide characters may mbstowcs store?
Date 2020-05-11 09:06 -0700
Organization A noiseless patient Spider
Message-ID <86mu6exxpw.fsf@linuxsc.com> (permalink)
References <r9bd16$nbt$1@solani.org> <r9bobp$fg5$1@news.albasani.net> <r9boue$25k$2@solani.org>

Show all headers | View raw

Philipp Klaus Krause <pkk@spth.de> writes:

> Am 11.05.20 um 16:44 schrieb Bonita Montero:
>
>> There's a POSIX-extension that if you pass nullptr for s, you get
>> the size of the buffer needed for s.  Maybe this will help you.
>> Otherwise: multibyte-characters are usually UTF-8-characters and
>> it should be easy to find code to convert these charaters into
>> wide-characters; but it should be also easy to write this yourself
>> in 20min.
>
> At the moment I want to figure out what to do about the problem.  File a
> bug against GCC in Ubuntu?  File a defect report / clarification request
> with WG14?

File a gcc bug report.  The gnu/gcc folks have misunderstood the
standard, and they are shooting their users in the foot.  Your
support/regression tests deserve thanks, and have provided a
public service.

It might also be useful to file a DR/CR, mainly to beat the gcc
people over the head when they push back and say it isn't a bug.
It is.

Here are the clues.

Your first posting asked about mbstowcs, but in the followup to 
Manfred you said the report you got was on wcstombs.  The wording
in the Standard for wcstombs is definite that conversion stops
when a null is encountered.  (I confirm that wcstombs exhibits
the erroneous behavior on Ubuntu 18.04 under gcc -O1 or -O2,
but not with clang, and not with gcc -O0.)

Second clue:  compare the man pages for mbstowcs and wcstombs.
The wording they give for what the length of the array should be
is essentially identical.  That description is obviously wrong
for wcstombs, and probably was simply copied from the wrong
wording in mbstowcs.  (Note that the Standard lists these
functions in the order mbstowcs then wcstombs, so the man pages
were probably written in that order.)

Third clue:  under -O0, gcc calls wcstombs, but under -O1 or -O2
gcc calls __wcstombs_chk.  Not only that, but the call passes an
extra argument, which is (surprise, surprise) the length of the
array into which the output is going.  Mostly likely the _chk
function simply compares the actual length against the requested
length, and then gives a buffer overflow message (as happens on
my Ubuntu system) because actual < requested.  Brilliant!

Fourth clue:  if the call to wcstombs is simply wrapped in another
function, so at the call site it isn't known how large the output
buffer is, gcc works fine.  The past-terminating-null array
positions are left unaffected.

Editorial comment:  I suspect, with no particular proof, that we
are seeing another manifestation of "overzealous optimization",
aka "anything that might possibly be undefined behavior gives us
license to screw things up however we want."  If that is the case
then gcc developers deserve to have their noses rubbed in this,
ahem, "helpful" interpretation.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 13:30 +0200
  Re: How many wide characters may mbstowcs store? Manfred <noname@add.invalid> - 2020-05-11 13:55 +0200
    Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 14:01 +0200
      Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-05-11 09:08 -0400
        Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 15:19 +0200
        Re: How many wide characters may mbstowcs store? Manfred <noname@add.invalid> - 2020-05-11 18:32 +0200
        Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 18:59 +0200
          Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 17:42 +0000
            Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 20:30 +0200
            Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-05-12 00:29 -0400
              Re: How many wide characters may mbstowcs store? Manfred <noname@add.invalid> - 2020-05-12 14:41 +0200
              Re: How many wide characters may mbstowcs store? Bonita Montero <Bonita.Montero@gmail.com> - 2020-05-12 16:19 +0200
  Re: How many wide characters may mbstowcs store? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2020-05-11 13:03 +0100
    Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 14:07 +0200
  Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 15:20 +0200
    Re: How many wide characters may mbstowcs store? Bonita Montero <Bonita.Montero@gmail.com> - 2020-05-11 16:31 +0200
    Re: How many wide characters may mbstowcs store? Barry Schwarz <schwarzb@delq.com> - 2020-05-11 10:06 -0700
  Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 15:58 +0200
    Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-05-11 10:24 -0400
    Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 16:52 +0200
      Re: How many wide characters may mbstowcs store? Manfred <noname@add.invalid> - 2020-05-11 18:55 +0200
    Re: How many wide characters may mbstowcs store? richard@cogsci.ed.ac.uk (Richard Tobin) - 2020-05-11 15:51 +0000
      Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 19:01 +0200
        Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 17:33 +0000
          Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 20:57 +0200
            Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 19:17 +0000
              Re: How many wide characters may mbstowcs store? richard@cogsci.ed.ac.uk (Richard Tobin) - 2020-05-11 19:41 +0000
                Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 20:01 +0000
            Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 19:19 +0000
    Re: How many wide characters may mbstowcs store? Florian Weimer <fw@deneb.enyo.de> - 2020-05-11 20:24 +0200
      Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 20:59 +0200
        Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 19:17 +0000
          Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 21:24 +0200
        Re: How many wide characters may mbstowcs store? Florian Weimer <fw@deneb.enyo.de> - 2020-05-11 22:30 +0200
  Re: How many wide characters may mbstowcs store? Bonita Montero <Bonita.Montero@gmail.com> - 2020-05-11 16:44 +0200
    Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 16:54 +0200
      Re: How many wide characters may mbstowcs store? Bonita Montero <Bonita.Montero@gmail.com> - 2020-05-11 16:57 +0200
        Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 17:07 +0200
          Re: How many wide characters may mbstowcs store? Bonita Montero <Bonita.Montero@gmail.com> - 2020-05-11 17:08 +0200
          Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-05-11 11:25 -0400
      Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-11 09:06 -0700
        Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 19:05 +0200
          Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 19:19 +0200
            Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-23 07:51 -0700
              Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-23 20:27 +0200
                Re: How many wide characters may mbstowcs store? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-05-23 14:25 -0700
                Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-26 07:09 -0700
                Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-26 07:14 -0700
                Re: How many wide characters may mbstowcs store? Spiros Bousbouras <spibou@gmail.com> - 2020-05-26 16:00 +0000
                Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-29 21:23 -0700
                Re: How many wide characters may mbstowcs store? Spiros Bousbouras <spibou@gmail.com> - 2020-05-30 20:08 +0000
                Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-06-03 08:46 -0700
                Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-06-03 10:18 -0700
                Re: How many wide characters may mbstowcs store? raltbos@xs4all.nl (Richard Bos) - 2020-06-04 20:34 +0000
        Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 17:39 +0000
          Re: How many wide characters may mbstowcs store? Autist <autist69@gmail.com> - 2020-05-11 19:42 +0200
          Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-11 20:28 +0200
            Re: How many wide characters may mbstowcs store? scott@slp53.sl.home (Scott Lurndal) - 2020-05-11 18:37 +0000
              Re: How many wide characters may mbstowcs store? Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2020-05-11 11:50 -0700
                Re: How many wide characters may mbstowcs store? Ben Bacarisse <ben.usenet@bsb.me.uk> - 2020-05-12 20:02 +0100
                Re: How many wide characters may mbstowcs store? Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2020-05-12 13:12 -0700
          Re: How many wide characters may mbstowcs store? richard@cogsci.ed.ac.uk (Richard Tobin) - 2020-05-11 19:56 +0000
          Re: How many wide characters may mbstowcs store? Tim Rentsch <tr.17687@z991.linuxsc.com> - 2020-05-24 16:49 -0700
      Re: How many wide characters may mbstowcs store? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-05-11 14:19 -0700
        Re: How many wide characters may mbstowcs store? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-05-11 14:22 -0700
          Re: How many wide characters may mbstowcs store? Philipp Klaus Krause <pkk@spth.de> - 2020-05-12 09:17 +0200
        Re: How many wide characters may mbstowcs store? raltbos@xs4all.nl (Richard Bos) - 2020-05-24 16:12 +0000
          Re: How many wide characters may mbstowcs store? Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2020-05-24 15:10 -0700
          Re: How many wide characters may mbstowcs store? James Kuyper <jameskuyper@alumni.caltech.edu> - 2020-05-24 22:58 -0400
  Re: How many wide characters may mbstowcs store? richard@cogsci.ed.ac.uk (Richard Tobin) - 2020-05-11 20:07 +0000

csiph-web