Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13627 > unrolled thread

How to check for error with write(2)

Back to article view | Back to comp.unix.programmer

Contents

  How to check for error with write(2) Steve Keller <keller.steve@gmx.de> - 2021-12-16 18:58 +0100
    Re: How to check for error with write(2) scott@slp53.sl.home (Scott Lurndal) - 2021-12-16 18:26 +0000
      Re: How to check for error with write(2) Geoff Clare <geoff@clare.See-My-Signature.invalid> - 2021-12-20 13:52 +0000
    Re: How to check for error with write(2) Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2021-12-16 18:49 +0000
      Re: How to check for error with write(2) Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-16 19:00 +0000
    Re: How to check for error with write(2) Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-16 19:06 +0000
    Re: How to check for error with write(2) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2021-12-16 22:34 +0000
      Re: How to check for error with write(2) Steve Keller <keller.steve@gmx.de> - 2021-12-17 04:38 +0100
        Re: How to check for error with write(2) Rainer Weikusat <rweikusat@talktalk.net> - 2021-12-17 15:14 +0000
        Re: How to check for error with write(2) Jorgen Grahn <grahn+nntp@snipabacken.se> - 2021-12-18 10:25 +0000
    Re: How to check for error with write(2) Philip Guenther <guenther@gmail.com> - 2021-12-16 15:04 -0800
      Re: How to check for error with write(2) Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2021-12-16 20:10 -0700
      Re: How to check for error with write(2) Steve Keller <keller.steve@gmx.de> - 2021-12-17 04:35 +0100
        Re: How to check for error with write(2) Philip Guenther <guenther@gmail.com> - 2021-12-18 14:58 -0800
          Re: How to check for error with write(2) scott@slp53.sl.home (Scott Lurndal) - 2021-12-19 16:07 +0000

#13627 — How to check for error with write(2)

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

        if ((fd = creat(name, 0666)) < 0) {
                perror(name);
                exit(1);
        }
        while (have_more_data) {
                if (write(fd, buffer, count) < 0) {
                        perror("write");
                        exit(1);
                }
        }
        if (close(fd) < 0) {
                perror("close");
                exit(1);
        }

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Steve

[toc] | [next] | [standalone]

#13628

Steve Keller <keller.steve@gmx.de> writes:
>A write() system call can return successfully and even the following
>close() syscall may not return an error, even if there was an error
>like EFBIG (File too large).
>
>        if ((fd = creat(name, 0666)) < 0) {
>                perror(name);
>                exit(1);
>        }
>        while (have_more_data) {
>                if (write(fd, buffer, count) < 0) {
>                        perror("write");
>                        exit(1);
>                }
>        }
>        if (close(fd) < 0) {
>                perror("close");
>                exit(1);
>        }
>
>This may complete successfully, e.g. when EFBIG occurs in the last
>write() syscall before the loop ends.
>
>So what is the best way to check for errors with write()?
>
>Steve


   ssize_t diag;
   bool    done = false;
   while (!done) {
       ssize_t diag = write(fd, buffer, count);
       if (diag == -1) {
            fprintf(stderr, "%s: Unable to write '%s': %s\n", argv[0], filename,
strerror(errno));
            done = true;
            continue;
       } else if (diag < count) {
            fprintf(stderr, "%s: Short write, %zd of %zu bytes written\n", argv[0], diag,
count);
            done = true;
            continue;
       }
       /* load buffer with next 'count' bytes of data */
   }
   int rval = fdatasync(fd);
   if (rval == -1) {
      fprintf(stderr, "%s: Unable to sync data to disk: %s\n", argv[0], strerror(errno));
   } 
   rval = close(fd);
   if (rval == -1) {
      fprintf(stderr, "%s: Close of '%s' failed: %s\n", argv[0], filename, strerror(errno));
   }
   fd = -1;

One may also catch the SIGXFSZ signal to ensure one detects file size resource limit violations
(see setrlimit(2), signal(7))

[toc] | [prev] | [next] | [standalone]

#13659

Scott Lurndal wrote:

> One may also catch the SIGXFSZ signal to ensure one detects file
> size resource limit violations (see setrlimit(2), signal(7))

Simpler just to set SIGXFSZ to SIG_IGN so that write() will fail
with EFBIG (which can then be handled in the normal way).

-- 
Geoff Clare <netnews@gclare.org.uk>

[toc] | [prev] | [next] | [standalone]

#13629

On Thu, 16 Dec 2021 18:58:28 +0100, Steve Keller wrote:

> A write() system call can return successfully and even the following
> close() syscall may not return an error, even if there was an error like
> EFBIG (File too large).
> 
>         if ((fd = creat(name, 0666)) < 0) {
>                 perror(name);
>                 exit(1);
>         }
>         while (have_more_data) {
>                 if (write(fd, buffer, count) < 0) {
>                         perror("write");
>                         exit(1);
>                 }
>         }
>         if (close(fd) < 0) {
>                 perror("close");
>                 exit(1);
>         }
> 
> This may complete successfully, e.g. when EFBIG occurs in the last
> write() syscall before the loop ends.
> 
> So what is the best way to check for errors with write()?
> 
According to the Linux write(2) manpage,
   A  successful return from write() does not make any guarantee that data
   has been committed to disk.  In fact, on some buggy implementations, it
   does  not  even guarantee that space has successfully been reserved for
   the data.  The only way to be sure is to call fsync(2)  after  you  are
   done writing all your data.

While this may not be true for other Unix and Unix-like systems, an 
appropriately-placed sync(2)/fsync(2) call wouldn't hurt.


-- 
Lew Pitcher
"In Skills, We Trust"

[toc] | [prev] | [next] | [standalone]

#13630

Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
> On Thu, 16 Dec 2021 18:58:28 +0100, Steve Keller wrote:
>
>> A write() system call can return successfully and even the following
>> close() syscall may not return an error, even if there was an error like
>> EFBIG (File too large).
>> 
>>         if ((fd = creat(name, 0666)) < 0) {
>>                 perror(name);
>>                 exit(1);
>>         }
>>         while (have_more_data) {
>>                 if (write(fd, buffer, count) < 0) {
>>                         perror("write");
>>                         exit(1);
>>                 }
>>         }
>>         if (close(fd) < 0) {
>>                 perror("close");
>>                 exit(1);
>>         }
>> 
>> This may complete successfully, e.g. when EFBIG occurs in the last
>> write() syscall before the loop ends.
>> 
>> So what is the best way to check for errors with write()?
>> 
> According to the Linux write(2) manpage,
>    A  successful return from write() does not make any guarantee that data
>    has been committed to disk.  In fact, on some buggy implementations, it
>    does  not  even guarantee that space has successfully been reserved for
>    the data.  The only way to be sure is to call fsync(2)  after  you  are
>    done writing all your data.
>
> While this may not be true for other Unix and Unix-like systems, an 
> appropriately-placed sync(2)/fsync(2) call wouldn't hurt.

Usually, it will hurt. The buffered I/O subsystem in the kernel ("page
cache") is there for a reason and it shouldn't be circumvented just
because it seems somehow suspicious. The default assumption should be
that the system will not suddenly stop and hence, that leaving it to the
kernel to decide when to write what will be ok.

[toc] | [prev] | [next] | [standalone]

#13631

Steve Keller <keller.steve@gmx.de> writes:
> A write() system call can return successfully and even the following
> close() syscall may not return an error, even if there was an error
> like EFBIG (File too large).
>
>         if ((fd = creat(name, 0666)) < 0) {
>                 perror(name);
>                 exit(1);
>         }
>         while (have_more_data) {
>                 if (write(fd, buffer, count) < 0) {
>                         perror("write");
>                         exit(1);
>                 }
>         }
>         if (close(fd) < 0) {
>                 perror("close");
>                 exit(1);
>         }
>
> This may complete successfully, e.g. when EFBIG occurs in the last
> write() syscall before the loop ends.
>
> So what is the best way to check for errors with write()?

Avoid checking the return value of close. The way to ensure that a
"physical write" took place is to use fsync before close (and checking
the return value of that) and this should usually be avoided.

The only thing checking the return value of close is actually good for
is to tell a user that "my program lost your data because I didn't use
fsync" --- not terribly useful.

[toc] | [prev] | [next] | [standalone]

#13632

On Thu, 2021-12-16, Steve Keller wrote:
> A write() system call can return successfully and even the following
> close() syscall may not return an error, even if there was an error
> like EFBIG (File too large).

The Linux write(2) man page says:

    RETURN VALUE
       On success, the number of bytes written is returned.  On error, -1  is  re-
       turned, and errno is set to indicate the cause of the error.

    ...

       EFBIG  An attempt was made to write a file that exceeds the implementation-
              defined maximum file size or the process's file size  limit,  or  to
              write at a position past the maximum allowed offset.

The OpenBSD one is similar. Are you talking about other Unix-like
systems, or are you saying the man page is wrong?  Because it seems to
say that in your scenario, the fatal write() will return -1 and set
EFBIG.

>         if ((fd = creat(name, 0666)) < 0) {
>                 perror(name);
>                 exit(1);
>         }
>         while (have_more_data) {
>                 if (write(fd, buffer, count) < 0) {
>                         perror("write");
>                         exit(1);
>                 }
>         }
>         if (close(fd) < 0) {
>                 perror("close");
>                 exit(1);
>         }
>
> This may complete successfully, e.g. when EFBIG occurs in the last
> write() syscall before the loop ends.
>
> So what is the best way to check for errors with write()?

Until I'm convinced it's wrong: according to the write(2) man page.

(I'm disregarding the fsync(2) angle, covered by others.)

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#13636

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:

> The OpenBSD one is similar. Are you talking about other Unix-like
> systems, or are you saying the man page is wrong?  Because it seems to
> say that in your scenario, the fatal write() will return -1 and set
> EFBIG.

I should have written more clearly, that I was talking about a "short
write".  I.e. the write() syscall doesn't return -1 and doesn't set
errno, but it could only write data partially.

I wasn't saying the man page is wrong, only that an error can occur
without write() returning -1 to indicate the error.  Repeating the
write() with the rest of the data to get the error, as Philip
suggested, was obviously too simple for me to get that idea :-)

> Until I'm convinced it's wrong: according to the write(2) man page.
> 
> (I'm disregarding the fsync(2) angle, covered by others.)

Yes, f*sync() is not what I was looking for.

Steve

[toc] | [prev] | [next] | [standalone]

#13643

Steve Keller <keller.steve@gmx.de> writes:
> Jorgen Grahn <grahn+nntp@snipabacken.se> writes:
>> The OpenBSD one is similar. Are you talking about other Unix-like
>> systems, or are you saying the man page is wrong?  Because it seems to
>> say that in your scenario, the fatal write() will return -1 and set
>> EFBIG.
>
> I should have written more clearly, that I was talking about a "short
> write".  I.e. the write() syscall doesn't return -1 and doesn't set
> errno, but it could only write data partially.
>
> I wasn't saying the man page is wrong, only that an error can occur
> without write() returning -1 to indicate the error.  Repeating the
> write() with the rest of the data to get the error, as Philip
> suggested, was obviously too simple for me to get that idea :-)

write generally needs to be called in a loop until either everything was
written or an error ocurred, subject to whatever special properties the
"output thing" the file descriptor refers to might have (eg, atomicity
of PIPE_BUF writes to a pipe, or preservation of message boundaries by
datagram sockets). But that's possibly not enough to detect errors
reliably: I've encountered people who believed that a write call which
cannot write anything due to the disk being full should return 0 and not
-1 and an ENOSPC error. I don't know if implementations exist which
actually work in this way.

[toc] | [prev] | [next] | [standalone]

#13647

On Fri, 2021-12-17, Steve Keller wrote:
> Jorgen Grahn <grahn+nntp@snipabacken.se> writes:
>
>> The OpenBSD one is similar. Are you talking about other Unix-like
>> systems, or are you saying the man page is wrong?  Because it seems to
>> say that in your scenario, the fatal write() will return -1 and set
>> EFBIG.
>
> I should have written more clearly, that I was talking about a "short
> write".  I.e. the write() syscall doesn't return -1 and doesn't set
> errno, but it could only write data partially.

Ah, and now I see this in write(2):

    Note that a successful write() may transfer fewer than count
    bytes.  Such partial writes can occur for various reasons; for
    example, because there was insufficient space on the disk device
    to write all of the requested bytes, or [more well-known reasons]

This is something I've not noticed before.  I assumed write() to disk
files used an "all or nothing" policy.

In my defense, I don't write to disk very often.

> I wasn't saying the man page is wrong, only that an error can occur
> without write() returning -1 to indicate the error.  Repeating the
> write() with the rest of the data to get the error, as Philip
> suggested, was obviously too simple for me to get that idea :-)
>
>> Until I'm convinced it's wrong: according to the write(2) man page.
>> 
>> (I'm disregarding the fsync(2) angle, covered by others.)
>
> Yes, f*sync() is not what I was looking for.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#13633

On Thursday, December 16, 2021 at 9:58:37 AM UTC-8, Steve Keller wrote:
> A write() system call can return successfully and even the following 
> close() syscall may not return an error, even if there was an error 
> like EFBIG (File too large). 
> 
> if ((fd = creat(name, 0666)) < 0) { 
> perror(name); 
> exit(1); 
> } 
> while (have_more_data) { 
> if (write(fd, buffer, count) < 0) { 
> perror("write"); 
> exit(1); 
> } 
> } 
> if (close(fd) < 0) { 
> perror("close"); 
> exit(1); 
> } 
> 
> This may complete successfully, e.g. when EFBIG occurs in the last 
> write() syscall before the loop ends. 
> 
> So what is the best way to check for errors with write()? 

Uh, actually look at the return value and see if it matches the number of bytes you asked to write: on partial success, write() will return less  than the requested count.  Note that a short write won't change errno; if you need to know why it was unable to write more than you need to keep calling write() until it completely fails (usually the next call) and check errno then.

To quote the (next issue) of the POSIX spec's description of write():
         If a write( ) requests that more bytes be written than there is room for (for example, the file size
         limit of the process or the physical end of a medium), only as many bytes as there is room for
         shall be written. For example, suppose there is space for 20 bytes more in a file before reaching a
         limit. A write of 512 bytes will return 20. The next write of a non-zero number of bytes would
         give a failure return (except as noted below).

         If the request would cause the file size to exceed the soft file size limit for the process and there
         is no room for any bytes to be written, the request shall fail and the implementation shall
         generate the SIGXFSZ signal for the thread.

And then in the ERRORS section:
ERRORS
            These functions shall fail if:
...
[EFBIG]      An attempt was made to write a file that exceeds the implementation-defined
             maximum file size and there was no room for any bytes to be written.

[EFBIG]  An attempt was made to write a file that exceeds the file size limit of the
             process, and there was no room for any bytes to be written. A SIGXFSZ signal
             shall also be sent to the thread.


Philip Guenther

[toc] | [prev] | [next] | [standalone]

#13634

Philip Guenther <guenther@gmail.com> writes:

<snip>

but down below the ERRORS section, in the NOTES:

       A successful return from write() does not make any guarantee that  data
       has  been  committed  to  disk.  On some filesystems, including NFS, it
       does not even guarantee that space has successfully been  reserved  for
       the  data.   In  this case, some errors might be delayed until a future
       write(), fsync(2), or even close(2).  The only way to  be  sure  is  to
       call fsync(2) after you are done writing all your data.

Successfully writing to the file system isn't the same thing as
successfully writing to actual nonvolatile storage (ie the disk).  Much
as we wish it were (and in all but a microscopic subset of cases we can
get away with pretending).

[toc] | [prev] | [next] | [standalone]

#13635

Philip Guenther <guenther@gmail.com> writes:

> Uh, actually look at the return value and see if it matches the
> number of bytes you asked to write: on partial success, write() will
> return less than the requested count.  Note that a short write won't
> change errno; if you need to know why it was unable to write more
> than you need to keep calling write() until it completely fails
> (usually the next call) and check errno then.
> 
> To quote the (next issue) of the POSIX spec's description of write():
>          If a write( ) requests that more bytes be written than
>          there is room for (for example, the file size limit of the
>          process or the physical end of a medium), only as many
>          bytes as there is room for shall be written. For example,
>          suppose there is space for 20 bytes more in a file before
>          reaching a limit. A write of 512 bytes will return 20. The
>          next write of a non-zero number of bytes would give a
>          failure return (except as noted below).
> 
>          If the request would cause the file size to exceed the soft
>          file size limit for the process and there is no room for
>          any bytes to be written, the request shall fail and the
>          implementation shall generate the SIGXFSZ signal for the
>          thread.
> 
> And then in the ERRORS section:
> ERRORS
>             These functions shall fail if:
> ...
> [EFBIG]      An attempt was made to write a file that exceeds the
>              implementation-defined maximum file size and there was
>              no room for any bytes to be written.
> 
> [EFBIG]      An attempt was made to write a file that exceeds the
>              file size limit of the process, and there was no room
>              for any bytes to be written. A SIGXFSZ signal shall
>              also be sent to the thread.

Ah, thanks.  How to handle a short write, i.e. with no error return
code, is exactly what I was looking for.  This information is not in
the Linux write(2) man page.

BTW, is it possible that after a short write the next write() syscall
for the remaining data will write some more data, possibly even the
whole remaining data successfully?  I think of something like a pipe
or a socket where not the whole data can be sent immediately.

Or will the write() syscall always block until all data could be
written?  I.e. will the next write syscall after a short write always
fail and return -1?

Steve

[toc] | [prev] | [next] | [standalone]

#13650

On Thursday, December 16, 2021 at 7:35:16 PM UTC-8, Steve Keller wrote:
> Philip Guenther <guen...@gmail.com> writes: 
> 
> > Uh, actually look at the return value and see if it matches the 
> > number of bytes you asked to write: on partial success, write() will 
> > return less than the requested count. Note that a short write won't 
> > change errno; if you need to know why it was unable to write more 
> > than you need to keep calling write() until it completely fails 
> > (usually the next call) and check errno then. 
> > 
> > To quote the (next issue) of the POSIX spec's description of write(): 
> > If a write( ) requests that more bytes be written than 
> > there is room for (for example, the file size limit of the 
> > process or the physical end of a medium), only as many 
> > bytes as there is room for shall be written. For example, 
> > suppose there is space for 20 bytes more in a file before 
> > reaching a limit. A write of 512 bytes will return 20. The 
> > next write of a non-zero number of bytes would give a 
> > failure return (except as noted below). 
> > 
> > If the request would cause the file size to exceed the soft 
> > file size limit for the process and there is no room for 
> > any bytes to be written, the request shall fail and the 
> > implementation shall generate the SIGXFSZ signal for the 
> > thread. 
> > 
> > And then in the ERRORS section: 
> > ERRORS 
> > These functions shall fail if: 
> > ... 
> > [EFBIG] An attempt was made to write a file that exceeds the 
> > implementation-defined maximum file size and there was 
> > no room for any bytes to be written. 
> > 
> > [EFBIG] An attempt was made to write a file that exceeds the 
> > file size limit of the process, and there was no room 
> > for any bytes to be written. A SIGXFSZ signal shall 
> > also be sent to the thread.
> Ah, thanks. How to handle a short write, i.e. with no error return 
> code, is exactly what I was looking for. This information is not in 
> the Linux write(2) man page. 
> 
> BTW, is it possible that after a short write the next write() syscall 
> for the remaining data will write some more data, possibly even the 
> whole remaining data successfully? I think of something like a pipe 
> or a socket where not the whole data can be sent immediately. 

If the condition that prevented writing more data has a change of state between the two calls, then sure, more data may be written successfully.  For example, if some other process or thread freed up disk space between the calls than an ENOSPC or EDQUOT condition that caused a short-write may go away, permitting more data to be written.

Could the system decide to do short-writes for no good reason and then permit more later, just because?  Given the first sentence of the wording I quoted, I think the answer is "no", but also think I'm uninterested in arguing about it and would just call such an implementation "low quality" and not use it.


Philip Guenther

[toc] | [prev] | [next] | [standalone]

#13653

Philip Guenther <guenther@gmail.com> writes:
>On Thursday, December 16, 2021 at 7:35:16 PM UTC-8, Steve Keller wrote:
>> Philip Guenther <guen...@gmail.com> writes:=20
>>=20
>> > Uh, actually look at the return value and see if it matches the=20
>> > number of bytes you asked to write: on partial success, write() will=20
>> > return less than the requested count. Note that a short write won't=20
>> > change errno; if you need to know why it was unable to write more=20
>> > than you need to keep calling write() until it completely fails=20
>> > (usually the next call) and check errno then.=20
>> >=20
>> > To quote the (next issue) of the POSIX spec's description of write():=
>=20
>> > If a write( ) requests that more bytes be written than=20
>> > there is room for (for example, the file size limit of the=20
>> > process or the physical end of a medium), only as many=20
>> > bytes as there is room for shall be written. For example,=20
>> > suppose there is space for 20 bytes more in a file before=20
>> > reaching a limit. A write of 512 bytes will return 20. The=20
>> > next write of a non-zero number of bytes would give a=20
>> > failure return (except as noted below).=20
>> >=20
>> > If the request would cause the file size to exceed the soft=20
>> > file size limit for the process and there is no room for=20
>> > any bytes to be written, the request shall fail and the=20
>> > implementation shall generate the SIGXFSZ signal for the=20
>> > thread.=20
>> >=20
>> > And then in the ERRORS section:=20
>> > ERRORS=20
>> > These functions shall fail if:=20
>> > ...=20
>> > [EFBIG] An attempt was made to write a file that exceeds the=20
>> > implementation-defined maximum file size and there was=20
>> > no room for any bytes to be written.=20
>> >=20
>> > [EFBIG] An attempt was made to write a file that exceeds the=20
>> > file size limit of the process, and there was no room=20
>> > for any bytes to be written. A SIGXFSZ signal shall=20
>> > also be sent to the thread.
>> Ah, thanks. How to handle a short write, i.e. with no error return=20
>> code, is exactly what I was looking for. This information is not in=20
>> the Linux write(2) man page.=20
>>=20
>> BTW, is it possible that after a short write the next write() syscall=20
>> for the remaining data will write some more data, possibly even the=20
>> whole remaining data successfully? I think of something like a pipe=20
>> or a socket where not the whole data can be sent immediately.=20
>
>If the condition that prevented writing more data has a change of state bet=
>ween the two calls, then sure, more data may be written successfully.  For =
>example, if some other process or thread freed up disk space between the ca=
>lls than an ENOSPC or EDQUOT condition that caused a short-write may go awa=
>y, permitting more data to be written.
>
>Could the system decide to do short-writes for no good reason and then perm=
>it more later, just because?  Given the first sentence of the wording I quo=
>ted, I think the answer is "no", but also think I'm uninterested in arguing=
> about it and would just call such an implementation "low quality" and not =
>use it.

It helps to remember that the write system call was used for more than
just disk-based files.    Consider a line printer, for example, that
supports 132 column lines where the write system call writes more than
132 characters - the driver might report that as a short write.

Short writes on a socket file descriptor are also possible.

Short reads are common on serial devices such as magnetic tape or
other unit record-style devices.

Granted most of those cases are now shrouded in the mists of history.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.unix.programmer

csiph-web