Path: csiph.com!xmission!news.snarked.org!news.linkpendium.com!news.linkpendium.com!panix!usenet.stanford.edu!not-for-mail From: gentoo_eshoes@tutanota.com Newsgroups: gnu.bash.bug Subject: Re: How can redirection operator aka ">" open output in append instead of ordinary write mode? Date: Fri, 17 Apr 2020 22:10:31 +0200 (CEST) Lines: 185 Approved: bug-bash@gnu.org Message-ID: References: NNTP-Posting-Host: lists.gnu.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_71243_1918066042.1587154231401" X-Trace: usenet.stanford.edu 1587154239 15377 209.51.188.17 (17 Apr 2020 20:10:39 GMT) X-Complaints-To: action@cs.stanford.edu To: bug-bash@gnu.org Envelope-to: bug-bash@gnu.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1587154231; s=s1; d=tutanota.com; h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Cc:Date:Date:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:Sender; bh=cgDIr5R73oFer8U44bvRmq721i6zSLwEuX2gOsc9c8A=; b=F+RPHizSwgQnUX4BI7D1ZSkqT5ss/ghBmOVlfHdlfr571DsDEaK56Q7I9wyvAnVI WaxBGSJAMknn6E4iLEjrYeGPwlCwW7SRJIcghtDUkrTu3xFqiTzkIPKAjJZqSJ/YyDq nDP2SGQjGlbENnln8XPdM+ceQgN92B6m3/PTgtebLYENB1JMBVj3GW1zUnuYi7WfG2H yZ98KEG7/rUZjVuAx0oOts1oZCNIrG5ez6NxK4OwlnIyYWxn8jSQAh8E5ZapEQxQdEy OH301CO86ZOXHQAA9WppgklBxypTp6DkiW8WMxh3fIbigg41a2C7SmsWNQYTRIf7h76 bukdojtayw== X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 81.3.6.162 X-BeenThere: bug-bash@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Bug reports for the GNU Bourne Again SHell List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Mailman-Original-Message-ID: Xref: csiph.com gnu.bash.bug:16154 ------=_Part_71243_1918066042.1587154231401 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable the only issue I've encountered with this so far is that 'dd' will fail som= e of its tests because it's not using of=3D=C2=A0 but instead using the red= ir operator and then it assumes it's NOT open in append mode ;)) so then it= tries to seek past EOF (which doesn't work in append mode) to make holes(o= f zeroes). to illustrate: $ rm /tmp/out;echo abcdefghijklm | dd bs=3D5 seek=3D8 oflag=3Dseek_bytes > = /tmp/out ; stat --format=3D%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.0001584 s, 88.4 kB/s Actually written to disk: '14' bytes (100.00%).=20 14 the normal output should be(ie. file size 22 bytes not 14):=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=20 $ rm /tmp/out;echo abcdefghijklm | dd bs=3D5 seek=3D8 oflag=3Dseek_bytes > = /tmp/out ; stat --format=3D%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.000112444 s, 125 kB/s Actually written to disk: '14' bytes (100.00%).=20 22 Not to worry though, I'm "sure" I pulled a Curly in Three Stooges episode t= itled "A Plumbing We Will Go." when I added a kernel patch for this (attach= ed) which ensures `cannot seek: Illegal seek` when trying to go beyond EOF = in O_APPEND mode. In a completely unrelated note, this cool Exolon theme/music, listening to = it right now: https://www.youtube.com/watch?v=3DGfw-CIEM6Zs ah memories:) ------=_Part_71243_1918066042.1587154231401 Content-Type: text/x-patch; charset=us-ascii; name=always_append_on_redirection.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=always_append_on_redirection.patch newer: now using kernel patch deny_lseek_past_EOF_when_O_APPEND.patch with this bash patch, testing to see how 'dd' reacts... better: $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out dd: 'standard output': cannot seek: Illegal seek 0+0 records in 0+0 records out 0 bytes copied, 0.000102177 s, 0.0 kB/s Actually written to disk: '0' bytes (-nan%). 0 new: ok this is bad: because it makes this fail: $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.0001584 s, 88.4 kB/s Actually written to disk: '14' bytes (100.00%). 14 the normal output should be(ie. file size 22 bytes not 14): $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.000112444 s, 125 kB/s Actually written to disk: '14' bytes (100.00%). 22 ok so because it's append only, dd cann't seek past the EOF, and thus lseek won't do it's job of skipping past EOF and the subsequent 'write' won't make those \0-es pop up in that gap! ie. dd assumes outfile is never opened in O_APPEND mode. meanwhile this will always work(the redirect became of=): $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes of=/tmp/out ; stat --format=%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.000137867 s, 102 kB/s Actually written to disk: '14' bytes (100.00%). 22 old info: https://github.com/libcheck/check/issues/188 diff --git a/make_cmd.c b/make_cmd.c index ecbbfd6e..5db799f2 100644 --- a/make_cmd.c +++ b/make_cmd.c @@ -700,7 +700,7 @@ make_redirection (source, instruction, dest_and_filename, flags) case r_output_direction: /* >foo */ case r_output_force: /* >| foo */ case r_err_and_out: /* &>filename */ - temp->flags = O_TRUNC | O_WRONLY | O_CREAT; + temp->flags = O_APPEND | O_TRUNC | O_WRONLY | O_CREAT; break; case r_appending_to: /* >>foo */ ------=_Part_71243_1918066042.1587154231401 Content-Type: text/x-patch; charset=us-ascii; name=deny_lseek_past_EOF_when_O_APPEND.patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=deny_lseek_past_EOF_when_O_APPEND.patch use this patch with bash patch always_append_on_redirection.patch this will fix 'dd' trying to seek past EOF into the file used for redirection while that file is open in O_APPEND mode now(with this patch and bash patch!) it's this: $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out dd: 'standard output': cannot seek: Illegal seek 0+0 records in 0+0 records out 0 bytes copied, 0.000102177 s, 0.0 kB/s Actually written to disk: '0' bytes (-nan%). 0 before patch: $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.0001584 s, 88.4 kB/s Actually written to disk: '14' bytes (100.00%). 14 should be(if no bash patch was used): $ rm /tmp/out;echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out 2+1 records in 2+1 records out 14 bytes copied, 0.000112444 s, 125 kB/s Actually written to disk: '14' bytes (100.00%). 22 diff --git a/fs/read_write.c b/fs/read_write.c index 7458fccc59e1..e46c2997a158 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -88,6 +88,10 @@ generic_file_llseek_size(struct file *file, loff_t offset, int whence, { switch (whence) { case SEEK_END: + if ((file->f_flags & O_APPEND) && (offset > 0)) { + //tried to seek past EOF in O_APPEND mode will have no effect! so, this 'dd' will fail: echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out ie. 14 written 14 bytes file size, instead of 22 bytes file size, if applied O_APPEND patch to bash's redirect operator, thus /tmp/out is O_APPEND mode! ok but actually it's not SEEK_END that's in effect here but SEEK_CUR instead! + return -ESPIPE; // ESPIPE 29 Illegal seek + } offset += eof; break; case SEEK_CUR: @@ -105,6 +109,13 @@ generic_file_llseek_size(struct file *file, loff_t offset, int whence, * like SEEK_SET. */ spin_lock(&file->f_lock); + + if ((file->f_flags & O_APPEND) && (offset + file->f_pos > eof)) { + //tried to seek past EOF in O_APPEND mode will have no effect! so, this 'dd' will fail: echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out ie. 14 written 14 bytes file size, instead of 22 bytes file size, if applied O_APPEND patch to bash's redirect operator, thus /tmp/out is O_APPEND mode! ok but actually it's not SEEK_END that's in effect here but SEEK_CUR instead! + spin_unlock(&file->f_lock); + return -ESPIPE; // ESPIPE 29 Illegal seek + } + offset = vfs_setpos(file, file->f_pos + offset, maxsize); spin_unlock(&file->f_lock); return offset; @@ -127,6 +138,10 @@ generic_file_llseek_size(struct file *file, loff_t offset, int whence, break; } + if ((file->f_flags & O_APPEND) && (offset > eof)) { + //tried to seek past EOF in O_APPEND mode will have no effect! so, this 'dd' will fail: echo abcdefghijklm | dd bs=5 seek=8 oflag=seek_bytes > /tmp/out ; stat --format=%s /tmp/out ie. 14 written 14 bytes file size, instead of 22 bytes file size, if applied O_APPEND patch to bash's redirect operator, thus /tmp/out is O_APPEND mode! ok but actually it's not SEEK_END that's in effect here but SEEK_CUR instead! + return -ESPIPE; // ESPIPE 29 Illegal seek + } return vfs_setpos(file, offset, maxsize); } EXPORT_SYMBOL(generic_file_llseek_size); ------=_Part_71243_1918066042.1587154231401--