Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #13340 > unrolled thread

Deleting first N lines from a text file

Back to article view | Back to comp.lang.c

Contents

  Deleting first N lines from a text file pozz <pozzugno@gmail.com> - 2011-11-14 16:02 -0800
    Re: Deleting first N lines from a text file Roberto Waltman <usenet@rwaltman.com> - 2011-11-14 19:21 -0500
      Re: Deleting first N lines from a text file Acid Washed China Blue Jeans <chine.bleu@yahoo.com> - 2011-11-14 16:31 -0800
        Re: Deleting first N lines from a text file blp@cs.stanford.edu (Ben Pfaff) - 2011-11-14 16:36 -0800
        Re: Deleting first N lines from a text file Roberto Waltman <usenet@rwaltman.com> - 2011-11-14 20:10 -0500
        Re: Deleting first N lines from a text file Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2011-11-16 11:04 +0200
    Re: Deleting first N lines from a text file gordonb.3hlvm@burditt.org (Gordon Burditt) - 2011-11-14 18:53 -0600
    Re: Deleting first N lines from a text file Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-11-14 22:06 -0500
      Re: Deleting first N lines from a text file Giuseppe <giuseppe.modugno@gmail.com> - 2011-11-15 16:57 -0800
        Re: Deleting first N lines from a text file Keith Thompson <kst-u@mib.org> - 2011-11-15 17:50 -0800
          Re: Deleting first N lines from a text file pozz <pozzugno@gmail.com> - 2011-11-15 23:11 -0800
            Re: Deleting first N lines from a text file jgharston <jgh@arcade.demon.co.uk> - 2011-11-16 04:11 -0800
        Re: Deleting first N lines from a text file Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-11-15 21:48 -0500
          Re: Deleting first N lines from a text file pozz <pozzugno@gmail.com> - 2011-11-15 23:14 -0800
            Re: Deleting first N lines from a text file "BartC" <bc@freeuk.com> - 2011-11-16 14:26 +0000
              Re: Deleting first N lines from a text file Eric Sosman <esosman@ieee-dot-org.invalid> - 2011-11-16 21:32 -0500
                Re: Deleting first N lines from a text file Ian Collins <ian-news@hotmail.com> - 2011-11-17 15:43 +1300
                  Re: Deleting first N lines from a text file "BartC" <bc@freeuk.com> - 2011-11-17 10:35 +0000
                    Re: Deleting first N lines from a text file Dr Nick <3-nospam@temporary-address.org.uk> - 2011-12-16 07:05 +0000
                Re: Deleting first N lines from a text file Robert Wessel <robertwessel2@yahoo.com> - 2011-11-16 22:46 -0600
        Re: Deleting first N lines from a text file gordonb.pa5u0@burditt.org (Gordon Burditt) - 2011-11-16 16:28 -0600
          Re: Deleting first N lines from a text file Kaz Kylheku <kaz@kylheku.com> - 2011-11-16 22:53 +0000
            Re: Deleting first N lines from a text file gordonb.z5i2o@burditt.org (Gordon Burditt) - 2011-11-16 18:34 -0600
    Re: Deleting first N lines from a text file jacob navia <jacob@spamsink.net> - 2011-11-15 12:48 +0100
      Re: Deleting first N lines from a text file "-.-" <none@none.none> - 2011-11-16 14:01 +0100
        Re: Deleting first N lines from a text file jacob navia <jacob@spamsink.net> - 2011-11-16 14:44 +0100
          Re: Deleting first N lines from a text file tom st denis <tom@iahu.ca> - 2011-11-17 10:43 -0800
            Re: Deleting first N lines from a text file jt@toerring.de (Jens Thoms Toerring) - 2011-11-18 23:12 +0000
    Re: Deleting first N lines from a text file jgharston <jgh@arcade.demon.co.uk> - 2011-11-16 06:42 -0800
      Re: Deleting first N lines from a text file jgharston <jgh@arcade.demon.co.uk> - 2011-11-16 07:43 -0800
      Re: Deleting first N lines from a text file Keith Thompson <kst-u@mib.org> - 2011-11-16 11:13 -0800
        Re: Deleting first N lines from a text file jgharston <jgh@arcade.demon.co.uk> - 2011-11-16 13:50 -0800

Page 2 of 2 — ← Prev page 1 [2]

#13524

> Do you think the process could be reduced launching an external script
> (for
> example, 'head' based) with system()?  If I redirect the output to the
> original
> filename I could avoid the time consuming process of copying the
> original
> to the temporary file.

A command like:

head -n 99 x.txt > x.txt

(This particular command does not delete the first N lines; it keeps
only the first N lines).

has a tendancy to produce a zero-length file because the shell opens
the output file (truncating it, as in fopen(..., "w") ) before the
"head" program starts running.  Then "head" reads an empty x.txt,
and outputs the first 99 lines (except only 0 exist) of it.

You need to sequence things so the file isn't truncated until *after*
you have retrieved any data that might be needed from it.  I also
consider the assumption that the contents of a file can fit in
memory (e.g. a 4.7GB video file fitting in memory on a 32-bit-address
machine) to be questionable unless you have prior knowledge of what
kinds of files will be fed to the program.

[toc] | [prev] | [next] | [standalone]

#13525

On 2011-11-16, Gordon Burditt <gordonb.pa5u0@burditt.org> wrote:
>> Do you think the process could be reduced launching an external script
>> (for
>> example, 'head' based) with system()?  If I redirect the output to the
>> original
>> filename I could avoid the time consuming process of copying the
>> original
>> to the temporary file.
>
> A command like:
>
> head -n 99 x.txt > x.txt
>
> (This particular command does not delete the first N lines; it keeps
> only the first N lines).
>
> has a tendancy to produce a zero-length file because the shell opens
> the output file (truncating it, as in fopen(..., "w") ) before the
> "head" program starts running.  Then "head" reads an empty x.txt,
> and outputs the first 99 lines (except only 0 exist) of it.

This could be fixed with a fictitious utility:

  head -n 99 x.txt | wne x.txt   # 'write if not empty'

wne reads standard input and copies to the named file, but does
not create the file if the input is empty.

> You need to sequence things so the file isn't truncated until *after*
> you have retrieved any data that might be needed from it. 

"wne" could also have the behavior of delayed truncation
in addition to delayed creation. ("dtc"?)

This could be used in any situation where a file is being
filtered in place such that characters written
to an earlier position are all derived from the same or
later positions.

[toc] | [prev] | [next] | [standalone]

#13529

>> A command like:
>>
>> head -n 99 x.txt > x.txt
>>
>> (This particular command does not delete the first N lines; it keeps
>> only the first N lines).
>>
>> has a tendancy to produce a zero-length file because the shell opens
>> the output file (truncating it, as in fopen(..., "w") ) before the
>> "head" program starts running.  Then "head" reads an empty x.txt,
>> and outputs the first 99 lines (except only 0 exist) of it.
> 
> This could be fixed with a fictitious utility:
> 
>  head -n 99 x.txt | wne x.txt   # 'write if not empty'

Or, more generally:
    arbitrary_filter x.txt | wne x.txt
as a replacement for:
    arbitrary_filter < x.txt > x.txt
which won't work because x.txt gets clobbered before it is read.

> wne reads standard input and copies to the named file, but does
> not create the file if the input is empty.

"wne" must not write to the file until it knows that whatever is sending
it data has finished reading it.  How can it know that?  When whatever
is sending it data closes the pipe, and it sees end-of-file on its stdin.

This means that "wne" (when the file is not empty) must keep reading and
storing (somewhere) data until it sees end-of-file, THEN it can write
the data.  If it won't fit in memory, then it has to use a temporary file.

Ok, count up the I/O.  
Original "copy and copy-back":  2 reads, 2 writes of whole file.
Alternative "copy and rename back":  1 read, 1 write of whole file.
Original "modify in place": 1 read, 1 write of whole file.
Method with external process: 2 reads, 2 writes of whole file if it
	fits in memory, otherwise 3 reads, 3 writes.

I think we just managed to replace something horribly slow with
something even worse.

"wne" also has a vulnerability to data loss:  if "arbitrary_filter"
crashes, then partial data written down the pipe gets put into the
output file, with no chance for "wne" to detect a problem unless it
knows what the data is supposed to look like.

I'll also challenge the function description "write if not empty".
If the objective is to run a file through an arbitrary filter, then
put the output back into the file, nobody said that putting an empty
file *in* gets you an empty file *out*.  (Example:  a program like
the UNIX "pr" which paginates files, with header lines, page numbers
and perhaps time stamps, will probably output something even for
empty files.  Or perhaps a cross-reference program that generates an
index of where every identifier is used, which still generates the
index page with an empty file.)

> This could be used in any situation where a file is being
> filtered in place such that characters written
> to an earlier position are all derived from the same or
> later positions.

The UNIX "sort" utility has a solution for this (and sorting something
with the output back in the original file is fairly common):

	sort x.txt > x.txt
will clobber x.txt, but 
	sort -o x.txt x.txt
will not.  "sort" is likely to use temporary files anyway (perhaps
hundreds of them depending on the size of the input), so it can
ensure it has read all of the input before opening the output.

[toc] | [prev] | [next] | [standalone]

#13374

Using the containers library (and if your file fits in memory)

#include <containers.h>
int main(int argc,char *argv[])
{
	if (argc != 3) {
		printf("Usage: deletelines <file> <N>\n");
		return -1;
	}
	strCollection *data = istrCollection.CreateFromFile(argv[1]);
	if (data == NULL) return -1;
	istrCollection.RemoveRange(data,0,atoi(argv[2]));
	istrCollection.WriteToFile(data,argv[1]);
	istrCollection.Finalize(data);
}

[toc] | [prev] | [next] | [standalone]

#13474

jacob navia was trying to save the world with his stuff:

> Using the containers library (and if your file fits in memory)
>
> #include <containers.h>

You self-celebrating fucko. There only exist your things to you:
that silly lcc-win and your funny containers.
Stop making this newsgroup your personal advertisements page.

[toc] | [prev] | [next] | [standalone]

#13478

Le 16/11/11 14:01, -.- a écrit :
> jacob navia was trying to save the world with his stuff:
>
>> Using the containers library (and if your file fits in memory)
>>
>> #include <containers.h>
>
> You self-celebrating fucko.

That is why you hide behind a pseudo, because you have the courage of
your opinions...

[toc] | [prev] | [next] | [standalone]

#13568

On Nov 16, 8:44 am, jacob navia <ja...@spamsink.net> wrote:
> Le 16/11/11 14:01, -.- a écrit :
>
> > jacob navia was trying to save the world with his stuff:
>
> >> Using the containers library (and if your file fits in memory)
>
> >> #include <containers.h>
>
> > You self-celebrating fucko.
>
> That is why you hide behind a pseudo, because you have the courage of
> your opinions...

While rude the guy has a point.  You can accomplish this goal easily
with something like

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
   int skip;
   char buf[2048];
   skip = 0;
   if (argc == 2) {
      skip = atoi(argv[1]);
   }
   while (skip-- && fgets(buf, sizeof buf, stdin) != NULL);
   while (fgets(buf, sizeof buf, stdin) != NULL) {
      fputs(buf, stdout);
   }
   return 0;
}

This doesn't require buffering the entire file, it's way more
portable, it's about the same length as your original "solution," etc.

IOW you're just posting to advertise your wares... in a C group where
your offerings are not C....

Stop spamming USENET and maybe people will treat you better.

Tom

[toc] | [prev] | [next] | [standalone]

#13606

tom st denis <tom@iahu.ca> wrote:
> On Nov 16, 8:44 am, jacob navia <ja...@spamsink.net> wrote:
> > Le 16/11/11 14:01, -.- a écrit :
> >
> > > jacob navia was trying to save the world with his stuff:
> >
> > >> Using the containers library (and if your file fits in memory)
> >
> > >> #include <containers.h>
> >
> > > You self-celebrating fucko.
> >
> > That is why you hide behind a pseudo, because you have the courage of
> > your opinions...

> While rude the guy has a point.  You can accomplish this goal easily
> with something like

> #include <stdio.h>
> #include <stdlib.h>

> int main(int argc, char **argv)
> {
>    int skip;
>    char buf[2048];
>    skip = 0;
>    if (argc == 2) {
>       skip = atoi(argv[1]);
>    }
>    while (skip-- && fgets(buf, sizeof buf, stdin) != NULL);
>    while (fgets(buf, sizeof buf, stdin) != NULL) {
>       fputs(buf, stdout);
>    }
>    return 0;
> }

> This doesn't require buffering the entire file, it's way more
> portable, it's about the same length as your original "solution," etc.

> IOW you're just posting to advertise your wares... in a C group where
> your offerings are not C....

> Stop spamming USENET and maybe people will treat you better.

I don't understand your complaints. As far as I can see Mr. Navia's
container library is written in standard compliant C - I just tried
to compile his program after minor modifications (added an include
for <stdio.h> and replaced the angle brackets around "containers.h"
by double quotes since the header file isn't in the systems include
directory on my system) with

gcc -std=c99 -pedantic -Wextra -Wall -Wwrite-strings

and there were no complaints at all. For C89 the main complaint
was the missing support for the 'long long' type used in the
library and a few minor niggles about two '//' comments in the
'containers.h" header file and about mixing of declarations and
code, finally, the missing return statement at the end - nothing
of any seriousness. (And if you try to compile the library it-
self in strict C89 mode all the compilers complaints seem to be
on the same level of seriousness, i.e. very low and not diffi-
cult to address.)

Further, the library is under a very permissive license, BSD,
so there are no strings attached.

So all I can see is that Mr. Navia proposed to use a library,
written by him and made available to the public, as a possible
solution for the problem the OP has. He didn't even mention
were his container library can be found - if the OP or others
are interested it's at 

http://code.google.com/p/ccl/

If that is spamming then a lot of other posters in this group
are guilty of the same when they dare to mention that they have
written some functions or library others might use freely for a
problem they want to solve. Just because Mr. Navia is also the
author of a compiler you can pay him money for IMHO shouldn't
bare him from mentioning code he wrote and made available for
free and taking part in trying to help others, or should it?

The only valid point I can see in your post is the question if
it's an optimal solution having to read in all of the file into
memory. But then this was already explicitly pointed out by Mr.
Navia himself as a requirement (and thus a possible shortcoming)
of using his solution - and for a lot of cases I don't think it's
a complete showstopper.

On the other hand your solution doesn't address the second re-
quirement of the OP, i.e. that the input file itself is to be
changed on exit of the program - your program reads from stdin
and writes to stdout, so it's a simple filter. In contrast, Mr.
Navia's program does also handle this other requirement of the
OP in changing the input file itself with a very similar number
of lines of code the user has to type...

Finally, I've got to say that I find Mr. Navia's solution rather
easy to comprehend without even having read the documentation for
his library (which I will do now;-). It seems to be a rather nice
example of how using it could make writing as well as understanding
certain types of programs in C quite a bit easier.

                               Regards, Jens
-- 
  \   Jens Thoms Toerring  ___      jt@toerring.de
   \__________________________      http://toerring.de

[toc] | [prev] | [next] | [standalone]

#13481

Try replacing:
>         while((c = fgetc(f)) != EOF) {
>                 fputc(c, ftmp);
>         }

with:
        bsize=m_free(0);
        buff=m_alloc(bsize);
        numread=-1;

        while(numread) {
          numread=fread(buff,1,bsize,f);
          fwrite(buff,1,numread,ftmp);
          }
        m_free(buff);

As with usenet tradition, completely untested.

JGH

[toc] | [prev] | [next] | [standalone]

#13482

jgharston wrote:
>         bsize=m_free(0);
>         buff=m_alloc(bsize);

Following up my own post, that call to m_free(0) is supposed to
return a size of a free block that can subsequently be claimed
with m_alloc(). A bit of a skim of through the web shows that
functionality isn't in any of the malloc libraries documented
there. All I can say is it worked 25 years ago! and inspired
me to include that functionality in my own malloc library.

Just replace bsize=m_free(0) with a suitable bsize=(some
method of deciding an amount of memory to claim).

JGH

[toc] | [prev] | [next] | [standalone]

#13501

jgharston <jgh@arcade.demon.co.uk> writes:
> Try replacing:
>>         while((c = fgetc(f)) != EOF) {
>>                 fputc(c, ftmp);
>>         }
>
> with:
>         bsize=m_free(0);
>         buff=m_alloc(bsize);
>         numread=-1;
>
>         while(numread) {
>           numread=fread(buff,1,bsize,f);
>           fwrite(buff,1,numread,ftmp);
>           }
>         m_free(buff);
>
> As with usenet tradition, completely untested.

Leaving aside the m_free and m_alloc calls, why do you assume that this
will be significantly faster than the fgetc/fputc loop?  stdio does its
own buffering.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#13523

Keith Thompson wrote:
> Leaving aside the m_free and m_alloc calls, why do you assume that this
> will be significantly faster than the fgetc/fputc loop?  stdio does its
> own buffering.

As I recall, this was a standard exam question back when I worra
litt'un.
If doing bulk data copying a program buffer is likely to be bigger
than stdio's buffer and bulk read/write/read/write is more efficient
for simple chucking of large lumps of data from one place to another,
one bit being the skipping of fgetc's unget functionality.

JGH

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web