Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c > #126789
| From | Ben Bacarisse <ben.usenet@bsb.me.uk> |
|---|---|
| Newsgroups | comp.lang.c |
| Subject | Re: Text Files |
| Date | 2018-02-16 14:31 +0000 |
| Organization | A noiseless patient Spider |
| Message-ID | <8737213t0w.fsf@bsb.me.uk> (permalink) |
| References | (3 earlier) <87mv097tcf.fsf@bsb.me.uk> <dsphC.15908$zy1.12073@fx30.am4> <p668md$gp3$1@dont-email.me> <87lgft4145.fsf@bsb.me.uk> <p66ho1$c2r$1@dont-email.me> |
David Brown <david.brown@hesbynett.no> writes:
> On 16/02/18 12:36, Ben Bacarisse wrote:
>> David Brown <david.brown@hesbynett.no> writes:
>>
>>> On 16/02/18 01:19, bartc wrote:
>>>> On 15/02/2018 22:58, Ben Bacarisse wrote:
>>>>> jacobnavia <jacob@jacob.remcomp.fr> writes:
>>>>
>>>>>> That last one is difficult to beat.
>>>>>
>>>>> Yes, I know what C's good at, but I was asking the OP. It's possible
>>>>> that none of that matters to them. It could be a quick throwaway
>>>>> program to find the line with the widest range:
>>>>>
>>>>> awk 'NF==6 { if ($6-$1 > r) { r = $6-$1; ln = NR }} END { print ln }'
>>>>>
>>>>> I'm shameless enough to say that I'm a pretty fast C programmer but,
>>>>> none the less, I'd write that sort of thing in AWK every time.
>>>>>
>>>>>> C is fast. Fast because it is just an abstraction of the machine.
>>>>>
>>>>> I changed my C version to do this calculation and found that the AWK
>>>>> version was faster.
>>>>
>>>> So, the task is to find the line where column 6 is bigger than column 1
>>>> by the maximum amount?
>>>>
>>>> I tried your code in gawk, on a 1-million line input file containing 6
>>>> columns of random numbers from 1 to 100,000.
>>>>
>>>> Gawk took 1.64 seconds.
>>>>
>>>> Then I threw something together with my scripting language, and it took
>>>> 1.50 seconds at the first attempt [code shown in my sig].
>>>
>>> Since we are having fun, I tested a simple python implementation.
>>>
>>> awk took 0.44s.
>>> Python2 and Python3 both took 0.9 seconds.
>>> pypy (JIT compiled python) took 0.37 seconds.
>>>
>>> All this shows that for a task like this, many languages are within
>>> spitting distance of the speed of C,
>>
>> I don't see how you can say that without a C data point on the same
>> machine. And, being very flexible, C code comes in with very different
>> timings depending on the exact job being done (for the "max range"
>> problem I got a fact of 10 between two simple C programs).
>
> "Within spitting distance" does not mean the same speed.
Yes, but I got about a factor 10. That's a long way to spit!
> It means fast
> enough to be usable for the same sort of task.
I don't think you can call that "within spitting distance". I'd take
anything from 0.01s to 10s for a once-off data analysis task, but I
would not call that within spitting distance in terms of speed.
If you want to try a C version to get a reference point for the other
times, this is the crudest, but fastest one I tried:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(void)
{
char line[1000];
size_t lnum = 0, lmax;
long int max = 0;
while (fgets(line, sizeof line, stdin)) {
lnum += 1;
long int d = strtol(strrchr(line, ' '), 0, 10) - strtol(line, 0, 10);
if (d > max) {
max = d;
lmax = lnum;
}
}
printf("%lu (%ld)\n", (unsigned long)lmax, max);
}
<snip>
> What we have shown in these few posts is that when you want to analyse
> lines of a text file with integer data, a language like Python (or awk,
> if the problem is simple enough, or even Bart's scripting language if he
> is the one solving the problem) is likely to give you a solid fraction
> of the speed of C with vastly less development effort.
Yes. What's more A quickly written C solution will be inflexible. More
and more work is needed to be able to alter the question you ask about
the data or to change the amount of checking applied to the input. Such
changes are usually quite small in languages aimed at this sort of task.
If I already had a collection of C functions to split lines on various
criteria and to convert text to numbers with proper error checking along
with some sort of fast associative array, I would consider using that
for large data sets, but there will always be a new question to be asked
of the data that will be more quickly answered with a slow AWK script.
--
Ben.
Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Text Files G B <gb@gb.com> - 2018-02-14 11:47 -0400
Re: Text Files Ed Prochak <edprochak@gmail.com> - 2018-02-14 08:45 -0800
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-14 08:47 -0800
Re: Text Files already5chosen@yahoo.com - 2018-02-15 01:17 -0800
Re: Text Files gordonb.ea2mq@burditt.org (Gordon Burditt) - 2018-02-14 11:10 -0600
Re: Text Files "James R. Kuyper" <jameskuyper@verizon.net> - 2018-02-14 12:28 -0500
Re: Text Files jt@toerring.de (Jens Thoms Toerring) - 2018-02-14 21:07 +0000
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-14 15:31 -0700
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-15 06:59 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 10:20 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-14 22:51 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-15 11:04 +0100
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-15 06:16 -0800
Re: Text Files "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2018-02-15 22:22 +0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-15 15:58 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 16:31 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-15 21:31 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 21:21 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 22:17 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 23:36 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 23:48 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 10:09 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 12:57 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 15:50 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 15:50 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 17:07 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 20:44 +0000
Re: Text Files Ian Collins <ian-news@hotmail.com> - 2018-02-17 11:22 +1300
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 17:48 +0000
Re: Text Files GOTHIER Nathan <nathan.gothier@gmail.com> - 2018-02-16 17:06 +0100
Re: Text Files Ian Collins <ian-news@hotmail.com> - 2018-02-17 11:10 +1300
Re: Text Files supercat@casperkitty.com - 2018-02-16 14:48 -0800
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-16 15:14 -0800
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 23:25 +0000
Re: Text Files supercat@casperkitty.com - 2018-02-16 15:50 -0800
Re: Text Files already5chosen@yahoo.com - 2018-02-16 07:19 -0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 17:13 +0100
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-19 17:55 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-19 19:14 +0100
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-19 11:48 -0800
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-16 08:14 -0800
Re: Text Files Robert Wessel <robertwessel2@yahoo.com> - 2018-02-17 18:32 -0600
Re: Text Files jacobnavia <jacob@jacob.remcomp.fr> - 2018-02-15 22:57 +0100
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 22:58 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 00:19 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 02:26 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 02:57 +0000
Re: Text Files asetofsymbols@gmail.com - 2018-02-16 00:21 -0800
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 11:30 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 10:36 +0100
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 11:36 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 13:10 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 12:25 +0000
Re: Text Files Ike Naar <ike@iceland.freeshell.org> - 2018-03-06 19:33 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-03-06 20:55 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-16 04:42 -0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 16:58 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 20:02 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 21:43 +0100
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-17 04:01 -0800
Re: Text Files bartc <bc@freeuk.com> - 2018-02-17 13:10 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-17 05:27 -0800
Re: Text Files bartc <bc@freeuk.com> - 2018-02-17 14:20 +0000
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-16 09:04 -0700
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 14:31 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-16 06:42 -0800
Re: Text Files bartc <bc@freeuk.com> - 2018-02-16 14:58 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-16 07:23 -0800
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-16 15:24 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-16 04:13 -0800
Re: Text Files Siri Cruise <chine.bleu@yahoo.com> - 2018-02-16 04:29 -0800
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-20 16:41 +0000
Re: Text Files Siri Cruise <chine.bleu@yahoo.com> - 2018-02-20 09:39 -0800
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-20 19:16 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 12:46 +0000
Re: Text Files GOTHIER Nathan <nathan.gothier@gmail.com> - 2018-02-15 14:52 +0100
Re: Text Files Anton Shepelev <anton.txt@g{oogle}mail.com> - 2018-02-15 17:07 +0300
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 14:23 +0000
Re: Text Files Anton Shepelev <anton.txt@g{oogle}mail.com> - 2018-02-15 18:52 +0300
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 14:39 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 15:43 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 20:32 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 22:06 +0000
Re: Text Files GOTHIER Nathan <nathan.gothier@gmail.com> - 2018-02-15 23:12 +0100
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-15 15:12 -0800
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 23:43 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 23:54 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 23:35 +0000
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-16 17:07 -0800
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-17 02:38 +0000
Re: Text Files Ian Collins <ian-news@hotmail.com> - 2018-02-17 17:26 +1300
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-17 11:22 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-17 12:07 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-17 04:17 -0800
Re: Text Files Anton Shepelev <anton.txt@gmail.com> - 2018-02-17 16:25 +0300
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-17 07:35 -0800
Re: Text Files GOTHIER Nathan <nathan.gothier@gmail.com> - 2018-02-17 15:24 +0100
Re: Text Files Ian Collins <ian-news@hotmail.com> - 2018-02-18 11:10 +1300
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-19 12:05 -0800
Re: Text Files already5chosen@yahoo.com - 2018-02-19 15:32 -0800
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-21 10:34 -0800
Re: Text Files already5chosen@yahoo.com - 2018-02-21 15:35 -0800
Re: Text Files bartc <bc@freeuk.com> - 2018-02-22 00:11 +0000
Re: Text Files Robert Wessel <robertwessel2@yahoo.com> - 2018-02-21 18:26 -0600
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-21 21:39 -0800
Re: Text Files Robert Wessel <robertwessel2@yahoo.com> - 2018-02-23 03:08 -0600
Re: Text Files supercat@casperkitty.com - 2018-02-23 16:16 -0800
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-23 16:46 -0800
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-24 02:49 -0800
Re: Text Files asetofsymbols@gmail.com - 2018-02-24 05:16 -0800
Re: Text Files "F. Russell" <fr@random.info> - 2018-02-24 14:35 +0000
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-24 08:48 -0700
Re: Text Files supercat@casperkitty.com - 2018-02-25 15:09 -0800
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-25 15:54 -0800
Re: Text Files supercat@casperkitty.com - 2018-02-26 11:04 -0800
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-26 11:52 -0800
Re: Text Files supercat@casperkitty.com - 2018-02-26 14:12 -0800
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-26 14:40 -0800
Re: Text Files already5chosen@yahoo.com - 2018-02-22 04:58 -0800
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-22 05:33 -0800
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-22 13:59 +0000
Re: Text Files Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-02-22 07:24 -0800
Re: Text Files supercat@casperkitty.com - 2018-02-20 12:14 -0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-15 21:37 +0100
Re: Text Files bartc <bc@freeuk.com> - 2018-02-15 21:34 +0000
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-15 14:15 -0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-16 10:47 +0100
Re: Text Files "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2018-02-15 22:20 +0800
Re: Text Files "F. Russell" <fr@random.info> - 2018-02-15 15:24 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-15 16:05 +0000
Re: Text Files "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2018-02-16 00:22 +0800
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-15 13:35 -0700
Re: Text Files Anton Shepelev <anton.txt@gmail.com> - 2018-02-25 00:13 +0300
Re: Text Files bartc <bc@freeuk.com> - 2018-02-24 21:24 +0000
Re: Text Files Anton Shepelev <anton.txt@gmail.com> - 2018-02-25 14:55 +0300
Re: Text Files bartc <bc@freeuk.com> - 2018-02-25 13:00 +0000
Re: Text Files Real Troll <Real.Troll@Trolls.com> - 2018-02-24 17:35 -0400
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-24 21:36 +0000
Re: Text Files Real Troll <Real.Troll@Trolls.com> - 2018-02-24 17:45 -0400
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-24 22:51 +0000
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-24 23:56 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-25 00:14 +0000
Re: Text Files bartc <bc@freeuk.com> - 2018-02-25 00:22 +0000
Re: Text Files Anton Shepelev <anton.txt@gmail.com> - 2018-02-25 15:16 +0300
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-25 15:20 -0800
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-25 20:27 -0700
Re: Text Files gazelle@shell.xmission.com (Kenny McCormack) - 2018-02-26 06:19 +0000
Re: Text Files Anton Shepelev <anton.txt@g{oogle}mail.com> - 2018-02-26 15:35 +0300
Re: Text Files Reinhardt Behm <rbehm@hushmail.com> - 2018-02-26 20:53 +0800
Grammar Patrol (Was: Text Files) gazelle@shell.xmission.com (Kenny McCormack) - 2018-02-26 17:54 +0000
Re: Text Files Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-02-26 15:27 +0000
Re: Text Files Anton Shepelev <anton.txt@g{oogle}mail.com> - 2018-02-26 18:50 +0300
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-26 09:26 -0800
Re: Text Files scott@slp53.sl.home (Scott Lurndal) - 2018-02-26 19:46 +0000
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-27 09:04 +0100
Re: Text Files already5chosen@yahoo.com - 2018-02-27 00:56 -0800
Re: Text Files Ian Collins <ian-news@hotmail.com> - 2018-02-27 22:07 +1300
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-27 14:37 +0100
Re: Text Files already5chosen@yahoo.com - 2018-02-27 07:38 -0800
Re: Text Files supercat@casperkitty.com - 2018-02-27 09:18 -0800
Re: Text Files David Brown <david.brown@hesbynett.no> - 2018-02-27 10:15 +0100
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-27 08:53 -0800
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-26 23:14 -0800
Re: Text Files Joe Pfeiffer <pfeiffer@cs.nmsu.edu> - 2018-02-26 08:32 -0700
Re: Text Files Keith Thompson <kst-u@mib.org> - 2018-02-26 09:17 -0800
Re: Text Files Tim Rentsch <txr@alumni.caltech.edu> - 2018-02-26 22:55 -0800
Re: Text Files Anton Shepelev <anton.txt@gmail.com> - 2018-02-25 15:02 +0300
Re: Text Files Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-02-25 20:12 +0000
csiph-web