Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #16962 > unrolled thread

seq command terribly slow

Back to article view | Back to comp.os.linux.misc

Contents

  seq command terribly slow Robert Riches <spamtrap42@jacob21819.net> - 2016-03-13 04:18 +0000
    Re: seq command terribly slow Richard Kettlewell <rjk@greenend.org.uk> - 2016-03-13 12:24 +0000
      Re: seq command terribly slow Robert Riches <spamtrap42@jacob21819.net> - 2016-03-13 16:42 +0000
        Re: seq command terribly slow "Anonymous Remailer (austria)" <mixmaster@remailer.privacy.at> - 2016-03-14 14:22 +0100
          Re: seq command terribly slow Robert Riches <spamtrap42@jacob21819.net> - 2016-03-15 02:38 +0000
    Re: seq command terribly slow Gernot Fink <g.fink@gmx.net> - 2016-03-13 18:21 +0100
      Re: seq command terribly slow Richard Kettlewell <rjk@greenend.org.uk> - 2016-03-13 17:45 +0000
    Re: seq command terribly slow Eli the Bearded <*@eli.users.panix.com> - 2016-03-14 21:58 +0000
      Re: seq command terribly slow Robert Riches <spamtrap42@jacob21819.net> - 2016-03-15 02:44 +0000
        Re: seq command terribly slow Eli the Bearded <*@eli.users.panix.com> - 2016-03-16 00:41 +0000

#16962 — seq command terribly slow

Any ideas why the seq command is so terribly slow?

In order to test whether a new (well, used, but that's another
story) disk, I tried using seq to generate a non-repeating
pattern.  After seeing seq consuming 100% CPU time and a very low
data rate being written to disk, I did some checking on a
few-years-old system with a Xeon W3680 running Debian 7/Wheezy.

This command shows about 15MB/s:

    seq 1e12 | pv > /dev/null

This command shows about 600MB/s, ~40X faster with much lower
total CPU use:

    yes '' | cat -n | tr -d ' \t' | head -c 1000000000000 | pv > /dev/null

Even this simple Python program shows about 12MB/s:
(indented for this posting)

    first     = 1.0
    last      = 1.0e12
    increment = 1.0
    
    i = first
    while (last >= i):
        print('%.0f' % i)
        i += increment

Yes, the seq command has a lot of formatting options.  They
should basically all be handled at initialization time and not
incur a significant CPU penalty in the main loop.

Any ideas why seq is to slow?

Thanks.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [next] | [standalone]

#16968

Robert Riches <spamtrap42@jacob21819.net> writes:
> Any ideas why the seq command is so terribly slow?
>
> In order to test whether a new (well, used, but that's another
> story) disk, I tried using seq to generate a non-repeating
> pattern.  After seeing seq consuming 100% CPU time and a very low
> data rate being written to disk, I did some checking on a
> few-years-old system with a Xeon W3680 running Debian 7/Wheezy.
>
> This command shows about 15MB/s:
>
>     seq 1e12 | pv > /dev/null
>
> This command shows about 600MB/s, ~40X faster with much lower
> total CPU use:
>
>     yes '' | cat -n | tr -d ' \t' | head -c 1000000000000 | pv > /dev/null

I see near-reverse behaviour in jessie (on a Q6600).  seq yields more
than 500MB/s while the pipeline is below 300MB/s.

Perhaps seq has been improved since the version in wheezy.

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#16970

On 2016-03-13, Richard Kettlewell <rjk@greenend.org.uk> wrote:
> Robert Riches <spamtrap42@jacob21819.net> writes:
>> Any ideas why the seq command is so terribly slow?
>>
>> In order to test whether a new (well, used, but that's another
>> story) disk, I tried using seq to generate a non-repeating
>> pattern.  After seeing seq consuming 100% CPU time and a very low
>> data rate being written to disk, I did some checking on a
>> few-years-old system with a Xeon W3680 running Debian 7/Wheezy.
>>
>> This command shows about 15MB/s:
>>
>>     seq 1e12 | pv > /dev/null
>>
>> This command shows about 600MB/s, ~40X faster with much lower
>> total CPU use:
>>
>>     yes '' | cat -n | tr -d ' \t' | head -c 1000000000000 | pv > /dev/null
>
> I see near-reverse behaviour in jessie (on a Q6600).  seq yields more
> than 500MB/s while the pipeline is below 300MB/s.
>
> Perhaps seq has been improved since the version in wheezy.

Good to hear it has apparently improved.  That would make it much
more useful.  I plan to switch to a different distribution soon
due to Wheezy's age, so I'll look forward to confirming that
improvement.

Thanks.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [prev] | [next] | [standalone]

#16981

Robert Riches <spamtra...@jacob21819.net> [RR]:

RR> This command shows about 600MB/s, ~40X faster with much lower
RR> total CPU use:
RR>   yes '' | cat -n | tr -d ' \t' | head -c 1000000000000 | pv > /dev/null
RR>
RR> I plan to switch to a different distribution soon due to Wheezy's age,
RR> so I'll look forward to confirming that improvement.

Since tr seems to run even faster than yes, you could in the meantime use
the following instead:

< /dev/zero tr '\0' '\n' | cat -n | tr  -d ' \t' |
	head -c 1000000000000 | pv > /dev/null

[toc] | [prev] | [next] | [standalone]

#16984

On 2016-03-14, Anonymous Remailer (austria) <mixmaster@remailer.privacy.at> wrote:
>
> Robert Riches <spamtra...@jacob21819.net> [RR]:
>
> RR> This command shows about 600MB/s, ~40X faster with much lower
> RR> total CPU use:
> RR>   yes '' | cat -n | tr -d ' \t' | head -c 1000000000000 | pv > /dev/null
> RR>
> RR> I plan to switch to a different distribution soon due to Wheezy's age,
> RR> so I'll look forward to confirming that improvement.
>
> Since tr seems to run even faster than yes, you could in the meantime use
> the following instead:
>
>< /dev/zero tr '\0' '\n' | cat -n | tr  -d ' \t' |
> 	head -c 1000000000000 | pv > /dev/null

That's very clever.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [prev] | [next] | [standalone]

#16971

I think its because the stream is line bufferd. Each line is handled
seperate.
 
In article <slrnne9qgo.5pk.spamtrap42@one.localnet>,
	Robert Riches <spamtrap42@jacob21819.net> writes:
> This command shows about 15MB/s:
> 
>     seq 1e12 | pv > /dev/null
> Any ideas why seq is to slow?

[toc] | [prev] | [next] | [standalone]

#16972

Gernot Fink <g.fink@gmx.net> writes:
> Robert Riches <spamtrap42@jacob21819.net> writes:
>> This command shows about 15MB/s:
>> 
>>     seq 1e12 | pv > /dev/null
>> Any ideas why seq is to slow?
>
> I think its because the stream is line bufferd. Each line is handled
> seperate.

Nope.

    $ seq --version
    seq (GNU coreutils) 8.5
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later
    <http://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Written by Ulrich Drepper.
    $ strace -o junk/trace seq 1e12 | cat > /dev/null
    ^C
    $ tail junk/trace
    write(1, "\n887151\n887152\n887153\n887154\n887"..., 4096) = 4096
    write(1, "887736\n887737\n887738\n887739\n8877"..., 4096) = 4096
    write(1, "88321\n888322\n888323\n888324\n88832"..., 4096) = 4096
    write(1, "8906\n888907\n888908\n888909\n888910"..., 4096) = 4096
    write(1, "491\n889492\n889493\n889494\n889495\n"..., 4096) = 4096
    write(1, "76\n890077\n890078\n890079\n890080\n8"..., 4096) = 4096
    write(1, "1\n890662\n890663\n890664\n890665\n89"..., 4096) = 4096
    write(1, "\n891247\n891248\n891249\n891250\n891"..., 4096) = 4096
    --- SIGINT (Interrupt) @ 0 (0) ---
    +++ killed by SIGINT +++

-- 
http://www.greenend.org.uk/rjk/

[toc] | [prev] | [next] | [standalone]

#16983

In comp.os.linux.misc, Robert Riches  <spamtrap42@jacob21819.net> wrote:
> This command shows about 15MB/s:
> 
>     seq 1e12 | pv > /dev/null

Does it store 1e12 as a floating point number and make conversion to int
for comparision every time through the loop? printf() format all output?

What do you get if you enable chargen in your inetd provider and then
"telnet localhost chargen"?

Elijah
------
cause that's the character generator an old-timer like me would use

[toc] | [prev] | [next] | [standalone]

#16985

On 2016-03-14, Eli the Bearded <*@eli.users.panix.com> wrote:
> In comp.os.linux.misc, Robert Riches  <spamtrap42@jacob21819.net> wrote:
>> This command shows about 15MB/s:
>> 
>>     seq 1e12 | pv > /dev/null
>
> Does it store 1e12 as a floating point number and make conversion to int
> for comparision every time through the loop? printf() format all output?
>
> What do you get if you enable chargen in your inetd provider and then
> "telnet localhost chargen"?
>
> Elijah
> ------
> cause that's the character generator an old-timer like me would use

The man page does seem to imply that seq uses floating point math
internally.  However, with a reasonably modern CPU, I would
expect it should do be able to do better than 15MB/s, even doing
double precision floating point math.  Another poster said a more
modern seq version is faster than the pipeline in his tests.

"whatis chargen" yields "chargen: nothing appropriate."  "locate
chargen" yields nothing.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

[toc] | [prev] | [next] | [standalone]

#17009

In comp.os.linux.misc, Robert Riches  <spamtrap42@jacob21819.net> wrote:
> On 2016-03-14, Eli the Bearded <*@eli.users.panix.com> wrote:
>> What do you get if you enable chargen in your inetd provider and then
>> "telnet localhost chargen"?
> "whatis chargen" yields "chargen: nothing appropriate."  "locate
> chargen" yields nothing.

On slackware, /etc/inetd.conf says "man 8 inetd" for more infomation.
When I uncomment line 20:

  # chargen	stream	tcp	nowait	root	internal

to

  chargen	stream	tcp	nowait	root	internal

And then "kill -HUP" inetd, then I can "telnet localhost chargen" and I
get throughput a little more than twice as fast as numbering lines from
"yes".

Off the top of my head, I don't know how to turn on chargen in xinetd,
but I suspect it can be done. Also look for the discard service if you
don't want to rely on external providers like:

https://devnull-as-a-service.com/

See also:

Discard protocol: https://tools.ietf.org/html/rfc863
Chargen protocol: https://tools.ietf.org/html/rfc864

Elijah
------
30.2 MB/s vs 71.0 MB/s as measured by "dd of=/dev/null"

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.misc

csiph-web