Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32200 > unrolled thread

better way for ' '.join(args) + '\n'?

Back to article view | Back to comp.lang.python

Contents

  better way for ' '.join(args) + '\n'? Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-10-26 09:49 +0200
    Re: better way for ' '.join(args) + '\n'? Peter Otten <__peter__@web.de> - 2012-10-26 10:58 +0200
    Re: better way for ' '.join(args) + '\n'? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-26 09:36 +0000
    Re: better way for ' '.join(args) + '\n'? Hubert Grünheidt <hgruenheidt@t-online.de> - 2012-10-26 18:21 +0200
    Re: better way for ' '.join(args) + '\n'? Tycho Andersen <tycho@tycho.ws> - 2012-10-26 16:26 -0500
    Re: better way for ' '.join(args) + '\n'? Dave Angel <d@davea.name> - 2012-10-26 17:36 -0400
    Re: better way for ' '.join(args) + '\n'? Tycho Andersen <tycho@tycho.ws> - 2012-10-26 16:42 -0500
      Re: better way for ' '.join(args) + '\n'? Ramchandra Apte <maniandram01@gmail.com> - 2012-11-03 01:26 -0700
      Re: better way for ' '.join(args) + '\n'? Ramchandra Apte <maniandram01@gmail.com> - 2012-11-03 01:26 -0700
    Re: better way for ' '.join(args) + '\n'? Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-10-27 11:26 +0200
      RE: better way for ' '.join(args) + '\n'? "Prasad, Ramit" <ramit.prasad@jpmorgan.com> - 2012-10-29 22:28 +0000

#32200 — better way for ' '.join(args) + '\n'?

Hi!

General advise when assembling strings is to not concatenate them 
repeatedly but instead use string's join() function, because it avoids 
repeated reallocations and is at least as expressive as any alternative.

What I have now is a case where I'm assembling lines of text for driving 
a program with a commandline interface. In this scenario, I'm currently 
doing this:

   args = ['foo', 'bar', 'baz']
   line = ' '.join(args) + '\n'

So, in other words, I'm avoiding all the unnecessary copying, just to 
make another copy to append the final newline.

The only way around this that I found involves creating an intermediate 
sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done 
rather cleanly with a generator:

   def helper(s):
       for i in s[:-1]:
            yield i
            yield ' '
       yield s[-1]
       yield '\n'
   line = ''.join(tmp(args))

Efficiency-wise, this is satisfactory. However, readability counts and 
that is where this version fails and that is the reason why I'm writing 
this message. So, dear fellow Pythonistas, any ideas to improve the 
original versions efficiency while preserving its expressiveness?

Oh, for all those that are tempted to tell me that this is not my 
bottleneck unless it's called in a very tight loop, you're right. 
Indeed, the overhead of the communication channel TCP between the two 
programs is by far dwarving the few microseconds I could save here. I'm 
still interested in learning new and better solutions though.


Cheers!

Uli

[toc] | [next] | [standalone]

#32202

Ulrich Eckhardt wrote:

> Hi!
> 
> General advise when assembling strings is to not concatenate them
> repeatedly but instead use string's join() function, because it avoids
> repeated reallocations and is at least as expressive as any alternative.
> 
> What I have now is a case where I'm assembling lines of text for driving
> a program with a commandline interface. In this scenario, I'm currently
> doing this:
> 
>    args = ['foo', 'bar', 'baz']
>    line = ' '.join(args) + '\n'
> 
> So, in other words, I'm avoiding all the unnecessary copying, just to
> make another copy to append the final newline.
> 
> The only way around this that I found involves creating an intermediate
> sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
> rather cleanly with a generator:
> 
>    def helper(s):
>        for i in s[:-1]:
>             yield i
>             yield ' '
>        yield s[-1]
>        yield '\n'
>    line = ''.join(tmp(args))
> 
> Efficiency-wise, this is satisfactory. 

No, it is not. In a quick timeit test it takes 5 to 10 times as long as the 
original. Remember that function calls are costly, and that with s[:-1] you 
are trading the extra string for an extra list. Also, you are doubling the 
loop implicit in str.join() with the explicit one in your oh-so-efficient 
generator.

> However, readability counts and
> that is where this version fails and that is the reason why I'm writing
> this message. So, dear fellow Pythonistas, any ideas to improve the
> original versions efficiency while preserving its expressiveness?
> 
> Oh, for all those that are tempted to tell me that this is not my
> bottleneck unless it's called in a very tight loop, you're right.
> Indeed, the overhead of the communication channel TCP between the two
> programs is by far dwarving the few microseconds I could save here. I'm
> still interested in learning new and better solutions though.

Even if it were the bottleneck the helper generator approach would still be 
unhelpful.

[toc] | [prev] | [next] | [standalone]

#32204

On Fri, 26 Oct 2012 09:49:50 +0200, Ulrich Eckhardt wrote:

> Hi!
> 
> General advise when assembling strings is to not concatenate them
> repeatedly but instead use string's join() function, because it avoids
> repeated reallocations and is at least as expressive as any alternative.
> 
> What I have now is a case where I'm assembling lines of text for driving
> a program with a commandline interface. In this scenario, I'm currently
> doing this:
> 
>    args = ['foo', 'bar', 'baz']
>    line = ' '.join(args) + '\n'
> 
> So, in other words, I'm avoiding all the unnecessary copying, just to
> make another copy to append the final newline.

*shrug*

The difference between ' '.join(sequence) and (' '.join(sequence) + '\n') 
is, in Big Oh analysis, insignificant. The first case does O(N) 
operations, the second does O(N) + O(N) = 2*O(N) operations, which is 
still O(N). In effect, the two differ only by an approximately constant 
factor.

If you really care, and you don't mind ending your last line with a 
space, just append '\n' to the sequence before calling join.


> The only way around this that I found involves creating an intermediate
> sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
> rather cleanly with a generator:
> 
>    def helper(s):
>        for i in s[:-1]:
>             yield i
>             yield ' '
>        yield s[-1]
>        yield '\n'
>    line = ''.join(tmp(args))
> 
> Efficiency-wise, this is satisfactory. 

Have you actually tested this? I would not be the least surprised if 
that's actually less efficient than the (' '.join(seq) + '\n') version.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#32223

Hi Ulrich,

is this acceptable?

     args = ['foo', 'bar', 'baz']
     args.append('\n')
     line = ' '.join(args)

Cheers,
       Hubert

On 10/26/2012 09:49 AM, Ulrich Eckhardt wrote:
> Hi!
>
> General advise when assembling strings is to not concatenate them
> repeatedly but instead use string's join() function, because it avoids
> repeated reallocations and is at least as expressive as any alternative.
>
> What I have now is a case where I'm assembling lines of text for driving
> a program with a commandline interface. In this scenario, I'm currently
> doing this:
>
>    args = ['foo', 'bar', 'baz']
>    line = ' '.join(args) + '\n'
>
> So, in other words, I'm avoiding all the unnecessary copying, just to
> make another copy to append the final newline.
>
> The only way around this that I found involves creating an intermediate
> sequence like ['foo', ' ', 'bar', ' ', 'baz', '\n']. This can be done
> rather cleanly with a generator:
>
>    def helper(s):
>        for i in s[:-1]:
>             yield i
>             yield ' '
>        yield s[-1]
>        yield '\n'
>    line = ''.join(tmp(args))
>
> Efficiency-wise, this is satisfactory. However, readability counts and
> that is where this version fails and that is the reason why I'm writing
> this message. So, dear fellow Pythonistas, any ideas to improve the
> original versions efficiency while preserving its expressiveness?
>
> Oh, for all those that are tempted to tell me that this is not my
> bottleneck unless it's called in a very tight loop, you're right.
> Indeed, the overhead of the communication channel TCP between the two
> programs is by far dwarving the few microseconds I could save here. I'm
> still interested in learning new and better solutions though.
>
>
> Cheers!
>
> Uli
>

[toc] | [prev] | [next] | [standalone]

#32234

On Fri, Oct 26, 2012 at 09:49:50AM +0200, Ulrich Eckhardt wrote:
> Hi!
> 
> General advise when assembling strings is to not concatenate them
> repeatedly but instead use string's join() function, because it
> avoids repeated reallocations and is at least as expressive as any
> alternative.
> 
> What I have now is a case where I'm assembling lines of text for
> driving a program with a commandline interface. In this scenario,
> I'm currently doing this:
> 
>   args = ['foo', 'bar', 'baz']
>   line = ' '.join(args) + '\n'

Assuming it's the length of the list that's the problem, not the
length of the strings in the list...

args = ['foo', 'bar', 'baz']
args[-1] = args[-1] + '\n'
line = ' '.join(args)

\t

[toc] | [prev] | [next] | [standalone]

#32235

On 10/26/2012 05:26 PM, Tycho Andersen wrote:
> On Fri, Oct 26, 2012 at 09:49:50AM +0200, Ulrich Eckhardt wrote:
>> Hi!
>>
>> General advise when assembling strings is to not concatenate them
>> repeatedly but instead use string's join() function, because it
>> avoids repeated reallocations and is at least as expressive as any
>> alternative.
>>
>> What I have now is a case where I'm assembling lines of text for
>> driving a program with a commandline interface. In this scenario,
>> I'm currently doing this:
>>
>>   args = ['foo', 'bar', 'baz']
>>   line = ' '.join(args) + '\n'
> Assuming it's the length of the list that's the problem, not the
> length of the strings in the list...
>
> args = ['foo', 'bar', 'baz']
> args[-1] = args[-1] + '\n'
> line = ' '.join(args)
>
> \t

Main problem with that is the trailing space before the newline.  If
that's not a problem, then fine.

Not sure why we try so hard to optimize something that's going to take
negligible time.

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#32236

On Fri, Oct 26, 2012 at 05:36:50PM -0400, Dave Angel wrote:
> On 10/26/2012 05:26 PM, Tycho Andersen wrote:
> > Assuming it's the length of the list that's the problem, not the
> > length of the strings in the list...
> >
> > args = ['foo', 'bar', 'baz']
> > args[-1] = args[-1] + '\n'
> > line = ' '.join(args)
> >
> > \t
> 
> Main problem with that is the trailing space before the newline.  If
> that's not a problem, then fine.

What trailing space before the newline? The other solutions have it,
the above does not. However, the above does mutate args, which isn't
all that great. Alas, if you want the performance of mutable
structures, you're probably going to have to mutate something. (In any
case, it's easy enough to change it back, though ugly.)

> Not sure why we try so hard to optimize something that's going to take
> negligible time.

The same reason some people enjoy sporting events: it's fun :-)

\t

[toc] | [prev] | [next] | [standalone]

#32682

On Saturday, 27 October 2012 03:12:31 UTC+5:30, Tycho Andersen  wrote:
> On Fri, Oct 26, 2012 at 05:36:50PM -0400, Dave Angel wrote:
> 
> > On 10/26/2012 05:26 PM, Tycho Andersen wrote:
> 
> > > Assuming it's the length of the list that's the problem, not the
> 
> > > length of the strings in the list...
> 
> > >
> 
> > > args = ['foo', 'bar', 'baz']
> 
> > > args[-1] = args[-1] + '\n'
> 
> > > line = ' '.join(args)
> 
> > >
> 
> > > \t
> 
> > 
> 
> > Main problem with that is the trailing space before the newline.  If
> 
> > that's not a problem, then fine.
> 
> 
> 
> What trailing space before the newline? The other solutions have it,
> 
> the above does not. However, the above does mutate args, which isn't
> 
> all that great. Alas, if you want the performance of mutable
> 
> structures, you're probably going to have to mutate something. (In any
> 
> case, it's easy enough to change it back, though ugly.)
> 
> 
> 
> > Not sure why we try so hard to optimize something that's going to take
> 
> > negligible time.
> 
> 
> 
> The same reason some people enjoy sporting events: it's fun :-)
Me too
> 
> 
> 
> \t

[toc] | [prev] | [next] | [standalone]

#32683

On Saturday, 27 October 2012 03:12:31 UTC+5:30, Tycho Andersen  wrote:
> On Fri, Oct 26, 2012 at 05:36:50PM -0400, Dave Angel wrote:
> 
> > On 10/26/2012 05:26 PM, Tycho Andersen wrote:
> 
> > > Assuming it's the length of the list that's the problem, not the
> 
> > > length of the strings in the list...
> 
> > >
> 
> > > args = ['foo', 'bar', 'baz']
> 
> > > args[-1] = args[-1] + '\n'
> 
> > > line = ' '.join(args)
> 
> > >
> 
> > > \t
> 
> > 
> 
> > Main problem with that is the trailing space before the newline.  If
> 
> > that's not a problem, then fine.
> 
> 
> 
> What trailing space before the newline? The other solutions have it,
> 
> the above does not. However, the above does mutate args, which isn't
> 
> all that great. Alas, if you want the performance of mutable
> 
> structures, you're probably going to have to mutate something. (In any
> 
> case, it's easy enough to change it back, though ugly.)
> 
> 
> 
> > Not sure why we try so hard to optimize something that's going to take
> 
> > negligible time.
> 
> 
> 
> The same reason some people enjoy sporting events: it's fun :-)
Me too
> 
> 
> 
> \t

[toc] | [prev] | [next] | [standalone]

#32269

Am 26.10.2012 09:49 schrieb Ulrich Eckhardt:
> Hi!
>
> General advise when assembling strings is to not concatenate them
> repeatedly but instead use string's join() function, because it avoids
> repeated reallocations and is at least as expressive as any alternative.
>
> What I have now is a case where I'm assembling lines of text for driving
> a program with a commandline interface.

Stop.

In this case, you think too complicated.

Just do

     subprocess.Popen(['prog', 'foo', 'bar', 'baz'])

-  is the most safest thing for this use case.

If it should not be possible for any reason, you should be aware of any 
traps you could catch - e.g., if you want to feed your string to a 
Bourne shell, you should escape the strings properly.

In such cases, I use


def shellquote(*strs):
	r"""Input: file names, output: ''-enclosed strings where every ' is 
replaced with '\''. Intended for usage with the shell."""
	# just take over everything except ';
	# replace ' with '\''
	# The shell sees ''' as ''\'''\'''\'''. Ugly, but works.
	return " ".join([
		"'"+st.replace("'","'\\''")+"'"
		for st in strs
	])


so I can use

shellquote('program name', 'argu"ment 1', '$arg 2',
     "even args containing a ' are ok")

For Windows, you'll have to modify this somehow.


HTH,

Thomas

[toc] | [prev] | [next] | [standalone]

#32429

Thomas Rachel wrote:
> Am 26.10.2012 09:49 schrieb Ulrich Eckhardt:
> > Hi!
> >
> > General advise when assembling strings is to not concatenate them
> > repeatedly but instead use string's join() function, because it avoids
> > repeated reallocations and is at least as expressive as any alternative.
> >
> > What I have now is a case where I'm assembling lines of text for driving
> > a program with a commandline interface.
> 
> Stop.
> 
> In this case, you think too complicated.
> 
> Just do
> 
>      subprocess.Popen(['prog', 'foo', 'bar', 'baz'])
> 
> -  is the most safest thing for this use case.
> 
> If it should not be possible for any reason, you should be aware of any
> traps you could catch - e.g., if you want to feed your string to a
> Bourne shell, you should escape the strings properly.
> 
> In such cases, I use
> 
> 
> def shellquote(*strs):
> 	r"""Input: file names, output: ''-enclosed strings where every ' is
> replaced with '\''. Intended for usage with the shell."""
> 	# just take over everything except ';
> 	# replace ' with '\''
> 	# The shell sees ''' as ''\'''\'''\'''. Ugly, but works.
> 	return " ".join([
> 		"'"+st.replace("'","'\\''")+"'"
> 		for st in strs
> 	])
> 
> 
> so I can use
> 
> shellquote('program name', 'argu"ment 1', '$arg 2',
>      "even args containing a ' are ok")
> 
> For Windows, you'll have to modify this somehow.
> 

The subprocess module suggests using pipes.quote for escaping.

>>> a
('program name', 'argu"ment 1', '$arg 2', "even args containing a ' are ok")
>>> import pipes
>>> map(pipes.quote, a)
["'program name'", '\'argu"ment 1\'', "'$arg 2'", '\'even args containing a \'"\'"\' are ok\'']
>>> ' '.join(a)
'\'program name\' \'argu"ment 1\' \'$arg 2\' \'even args containing a \'\\\'\' are ok\''


Ramit Prasad


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web