Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #20748 > unrolled thread

sum() requires number, not simply __add__

Back to article view | Back to comp.lang.python

Contents

  sum() requires number, not simply __add__ Buck Golemon <buck@yelp.com> - 2012-02-23 13:19 -0800
    Re: sum() requires number, not simply __add__ Buck Golemon <buck@yelp.com> - 2012-02-23 13:23 -0800
      Re: sum() requires number, not simply __add__ Arnaud Delobelle <arnodel@gmail.com> - 2012-02-23 21:41 +0000
      Re: sum() requires number, not simply __add__ Chris Angelico <rosuav@gmail.com> - 2012-02-24 08:53 +1100
        Re: sum() requires number, not simply __add__ Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-02-23 23:33 +0000
          Re: sum() requires number, not simply __add__ Chris Angelico <rosuav@gmail.com> - 2012-02-24 10:39 +1100
          Re: sum() requires number, not simply __add__ Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2012-02-24 12:41 +0100
            Re: sum() requires number, not simply __add__ Roy Smith <roy@panix.com> - 2012-02-24 08:23 -0500
              Re: sum() requires number, not simply __add__ Terry Reedy <tjreedy@udel.edu> - 2012-02-24 16:58 -0500
      Re: sum() requires number, not simply __add__ Arnaud Delobelle <arnodel@gmail.com> - 2012-02-23 21:59 +0000
      Re: sum() requires number, not simply __add__ Ian Kelly <ian.g.kelly@gmail.com> - 2012-02-23 15:00 -0700
      Re: sum() requires number, not simply __add__ Chris Angelico <rosuav@gmail.com> - 2012-02-24 09:04 +1100
      Re: sum() requires number, not simply __add__ Arnaud Delobelle <arnodel@gmail.com> - 2012-02-23 22:09 +0000
    Re: sum() requires number, not simply __add__ Arnaud Delobelle <arnodel@gmail.com> - 2012-02-23 21:32 +0000
    Re: sum() requires number, not simply __add__ Chris Rebert <clp2@rebertia.com> - 2012-02-23 13:32 -0800
      Re: sum() requires number, not simply __add__ Buck Golemon <buck@yelp.com> - 2012-02-23 13:38 -0800
        Re: sum() requires number, not simply __add__ Ian Kelly <ian.g.kelly@gmail.com> - 2012-02-23 14:54 -0700
    Re: sum() requires number, not simply __add__ Stefan Behnel <stefan_ml@behnel.de> - 2012-02-23 22:42 +0100
      Re: sum() requires number, not simply __add__ Duncan Booth <duncan.booth@invalid.invalid> - 2012-02-24 11:40 +0000
    Re: sum() requires number, not simply __add__ Peter Otten <__peter__@web.de> - 2012-02-24 09:29 +0100

#20748 — sum() requires number, not simply __add__

I feel like the design of sum() is inconsistent with other language
features of python. Often python doesn't require a specific type, only
that the type implement certain methods.

Given a class that implements __add__ why should sum() not be able to
operate on that class?

We can fix this in a backward-compatible way, I believe.

Demonstration:
    I'd expect these two error messages to be identical, but they are
not.

     >>> class C(object): pass
     >>> c = C()
     >>> sum((c,c))
    TypeError: unsupported operand type(s) for +: 'int' and 'C'
    >>> c + c
    TypeError: unsupported operand type(s) for +: 'C' and 'C'

[toc] | [next] | [standalone]

#20749

On Feb 23, 1:19 pm, Buck Golemon <b...@yelp.com> wrote:
> I feel like the design of sum() is inconsistent with other language
> features of python. Often python doesn't require a specific type, only
> that the type implement certain methods.
>
> Given a class that implements __add__ why should sum() not be able to
> operate on that class?
>
> We can fix this in a backward-compatible way, I believe.
>
> Demonstration:
>     I'd expect these two error messages to be identical, but they are
> not.
>
>      >>> class C(object): pass
>      >>> c = C()
>      >>> sum((c,c))
>     TypeError: unsupported operand type(s) for +: 'int' and 'C'
>     >>> c + c
>     TypeError: unsupported operand type(s) for +: 'C' and 'C'

Proposal:

def sum(values,
base=0):
      values =
iter(values)

      try:
          result = values.next()
      except StopIteration:
          return base

      for value in values:
          result += value
      return result

[toc] | [prev] | [next] | [standalone]

#20753

On 23 February 2012 21:23, Buck Golemon <buck@yelp.com> wrote:
> def sum(values,
> base=0):
>      values =
> iter(values)
>
>      try:
>          result = values.next()
>      except StopIteration:
>          return base
>
>      for value in values:
>          result += value
>      return result

This is definitely not backward compatible.  To get something that has
a better chance of working with existing code, try this (untested):

_sentinel = object()

def sum(iterable, start=_sentinel):
    if start is _sentinel:
        iterable = iter(iterable)
        try:
            start = iterable.next()
        except StopIteration:
            return 0
    for x in iterable:
        start += x
    return start

del _sentinel

-- 
Arnaud

[toc] | [prev] | [next] | [standalone]

#20755

On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
> _sentinel = object()
>
> def sum(iterable, start=_sentinel):
>    if start is _sentinel:
>
> del _sentinel

Somewhat off-topic: Doesn't the if statement there do a lookup for a
global, which would mean that 'del _sentinel' will cause it to fail?
Or have I missed something here?

ChrisA

[toc] | [prev] | [next] | [standalone]

#20765

On Fri, 24 Feb 2012 08:53:49 +1100, Chris Angelico wrote:

> On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <arnodel@gmail.com>
> wrote:
>> _sentinel = object()
>>
>> def sum(iterable, start=_sentinel):
>>    if start is _sentinel:
>>
>> del _sentinel
> 
> Somewhat off-topic: Doesn't the if statement there do a lookup for a
> global, which would mean that 'del _sentinel' will cause it to fail? Or
> have I missed something here?

Yes, deleting _sentinel will cause the custom sum to fail, and yes, you 
have missed something.

If the caller wants to mess with your library and break it, they have 
many, many ways to do so apart from deleting your private variables.


del _sentinel
_sentinel = "something else"
sum.__defaults__ = (42,)  # mess with the function defaults
sum.__code__ = (lambda a, b=None: 100).__code__  # and with func internals
sum = None  # change your custom sum to something else
del sum  # or just delete it completely
len = 42  # shadow a built-in
import builtins; del builtins.range  # really screw with you


If your application stops working after you carelessly mess with 
components your application relies on, the right answer is usually:

"Don't do that then."

Python doesn't try to prevent people from shooting themselves in the foot.


Monkey-patching-by-actual-monkeys-for-fun-and-profit-ly y'rs,


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#20766

On Fri, Feb 24, 2012 at 10:33 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Yes, deleting _sentinel will cause the custom sum to fail, and yes, you
> have missed something.
>
> If the caller wants to mess with your library and break it, they have
> many, many ways to do so apart from deleting your private variables.

I was looking at the module breaking itself, though, not even waiting
for the caller to do it.

ChrisA

[toc] | [prev] | [next] | [standalone]

#20800

On 02/24/2012 12:33 AM, Steven D'Aprano wrote:
> If your application stops working after you carelessly mess with
> components your application relies on, the right answer is usually:
>
> "Don't do that then."
>
> Python doesn't try to prevent people from shooting themselves in the foot.
>    
Yes it does! A simple example is None as a keyword to prevent 
assignments to it.

-- 
Antoon Pardon

[toc] | [prev] | [next] | [standalone]

#20810

In article <mailman.123.1330083762.3037.python-list@python.org>,
 Antoon Pardon <antoon.pardon@rece.vub.ac.be> wrote:

> > Python doesn't try to prevent people from shooting themselves in the foot.
> >    
> Yes it does! A simple example is None as a keyword to prevent 
> assignments to it.

Hmmm.  Just playing around with some bizarre things to do with None, and 
discovered this:

>>> import sys as None

doesn't give an error, but also doesn't assign the module to the symbol 
'None'.  Weird.

[toc] | [prev] | [next] | [standalone]

#20834

On 2/24/2012 8:23 AM, Roy Smith wrote:
> In article<mailman.123.1330083762.3037.python-list@python.org>,
>   Antoon Pardon<antoon.pardon@rece.vub.ac.be>  wrote:
>
>>> Python doesn't try to prevent people from shooting themselves in the foot.
>>>
>> Yes it does! A simple example is None as a keyword to prevent
>> assignments to it.
>
> Hmmm.  Just playing around with some bizarre things to do with None, and
> discovered this:
>
>>>> import sys as None
>
> doesn't give an error, but also doesn't assign the module to the symbol
> 'None'.  Weird.

In 3.2
 >>> import sys as None
SyntaxError: invalid syntax

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#20757

On 23 February 2012 21:53, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
>> _sentinel = object()
>>
>> def sum(iterable, start=_sentinel):
>>    if start is _sentinel:
>>
>> del _sentinel
>
> Somewhat off-topic: Doesn't the if statement there do a lookup for a
> global, which would mean that 'del _sentinel' will cause it to fail?
> Or have I missed something here?

Yes, you're right :)  Change the signature to

def sum(iterable, start=_sentinel, _sentinel=_sentinel):

This is not pretty...

-- 
Arnaud

[toc] | [prev] | [next] | [standalone]

#20758

On Thu, Feb 23, 2012 at 2:53 PM, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
>> _sentinel = object()
>>
>> def sum(iterable, start=_sentinel):
>>    if start is _sentinel:
>>
>> del _sentinel
>
> Somewhat off-topic: Doesn't the if statement there do a lookup for a
> global, which would mean that 'del _sentinel' will cause it to fail?
> Or have I missed something here?

I believe you're correct.  If you really want to delete the _sentinel
reference though, you could do:

def sum(iterable, start=object()):
    if start is sum.func_defaults[0]:
        ...

Cheers,
Ian

[toc] | [prev] | [next] | [standalone]

#20759

On Fri, Feb 24, 2012 at 8:59 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
> def sum(iterable, start=_sentinel, _sentinel=_sentinel):

Is this a reason for Python to introduce a new syntax, such as:

def foo(blah, optional=del):
    if optional is del: print("No argument was provided")

Basically, 'del' is treated like a unique non-providable object, only
possible in an argument list and only if the argument was omitted. No
more proliferation of individual sentinels... what do you think?

(I picked "del" because it's an existing keyword. Fairly arbitrary
choice though.)

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#20760

On 23 February 2012 22:04, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 8:59 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
>> def sum(iterable, start=_sentinel, _sentinel=_sentinel):
>
> Is this a reason for Python to introduce a new syntax, such as:
>
> def foo(blah, optional=del):
>    if optional is del: print("No argument was provided")
>
> Basically, 'del' is treated like a unique non-providable object, only
> possible in an argument list and only if the argument was omitted. No
> more proliferation of individual sentinels... what do you think?

The problem with these proposals is to avoid the leakage of 'del'.
Here you could do:

def get_del(x=del):
    return x

And then you're in trouble again.

-- 
Arnaud

[toc] | [prev] | [next] | [standalone]

#20750

On 23 February 2012 21:19, Buck Golemon <buck@yelp.com> wrote:
> I feel like the design of sum() is inconsistent with other language
> features of python. Often python doesn't require a specific type, only
> that the type implement certain methods.
>
> Given a class that implements __add__ why should sum() not be able to
> operate on that class?

It can.  You need to pass a second argument which will be the start
value.  Try help(sum) for details.

-- 
Arnaud

[toc] | [prev] | [next] | [standalone]

#20751

On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <buck@yelp.com> wrote:
> I feel like the design of sum() is inconsistent with other language
> features of python. Often python doesn't require a specific type, only
> that the type implement certain methods.
>
> Given a class that implements __add__ why should sum() not be able to
> operate on that class?

The time machine strikes again! sum() already can. You just need to
specify an appropriate initial value (the empty list in this example)
for the accumulator :

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> sum([[1,2],[3,4]], [])
[1, 2, 3, 4]

Cheers,
Chris
--
http://rebertia.com

[toc] | [prev] | [next] | [standalone]

#20752

On Feb 23, 1:32 pm, Chris Rebert <c...@rebertia.com> wrote:
> On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <b...@yelp.com> wrote:
> > I feel like the design of sum() is inconsistent with other language
> > features of python. Often python doesn't require a specific type, only
> > that the type implement certain methods.
>
> > Given a class that implements __add__ why should sum() not be able to
> > operate on that class?
>
> The time machine strikes again! sum() already can. You just need to
> specify an appropriate initial value (the empty list in this example)
> for the accumulator :
>
> Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.>>> sum([[1,2],[3,4]], [])
>
> [1, 2, 3, 4]
>
> Cheers,
> Chris
> --http://rebertia.com

Thanks. I did not know that!

My proposal is still *slightly* superior in two ways:

1) It reduces the number of __add__ operations by one
2) The second argument isn't strictly necessary, if you don't mind
that the 'null sum' will produce zero.

def sum(values, base=0):
      values = iter(values)

      try:
          result = values.next()
      except StopIteration:
          return base

      for value in values:
          result += value

      return result

[toc] | [prev] | [next] | [standalone]

#20756

On Thu, Feb 23, 2012 at 2:38 PM, Buck Golemon <buck@yelp.com> wrote:
> My proposal is still *slightly* superior in two ways:
>
> 1) It reduces the number of __add__ operations by one
> 2) The second argument isn't strictly necessary, if you don't mind
> that the 'null sum' will produce zero.

It produces the wrong result, though:

>>> sum([3,4], base=12)
7

If I'm starting with 12 and summing 3 and 4, I expect to get 19.

Ideally the second argument should be ignored only if it isn't passed
in at all, and I don't know off-hand why the built-in sum doesn't do
this.  We really don't need to replace it, though.  If you want a
different sum behavior, just write your own.

def sum(iterable, *args):
    return reduce(operator.add, iterable, *args)

>>> sum([3,4])
7
>>> sum([3,4], 12)
19
>>> sum(['hello', 'world'])
'helloworld'

Cheers,
Ian

[toc] | [prev] | [next] | [standalone]

#20754

Chris Rebert, 23.02.2012 22:32:
> On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <buck@yelp.com> wrote:
>> I feel like the design of sum() is inconsistent with other language
>> features of python. Often python doesn't require a specific type, only
>> that the type implement certain methods.
>>
>> Given a class that implements __add__ why should sum() not be able to
>> operate on that class?
> 
> The time machine strikes again! sum() already can. You just need to
> specify an appropriate initial value (the empty list in this example)
> for the accumulator :
> 
> Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> sum([[1,2],[3,4]], [])
> [1, 2, 3, 4]

I know that you just meant this as an example, but it's worth mentioning in
this context that it's not exactly efficient to "sum up" lists this way
because there is a lot of copying involved. Each adding of two lists
creates a third one and copies all elements into it. So it eats a lot of
time and space.

Stefan

[toc] | [prev] | [next] | [standalone]

#20799

Stefan Behnel <stefan_ml@behnel.de> wrote:

> I know that you just meant this as an example, but it's worth
> mentioning in this context that it's not exactly efficient to "sum up"
> lists this way because there is a lot of copying involved. Each adding
> of two lists creates a third one and copies all elements into it. So
> it eats a lot of time and space.

If you search back through this group far enough you can find an 
alternative implementation of sum that I suggested which doesn't have the 
same performance problem with lists or strings and also improves the 
accuracy of the result with floats.

In effect what it does is instead of:
  (((((a + b) + c) + d) + e) + f)
it calculates the sum as:
  ((a + b) + (c + d)) + (e + f)

i.e. in as balanced a manner as it can given that it still has to work from 
left to right.

Of course that could still change the final result for some user defined 
types and never having converted my code to C I have no idea whether or not 
the performance for the intended case would be competitive with the builtin 
sum though I don't see why it wouldn't be.

-- 
Duncan Booth http://kupuguy.blogspot.com

[toc] | [prev] | [next] | [standalone]

#20788

Buck Golemon wrote:

> I feel like the design of sum() is inconsistent with other language
> features of python. Often python doesn't require a specific type, only
> that the type implement certain methods.
> 
> Given a class that implements __add__ why should sum() not be able to
> operate on that class?
> 
> We can fix this in a backward-compatible way, I believe.
> 
> Demonstration:
>     I'd expect these two error messages to be identical, but they are
> not.
> 
>      >>> class C(object): pass
>      >>> c = C()
>      >>> sum((c,c))
>     TypeError: unsupported operand type(s) for +: 'int' and 'C'
>     >>> c + c
>     TypeError: unsupported operand type(s) for +: 'C' and 'C'

You could explicitly provide a null object:

>>> class Null(object):
...     def __add__(self, other):
...             return other
...
>>> null = Null()
>>> class A(object):
...     def __init__(self, v):
...             self.v = v
...     def __add__(self, other):
...             return A("%s+%s" % (self, other))
...     def __str__(self):
...             return self.v
...     def __repr__(self):
...             return "A(%r)" % self. v
...
>>> sum(map(A, "abc"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'A'
>>> sum(map(A, "abc"), null)
A('a+b+c')

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web