Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #56571 > unrolled thread

Skipping decorators in unit tests

Back to article view | Back to comp.lang.python

Contents

  Skipping decorators in unit tests Gilles Lenfant <gilles.lenfant@gmail.com> - 2013-10-10 07:00 -0700
    Re: Skipping decorators in unit tests Cameron Simpson <cs@zip.com.au> - 2013-10-11 09:12 +1100
      Re: Skipping decorators in unit tests Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-11 02:55 +0000
        Re: Skipping decorators in unit tests Cameron Simpson <cs@zip.com.au> - 2013-10-11 14:13 +1100
          Re: Skipping decorators in unit tests Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-11 04:36 +0000
            Re: Skipping decorators in unit tests Terry Reedy <tjreedy@udel.edu> - 2013-10-11 04:23 -0400
        Re: Skipping decorators in unit tests Ben Finney <ben+python@benfinney.id.au> - 2013-10-11 14:42 +1100
        Re: Skipping decorators in unit tests Terry Reedy <tjreedy@udel.edu> - 2013-10-11 04:17 -0400
        Re: Skipping decorators in unit tests Terry Reedy <tjreedy@udel.edu> - 2013-10-11 04:25 -0400
        Re: Skipping decorators in unit tests Terry Reedy <tjreedy@udel.edu> - 2013-10-11 04:32 -0400
        Re: Skipping decorators in unit tests Ethan Furman <ethan@stoneleaf.us> - 2013-10-11 10:51 -0700
    Re: Skipping decorators in unit tests Ned Batchelder <ned@nedbatchelder.com> - 2013-10-10 19:44 -0400
    Re: Skipping decorators in unit tests Terry Reedy <tjreedy@udel.edu> - 2013-10-10 21:12 -0400
    Re: Skipping decorators in unit tests Gilles Lenfant <gilles.lenfant@gmail.com> - 2013-10-11 02:37 -0700
      Re: Skipping decorators in unit tests Cameron Simpson <cs@zip.com.au> - 2013-10-12 08:38 +1100

#56571 — Skipping decorators in unit tests

Hi,

(explaining the title) : my app has functions and methods (and maybe classes in the future) that are decorated by decorators provided by the standard library or 3rd party packages.

But I need to test "undecorated" functions and methods in my unit tests, preferably without adding "special stuffs" in my target tested modules.

Can someone point out good practices or dedicated tools that "remove temporarily" the decorations.

I pasted a small example of what I heed at http://pastebin.com/20CmHQ7Y

Many thanks in advance
-- 
Gilles Lenfant

[toc] | [next] | [standalone]

#56612

On 10Oct2013 07:00, Gilles Lenfant <gilles.lenfant@gmail.com> wrote:
> (explaining the title) : my app has functions and methods (and
> maybe classes in the future) that are decorated by decorators
> provided by the standard library or 3rd party packages.
> 
> But I need to test "undecorated" functions and methods in my unit tests, preferably without adding "special stuffs" in my target tested modules.
> 
> Can someone point out good practices or dedicated tools that "remove temporarily" the decorations.
> I pasted a small example of what I heed at http://pastebin.com/20CmHQ7Y

Speaking for myself, I would be include to recast this code:

  @absolutize
  def addition(a, b):
      return a + b

into:

  def _addition(a, b):
      return a + b

  addition = absolutize(_addition)

Then you can unit test both _addition() and addition().

And so forth.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

1st Law Economists: For every economist there exists an equal and opposite
                    economist.
2nd Law Economists: They're both always wrong!

[toc] | [prev] | [next] | [standalone]

#56637

On Fri, 11 Oct 2013 09:12:38 +1100, Cameron Simpson wrote:

> On 10Oct2013 07:00, Gilles Lenfant <gilles.lenfant@gmail.com> wrote:
>> (explaining the title) : my app has functions and methods (and maybe
>> classes in the future) that are decorated by decorators provided by the
>> standard library or 3rd party packages.
>> 
>> But I need to test "undecorated" functions and methods in my unit
>> tests, preferably without adding "special stuffs" in my target tested
>> modules.
>> 
>> Can someone point out good practices or dedicated tools that "remove
>> temporarily" the decorations. I pasted a small example of what I heed
>> at http://pastebin.com/20CmHQ7Y
> 
> Speaking for myself, I would be include to recast this code:
> 
>   @absolutize
>   def addition(a, b):
>       return a + b
> 
> into:
> 
>   def _addition(a, b):
>       return a + b
> 
>   addition = absolutize(_addition)
> 
> Then you can unit test both _addition() and addition().

*shudders*

Ew ew ew ew.


I would much rather do something like this:


def undecorate(f):
    """Return the undecorated inner function from function f."""
    return f.func_closure[0].cell_contents

def decorate(func):
    def inner(arg):
        return func(arg) + 1
    return inner

@decorate
def f(x):
    return 2*x



And in use:


py> f(100)
201
py> undecorate(f)(100)
200



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#56640

On 11Oct2013 02:55, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> On Fri, 11 Oct 2013 09:12:38 +1100, Cameron Simpson wrote:
> > Speaking for myself, I would be include to recast this code:
> > 
> >   @absolutize
> >   def addition(a, b):
> >       return a + b
> > 
> > into:
> > 
> >   def _addition(a, b):
> >       return a + b
> >   addition = absolutize(_addition)
> > 
> > Then you can unit test both _addition() and addition().
> 
> *shudders*
> Ew ew ew ew.

Care to provide some technical discourse here? Aside from losing the neat
and evocative @decorator syntax, the above is simple and overt.

> I would much rather do something like this:
> 
> def undecorate(f):
>     """Return the undecorated inner function from function f."""
>     return f.func_closure[0].cell_contents

Whereas this feels like black magic. Is this portable to any decorated
function? If so, I'd have hoped it was in the stdlib. If not: black magic.

> And in use:
> 
> py> f(100)
> 201
> py> undecorate(f)(100)
> 200

All lovely, provided you can convince me that undecorate() is robust.
(And if you can, I'll certainly be filing it away in my funcutils
module for later use.)

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

DRM doesn't inconvenience pirates ¿ indeed, over time it trains
law-abiding users to become pirates out of sheer frustration.
- Charles Stross

[toc] | [prev] | [next] | [standalone]

#56643

On Fri, 11 Oct 2013 14:13:19 +1100, Cameron Simpson wrote:

> On 11Oct2013 02:55, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Fri, 11 Oct 2013 09:12:38 +1100, Cameron Simpson wrote:
>> > Speaking for myself, I would be include to recast this code:
>> > 
>> >   @absolutize
>> >   def addition(a, b):
>> >       return a + b
>> > 
>> > into:
>> > 
>> >   def _addition(a, b):
>> >       return a + b
>> >   addition = absolutize(_addition)
>> > 
>> > Then you can unit test both _addition() and addition().
>> 
>> *shudders*
>> Ew ew ew ew.
> 
> Care to provide some technical discourse here? Aside from losing the
> neat and evocative @decorator syntax, the above is simple and overt.


What part of "Ew ew ew ew" was not technical enough for you? Would it 
help if I add a few extra "ew"s? 

*wink*

But seriously, I don't like doubling the number of names in the namespace 
just for the sake of white-box testing. That means you have to somehow 
document each and every one of the private functions that they aren't for 
using, just for testing.

For the avoidance of doubt, I understand you flagged them as private. But 
even for private use within the module, they're not supposed to be used. 
They are only for testing. So now you have to have a naming convention 
and/or documentation to ensure that they aren't used internally.

If there really is a good use-case for using both the decorated and 
undecorated version of the function (whether internally, or as part of 
the public API), then I'm completely with you. We don't have to use 
decorator syntax if we have good reason to keep the wrapped and unwrapped 
functions separate, just bind them to separate names.

I also like Terry Reedy's suggestion of having the decorator 
automatically add the unwrapped function to the wrapped function as an 
attribute:

def decorate(func):
    @functools.wraps(func)
    def inner(arg):
        blah blah
    inner._unwrapped = func  # make it public if you prefer
    return inner

which makes it all nice and clean and above board. (I seem to recall a 
proposal to have functools.wraps do this automatically...)


>> I would much rather do something like this:
>> 
>> def undecorate(f):
>>     """Return the undecorated inner function from function f.""" return
>>     f.func_closure[0].cell_contents
> 
> Whereas this feels like black magic. Is this portable to any decorated
> function? If so, I'd have hoped it was in the stdlib. If not: black
> magic.

Not every one-line function needs to be in the std lib :-)

To go into the std lib, it would need to be a tad more bullet-proof. For 
instance, it should have better error checking for the case where 
func_closure is None, rather than just raise the cryptic error message:

TypeError: 'NoneType' object is not subscriptable

If would also need to deal with arbitrary closures, where item 0 is not 
necessarily a function, or where there might be multiple functions, or no 
functions at all.

But for purely internal use within a test suite, we can afford to be a 
little more ad hoc and just deal with those cases when and if they occur. 
Since we're white-box testing, we presumably know which functions are 
decorated and which ones aren't (we can read the source code!), and will 
only call undecorate on those which are decorated.

>> And in use:
>> 
>> py> f(100)
>> 201
>> py> undecorate(f)(100)
>> 200
> 
> All lovely, provided you can convince me that undecorate() is robust.
> (And if you can, I'll certainly be filing it away in my funcutils module
> for later use.)

It needs error handling. It assumes that the closure only references a 
single object, namely the function being wrapped. In that sense, it's not 
ready for production as a public utility function. But as a private 
function for use only in testing, under controlled conditions, I think it 
is robust enough, it works in CPython 2.5 through 2.7 and IronPython 2.6. 
In Python 3.x, you have to change func_closure to __closure__, but 
otherwise it works in 3.2 and 3.3. Jython 2.5 seems to choke on the 
decorator syntax:


>>> @decorate
  File "<stdin>", line 1
    @decorate
            ^
SyntaxError: mismatched input '<EOF>' expecting CLASS


which surely is a bug in Jython. If I apply the decorator manually, it 
works.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#56660

On 10/11/2013 12:36 AM, Steven D'Aprano wrote:

> I also like Terry Reedy's suggestion of having the decorator
> automatically add the unwrapped function to the wrapped function as an
> attribute:
>
> def decorate(func):
>      @functools.wraps(func)
>      def inner(arg):
>          blah blah
>      inner._unwrapped = func  # make it public if you prefer
>      return inner
>
> which makes it all nice and clean and above board. (I seem to recall a
> proposal to have functools.wraps do this automatically...)

The explicit attribute can also be set by class rather than closure 
based decorators.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56641

Cameron Simpson <cs@zip.com.au> writes:

> On 11Oct2013 02:55, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> > def undecorate(f):
> >     """Return the undecorated inner function from function f."""
> >     return f.func_closure[0].cell_contents
>
> Whereas this feels like black magic. Is this portable to any decorated
> function? If so, I'd have hoped it was in the stdlib. If not: black
> magic.

What would you expect? The purpose of decorating functions is to do
magic to make it appear as though the original function isn't there any
more. Any technique to getting at that original function anyway is *of
course* going to look like black magic at the implementation level.

-- 
 \        “What if the Hokey Pokey IS what it's all about?” —anonymous |
  `\                                                                   |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#56658

On 10/10/2013 11:13 PM, Cameron Simpson wrote:
> On 11Oct2013 02:55, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

>> def undecorate(f):
>>      """Return the undecorated inner function from function f."""
>>      return f.func_closure[0].cell_contents
>
> Whereas this feels like black magic. Is this portable to any decorated
> function? If so, I'd have hoped it was in the stdlib. If not: black magic.
>
>> And in use:
>>
>> py> f(100)
>> 201
>> py> undecorate(f)(100)
>> 200
>
> All lovely, provided you can convince me that undecorate() is robust.
> (And if you can, I'll certainly be filing it away in my funcutils
> module for later use.)

It only works if the decorator returns a closure with the original 
function as the first member (of func_closure). Often true, but not at 
all a requirement.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56661

On 10/11/2013 4:17 AM, Terry Reedy wrote:
> On 10/10/2013 11:13 PM, Cameron Simpson wrote:
>> On 11Oct2013 02:55, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>
>>> def undecorate(f):
>>>      """Return the undecorated inner function from function f."""
>>>      return f.func_closure[0].cell_contents
>>
>> Whereas this feels like black magic. Is this portable to any decorated
>> function? If so, I'd have hoped it was in the stdlib. If not: black
>> magic.
>>
>>> And in use:
>>>
>>> py> f(100)
>>> 201
>>> py> undecorate(f)(100)
>>> 200
>>
>> All lovely, provided you can convince me that undecorate() is robust.
>> (And if you can, I'll certainly be filing it away in my funcutils
>> module for later use.)
>
> It only works if the decorator returns a closure with the original
> function as the first member (of func_closure). Often true, but not at
> all a requirement.
>


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56663

On 10/11/2013 4:17 AM, Terry Reedy wrote:
> On 10/10/2013 11:13 PM, Cameron Simpson wrote:
>> On 11Oct2013 02:55, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>
>>> def undecorate(f):
>>>      """Return the undecorated inner function from function f."""
>>>      return f.func_closure[0].cell_contents
>>
>> Whereas this feels like black magic. Is this portable to any decorated
>> function? If so, I'd have hoped it was in the stdlib. If not: black
>> magic.
>>
>>> And in use:
>>>
>>> py> f(100)
>>> 201
>>> py> undecorate(f)(100)
>>> 200
>>
>> All lovely, provided you can convince me that undecorate() is robust.
>> (And if you can, I'll certainly be filing it away in my funcutils
>> module for later use.)
>
> It only works if the decorator returns a closure with the original
> function as the first member (of func_closure). Often true, but not at
> all a requirement.

Another standard decorator method is to write a class with a .__call__ 
method and attach the original function to instances as an attribute. 
(Indeed, decorators were borrowed from class-happy Java ;-). But there 
is no standard as to what the function attribute of instances is called. 
The OP's request for accessing the function without modifying the tested 
code cannot be met in general. One must have access to the tested code.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56701

On 10/10/2013 08:13 PM, Cameron Simpson wrote:
> On 11Oct2013 02:55, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
>> On Fri, 11 Oct 2013 09:12:38 +1100, Cameron Simpson wrote:
>>> Speaking for myself, I would be include to recast this code:
>>>
>>>    @absolutize
>>>    def addition(a, b):
>>>        return a + b
>>>
>>> into:
>>>
>>>    def _addition(a, b):
>>>        return a + b
>>>    addition = absolutize(_addition)
>>>
>>> Then you can unit test both _addition() and addition().
>>
>> *shudders*
>> Ew ew ew ew.
>
> Care to provide some technical discourse here? Aside from losing the neat
> and evocative @decorator syntax, the above is simple and overt.

And completely dismisses the whole point of adding @decorator to the 
language:  easy to use, easy to see == folks will actually use it.

>> I would much rather do something like this:
>>
>> def undecorate(f):
>>      """Return the undecorated inner function from function f."""
>>      return f.func_closure[0].cell_contents
>
> Whereas this feels like black magic. Is this portable to any decorated
> function? If so, I'd have hoped it was in the stdlib. If not: black magic.

Probably black magic.  But you can go with the decorator.wrapped route; 
after all, you're testing your own stuff so you should have control of 
your own decorators (okay, you may have to adapt a few others ;) .

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#56617

On 10/10/13 6:12 PM, Cameron Simpson wrote:
> On 10Oct2013 07:00, Gilles Lenfant <gilles.lenfant@gmail.com> wrote:
>> (explaining the title) : my app has functions and methods (and
>> maybe classes in the future) that are decorated by decorators
>> provided by the standard library or 3rd party packages.
>>
>> But I need to test "undecorated" functions and methods in my unit tests, preferably without adding "special stuffs" in my target tested modules.
>>
>> Can someone point out good practices or dedicated tools that "remove temporarily" the decorations.
>> I pasted a small example of what I heed at http://pastebin.com/20CmHQ7Y
> Speaking for myself, I would be include to recast this code:
>
>    @absolutize
>    def addition(a, b):
>        return a + b
>
> into:
>
>    def _addition(a, b):
>        return a + b
>
>    addition = absolutize(_addition)
>
> Then you can unit test both _addition() and addition().
>
> And so forth.
>
> Cheers,

I have to admit I'm having a hard time understanding why you'd need to 
test the undecorated functions.  After all, the undecorated functions 
aren't available to anyone.  All that matters is how they behave with 
the decorators.

But my imagination is weak: do you mind explaining more about what the 
functions do, what the decorators do, and why you need to test the 
undecorated functions?  I'll learn something, and with more information, 
we might be able to find a better solution.

--Ned.

[toc] | [prev] | [next] | [standalone]

#56627

On 10/10/2013 10:00 AM, Gilles Lenfant wrote:

To add to the other two responses so far...

> (explaining the title) : my app has functions and methods (and maybe classes in the future) that are decorated by decorators provided by the standard library or 3rd party packages.
>
> But I need to test "undecorated" functions and methods in my unit tests, preferably without adding "special stuffs" in my target tested modules.

Let's assume that the decorator wraps the function in a way that the 
wrapper has a reference to the original function, so it does not disappear.

> Can someone point out good practices or dedicated tools that "remove temporarily" the decorations.

The easiest thing would be to have the decorator add the original 
function as an attribute .wrapped to the wrapper. Then test foo.wrapped. 
If you do not like this 'special stuff', then you would have to 
introspect the wrapper to access the wrapped function. How to do that 
depends on the wrapper.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56672

Cameron, Steven, Ben, Ned, Terry, Roy. Many thanks for this interesting discussion.

I ended up... mixing some solutions provided by your hints :

* Adding an "__original__" attribute to the wrapper func in the decorators of my own
* Playing with "func_closure" to test functions/methods provided by 3rd party tools

Cheers and thanks again for taking time to help me.

-- 
Gilles Lenfant

[toc] | [prev] | [next] | [standalone]

#56714

On 11Oct2013 02:37, Gilles Lenfant <gilles.lenfant@gmail.com> wrote:
> * Adding an "__original__" attribute to the wrapper func in the decorators of my own

Just one remark: Call this __original or _original (or even original).
The __x__ names are reserved for python operations (like __add__, supporting "+").

Cheers,
-- 
Cameron Simpson <cs@zip.com.au>

I very strongly suggest that you periodically place ice packs over the abused
areas.  - Steve Garnier

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web