Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #49315 > unrolled thread

Why is the argparse module so inflexible?

Back to article view | Back to comp.lang.python

Contents

  Why is the argparse module so inflexible? Andrew Berg <robotsondrugs@gmail.com> - 2013-06-27 07:54 -0500
    Re: Why is the argparse module so inflexible? Roy Smith <roy@panix.com> - 2013-06-27 09:08 -0400
      Re: Why is the argparse module so inflexible? Andrew Berg <robotsondrugs@gmail.com> - 2013-06-27 08:49 -0500
      Re: Why is the argparse module so inflexible? Dave Angel <davea@davea.name> - 2013-06-27 12:02 -0400
        Re: Why is the argparse module so inflexible? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-27 22:49 +0000
          Re: Why is the argparse module so inflexible? Cameron Simpson <cs@zip.com.au> - 2013-06-28 09:34 +1000
          Re: Why is the argparse module so inflexible? Ethan Furman <ethan@stoneleaf.us> - 2013-06-28 18:36 -0700
            Re: Why is the argparse module so inflexible? rusi <rustompmody@gmail.com> - 2013-06-28 21:12 -0700
              Re: Why is the argparse module so inflexible? Terry Reedy <tjreedy@udel.edu> - 2013-06-29 00:37 -0400
                Re: Why is the argparse module so inflexible? Roy Smith <roy@panix.com> - 2013-06-29 10:12 -0400
                  Re: Why is the argparse module so inflexible? Andrew Berg <robotsondrugs@gmail.com> - 2013-06-29 09:17 -0500
            Re: Why is the argparse module so inflexible? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-06-29 05:28 +0000
              Re: Why is the argparse module so inflexible? Marcin Szamotulski <mszamot@gmail.com> - 2013-06-29 13:38 +0100
              Re: Why is the argparse module so inflexible? MRAB <python@mrabarnett.plus.com> - 2013-06-29 16:58 +0100
              Re: Why is the argparse module so inflexible? Ethan Furman <ethan@stoneleaf.us> - 2013-06-29 12:17 -0700
          Re: Why is the argparse module so inflexible? Modulok <modulok@gmail.com> - 2013-06-28 19:39 -0600
          Re: Why is the argparse module so inflexible? Isaac To <isaac.to@gmail.com> - 2013-06-29 12:37 +0800
      Re: Why is the argparse module so inflexible? Robert Kern <robert.kern@gmail.com> - 2013-06-27 20:47 +0100

#49315 — Why is the argparse module so inflexible?

I've begun writing a program with an interactive prompt, and it needs to parse input from the user. I thought the argparse module would be
great for this, but unfortunately it insists on calling sys.exit() at any sign of trouble instead of letting its ArgumentError exception
propagate so that I can handle it. Overriding ArgumentParser.error doesn't really help since methods like parse_known_args just send a
message to be relayed to the user as an argument after swallowing ArgumentError (which does have useful information in its attributes). If I
wanted to figure out what actually caused the exception to be raised, I'd have to parse the message, which is ugly at best. I understand
that most people do want argparse to just display a message and terminate if the arguments supplied aren't useful, but there's a lot of
potential in the module that is crippled outside the main use case. I have to wonder why a module that is meant to be imported would ever
call sys.exit(), even if that is what the caller would normally do if presented with an exception.
-- 
CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1

[toc] | [next] | [standalone]

#49318

In article <mailman.3924.1372337705.3114.python-list@python.org>,
 Andrew Berg <robotsondrugs@gmail.com> wrote:

> I've begun writing a program with an interactive prompt, and it needs to 
> parse input from the user. I thought the argparse module would be
> great for this, but unfortunately it insists on calling sys.exit() at any 
> sign of trouble instead of letting its ArgumentError exception
> propagate so that I can handle it.
>
> [...] there's a lot of potential in the module that is crippled 
> outside the main use case.

Having used (and written) a number of different ways of dealing with CLI 
parsing (in several languages), I can tell you that argparse is pretty 
cool.  CLI parsing is amazingly complicated.  Argparse turns that into a 
job which is only moderately complicated and slightly annoying.  
Compared to the alternatives, that's a big win.

Can you give us a concrete example of what you're trying to do?

You might look into "type=".  It's normally used for things like 
"type=int" or "type=float", but it could give it any user-defined 
function as a type and this essentially becomes a hook to insert your 
own code into the middle of the processing.  Sometimes that can be 
warped into doing all sorts of useful things.

[toc] | [prev] | [next] | [standalone]

#49326

On 2013.06.27 08:08, Roy Smith wrote:
> Can you give us a concrete example of what you're trying to do?
The actual code I've written so far isn't easily condensed into a short simple snippet.
I'm trying to use argparse to handle all the little details of parsing and verifying arguments in the precmd hook for a cmd.Cmd child class.
argparse's help system is more sophisticated than cmd's help and does all the work of verifying arguments.
The problem I keep running into is that I can't handle any bad input very well. I would have to override every method that catches
ArgumentError in order to get a useful exception that I would then handle.
If I input something that begins with '-' that isn't recognized, parse_args doesn't even raise the exception; it just quits. In this case,
the message gets mangled if error is overridden, and I don't know why.

> You might look into "type=".  It's normally used for things like 
> "type=int" or "type=float", but it could give it any user-defined 
> function as a type and this essentially becomes a hook to insert your 
> own code into the middle of the processing.  Sometimes that can be 
> warped into doing all sorts of useful things.
I don't think that would solve my problem, but it would probably be quite useful.
-- 
CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1

[toc] | [prev] | [next] | [standalone]

#49334

On 06/27/2013 09:49 AM, Andrew Berg wrote:
> On 2013.06.27 08:08, Roy Smith wrote:
>> Can you give us a concrete example of what you're trying to do?
> The actual code I've written so far isn't easily condensed into a short simple snippet.
> I'm trying to use argparse to handle all the little details of parsing and verifying arguments in the precmd hook for a cmd.Cmd child class.
> argparse's help system is more sophisticated than cmd's help and does all the work of verifying arguments.
> The problem I keep running into is that I can't handle any bad input very well. I would have to override every method that catches
> ArgumentError in order to get a useful exception that I would then handle.
> If I input something that begins with '-' that isn't recognized, parse_args doesn't even raise the exception; it just quits.

No, it raises the SystemExit exception.  if you don't catch it, then the 
program exits.  Perhaps it's not clear to you, but sys.exit() just 
raises the SystemExit exception, as  Joshua pointed out.

> In this case,
> the message gets mangled if error is overridden, and I don't know why.
>

I can't help there.


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#49349

On Thu, 27 Jun 2013 12:02:22 -0400, Dave Angel wrote:

> On 06/27/2013 09:49 AM, Andrew Berg wrote:
>> On 2013.06.27 08:08, Roy Smith wrote:
>>> Can you give us a concrete example of what you're trying to do?
>> The actual code I've written so far isn't easily condensed into a short
>> simple snippet. I'm trying to use argparse to handle all the little
>> details of parsing and verifying arguments in the precmd hook for a
>> cmd.Cmd child class. argparse's help system is more sophisticated than
>> cmd's help and does all the work of verifying arguments. The problem I
>> keep running into is that I can't handle any bad input very well. I
>> would have to override every method that catches ArgumentError in order
>> to get a useful exception that I would then handle. If I input
>> something that begins with '-' that isn't recognized, parse_args
>> doesn't even raise the exception; it just quits.
> 
> No, it raises the SystemExit exception.  if you don't catch it, then the
> program exits.  Perhaps it's not clear to you, but sys.exit() just
> raises the SystemExit exception, as  Joshua pointed out.

[rant]
I think it is lousy design for a framework like argparse to raise a 
custom ArgumentError in one part of the code, only to catch it elsewhere 
and call sys.exit. At the very least, that ought to be a config option, 
and off by default.

Libraries should not call sys.exit, or raise SystemExit. Whether to quit 
or not is not the library's decision to make, that decision belongs to 
the application layer. Yes, the application could always catch 
SystemExit, but it shouldn't have to.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#49352

On 27Jun2013 22:49, Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
| [rant]
| I think it is lousy design for a framework like argparse to raise a 
| custom ArgumentError in one part of the code, only to catch it elsewhere 
| and call sys.exit. At the very least, that ought to be a config option, 
| and off by default.
| 
| Libraries should not call sys.exit, or raise SystemExit. Whether to quit 
| or not is not the library's decision to make, that decision belongs to 
| the application layer. Yes, the application could always catch 
| SystemExit, but it shouldn't have to.

+1
-- 
Cameron Simpson <cs@zip.com.au>

There is more stupidity than hydrogen in the universe, and it has a
longer shelf life.      - Frank Zappa

[toc] | [prev] | [next] | [standalone]

#49396

On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>
> [rant]
> I think it is lousy design for a framework like argparse to raise a
> custom ArgumentError in one part of the code, only to catch it elsewhere
> and call sys.exit. At the very least, that ought to be a config option,
> and off by default.
>
> Libraries should not call sys.exit, or raise SystemExit. Whether to quit
> or not is not the library's decision to make, that decision belongs to
> the application layer. Yes, the application could always catch
> SystemExit, but it shouldn't have to.

So a library that is explicitly designed to make command-line scripts easier and friendlier should quit with a traceback?

Really?

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#49402

On Saturday, June 29, 2013 7:06:37 AM UTC+5:30, Ethan Furman wrote:
> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
> > [rant]
> > I think it is lousy design for a framework like argparse to raise a
> > custom ArgumentError in one part of the code, only to catch it elsewhere
> > and call sys.exit. At the very least, that ought to be a config option,
> > and off by default.
> >
> > Libraries should not call sys.exit, or raise SystemExit. Whether to quit
> > or not is not the library's decision to make, that decision belongs to
> > the application layer. Yes, the application could always catch
> > SystemExit, but it shouldn't have to.
> 
> 
> So a library that is explicitly designed to make command-line scripts easier 
> and friendlier should quit with a traceback?
> 
> Really?

So a library that behaves like an app is OK?

Really?

I would have thought that with the sophistication of python's exception mechanism in place there would be corresponding peps in place proscribing such behavior.  In the same way that if I submitted my super-duper library for inclusion into python with 7 spaces indentation, some kind dev would reply with

http://www.python.org/dev/peps/pep-0008/#indentation

[toc] | [prev] | [next] | [standalone]

#49404

On 6/29/2013 12:12 AM, rusi wrote:
> On Saturday, June 29, 2013 7:06:37 AM UTC+5:30, Ethan Furman wrote:
>> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>>> [rant]
>>> I think it is lousy design for a framework like argparse to raise a
>>> custom ArgumentError in one part of the code, only to catch it elsewhere
>>> and call sys.exit. At the very least, that ought to be a config option,
>>> and off by default.
>>>
>>> Libraries should not call sys.exit, or raise SystemExit. Whether to quit
>>> or not is not the library's decision to make, that decision belongs to
>>> the application layer. Yes, the application could always catch
>>> SystemExit, but it shouldn't have to.
>>
>>
>> So a library that is explicitly designed to make command-line scripts easier
>> and friendlier should quit with a traceback?
>>
>> Really?
>
> So a library that behaves like an app is OK?

No, Steven is right as a general rule (do not raise SystemExit), but 
argparse was considered an exception because its purpose is to turn a 
module into an app. With the responses I have seen here, I agree that 
this is a bit short-sighted, as inflexible behavior. The tracker issue 
could use more review and comment.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#49422

In article <mailman.3980.1372480662.3114.python-list@python.org>,
 Terry Reedy <tjreedy@udel.edu> wrote:

> > So a library that behaves like an app is OK?
> 
> No, Steven is right as a general rule (do not raise SystemExit), but 
> argparse was considered an exception because its purpose is to turn a 
> module into an app. With the responses I have seen here, I agree that 
> this is a bit short-sighted, as inflexible behavior. The tracker issue 
> could use more review and comment.

What is the tracker issue number or url?

[toc] | [prev] | [next] | [standalone]

#49423

On 2013.06.29 09:12, Roy Smith wrote:
> What is the tracker issue number or url?
http://bugs.python.org/issue9938
-- 
CPython 3.3.2 | Windows NT 6.2.9200 / FreeBSD 9.1

[toc] | [prev] | [next] | [standalone]

#49405

On Fri, 28 Jun 2013 18:36:37 -0700, Ethan Furman wrote:

> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>>
>> [rant]
>> I think it is lousy design for a framework like argparse to raise a
>> custom ArgumentError in one part of the code, only to catch it
>> elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
>> CONFIG OPTION, and OFF BY DEFAULT.

[emphasis added]

>> Libraries should not call sys.exit, or raise SystemExit. Whether to
>> quit or not is not the library's decision to make, that decision
>> belongs to the application layer. Yes, the application could always
>> catch SystemExit, but it shouldn't have to.
> 
> So a library that is explicitly designed to make command-line scripts
> easier and friendlier should quit with a traceback?
> 
> Really?

Yes, really.

Tracebacks are not that unfriendly, generally speaking. In my experience, 
the average non-technical person is no more confused and distressed by a 
traceback extending over thirty lines than they are by a one line error 
message. As the developer, I should see the tracebacks by default[1]. If 
I want to suppress or simplify them, then I should take explicit steps to 
do so, either by catching the exception and calling sys.exit myself, or 
at least by setting a runtime config option to the library.

This also allows me to enable debugging in my app by showing tracebacks, 
or disable it by hiding them. That should be my decision, not the 
library. If the library catches exceptions then exits, throwing away 
potentially useful information, that makes it difficult to debug anything 
relying on the library.

I'm willing to concede that, just maybe, something like argparse could 
default to "catch exceptions and exit" ON rather than OFF. 


[1] There's something in the Zen of Python about that...


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#49416

On 05:28 Sat 29 Jun     , Steven D'Aprano wrote:
> On Fri, 28 Jun 2013 18:36:37 -0700, Ethan Furman wrote:
> 
> > On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
> >>
> >> [rant]
> >> I think it is lousy design for a framework like argparse to raise a
> >> custom ArgumentError in one part of the code, only to catch it
> >> elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
> >> CONFIG OPTION, and OFF BY DEFAULT.
> 
> [emphasis added]
> 
> >> Libraries should not call sys.exit, or raise SystemExit. Whether to
> >> quit or not is not the library's decision to make, that decision
> >> belongs to the application layer. Yes, the application could always
> >> catch SystemExit, but it shouldn't have to.
> > 
> > So a library that is explicitly designed to make command-line scripts
> > easier and friendlier should quit with a traceback?
> > 
> > Really?
> 
> Yes, really.
> 
> Tracebacks are not that unfriendly, generally speaking. In my experience, 
> the average non-technical person is no more confused and distressed by a 
> traceback extending over thirty lines than they are by a one line error 
> message. As the developer, I should see the tracebacks by default[1]. If 
> I want to suppress or simplify them, then I should take explicit steps to 
> do so, either by catching the exception and calling sys.exit myself, or 
> at least by setting a runtime config option to the library.
> 
> This also allows me to enable debugging in my app by showing tracebacks, 
> or disable it by hiding them. That should be my decision, not the 
> library. If the library catches exceptions then exits, throwing away 
> potentially useful information, that makes it difficult to debug anything 
> relying on the library.
> 
> I'm willing to concede that, just maybe, something like argparse could 
> default to "catch exceptions and exit" ON rather than OFF. 
> 
> 
> [1] There's something in the Zen of Python about that...
> 
> 
> -- 
> Steven
> -- 
>

Although I got confused at the first time I was using argparse (or
optparse which is now obsolte and also has this feature), I see the
value when you write scripts.  It is mostly annoying when playing with
it in a console, but there is a very easy (but partial) fix for that:
just subclass argparse.ArgumentParser:

import sys
class ArgumentParser(argparse.ArgumentParser):
    def  exit(self, status=0, message=None):
        if message:
            self._print_message(message, sys.stderr)

now the parser will not exit, though there is no ease fix to get the
traceback: self.exit() is called in various places sometimes inside
a try block.

Best regards,
Marcin http://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#49429

On 29/06/2013 06:28, Steven D'Aprano wrote:
> On Fri, 28 Jun 2013 18:36:37 -0700, Ethan Furman wrote:
>
>> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>>>
>>> [rant]
>>> I think it is lousy design for a framework like argparse to raise a
>>> custom ArgumentError in one part of the code, only to catch it
>>> elsewhere and call sys.exit. At the very least, that OUGHT TO BE A
>>> CONFIG OPTION, and OFF BY DEFAULT.
>
> [emphasis added]
>
>>> Libraries should not call sys.exit, or raise SystemExit. Whether to
>>> quit or not is not the library's decision to make, that decision
>>> belongs to the application layer. Yes, the application could always
>>> catch SystemExit, but it shouldn't have to.
>>
>> So a library that is explicitly designed to make command-line scripts
>> easier and friendlier should quit with a traceback?
>>
>> Really?
>
> Yes, really.
>
[snip]
+1

It's the job of argparse to parse the arguments. What should happen if
they're invalid is for its caller to decide.

[toc] | [prev] | [next] | [standalone]

#49458

On 06/28/2013 10:28 PM, Steven D'Aprano wrote:
>
> I'm willing to concede that, just maybe, something like argparse could
> default to "catch exceptions and exit" ON rather than OFF.

On this we can agree.  :)

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#49397

[Multipart message — attachments visible in raw view] — view raw

Have you looked into docopt?

-Modulok-


On Fri, Jun 28, 2013 at 7:36 PM, Ethan Furman <ethan@stoneleaf.us> wrote:

> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>
>>
>> [rant]
>> I think it is lousy design for a framework like argparse to raise a
>> custom ArgumentError in one part of the code, only to catch it elsewhere
>> and call sys.exit. At the very least, that ought to be a config option,
>> and off by default.
>>
>> Libraries should not call sys.exit, or raise SystemExit. Whether to quit
>> or not is not the library's decision to make, that decision belongs to
>> the application layer. Yes, the application could always catch
>> SystemExit, but it shouldn't have to.
>>
>
> So a library that is explicitly designed to make command-line scripts
> easier and friendlier should quit with a traceback?
>
> Really?
>
> --
> ~Ethan~
> --
> http://mail.python.org/**mailman/listinfo/python-list<http://mail.python.org/mailman/listinfo/python-list>
>

[toc] | [prev] | [next] | [standalone]

#49403

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jun 29, 2013 at 9:36 AM, Ethan Furman <ethan@stoneleaf.us> wrote:

> On 06/27/2013 03:49 PM, Steven D'Aprano wrote:
>
>>
>> Libraries should not call sys.exit, or raise SystemExit. Whether to quit
>> or not is not the library's decision to make, that decision belongs to
>> the application layer. Yes, the application could always catch
>> SystemExit, but it shouldn't have to.
>>
>
> So a library that is explicitly designed to make command-line scripts
> easier and friendlier should quit with a traceback?
>
> Really?
>

Perhaps put the functionality "handling of the exception of library to
sys.exit with a message" into a method so that the user can override it
(e.g., so that it just throw the same exception to the caller of the
library)?

[toc] | [prev] | [next] | [standalone]

#49345

On 2013-06-27 17:02, Dave Angel wrote:
> On 06/27/2013 09:49 AM, Andrew Berg wrote:
>> On 2013.06.27 08:08, Roy Smith wrote:
>>> Can you give us a concrete example of what you're trying to do?
>> The actual code I've written so far isn't easily condensed into a short simple
>> snippet.
>> I'm trying to use argparse to handle all the little details of parsing and
>> verifying arguments in the precmd hook for a cmd.Cmd child class.
>> argparse's help system is more sophisticated than cmd's help and does all the
>> work of verifying arguments.
>> The problem I keep running into is that I can't handle any bad input very
>> well. I would have to override every method that catches
>> ArgumentError in order to get a useful exception that I would then handle.
>> If I input something that begins with '-' that isn't recognized, parse_args
>> doesn't even raise the exception; it just quits.
>
> No, it raises the SystemExit exception.  if you don't catch it, then the program
> exits.  Perhaps it's not clear to you, but sys.exit() just raises the SystemExit
> exception, as  Joshua pointed out.

Internally, the parser raises ArgumentError which has some useful pieces of 
information, specifically, the name of the argument that failed to parse. 
Unfortunately, it catches that error in parse_known_args(), then formats that 
information into a message string to pass to the error() method, which by 
default raises SystemExit with just that message string. It is somewhat 
difficult to override the parse_known_args() to not lose that information from 
the ArgumentError because you will have to copy-paste the rest of the code in there.

So yes, you can override the error() method or catch SystemExit if all you want 
is the formatted message string, but I have to agree that there is a missed 
opportunity to make argparse more widely usable for other command-line like 
parsing tasks[1].

[1] As an existence proof, I offer you one that I wrote for handling IPython's 
%magic commands. I just needed the formatted string, so I could just get away 
with overriding error().

https://github.com/ipython/ipython/blob/master/IPython/core/magic_arguments.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web