Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109185 > unrolled thread

How can I debug silent failure - print no output

Back to article view | Back to comp.lang.python

Contents

  How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-27 19:41 -0700
    Re: How can I debug silent failure - print no output Michael Torrie <torriem@gmail.com> - 2016-05-27 21:06 -0600
      Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-27 21:02 -0700
        Re: How can I debug silent failure - print no output cs@zip.com.au - 2016-05-28 14:35 +1000
          Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-27 23:35 -0700
            Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-28 00:14 -0700
              Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-28 01:01 -0700
                Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-28 02:44 -0700
                  Re: How can I debug silent failure - print no output Sayth Renshaw <flebber.crue@gmail.com> - 2016-05-28 03:32 -0700
                    Re: How can I debug silent failure - print no output cs@zip.com.au - 2016-05-29 08:43 +1000
            Re: How can I debug silent failure - print no output Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-05-28 12:52 -0400
    Re: How can I debug silent failure - print no output Jason Friedman <jsf80238@gmail.com> - 2016-05-27 22:02 -0600

#109185 — How can I debug silent failure - print no output

Afternoon

I am looking for help with this excerpt of code. The issue is that it completes with no error and with other code in I can see it connects to postgres however it is not picking up or parsing file, also no error though so I am not sure of the exact problem.

I use prints to try and see what it is picking up from terminal however they also print nothing.

How can i find the error to troubleshoot.

This is my terminal and directory structure.

sayth@localhost pyXML]$ ls -a
.  ..  data  .git  racemeeting.py  README.md


sayth@localhost data]$ ls -a
.  ..  20160528RAND0.xml


I execute the file as such

[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
[sayth@localhost pyXML]$ 

As you can see no output or failure.


from pyquery import PyQuery as pq
import psycopg2
import argparse
import os
import glob


def GetArgs():
    '''parse XML from command line'''
    parser = argparse.ArgumentParser()

    parser.add_argument("path", nargs="+")
    parser.add_argument('-e', '--extension', default='',
                        help='File extension to filter by.')
    args = parser.parse_args()

    files = set()
    name_pattern = "*" + args.extension
    for path in args.path:
        files.update(glob.glob(os.path.join(path, name_pattern)))
    return files



    # Now walk the tree and insert data.
    for filename in sorted(GetArgs()):
        for meeting in pq(filename=filename):
            print(filename)
            print(meeting)
            meetdata = [meeting.get(attr) for attr in meetattrs]
            cur.execute("insert into meetings valueme in GetArgs():s (" +
                        ",".join(["%s"]*len(meetattrs)) + ")", meetdata)
            for race in meeting.findall("race"):
                race.set("meeting_id", meeting.get("id"))
                racedata = [race.get(attr) for attr in raceattrs]
                cur.execute("insert into races values (" +
                            ",".join(["%s"]*len(raceattrs)) + ")", racedata)
                for horse in race.findall("nomination"):
                    horse.set("race_id", race.get("id"))
                    horsedata = [horse.get(attr) for attr in horseattrs]
                    cur.execute("insert into horses values (" +
                                ",".join(["%s"]*len(horseattrs)) + ")", horsedata)


Thanks in advance

Sayth

[toc] | [next] | [standalone]

#109187

On 05/27/2016 08:41 PM, Sayth Renshaw wrote:
> This is my terminal and directory structure.

Add more print() calls. Offhand I'd say that pq(filename=filename) is
returning an empty list so that for loop is not doing anything.  Hence
your debugging print() calls never happen.

Add sanity print()'s earlier in your program, and make sure everything
you are iterating over is what you expect.

[toc] | [prev] | [next] | [standalone]

#109189

On Saturday, 28 May 2016 13:06:59 UTC+10, Michael Torrie  wrote:

> Add more print() calls. Offhand I'd say that pq(filename=filename) is
> returning an empty list so that for loop is not doing anything.  Hence
> your debugging print() calls never happen.
> 
> Add sanity print()'s earlier in your program, and make sure everything
> you are iterating over is what you expect.

Ok after printing a few things i have found an error.

def GetArgs():
    '''parse XML from command line'''
    parser = argparse.ArgumentParser()

    parser.add_argument("path", nargs="+")
    parser.add_argument('-e', '--extension', default='',
                        help='File extension to filter by.')
    args = parser.parse_args()

    files = set()
    name_pattern = "*" + args.extension
    for path in args.path:
        files.update(glob.glob(os.path.join(path, name_pattern)))

    print(files)
    return files

a = GetArgs()
print(a)

so printing the files or the call to the function returns set() not the actual files.

[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
set()
set()
set()


Sayth

[toc] | [prev] | [next] | [standalone]

#109192

On 27May2016 21:02, Sayth Renshaw <flebber.crue@gmail.com> wrote:
>On Saturday, 28 May 2016 13:06:59 UTC+10, Michael Torrie  wrote:
>> Add more print() calls. Offhand I'd say that pq(filename=filename) is
>> returning an empty list so that for loop is not doing anything.  Hence
>> your debugging print() calls never happen.
>>
>> Add sanity print()'s earlier in your program, and make sure everything
>> you are iterating over is what you expect.
>
>Ok after printing a few things i have found an error.
>
>def GetArgs():
>    '''parse XML from command line'''
>    parser = argparse.ArgumentParser()
>
>    parser.add_argument("path", nargs="+")
>    parser.add_argument('-e', '--extension', default='',
>                        help='File extension to filter by.')
>    args = parser.parse_args()
>
>    files = set()
>    name_pattern = "*" + args.extension
>    for path in args.path:
>        files.update(glob.glob(os.path.join(path, name_pattern)))
>
>    print(files)
>    return files
>
>a = GetArgs()
>print(a)
>
>so printing the files or the call to the function returns set() not the actual files.

Since you're constructing a set of filenames, this means it is probably 
returning the right kind of thing, but it is empty. That points to the glob not 
doing what you want or the for-loop not doing anything.

>[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
>set()
>set()
>set()

So...  Add more prints!

Specificly, print(args) right after it is set, and put a print() _inside_ the 
loop before the call to files.update, probably printing "path", eg print("path 
=", path).

Then see what you learn.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#109193

> >
> >Ok after printing a few things i have found an error.
> >
> >def GetArgs():
> >    '''parse XML from command line'''
> >    parser = argparse.ArgumentParser()
> >
> >    parser.add_argument("path", nargs="+")
> >    parser.add_argument('-e', '--extension', default='',
> >                        help='File extension to filter by.')
> >    args = parser.parse_args()
> >
> >    files = set()
> >    name_pattern = "*" + args.extension
> >    for path in args.path:
> >        files.update(glob.glob(os.path.join(path, name_pattern)))
> >
> >    print(files)
> >    return files
> >
> >a = GetArgs()
> >print(a)
> >
> >so printing the files or the call to the function returns set() not the actual files.
> 
> Since you're constructing a set of filenames, this means it is probably 
> returning the right kind of thing, but it is empty. That points to the glob not 
> doing what you want or the for-loop not doing anything.
> 
> >[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> >set()
> >set()
> >set()
> 
> So...  Add more prints!
> 
> Specificly, print(args) right after it is set, and put a print() _inside_ the 
> loop before the call to files.update, probably printing "path", eg print("path 
> =", path).
> 
> Then see what you learn.
> 
> Cheers,
> Cameron Simpson 

Having done extra prints 

name_pattern = "*" + args.extension
    for path in args.path:
        print(args.path)
        print(path)
        files.update(glob.glob(os.path.join(path, name_pattern)))

it is getting the path and file however I think it is keeping the directory so i am not getting files.

[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
['data/20160528RAND0.xml']
data/20160528RAND0.xml
set()
set()
['data/20160528RAND0.xml']
data/20160528RAND0.xml

Sayth

[toc] | [prev] | [next] | [standalone]

#109194

On Saturday, 28 May 2016 16:35:35 UTC+10, Sayth Renshaw  wrote:
> > >
> > >Ok after printing a few things i have found an error.
> > >
> > >def GetArgs():
> > >    '''parse XML from command line'''
> > >    parser = argparse.ArgumentParser()
> > >
> > >    parser.add_argument("path", nargs="+")
> > >    parser.add_argument('-e', '--extension', default='',
> > >                        help='File extension to filter by.')
> > >    args = parser.parse_args()
> > >
> > >    files = set()
> > >    name_pattern = "*" + args.extension
> > >    for path in args.path:
> > >        files.update(glob.glob(os.path.join(path, name_pattern)))
> > >
> > >    print(files)
> > >    return files
> > >
> > >a = GetArgs()
> > >print(a)
> > >
> > >so printing the files or the call to the function returns set() not the actual files.
> > 
> > Since you're constructing a set of filenames, this means it is probably 
> > returning the right kind of thing, but it is empty. That points to the glob not 
> > doing what you want or the for-loop not doing anything.
> > 
> > >[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> > >set()
> > >set()
> > >set()
> > 
> > So...  Add more prints!
> > 
> > Specificly, print(args) right after it is set, and put a print() _inside_ the 
> > loop before the call to files.update, probably printing "path", eg print("path 
> > =", path).
> > 
> > Then see what you learn.
> > 
> > Cheers,
> > Cameron Simpson 
> 
> Having done extra prints 
> 
> name_pattern = "*" + args.extension
>     for path in args.path:
>         print(args.path)
>         print(path)
>         files.update(glob.glob(os.path.join(path, name_pattern)))
> 
> it is getting the path and file however I think it is keeping the directory so i am not getting files.
> 
> [sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> ['data/20160528RAND0.xml']
> data/20160528RAND0.xml
> set()
> set()
> ['data/20160528RAND0.xml']
> data/20160528RAND0.xml
> 
> Sayth
Actually think I have found the cause and its really small but on way its called.
I was calling
python3 racemeeting.py data/*.xml
which gives the directory and file as the path 
['data/20160528RAND0.xml']

But with arguments separated by a space I actually receive what i thought I would get a path and extension such as

sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
Namespace(extension='', path=['data/', '*.xml'])
Traceback (most recent call last):
  File "racemeeting.py", line 35, in <module>

Sayth

[toc] | [prev] | [next] | [standalone]

#109196

So how do i get argparse to read the file arguments correctly?

Looking at the namespace it all gets pushed into path and extension remains empty.

[sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
Namespace(extension='', path=['data/', '*.xml'])

This is the section I am running

parser = argparse.ArgumentParser()
parser.add_argument("path", nargs="+")
parser.add_argument('-e', '--extension', default='',
                    help='File extension to filter by.')

args = parser.parse_args()
name_pattern = "*" + args.extension
print(args)

Sayth

[toc] | [prev] | [next] | [standalone]

#109198

On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
> So how do i get argparse to read the file arguments correctly?
> 
> Looking at the namespace it all gets pushed into path and extension remains empty.
> 
> [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
> Namespace(extension='', path=['data/', '*.xml'])
> 
> This is the section I am running
> 
> parser = argparse.ArgumentParser()
> parser.add_argument("path", nargs="+")
> parser.add_argument('-e', '--extension', default='',
>                     help='File extension to filter by.')
> 
> args = parser.parse_args()
> name_pattern = "*" + args.extension
> print(args)
> 
> Sayth

Ah if only i used argparse properly

python racemeeting.py data/ -e *.xml

Sayth

[toc] | [prev] | [next] | [standalone]

#109200

On Saturday, 28 May 2016 19:44:53 UTC+10, Sayth Renshaw  wrote:
> On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
> > So how do i get argparse to read the file arguments correctly?
> > 
> > Looking at the namespace it all gets pushed into path and extension remains empty.
> > 
> > [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
> > Namespace(extension='', path=['data/', '*.xml'])
> > 
> > This is the section I am running
> > 
> > parser = argparse.ArgumentParser()
> > parser.add_argument("path", nargs="+")
> > parser.add_argument('-e', '--extension', default='',
> >                     help='File extension to filter by.')
> > 
> > args = parser.parse_args()
> > name_pattern = "*" + args.extension
> > print(args)
> > 
> > Sayth
> 
> Ah if only i used argparse properly
> 
> python racemeeting.py data/ -e *.xml
> 
> Sayth

Which means I can rewrite it like this.

parser = argparse.ArgumentParser()
parser.add_argument("path", type=str, nargs="+")
parser.add_argument('-e', '--extension', default='',
                    help='File extension to filter by.')

args = parser.parse_args()
name_pattern = "*" + args.extension
my_dir = args.path[0]

for dir_path, subdir_list, file_list in os.walk(my_dir):
    for name_pattern in file_list:
        full_path = os.path.join(dir_path, name_pattern)

Cheers

Sayth

[toc] | [prev] | [next] | [standalone]

#109222

On 28May2016 03:32, Sayth Renshaw <flebber.crue@gmail.com> wrote:
>On Saturday, 28 May 2016 19:44:53 UTC+10, Sayth Renshaw  wrote:
>> On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
>> > So how do i get argparse to read the file arguments correctly?
>> >
>> > Looking at the namespace it all gets pushed into path and extension remains empty.
>> >
>> > [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
>> > Namespace(extension='', path=['data/', '*.xml'])
>> > This is the section I am running
>> >
>> > parser = argparse.ArgumentParser()
>> > parser.add_argument("path", nargs="+")
>> > parser.add_argument('-e', '--extension', default='',
>> >                     help='File extension to filter by.')
>> >
>> > args = parser.parse_args()
>> > name_pattern = "*" + args.extension
>> > print(args)
>>
>> Ah if only i used argparse properly
>>
>> python racemeeting.py data/ -e *.xml

There are a couple of things here.

First is that a normal UNIX command puts all the options first, so you should 
be desiging your command line to run like this:

  python racemeeting.py -e *.xml data

or perhaps:

  python racemeeting.py -d data -e *.xml

It is traditional to stop parsing options such as -e when you reach the first 
non-option, because that lets one put whatever is necessary safely _after_ the 
options without fear that one of the arguments will resemble an option. For 
example, suppoing you have a file with the name "-e" and said:

  somecommand -f foo dir *

intending to use all the local filenames after "dir". In your current scheme 
(accepting options after "dir") a "-e" appearing later would be misinterpreted.

The second is to be aware that the shell expands globs _before_ invoking the 
command. This is extremely useful because it means that (a) commands usually 
don't need to do their own glob expansion and (b) all commands end up using the 
same glob syntax because it is common to the shell, not a command-specific 
special syntax. Which makes everything easier to use.

The upshot of that is that you should _either_ be quoting "*.xml" on your 
command line to prevent expansion, _or_ you should not be bothering with he 
asterisk, instead passing the argument ".xml" or even just "xml".

As an experiment, make two dummy XML files in your current directory (where 
your script is):

  >>dummy1.xml
  >>dummy2.xml

and run your script passing *.xml as you currently do, and see what your print 
statements say. This should make the effect obvious.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#109215

On Fri, 27 May 2016 23:35:21 -0700 (PDT), Sayth Renshaw
<flebber.crue@gmail.com> declaimed the following:

>[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml

	You're on a Linux type OS, no?

	The SHELL is EXPANDING data/*.xml with the result that you are getting
a list of actual file names
>['data/20160528RAND0.xml']
>data/20160528RAND0.xml

	One file in the list. And your code is then joining it with "*" to
create:
	data/20160528RAND0.xml/*
which will not be found by the subsequent glob.glob
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#109188

>
> def GetArgs():
>     '''parse XML from command line'''
>     parser = argparse.ArgumentParser()
>
>     parser.add_argument("path", nargs="+")
>     parser.add_argument('-e', '--extension', default='',
>                         help='File extension to filter by.')
>     args = parser.parse_args()
>
>     files = set()
>     name_pattern = "*" + args.extension
>     for path in args.path:
>         files.update(glob.glob(os.path.join(path, name_pattern)))
>     return files
>
>
>
>     # Now walk the tree and insert data.
>     for filename in sorted(GetArgs()):
>         for meeting in pq(filename=filename):
>             print(filename)
>             print(meeting)
>             meetdata = [meeting.get(attr) for attr in meetattrs]
>             cur.execute("insert into meetings valueme in GetArgs():s (" +
>                         ",".join(["%s"]*len(meetattrs)) + ")", meetdata)
>             for race in meeting.findall("race"):
>                 race.set("meeting_id", meeting.get("id"))
>                 racedata = [race.get(attr) for attr in raceattrs]
>                 cur.execute("insert into races values (" +
>                             ",".join(["%s"]*len(raceattrs)) + ")",
> racedata)
>                 for horse in race.findall("nomination"):
>                     horse.set("race_id", race.get("id"))
>                     horsedata = [horse.get(attr) for attr in horseattrs]
>                     cur.execute("insert into horses values (" +
>                                 ",".join(["%s"]*len(horseattrs)) + ")",
> horsedata)
>
> If your actual indentation matches what I see in your post, is your

for filename in sorted(GetArgs())

line within the definition of GetArgs?

If yes, it will not be executed.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web