Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <57693d65-e683-4972-ac8d-97b2feace3bb@googlegroups.com>
References: <6203299c-f9b2-4169-9d68-4c92e0f7b32f@googlegroups.com> <mailman.18.1427977502.12925.python-list@python.org> <57693d65-e683-4972-ac8d-97b2feace3bb@googlegroups.com>
Date: Fri, 3 Apr 2015 00:06:31 +1100
Subject: Re: New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
From: Chris Angelico <rosuav@gmail.com>
Cc: "python-list@python.org" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.21.1427979994.12925.python-list@python.org>
Lines: 69
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:88447

On Thu, Apr 2, 2015 at 11:46 PM, Saran A <ahlusar.ahluwalia@gmail.com> wrote:
> @ChrisA - this is a smaller function that will take the most updated file. My intention is the following:
>
> * Monitor a folder for files that are dropped throughout the day
>
> * When a file is dropped in the folder the program should scan the file
>
> o IF all the contents in the file have the same length (let's assume line length)
>
> o THEN the file should be moved to a "success" folder and a text file written indicating the total number of records/lines/words processed
>
> o IF the file is empty OR the contents are not all of the same length
>
> o THEN the file should be moved to a "failure" folder and a text file written indicating the cause for failure (for example: Empty file or line 100 was not the same length as the rest).
>

Sounds like a perfect job for inotify, then. Your function will be
called whenever there's a new file.

> Here is the code I have written:
>
> def initialize_logger(output_dir):
>     logger = logging.getLogger()
>     ...
>     def file_len(filename
>         with open(filename) as f:
>             for i, l in enumerate(f):
>                 pass
>             return i + 1

These functions are all getting defined inside your
initialize_logger() function. I suspect you want them to be flush left
instead.

>     def copyFile(src, dest):
>         try:
>             shutil.copy(src, dest)
>         # eg. src and dest are the same file
>         except shutil.Error as e:
>             print('Error: %s' % e)
>         # eg. source or destination doesn't exist
>         except IOError as e:
>             print('Error: %s' % e.strerror)

Recommendation: Skip the try/except, and just let exceptions bubble
up. Don't just print out messages and keep going.

> def move_to_failure_folder_and_return_error_file():
>     os.mkdir('Failure')
>     copyFile(filename, 'Failure')
>     initialize_logger('rootdir/Failure')
>     logging.error("Either this file is empty or the lines")

This doesn't move the file, it copies it. Is that your intention?

Moving a file is pretty easy. Just use os.rename().

> if __name__ == '__main__':
>    import sys
>    validate_files(sys.argv[1:])

I've no idea what validate_files() does, as you haven't included that.

I think you could code this fairly efficiently as a simple callback
off pyinotify, or if you're not on Linux, with one of the equivalent
services. What you're doing here (watching for files, looking inside
them, and moving them when done) is pretty common around the world.

ChrisA