Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #9561 > unrolled thread

Looking for general advice on complex program

Back to article view | Back to comp.lang.python

Contents

  Looking for general advice on complex program Josh English <Joshua.R.English@gmail.com> - 2011-07-15 12:47 -0700
    Re: Looking for general advice on complex program Billy Mays <81282ed9a88799d21e77957df2d84bd6514d9af6@myhashismyemail.com> - 2011-07-15 16:03 -0400
      Re: Looking for general advice on complex program Cameron Simpson <cs@zip.com.au> - 2011-07-16 08:37 +1000
      Re: Looking for general advice on complex program Chris Angelico <rosuav@gmail.com> - 2011-07-16 10:01 +1000
    Re: Looking for general advice on complex program geremy condra <debatem1@gmail.com> - 2011-07-16 14:08 -0400

#9561 — Looking for general advice on complex program

Maybe not to the gurus here, but for me, this is a complex problem and I want to make sure I understand the real problem.

All of this is in Python 2.7 and wxPython

I have several XML files on a shared drive.
I have applications on other machines that need to access this XML file.
These applications need to read and write to this file.
These applications need to a) be alerted to a file change, or b) monitor the file for changes and regular intervals.

In my code, I have XManager classes (using a Singleton pattern) that reads each XML file into a tree (using ElementTree). The XManager class can read the file, make changes to the tree, and write the file as needed.

Now I'm expanding to the multiple application level, and I think I understand what I need to do, and I'm unsure about the exact processes.

I've been trying to have the XManagers check periodically if the XML file they monitor has changed. Since I don't want to mess up the GUI with constant hanging, I think I can use the thread or threading modules to create a recurring timed check, and then I need a separate check to see if the file is in use before reading or writing. 

I also need, I think, to have a way to check if the GUI is editing a node before the XManager reads the file, so the thread needs to be interrupted, or paused, because I don't know if threads would stop if a wxDialog is being show modally or not.

So, am I on the right track here? 

josh

[toc] | [next] | [standalone]

#9562

On 07/15/2011 03:47 PM, Josh English wrote:
>

I remember reading that file locking doesn't work on network mounted 
drives (specifically nfs mounts), but you might be able to simply create 
a 'lock' (mydoc.xml.lock or the like) file for the XML doc in question. 
  If that file exists you could either hang or silently give up.  Not 
sure if that helps.

--
Bill

[toc] | [prev] | [next] | [standalone]

#9576

On 15Jul2011 16:03, Billy Mays <81282ed9a88799d21e77957df2d84bd6514d9af6@myhashismyemail.com> wrote:
| I remember reading that file locking doesn't work on network mounted
| drives (specifically nfs mounts), but you might be able to simply
| create a 'lock' (mydoc.xml.lock or the like) file for the XML doc in
| question.  If that file exists you could either hang or silently
| give up.  Not sure if that helps.

There are two approaches to this. Plain old make-a-file won't work - it
is racy (and as mentioned, you can't rely on the various lock
facilities).

You can create a file while your umask is 0777; it will be non-writable
immediately (no chmod required), preventing another attempt to make it.

My personal habit is to make a directory for the lock; mkdir
also can't happen twice to the same name, you don't need to fiddle you
umask (racy and annoying, and problematic if you're using multiple
threads), _and_ you can put meta info inside it, like pid files etc.

Cheers,
-- 
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I had a wierd dream with Ken Thompson in it once.
        - George Politis <george@research.canon.com.au>

[toc] | [prev] | [next] | [standalone]

#9586

On Sat, Jul 16, 2011 at 8:37 AM, Cameron Simpson <cs@zip.com.au> wrote:
> There are two approaches to this.
> You can create a file while your umask is 0777... [or]
> My personal habit is to make a directory for the lock

Both viable options; I'd be inclined toward the second. Or, here's a
third option. Instead of writing to a shared network drive, submit
requests on a TCP socket direct to the monitor program. Spin off a
thread that does this:

* Wait for incoming socket connection
* Set overall cutoff timer; if (say) 2 seconds pass, kill the connection
* Authenticate the client (if applicable)
* Accept the update data, sanitize if necessary
* Write the file to disk
* Notify the XManager
* Loop.

Do all this on *one thread* and then you eliminate all race
conditions. Good use of a TCP listen queue and the cutoff timer will
mean that applications aren't actually kept waiting, but they're still
rigidly locked into a queue - depending on how frequent your updates
are, this could be a problem. If you need simultaneous updates, spin
off a new thread for each socket connection, and then use something
simple like a mapping of file name to semaphore to ensure no two try
to update the same file at once.

By moving the actual file read/writes to a single computer, you
simplify the task of notifying the parent. In fact, if there's only
one process that needs to be made aware of the change, the job's even
easier - all you need to do is change a variable or call a method or
whatever it be, right there in the socket handler.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#9639

On Fri, Jul 15, 2011 at 3:47 PM, Josh English
<Joshua.R.English@gmail.com> wrote:
> Maybe not to the gurus here, but for me, this is a complex problem and I want to make sure I understand the real problem.
>
> All of this is in Python 2.7 and wxPython
>
> I have several XML files on a shared drive.
> I have applications on other machines that need to access this XML file.
> These applications need to read and write to this file.
> These applications need to a) be alerted to a file change, or b) monitor the file for changes and regular intervals.
>
> In my code, I have XManager classes (using a Singleton pattern) that reads each XML file into a tree (using ElementTree). The XManager class can read the file, make changes to the tree, and write the file as needed.
>
> Now I'm expanding to the multiple application level, and I think I understand what I need to do, and I'm unsure about the exact processes.
>
> I've been trying to have the XManagers check periodically if the XML file they monitor has changed. Since I don't want to mess up the GUI with constant hanging, I think I can use the thread or threading modules to create a recurring timed check, and then I need a separate check to see if the file is in use before reading or writing.
>
> I also need, I think, to have a way to check if the GUI is editing a node before the XManager reads the file, so the thread needs to be interrupted, or paused, because I don't know if threads would stop if a wxDialog is being show modally or not.
>
> So, am I on the right track here?

I'd try watchdog[0] before I went to the trouble of rolling my own.

Geremy Condra

[0]: http://packages.python.org/watchdog/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web