Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #27630 > unrolled thread

psphere: how to make thread safe

Started bysajuptpm <sajuptpm@gmail.com>
First post2012-08-22 05:03 -0700
Last post2012-08-26 05:57 -0700
Articles 4 — 3 participants

Back to article view | Back to comp.lang.python

Contents

  psphere: how to make thread safe sajuptpm <sajuptpm@gmail.com> - 2012-08-22 05:03 -0700
    Re: psphere: how to make thread safe Ramchandra Apte <maniandram01@gmail.com> - 2012-08-24 07:45 -0700
    Re: psphere: how to make thread safe jonathan.kinred@gmail.com - 2012-08-26 05:45 -0700
      Re: psphere: how to make thread safe jonathan.kinred@gmail.com - 2012-08-26 05:57 -0700

#27630 — psphere: how to make thread safe

Fromsajuptpm <sajuptpm@gmail.com>
Date2012-08-22 05:03 -0700
Subjectpsphere: how to make thread safe
Message-ID<578d6c60-7b64-4d4e-9d38-4b26f9e4ad59@googlegroups.com>
Hi,

psphere: Python interface for the VMware vSphere Web Services SDK

I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.


=======Test Code ========

import psphere
from psphere.client import Client
from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
client = Client("192.168.0.114", "root", "vmware1") ##vCenter
print "\nSucessfully connected to vCenter.\n"

from threading import Thread

def myfunc(i):
	host1 = HostSystem.get(client, name="192.168.0.134")
	host2 = HostSystem.get(client, name="192.168.0.113")
	print "----i------",i
	while True:
		#host1.update(properties=["config", "vm"])
		#host2.update(properties=["config", "vm"])
		c = type(host1.config.network)
		v = type(host2.config.network)
		for vm in host1.vm:
			k = vm.config.template
		for vm in host2.vm:
			p = vm.config.template


for i in range(10):	
    t = Thread(target=myfunc, args=(i,))
    t.start()


"""
OUTPUT 
=======
Sucessfully connected to vCenter.

----i------ 1
----i------ 3
----i------ 5
----i------ 0
 ----i------ 4
----i------ 2----i------ 7
----i------ 
9
----i------ 6
----i------ 8
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "vcenter_test1.py", line 19, in myfunc
    k = vm.config.template
  File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
    value = self.fget(inst)
  File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
    return self._get_dataobject("config", False)
  File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
    return self._cache[name][0]
KeyError: 'config'

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "vcenter_test1.py", line 17, in myfunc
    v = type(host2.config.network)
AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'



"""

[toc] | [next] | [standalone]

#27806

FromRamchandra Apte <maniandram01@gmail.com>
Date2012-08-24 07:45 -0700
Message-ID<85ee8513-648c-4322-af33-0448e7a35c1e@googlegroups.com>
In reply to#27630
On Wednesday, 22 August 2012 17:33:48 UTC+5:30, sajuptpm  wrote:
> Hi,
> 
> 
> 
> psphere: Python interface for the VMware vSphere Web Services SDK
> 
> 
> 
> I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
> 
> 
> 
> 
> 
> =======Test Code ========
> 
> 
> 
> import psphere
> 
> from psphere.client import Client
> 
> from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
> 
> client = Client("192.168.0.114", "root", "vmware1") ##vCenter
> 
> print "\nSucessfully connected to vCenter.\n"
> 
> 
> 
> from threading import Thread
> 
> 
> 
> def myfunc(i):
> 
> 	host1 = HostSystem.get(client, name="192.168.0.134")
> 
> 	host2 = HostSystem.get(client, name="192.168.0.113")
> 
> 	print "----i------",i
> 
> 	while True:
> 
> 		#host1.update(properties=["config", "vm"])
> 
> 		#host2.update(properties=["config", "vm"])
> 
> 		c = type(host1.config.network)
> 
> 		v = type(host2.config.network)
> 
> 		for vm in host1.vm:
> 
> 			k = vm.config.template
> 
> 		for vm in host2.vm:
> 
> 			p = vm.config.template
> 
> 
> 
> 
> 
> for i in range(10):	
> 
>     t = Thread(target=myfunc, args=(i,))
> 
>     t.start()
> 
> 
> 
> 
> 
> """
> 
> OUTPUT 
> 
> =======
> 
> Sucessfully connected to vCenter.
> 
> 
> 
> ----i------ 1
> 
> ----i------ 3
> 
> ----i------ 5
> 
> ----i------ 0
> 
>  ----i------ 4
> 
> ----i------ 2----i------ 7
> 
> ----i------ 
> 
> 9
> 
> ----i------ 6
> 
> ----i------ 8
> 
> Exception in thread Thread-4:
> 
> Traceback (most recent call last):
> 
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
>     self.run()
> 
>   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
>     self.__target(*self.__args, **self.__kwargs)
> 
>   File "vcenter_test1.py", line 19, in myfunc
> 
>     k = vm.config.template
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
> 
>     value = self.fget(inst)
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
> 
>     return self._get_dataobject("config", False)
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
> 
>     return self._cache[name][0]
> 
> KeyError: 'config'
> 
> 
> 
> Exception in thread Thread-6:
> 
> Traceback (most recent call last):
> 
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
>     self.run()
> 
>   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
>     self.__target(*self.__args, **self.__kwargs)
> 
>   File "vcenter_test1.py", line 17, in myfunc
> 
>     v = type(host2.config.network)
> 
> AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
> 
> 
> 
> 
> 
> 
> 
> """

use locks please on the cache

[toc] | [prev] | [next] | [standalone]

#27916

Fromjonathan.kinred@gmail.com
Date2012-08-26 05:45 -0700
Message-ID<e6752d78-c21e-42e0-ab47-3acc2c5e5468@googlegroups.com>
In reply to#27630
On Wednesday, 22 August 2012 22:03:48 UTC+10, sajuptpm  wrote:
> Hi,
> 
> 
> 
> psphere: Python interface for the VMware vSphere Web Services SDK
> 
> 
> 
> I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
> 
> 
> 
> 
> 
> =======Test Code ========
> 
> 
> 
> import psphere
> 
> from psphere.client import Client
> 
> from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
> 
> client = Client("192.168.0.114", "root", "vmware1") ##vCenter
> 
> print "\nSucessfully connected to vCenter.\n"
> 
> 
> 
> from threading import Thread
> 
> 
> 
> def myfunc(i):
> 
> 	host1 = HostSystem.get(client, name="192.168.0.134")
> 
> 	host2 = HostSystem.get(client, name="192.168.0.113")
> 
> 	print "----i------",i
> 
> 	while True:
> 
> 		#host1.update(properties=["config", "vm"])
> 
> 		#host2.update(properties=["config", "vm"])
> 
> 		c = type(host1.config.network)
> 
> 		v = type(host2.config.network)
> 
> 		for vm in host1.vm:
> 
> 			k = vm.config.template
> 
> 		for vm in host2.vm:
> 
> 			p = vm.config.template
> 
> 
> 
> 
> 
> for i in range(10):	
> 
>     t = Thread(target=myfunc, args=(i,))
> 
>     t.start()
> 
> 
> 
> 
> 
> """
> 
> OUTPUT 
> 
> =======
> 
> Sucessfully connected to vCenter.
> 
> 
> 
> ----i------ 1
> 
> ----i------ 3
> 
> ----i------ 5
> 
> ----i------ 0
> 
>  ----i------ 4
> 
> ----i------ 2----i------ 7
> 
> ----i------ 
> 
> 9
> 
> ----i------ 6
> 
> ----i------ 8
> 
> Exception in thread Thread-4:
> 
> Traceback (most recent call last):
> 
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
>     self.run()
> 
>   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
>     self.__target(*self.__args, **self.__kwargs)
> 
>   File "vcenter_test1.py", line 19, in myfunc
> 
>     k = vm.config.template
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
> 
>     value = self.fget(inst)
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
> 
>     return self._get_dataobject("config", False)
> 
>   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
> 
>     return self._cache[name][0]
> 
> KeyError: 'config'
> 
> 
> 
> Exception in thread Thread-6:
> 
> Traceback (most recent call last):
> 
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
>     self.run()
> 
>   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
>     self.__target(*self.__args, **self.__kwargs)
> 
>   File "vcenter_test1.py", line 17, in myfunc
> 
>     v = type(host2.config.network)
> 
> AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
> 
> 
> 
> 
> 
> 
> 
> """

I'm the author of psphere. I'd be really interested to get some experienced insight into this as I really don't know the first thing about thread safety.

psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.

What I'm suspecting (and this only came to me while writing this post) is that I have a "client" object which is passed into each object and used to retrieve data from the server. See this gist:
https://gist.github.com/3478641

When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?

A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like this I'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!

[toc] | [prev] | [next] | [standalone]

#27918

Fromjonathan.kinred@gmail.com
Date2012-08-26 05:57 -0700
Message-ID<97692dc5-b8c1-4573-84e9-2bc76b4c188f@googlegroups.com>
In reply to#27916
On Sunday, 26 August 2012 22:45:25 UTC+10, jonatha...@gmail.com  wrote:
> On Wednesday, 22 August 2012 22:03:48 UTC+10, sajuptpm  wrote:
> 
> > Hi,
> 
> > 
> 
> > 
> 
> > 
> 
> > psphere: Python interface for the VMware vSphere Web Services SDK
> 
> > 
> 
> > 
> 
> > 
> 
> > I already developed an app using https://bitbucket.org/jkinred/psphere. But getting lot of errors since psphere is not thread safe (I think). So i wrote couple of scripts to test it (See attached files) and found that caching mechanism used by psphere is not thread safe. Could someone please give me some suggestion to make it thread safe.
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > =======Test Code ========
> 
> > 
> 
> > 
> 
> > 
> 
> > import psphere
> 
> > 
> 
> > from psphere.client import Client
> 
> > 
> 
> > from psphere.managedobjects import HostSystem, VirtualMachine, ComputeResource
> 
> > 
> 
> > client = Client("192.168.0.114", "root", "vmware1") ##vCenter
> 
> > 
> 
> > print "\nSucessfully connected to vCenter.\n"
> 
> > 
> 
> > 
> 
> > 
> 
> > from threading import Thread
> 
> > 
> 
> > 
> 
> > 
> 
> > def myfunc(i):
> 
> > 
> 
> > 	host1 = HostSystem.get(client, name="192.168.0.134")
> 
> > 
> 
> > 	host2 = HostSystem.get(client, name="192.168.0.113")
> 
> > 
> 
> > 	print "----i------",i
> 
> > 
> 
> > 	while True:
> 
> > 
> 
> > 		#host1.update(properties=["config", "vm"])
> 
> > 
> 
> > 		#host2.update(properties=["config", "vm"])
> 
> > 
> 
> > 		c = type(host1.config.network)
> 
> > 
> 
> > 		v = type(host2.config.network)
> 
> > 
> 
> > 		for vm in host1.vm:
> 
> > 
> 
> > 			k = vm.config.template
> 
> > 
> 
> > 		for vm in host2.vm:
> 
> > 
> 
> > 			p = vm.config.template
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > for i in range(10):	
> 
> > 
> 
> >     t = Thread(target=myfunc, args=(i,))
> 
> > 
> 
> >     t.start()
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > """
> 
> > 
> 
> > OUTPUT 
> 
> > 
> 
> > =======
> 
> > 
> 
> > Sucessfully connected to vCenter.
> 
> > 
> 
> > 
> 
> > 
> 
> > ----i------ 1
> 
> > 
> 
> > ----i------ 3
> 
> > 
> 
> > ----i------ 5
> 
> > 
> 
> > ----i------ 0
> 
> > 
> 
> >  ----i------ 4
> 
> > 
> 
> > ----i------ 2----i------ 7
> 
> > 
> 
> > ----i------ 
> 
> > 
> 
> > 9
> 
> > 
> 
> > ----i------ 6
> 
> > 
> 
> > ----i------ 8
> 
> > 
> 
> > Exception in thread Thread-4:
> 
> > 
> 
> > Traceback (most recent call last):
> 
> > 
> 
> >   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
> > 
> 
> >     self.run()
> 
> > 
> 
> >   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
> > 
> 
> >     self.__target(*self.__args, **self.__kwargs)
> 
> > 
> 
> >   File "vcenter_test1.py", line 19, in myfunc
> 
> > 
> 
> >     k = vm.config.template
> 
> > 
> 
> >   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 79, in __get__
> 
> > 
> 
> >     value = self.fget(inst)
> 
> > 
> 
> >   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/managedobjects.py", line 1236, in config
> 
> > 
> 
> >     return self._get_dataobject("config", False)
> 
> > 
> 
> >   File "/home/saju/cvt/trunk/src/cvt/web/cvt/psphere/__init__.py", line 116, in _get_dataobject
> 
> > 
> 
> >     return self._cache[name][0]
> 
> > 
> 
> > KeyError: 'config'
> 
> > 
> 
> > 
> 
> > 
> 
> > Exception in thread Thread-6:
> 
> > 
> 
> > Traceback (most recent call last):
> 
> > 
> 
> >   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> 
> > 
> 
> >     self.run()
> 
> > 
> 
> >   File "/usr/lib/python2.7/threading.py", line 504, in run
> 
> > 
> 
> >     self.__target(*self.__args, **self.__kwargs)
> 
> > 
> 
> >   File "vcenter_test1.py", line 17, in myfunc
> 
> > 
> 
> >     v = type(host2.config.network)
> 
> > 
> 
> > AttributeError: VirtualMachineConfigInfo instance has no attribute 'network'
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > """
> 
> 
> 
> I'm the author of psphere. I'd be really interested to get some experienced insight into this as I really don't know the first thing about thread safety.
> 
> 
> 
> psphere does use a cache, but I'm confused as to how the cache would be thread unsafe as the cache is just a property of an object.
> 
> 
> 
> What I'm suspecting (and this only came to me while writing this post) is that I have a "client" object which is passed into each object and used to retrieve data from the server. See this gist:
> 
> https://gist.github.com/3478641
> 
> 
> 
> When used in a threaded scenario could the client be used by multiple objects simultaneously and the replies would come in out of order?
> 
> 
> 
> A couple of years ago I started psphere to learn Python and I always hoped that I would have an experienced developer join me. I did learn Python but I never got that experienced developer so when it comes to issues like this I'm definitely out of my depth! Please if someone could spend the time with me I would really appreciate it!

Just to clarify, the real Client is here:
https://github.com/jkinred/psphere/blob/master/psphere/client.py#L45

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web