Load averages and Python threads on different Linux cloud provider setup

Path	csiph.com!x330-a1.tempe.blueboxinc.net!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<mohangk@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.000
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'oct': 0.02; 'else:': 0.03; 'debug': 0.03; 'filename:fname piece:py': 0.04; 'ubuntu': 0.04; 'sys': 0.05; 'subject:Python': 0.06; 'run,': 0.07; 'tue': 0.07; 'python': 0.08; '__name__': 0.09; 'finally:': 0.09; 'loop.': 0.09; 'method:': 0.09; 'received:mail-bw0-f46.google.com': 0.09; 'x86_64': 0.09; 'def': 0.12; 'written': 0.14; 'users,': 0.14; 'defined': 0.14; "'__main__':": 0.16; 'attached,': 0.16; 'except:': 0.16; 'loads.': 0.16; 'os:': 0.16; 'parameters,': 0.16; 'polls': 0.16; 'ram;': 0.16; 'received:209.85.214.46': 0.16; 'run(self):': 0.16; 'sleeping': 0.16; 'specs': 0.16; 'subject:cloud': 0.16; 'subject:setup': 0.16; 'subject:threads': 0.16; 'sure,': 0.16; 'sys.exit(1)': 0.16; 'threading': 0.16; 'url:pastebin': 0.16; 'utc': 0.16; '"python': 0.16; '0.1': 0.23; 'fri': 0.23; 'topic.': 0.23; 'load': 0.24; 'suggestion': 0.26; 'script': 0.27; 'testing': 0.27; 'message-id:@mail.gmail.com': 0.28; 'seeing': 0.28; 'received:209.85.214': 0.28; 'cloud': 0.29; 'e.g.': 0.29; 'import': 0.29; 'server': 0.29; 'class': 0.29; 'bare': 0.30; 'ran': 0.30; 'threads': 0.30; 'url:net)': 0.30; 'looks': 0.31; 'print': 0.31; 'apr': 0.32; 'to:addr:python-list': 0.33; 'post': 0.33; 'break': 0.33; 'greatly': 0.33; 'handling': 0.33; 'starting': 0.33; 'error': 0.33; 'try:': 0.35; 'using': 0.35; 'test': 0.35; 'charset:us-ascii': 0.36; 'log': 0.36; 'running': 0.37; 'takes': 0.37; 'received:google.com': 0.37; 'received:209.85': 0.37; 'despite': 0.37; 'logging': 0.37; 'thread': 0.37; 'think': 0.38; 'could': 0.38; 'hello,': 0.38; 'run': 0.38; 'but': 0.38; 'user': 0.39; 'received:209': 0.39; 'version:': 0.39; 'to:addr:python.org': 0.39; 'appreciated.': 0.40; 'more': 0.60; 'below': 0.61; 'skip:2 10': 0.61; 'our': 0.63; 'details': 0.64; 'further': 0.65; 'minutes': 0.67; 'period.': 0.73; 'average': 0.77; '100%': 0.80; 'subject:different': 0.84; 'uptime': 0.84; 'metal': 0.91
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=1pGBfH7G6miywwq11pqRUPbAjc3K6ltLZTYsu914p/w=; b=l9E0y+exnFdXKqgRGzHuGJ462eej0b0S0ya6c5ZExAallbAf6u596/Gy/DYqmb6pqf zdGJ6G6M3+tukWu8vZqoUTVM3ose/sHeSTtQtxFRyRr3uSwjKWF4OgLBjRX+ZjnaC5zr lL2mlld2Tq6vOVgLuU8LEVtGda8uFFQw65x/w=
DomainKey-Signature	a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=es79kmWNCYWTaoP3ie7R80l+movaUCWXTqQSQN6SeeBwuar9Qy/uw3g1UU3sASvNz5 wPRxWvj2nYLBSC+7ltAakOlZt8aUkbL8dxjpkX8A4YEdCeULUQbsLu0pqzL7rR3s/UCE QonaHHSBJuu3nnc+UkxHs1z1AncEEbpfXptqE=
MIME-Version	1.0
Date	Tue, 31 May 2011 19:25:31 +0800
Subject	Load averages and Python threads on different Linux cloud provider setup
From	Mohanaraj Gopala Krishnan <mohangk@gmail.com>
To	python-list@python.org
Content-Type	multipart/mixed; boundary=0016e6db5be874b38804a490a9d7
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.2314.1306841134.9059.python-list@python.org> (permalink)
Lines	100
NNTP-Posting-Host	82.94.164.166
X-Trace	1306841134 news.xs4all.nl 49042 [::ffff:82.94.164.166]:49679
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:6726

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

Hello,

Apologies if this post is off topic.

We have a Python application that polls directories using threads and
inotify watchers. We have always run this application in a cloud
server provided by Voxel (http://www.voxel.net). We are currently
testing a different cloud server provider StormOnDemand
(http://stormondemand.com) and when we ran our application, our load
averages were a lot higher then they were when they were running on
the Voxel cloud server despite the specs being about the same (Refer
below for more details on setup). We have also ensured then when
testing the server was not handling any other loads.

I have written a simple test application (test_threads.py - attached,
or refer to http://pastebin.com/xGQU7JD0) that simulates the issues we
are seeing by starting up  threads that loops, sleeping for a user
defined time on each loop. It takes 2 parameters, the amount of
threads to start and the interval period.

When I run, "python test_threads.py 50 0.1" for about 10 minutes

Load average results:

StormOnDemand:
$ uptime
 18:46:22 up  7:29,  6 users,  load average: 4.43, 4.16, 2.93

voxel.net
$ uptime
 18:48:14 up 9 days, 15:09,  9 users,  load average: 0.51, 0.47, 0.43

The load average on the StormOnDemand server is a lot higher.

Python version:
StormOnDemand - 2.6.5
Voxel - 2.6.5

Server spec:
StormOnDemand - 8 x Intel(R) Xeon(R) CPU E5506 @ 2.13GHz; 16GB RAM;
230GB HDD (Storm Bare Metal servers)
Voxel - 7 x Intel(R) Xeon(R) CPU L5640 @ 2.27GHz; 14GB RAM; 200GB HDD
(VoxCloud servers)

OS:
StormOnDemand - Ubuntu 10.04 - 2.6.36-rc8101910 #1 SMP Tue Oct 19
19:18:34 UTC 2010 x86_64 GNU/Linux
Voxel - Ubuntu 10.04 -  2.6.32-31-server #61-Ubuntu SMP Fri Apr 8
19:44:42 UTC 2011 x86_64 GNU/Linux

Virtualisation method:
StormOnDemand - Not 100% sure, but I think they use Xen
Voxel - Not sure, but the image that we are using looks to us like a
stock standard Ubuntu 10.04 server

Any suggestion on why the load would be a lot higher or how I could
debug this further is greatly appreciated.


-- 
Mohan

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Load averages and Python threads on different Linux cloud provider setup Mohanaraj Gopala Krishnan <mohangk@gmail.com> - 2011-05-31 19:25 +0800

csiph-web