Path: csiph.com!x330-a1.tempe.blueboxinc.net!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'oct': 0.02; 'else:': 0.03; 'debug': 0.03; 'filename:fname piece:py': 0.04; 'ubuntu': 0.04; 'sys': 0.05; 'subject:Python': 0.06; 'run,': 0.07; 'tue': 0.07; 'python': 0.08; '__name__': 0.09; 'finally:': 0.09; 'loop.': 0.09; 'method:': 0.09; 'received:mail-bw0-f46.google.com': 0.09; 'x86_64': 0.09; 'def': 0.12; 'written': 0.14; 'users,': 0.14; 'defined': 0.14; "'__main__':": 0.16; 'attached,': 0.16; 'except:': 0.16; 'loads.': 0.16; 'os:': 0.16; 'parameters,': 0.16; 'polls': 0.16; 'ram;': 0.16; 'received:209.85.214.46': 0.16; 'run(self):': 0.16; 'sleeping': 0.16; 'specs': 0.16; 'subject:cloud': 0.16; 'subject:setup': 0.16; 'subject:threads': 0.16; 'sure,': 0.16; 'sys.exit(1)': 0.16; 'threading': 0.16; 'url:pastebin': 0.16; 'utc': 0.16; '"python': 0.16; '0.1': 0.23; 'fri': 0.23; 'topic.': 0.23; 'load': 0.24; 'suggestion': 0.26; 'script': 0.27; 'testing': 0.27; 'message-id:@mail.gmail.com': 0.28; 'seeing': 0.28; 'received:209.85.214': 0.28; 'cloud': 0.29; 'e.g.': 0.29; 'import': 0.29; 'server': 0.29; 'class': 0.29; 'bare': 0.30; 'ran': 0.30; 'threads': 0.30; 'url:net)': 0.30; 'looks': 0.31; 'print': 0.31; 'apr': 0.32; 'to:addr:python-list': 0.33; 'post': 0.33; 'break': 0.33; 'greatly': 0.33; 'handling': 0.33; 'starting': 0.33; 'error': 0.33; 'try:': 0.35; 'using': 0.35; 'test': 0.35; 'charset:us-ascii': 0.36; 'log': 0.36; 'running': 0.37; 'takes': 0.37; 'received:google.com': 0.37; 'received:209.85': 0.37; 'despite': 0.37; 'logging': 0.37; 'thread': 0.37; 'think': 0.38; 'could': 0.38; 'hello,': 0.38; 'run': 0.38; 'but': 0.38; 'user': 0.39; 'received:209': 0.39; 'version:': 0.39; 'to:addr:python.org': 0.39; 'appreciated.': 0.40; 'more': 0.60; 'below': 0.61; 'skip:2 10': 0.61; 'our': 0.63; 'details': 0.64; 'further': 0.65; 'minutes': 0.67; 'period.': 0.73; 'average': 0.77; '100%': 0.80; 'subject:different': 0.84; 'uptime': 0.84; 'metal': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=1pGBfH7G6miywwq11pqRUPbAjc3K6ltLZTYsu914p/w=; b=l9E0y+exnFdXKqgRGzHuGJ462eej0b0S0ya6c5ZExAallbAf6u596/Gy/DYqmb6pqf zdGJ6G6M3+tukWu8vZqoUTVM3ose/sHeSTtQtxFRyRr3uSwjKWF4OgLBjRX+ZjnaC5zr lL2mlld2Tq6vOVgLuU8LEVtGda8uFFQw65x/w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=es79kmWNCYWTaoP3ie7R80l+movaUCWXTqQSQN6SeeBwuar9Qy/uw3g1UU3sASvNz5 wPRxWvj2nYLBSC+7ltAakOlZt8aUkbL8dxjpkX8A4YEdCeULUQbsLu0pqzL7rR3s/UCE QonaHHSBJuu3nnc+UkxHs1z1AncEEbpfXptqE= MIME-Version: 1.0 Date: Tue, 31 May 2011 19:25:31 +0800 Subject: Load averages and Python threads on different Linux cloud provider setup From: Mohanaraj Gopala Krishnan To: python-list@python.org Content-Type: multipart/mixed; boundary=0016e6db5be874b38804a490a9d7 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 100 NNTP-Posting-Host: 82.94.164.166 X-Trace: 1306841134 news.xs4all.nl 49042 [::ffff:82.94.164.166]:49679 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:6726 --0016e6db5be874b38804a490a9d7 Content-Type: text/plain; charset=ISO-8859-1 Hello, Apologies if this post is off topic. We have a Python application that polls directories using threads and inotify watchers. We have always run this application in a cloud server provided by Voxel (http://www.voxel.net). We are currently testing a different cloud server provider StormOnDemand (http://stormondemand.com) and when we ran our application, our load averages were a lot higher then they were when they were running on the Voxel cloud server despite the specs being about the same (Refer below for more details on setup). We have also ensured then when testing the server was not handling any other loads. I have written a simple test application (test_threads.py - attached, or refer to http://pastebin.com/xGQU7JD0) that simulates the issues we are seeing by starting up threads that loops, sleeping for a user defined time on each loop. It takes 2 parameters, the amount of threads to start and the interval period. When I run, "python test_threads.py 50 0.1" for about 10 minutes Load average results: StormOnDemand: $ uptime 18:46:22 up 7:29, 6 users, load average: 4.43, 4.16, 2.93 voxel.net $ uptime 18:48:14 up 9 days, 15:09, 9 users, load average: 0.51, 0.47, 0.43 The load average on the StormOnDemand server is a lot higher. Python version: StormOnDemand - 2.6.5 Voxel - 2.6.5 Server spec: StormOnDemand - 8 x Intel(R) Xeon(R) CPU E5506 @ 2.13GHz; 16GB RAM; 230GB HDD (Storm Bare Metal servers) Voxel - 7 x Intel(R) Xeon(R) CPU L5640 @ 2.27GHz; 14GB RAM; 200GB HDD (VoxCloud servers) OS: StormOnDemand - Ubuntu 10.04 - 2.6.36-rc8101910 #1 SMP Tue Oct 19 19:18:34 UTC 2010 x86_64 GNU/Linux Voxel - Ubuntu 10.04 - 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux Virtualisation method: StormOnDemand - Not 100% sure, but I think they use Xen Voxel - Not sure, but the image that we are using looks to us like a stock standard Ubuntu 10.04 server Any suggestion on why the load would be a lot higher or how I could debug this further is greatly appreciated. -- Mohan --0016e6db5be874b38804a490a9d7 Content-Type: text/x-python; charset=US-ASCII; name="test_threads.py" Content-Disposition: attachment; filename="test_threads.py" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gocrgkm70 I1NpbXBsZSBzY3JpcHQgdG8gdGVzdCBsb2FkIHdoZW4gcnVubmluZyBweXRob24gdGhyZWFkcwoj IHVzYWdlOgojIHB5dGhvbiB0ZXN0X3RocmVhZHMucHkgPG51bWJlciBvZiB0aHJlYWRzPiA8dGhy ZWFkIGxvb3AgaW50ZXJ2YWw+CiMgZS5nLiA6CiMgcHl0aG9uIHRlc3RfdGhyZWFkcy5weSA1MCAw LjEKIwoKaW1wb3J0IHN5cwpmcm9tIHRocmVhZGluZyBpbXBvcnQgTG9jaywgVGhyZWFkCmltcG9y dCBsb2dnaW5nCmltcG9ydCB0aW1lCgpsb2dnaW5nLmJhc2ljQ29uZmlnKCkKCmxvZyA9IGxvZ2dp bmcuZ2V0TG9nZ2VyKCJ0ZXN0IikKbG9nLnNldExldmVsKGxvZ2dpbmcuREVCVUcpIAoKY2xhc3Mg TW9uaXRvckRpcihUaHJlYWQpOgogICAgZGVmIF9faW5pdF9fKHNlbGYsIGlkLCBpbnRlcnZhbCk6 CiAgICAgICAgc3VwZXIoTW9uaXRvckRpciwgc2VsZikuX19pbml0X18oKQogICAgICAJc2VsZi5p bnRlcnZhbCA9IGludGVydmFsIAogICAgICAgIHNlbGYuaWQgPSBpZCAKICAgICAgICBzZWxmLl90 ZXJtaW5hdGVkID0gRmFsc2UKICAgIAogICAgZGVmIGZpbmlzaChzZWxmKToKICAgICAgICBpZiBu b3Qgc2VsZi5fdGVybWluYXRlZDoKICAgICAgICAgICAgbG9nLmluZm8oJ1Rlcm1pbmF0aW5nLi4u ICVpJyVzZWxmLmlkKQogICAgICAgICAgICBzZWxmLl90ZXJtaW5hdGVkID0gVHJ1ZQogICAgCiAg ICBkZWYgcnVuKHNlbGYpOgogICAgICAgIHByaW50ICdTdGFydCB0aHJlYWRpICVpJyVzZWxmLmlk CiAgICAgICAgCiAgICAgICAgdHJ5OgogICAgICAgICAgICB3aGlsZSBub3Qgc2VsZi5fdGVybWlu YXRlZDoKICAgICAgICAgICAgICAgIGxvZy5pbmZvKCdpbiBpZCAtICVpIFxuJyVzZWxmLmlkKQog ICAgICAgICAgICAgICAgdGltZS5zbGVlcChzZWxmLmludGVydmFsKQogICAgICAgIGV4Y2VwdDoK ICAgICAgICAgICAgbG9nLmV4Y2VwdGlvbignVW5leHBlY3RlZCBlcnJvciB3aGlsZSBtb25pdG9y aW5nJykKICAgICAgICBmaW5hbGx5OgogICAgICAgICAgICBsb2cuaW5mbygnRG9uZSBtb25pdG9y aW5nJykKICAgIAogICAgICAgIAppZiBfX25hbWVfXyA9PSAnX19tYWluX18nOgogICAgbVRocmVh ZHMgPSBbXQogICAgZm9yIGkgaW4gcmFuZ2UoaW50KHN5cy5hcmd2WzFdKSk6CiAgICAgICAgcHJp bnQgImNyZWF0ZSB0aHJlYWQgJWkiJWkKCW1UaHJlYWQgPSBNb25pdG9yRGlyKGksIGZsb2F0KHN5 cy5hcmd2WzJdKSkKCW1UaHJlYWRzLmFwcGVuZChtVGhyZWFkKQoJbVRocmVhZC5zdGFydCgpCgog ICAgd2hpbGUgYW55KHguaXNBbGl2ZSgpIGZvciB4IGluIG1UaHJlYWRzKToKICAgICAgICB0cnk6 CgkgICAgdGltZS5zbGVlcCgxKQogICAgICAgIGV4Y2VwdCBLZXlib2FyZEludGVycnVwdDoKCSAg ICBsb2cuZGVidWcoJ0V4aXRpbmcuLi4nKQoJICAgIGJyZWFrCiAgICBlbHNlOgogICAgICAgc3lz LmV4aXQoMSkgCgogICAgZm9yIG1UaHJlYWQgaW4gbVRocmVhZHM6CiAgICAgICAgbVRocmVhZC5m aW5pc2goKQoKICAgIGZvciBtVGhyZWFkIGluIG1UaHJlYWRzOgogICAgICAgIG1UaHJlYWQuam9p bigpCgogICAgc3lzLmV4aXQoMCkK --0016e6db5be874b38804a490a9d7--