Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #2652 > unrolled thread

How to program in Python to run system commands in 1000s of servers

Back to article view | Back to comp.lang.python

Contents

  How to program in Python to run system commands in 1000s of servers Babu <babukk@gmail.com> - 2011-04-05 07:51 -0700
    Re: How to program in Python to run system commands in 1000s of servers geremy condra <debatem1@gmail.com> - 2011-04-05 09:59 -0700
      Re: How to program in Python to run system commands in 1000s of servers Roy Smith <roy@panix.com> - 2011-04-05 21:24 -0400
        Re: How to program in Python to run system commands in 1000s of servers Anssi Saari <as@sci.fi> - 2011-04-07 14:27 +0300
          Re: How to program in Python to run system commands in 1000s of servers Chris Angelico <rosuav@gmail.com> - 2011-04-08 00:13 +1000
            Re: How to program in Python to run system commands in 1000s of servers Anssi Saari <as@sci.fi> - 2011-04-07 22:14 +0300
              Re: How to program in Python to run system commands in 1000s of servers Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-04-07 22:40 +0200
                Re: How to program in Python to run system commands in 1000s of servers Babu <babukk@gmail.com> - 2011-04-10 07:22 -0700
                  Re: How to program in Python to run system commands in 1000s of servers Chris Angelico <rosuav@gmail.com> - 2011-04-11 00:42 +1000

#2652 — How to program in Python to run system commands in 1000s of servers

Here is my problem:  Want to program in python to run sysadmin
commands across 1000s of servers and gather the result in one place.
Many times the commands need to be run as root.  We cannot use ssh as
root remote connectivity as well.  What are the different ways of
programming in python to achieve this?

[toc] | [next] | [standalone]

#2662

On Tue, Apr 5, 2011 at 7:51 AM, Babu <babukk@gmail.com> wrote:
>
> Here is my problem:  Want to program in python to run sysadmin
> commands across 1000s of servers and gather the result in one place.
> Many times the commands need to be run as root.  We cannot use ssh as
> root remote connectivity as well.  What are the different ways of
> programming in python to achieve this?

There are a bajillion ways to do it badly, but SSH sounds like the
right tool for the job here. You really don't want your remote admin
system compromised, and fabric makes this kind of thing really much
less painful.

Geremy Condra

[toc] | [prev] | [next] | [standalone]

#2681

In article <mailman.52.1302022780.9059.python-list@python.org>,
 geremy condra <debatem1@gmail.com> wrote:

> On Tue, Apr 5, 2011 at 7:51 AM, Babu <babukk@gmail.com> wrote:
> >
> > Here is my problem:  Want to program in python to run sysadmin
> > commands across 1000s of servers and gather the result in one place.
> > Many times the commands need to be run as root.  We cannot use ssh as
> > root remote connectivity as well.  What are the different ways of
> > programming in python to achieve this?
> 
> There are a bajillion ways to do it badly, but SSH sounds like the
> right tool for the job here. You really don't want your remote admin
> system compromised, and fabric makes this kind of thing really much
> less painful.

Agreed on the fabric (fabfile.org) recommendation.  We've been using it 
for about 6 months.  Very handy.

I'm not sure how to parse:

> We cannot use ssh as root remote connectivity as well.

but with 1000's of servers, I really don't see any alternative to ssh, 
with key authentication.  You don't really propose to type passwords at 
1000's of machines, do you?

As far as fabric goes, it's not perfect, but it's pretty good and if you 
try to roll your own alternative, you will likely 1) waste a lot of time 
and money and 2) end up with an inferior solution.

[toc] | [prev] | [next] | [standalone]

#2768

Roy Smith <roy@panix.com> writes:

> I'm not sure how to parse:

>> We cannot use ssh as root remote connectivity as well.
>
> but with 1000's of servers, I really don't see any alternative to ssh, 
> with key authentication.  You don't really propose to type passwords at 
> 1000's of machines, do you?

I guess it might mean someone decided to config sshd with
PermitRootLogin no... I believe this is common? I don't think it's a
particularly good idea, especially for a large scale deployment.

So I guess there may be some config needed for the machines before
they can be remotely administrated in an automatic fashion.

[toc] | [prev] | [next] | [standalone]

#2776

On Thu, Apr 7, 2011 at 9:27 PM, Anssi Saari <as@sci.fi> wrote:
> Roy Smith <roy@panix.com> writes:
>
>>> We cannot use ssh as root remote connectivity as well.
>>
>> but with 1000's of servers, I really don't see any alternative to ssh,
>> with key authentication.  You don't really propose to type passwords at
>> 1000's of machines, do you?
>
> I guess it might mean someone decided to config sshd with
> PermitRootLogin no... I believe this is common? I don't think it's a
> particularly good idea, especially for a large scale deployment.
>
> So I guess there may be some config needed for the machines before
> they can be remotely administrated in an automatic fashion.

Depending on what exactly is needed, it might be easier to run a
separate daemon on the computers, one whose sole purpose is to do the
task / get the statistics needed and return them. Then the Python
script need only collect each program's returned response.

Alternatively, if the program needs to be run periodically anyway, it
might be easier to simply cron it on every computer it needs to run
on, and then log the results to some central server (maybe a MySQL
database, or something). Then whenever you want stats, you just query
that server.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#2801

Chris Angelico <rosuav@gmail.com> writes:

> Depending on what exactly is needed, it might be easier to run a
> separate daemon on the computers, one whose sole purpose is to do the
> task / get the statistics needed and return them. Then the Python
> script need only collect each program's returned response.

Those would still need to be deployed somehow to the thousands of
machines though. 

I realized after posting that something like pexpect might work for
stuffing the keystrokes needed to root login via ssh to all machines
and such... If that's what he needs to do, since it wasn't very clear.

[toc] | [prev] | [next] | [standalone]

#2808

Am 07.04.2011 21:14, schrieb Anssi Saari:
> Chris Angelico<rosuav@gmail.com>  writes:
>
>> Depending on what exactly is needed, it might be easier to run a
>> separate daemon on the computers, one whose sole purpose is to do the
>> task / get the statistics needed and return them. Then the Python
>> script need only collect each program's returned response.
>
> Those would still need to be deployed somehow to the thousands of
> machines though.

But only once...


> I realized after posting that something like pexpect might work for
> stuffing the keystrokes needed to root login via ssh to all machines
> and such... If that's what he needs to do, since it wasn't very clear.

Maybe that works. But it is much, much worse than using keys...


Thomas

[toc] | [prev] | [next] | [standalone]

#2945

On Apr 8, 5:40 am, Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-
a470-7603bd3aa...@spamschutz.glglgl.de> wrote:
> Am 07.04.2011 21:14, schrieb Anssi Saari:
>
> > Chris Angelico<ros...@gmail.com>  writes:
>
> >> Depending on what exactly is needed, it might be easier to run a
> >> separate daemon on the computers, one whose sole purpose is to do the
> >> task / get the statistics needed and return them. Then the Python
> >> script need only collect each program's returned response.
>
> > Those would still need to be deployed somehow to the thousands of
> > machines though.
>
> But only once...
>
> > I realized after posting that something like pexpect might work for
> > stuffing the keystrokes needed to root login via ssh to all machines
> > and such... If that's what he needs to do, since it wasn't very clear.
>
> Maybe that works. But it is much, much worse than using keys...
>
> Thomas

Thank you all for various ideas.  Let me give some background and more
information here.  Reason that we cannot use root trusted ssh is a
Internal Information Security decision.  Given that we have this
restriction, I wanted to explore what other creative options we have
so that we can still accomplish this.

In our enterprise environment, quick production support is very
important.  An application problem troubleshooting might require
we check various status on multiple servers quickly.  So we need to
execute commands depending on the situation.  Let me summarize some of
the ideas presented in this thread.
  1. Use pexpect to login and become root(or sudo - yes sudo is
allowed) on the remote machines
  2. run a daemon on each server, which will respond to client
requests
  3. run your program through cron and collect data and dump into a
database which can be used for query later [ yes - this is on
      plate ]
  4. Use fabric (fabile.org) for developing program.  Does this assume
that ssh root trust is already in place?

Are there any more different approaches?  I suppose if we take the
daemon approach then we can make it as a webservice as well?

[toc] | [prev] | [next] | [standalone]

#2946

On Mon, Apr 11, 2011 at 12:22 AM, Babu <babukk@gmail.com> wrote:
> Are there any more different approaches?  I suppose if we take the
> daemon approach then we can make it as a webservice as well?

Yes, your daemon could function via HTTP. But if you go that route,
you would need some way to collect all the different computers'
results.

For example, suppose you build your daemon to respond to HTTP requests
on port 8000, with a document name like "/status". You could then
retrieve _one_ computer's status by pointing your browser to
http://computername/status - but that's only one. You would then need
a wrapper somewhere to collect them, for instance:


<iframe src="http://computer1/status"></iframe>
<iframe src="http://computer2/status"></iframe>
<iframe src="http://computer3/status"></iframe>

etc. If you're always getting status on the same set of computers (or
a few standard sets of computers), this could be a simple .HTML file
that you have on your hard disk; otherwise, you may want to consider
another web server that lets you tick which ones to query, and builds
an iframe list from your selections.

Chris Angelico

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web