Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #676

Re: Best method for creating a global counter

Show all headers | View raw

Tom  <tomsellner@gmail.com> wrote:
>Hi - I'm curious what the best method is for creating a global counter
>for an ETL system we have.

How "global" do you want to be?  Across a process, a host, a subnet?
Currently processing, or in the last day, or historical?  Transactional or
best-efforts?

>Currently, at run time in our ksh script,
>before starting a new a process, we do something like ps -ef | grep -c
>"my_exe".  If it's greater than some count, then we sleep and wait for
>existing processes to finish.

Oh, this isn't about a counter for reporting or usage?  It's about load
throttling?

>This is a somewhat expensive process,

Not only that, but it probably doesn't do a very good job of throttling usage.
What resource exactly are you trying to protect?  My guess is database
performance or maybe host memory or CPU.  Certainly not host process table
entries.

>and I'm wondering what other people suggest.

For systems based on a database, you should at least consider using the
database.  It's a higher level of IPC than you often need, but as long as it's
there, it makes things easy.  And it works across hosts as easily as across
processes on a host.  And it's durable against most types of failure.  And it
has built-in locking.

>Perhaps a shared memory array? 

Works fine (assuming you use mutexes correctly to avoid overwriting each
other).  Also just writing/reading a file (with locking, as needed) is
reasonable.

Or, instead of having each job be a new process, have a singleton (per-host)
parent process that keeps track of its children and has a work queue.  Rather
than starting a new process each time, you just send a request to this
executor.  That means your IPC needs are for messaging, rather than for
state, and that's often much simpler.

>I guess the question is - what do real programmers use for IPC?

Real programmers use whatever fits their needs with the least amount of
additional complexity and twiddling.  Generally, that starts with "what're the
minimum requirements, and what's the simplest thing that could possibly work".
--
Mark Rafn    dagon@dagon.net    <http://www.dagon.net/>  

Back to comp.unix.programmer | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Best method for creating a global counter Tom <tomsellner@gmail.com> - 2011-05-31 08:40 -0700
  Re: Best method for creating a global counter Alex Fraser <me@privacy.net> - 2011-05-31 19:39 +0100
  Re: Best method for creating a global counter dagon@dagon.net (Dagon) - 2011-05-31 15:18 -0700

csiph-web