Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #65548 > unrolled thread

Logging from a multiprocess application

Back to article view | Back to comp.lang.python

Contents

  Logging from a multiprocess application "Joseph L. Casale" <jcasale@activenetwerx.com> - 2014-02-06 18:24 +0000
    Re: Logging from a multiprocess application Mark Betz <betz.mark@gmail.com> - 2014-02-06 11:45 -0800
      RE: Logging from a multiprocess application "Joseph L. Casale" <jcasale@activenetwerx.com> - 2014-02-06 21:38 +0000

#65548 — Logging from a multiprocess application

I have a module that has one operation that benefits greatly from being multiprocessed.
Its a console based module and as such I have a stream handler and filter associated to
the console, obviously the mp based instances need special handling, so I have been
experimenting with a socket server in a thread in order for the remaining application to
carry on.

How have others tackled this problem? The portion of the code made to use multiprocessing
can not be switched to threading as it performs worse than simply serializing each task.

Thanks,
jlc

[toc] | [next] | [standalone]

#65554

On Thursday, February 6, 2014 1:24:17 PM UTC-5, Joseph L. Casale wrote:
> I have a module that has one operation that benefits greatly from being multiprocessed.
> Its a console based module and as such I have a stream handler and filter associated to
> the console, obviously the mp based instances need special handling, so I have been
> experimenting with a socket server in a thread in order for the remaining application to
> carry on.
> 
> How have others tackled this problem? The portion of the code made to use multiprocessing
> can not be switched to threading as it performs worse than simply serializing each task.
> 
> Thanks,
> jlc

Maybe check out logstash (http://logstash.net/). It's part of elastic search now. Using the python-logstash module (https://pypi.python.org/pypi/python-logstash) you can either log directly to the logstash daemon using the provided event formatter and http, or you can do as I did and set up a redis server to host a list. Logstash has an included redis input plugin that will blpop a list to get events. I use the formatter from the logstash module to format the events and then write them into redis as json. Logstash picks them up from redis and indexes them into its embedded elastic search instance. You can then connect to the embedded kibana web server to view a dashboard with event information. It's pretty cool and doesn't take long to set up.

[toc] | [prev] | [next] | [standalone]

#65558

> Maybe check out logstash (http://logstash.net/).

That looks pretty slick, I am constrained to using something provided by the packaged modules
in this scenario.

I think I have it pretty close except for the fact that the LogRecordStreamHandler from the cookbook
excepts when the sending process ends, aside from the obvious, I am sure there is a more robust
performance oriented approach to this. I've yet to work with the SocketServer module so I am lacking
any experience.

Thanks for the pointer,
jlc

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web