Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #196375 > unrolled thread

Best (simplest) way to share data between processes

Back to article view | Back to comp.lang.python

Contents

  Best (simplest) way to share data between processes Chris Green <cl@isbd.net> - 2024-07-06 08:28 +0100
    Re: Best (simplest) way to share data between processes Gordinator <gordinator@gordinator.org> - 2024-07-06 20:03 +0100
    Re: Best (simplest) way to share data between processes Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2024-07-06 21:32 +0200
      Re: Best (simplest) way to share data between processes Chris Green <cl@isbd.net> - 2024-07-08 13:52 +0100
    Re: Best (simplest) way to share data (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-07-07 01:43 +0000
      Re: Best (simplest) way to share data (Posting On Python-List Prohibited) Chris Green <cl@isbd.net> - 2024-07-08 13:56 +0100
        Re: Best (simplest) way to share data (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-07-09 05:52 +0000
        Re: Best (simplest) way to share data Chris Green <cl@isbd.net> - 2024-07-09 11:02 +0100
    Re: Best (simplest) way to share data between processes Barry <barry@barrys-emacs.org> - 2024-07-07 23:27 +0100
    Re: Best (simplest) way to share data between processes MRAB <python@mrabarnett.plus.com> - 2024-07-07 23:47 +0100
    Re: Best (simplest) way to share data between processes Barry Scott <barry@barrys-emacs.org> - 2024-07-08 09:34 +0100
    Re: Best (simplest) way to share data between processes Left Right <olegsivokon@gmail.com> - 2024-07-07 23:55 +0200

#196375 — Best (simplest) way to share data between processes

I have a Raspberry Pi in my boat that uses I2C to read a number of
voltages and currents (using ADS1115 A2D) so I can monitor the battery
condition etc.

At present various different scripts (i.e. processes) just read the
values using the I2C bus whenever they need to but I'm pretty sure
this (quite rarely) results in false readings because two processes
try to read at the same time.

Thus I'm looking for ways to prevent simultaneous access.

One fairly obvious way is to have single process/script which reads
the A2D values continuously and writes them to a file.  All other
scripts then read from the file as needed, a simple file lock can then
be used to prevent simultaneous access (well, simultaneous access when
the writing process is writing).

Is this the simplest approach?  Are there better ways using
multiprocess?  (They look more complicated though).

The I2C bus itself has a mutex but I don't think this guarantees that
(for example) an A2D reading is atomic because one reading takes more
than one I2C bus access.

Would a mutex of some sort around each I2C transaction (i.e. complete
A2D reading) be a better way to go?

-- 
Chris Green
·

[toc] | [next] | [standalone]

#196385

On 06/07/2024 12:32, Stefan Ram wrote:
>    But why overengineer? If you feel comfortable with the file
>    solution, go for it! The only drawback might be that it's a
>    bit slower than other approaches.

I absolutely agree. Overengineering is generally a bad idea because 
you're using a complex solution to solve a simple problem, which leads 
to unexpected breakage.

The file solution is perfectly fine, and file locks are used by really 
important software (i.e package managers and such) because its 
simplicity makes it almost never fail.

So yeah, go for it!

[toc] | [prev] | [next] | [standalone]

#196386

On 06/07/2024 09.28, Chris Green wrote:
> I have a Raspberry Pi in my boat that uses I2C to read a number of
> voltages and currents (using ADS1115 A2D) so I can monitor the battery
> condition etc.
> 
> At present various different scripts (i.e. processes) just read the
> values using the I2C bus whenever they need to but I'm pretty sure
> this (quite rarely) results in false readings because two processes
> try to read at the same time.
> 
> Thus I'm looking for ways to prevent simultaneous access.

Why using "different scripts"?
Is it there any particular reason?

Maybe it would be better, if possible, to have
a single script, which, sequentially, reads
whatever needs to be read (or written).
In a loop.

This is even simpler than using a file.

bye,

pg

> One fairly obvious way is to have single process/script which reads
> the A2D values continuously and writes them to a file.  All other
> scripts then read from the file as needed, a simple file lock can then
> be used to prevent simultaneous access (well, simultaneous access when
> the writing process is writing).
> 
> Is this the simplest approach?  Are there better ways using
> multiprocess?  (They look more complicated though).
> 
> The I2C bus itself has a mutex but I don't think this guarantees that
> (for example) an A2D reading is atomic because one reading takes more
> than one I2C bus access.
> 
> Would a mutex of some sort around each I2C transaction (i.e. complete
> A2D reading) be a better way to go?
> 

-- 

piergiorgio

[toc] | [prev] | [next] | [standalone]

#196406

Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.removethis.de> wrote:
> On 06/07/2024 09.28, Chris Green wrote:
> > I have a Raspberry Pi in my boat that uses I2C to read a number of
> > voltages and currents (using ADS1115 A2D) so I can monitor the battery
> > condition etc.
> > 
> > At present various different scripts (i.e. processes) just read the
> > values using the I2C bus whenever they need to but I'm pretty sure
> > this (quite rarely) results in false readings because two processes
> > try to read at the same time.
> > 
> > Thus I'm looking for ways to prevent simultaneous access.
> 
> Why using "different scripts"?
> Is it there any particular reason?
> 
> Maybe it would be better, if possible, to have
> a single script, which, sequentially, reads
> whatever needs to be read (or written).
> In a loop.
> 
> This is even simpler than using a file.
> 
Yes, but it's conceptually (and programming wise) much simpler to have
separate scripts.  Some of them are simple 'on demand' scripts that I
run from the command line when I want to know something.  Others are
scripts that drive displays on control panels.

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#196390 — Re: Best (simplest) way to share data (Posting On Python-List Prohibited)

On Sat, 6 Jul 2024 08:28:41 +0100, Chris Green wrote:

> One fairly obvious way is to have single process/script which reads the
> A2D values continuously and writes them to a file.  All other scripts
> then read from the file as needed, a simple file lock can then be used
> to prevent simultaneous access (well, simultaneous access when the
> writing process is writing).

The thing with a file is, it persists even when the collector process is 
not running. Do you want data that persists when the collector process is 
not running?

Is this a history of values, or just a snapshot of current values? A 
history of values could be written to a database. Databases provide their 
own transactions and interlocking to prevent readers from reading partial 
updates.

If it’s a snapshot of current values, that does not persist when the 
collector process is not running, then why not just keep the data in the 
memory of the collector process, and have it concurrently listen on a 
socket for connections from readers requesting a copy of the current data?

[toc] | [prev] | [next] | [standalone]

#196407 — Re: Best (simplest) way to share data (Posting On Python-List Prohibited)

Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> On Sat, 6 Jul 2024 08:28:41 +0100, Chris Green wrote:
> 
> > One fairly obvious way is to have single process/script which reads the
> > A2D values continuously and writes them to a file.  All other scripts
> > then read from the file as needed, a simple file lock can then be used
> > to prevent simultaneous access (well, simultaneous access when the
> > writing process is writing).
> 
> The thing with a file is, it persists even when the collector process is 
> not running. Do you want data that persists when the collector process is 
> not running?
> 
> Is this a history of values, or just a snapshot of current values? A 
> history of values could be written to a database. Databases provide their 
> own transactions and interlocking to prevent readers from reading partial 
> updates.
> 
There's a separate (crontab driven) process that writes the history to
a sqlite3 database,


> If it’s a snapshot of current values, that does not persist when the 
> collector process is not running, then why not just keep the data in the 
> memory of the collector process, and have it concurrently listen on a 
> socket for connections from readers requesting a copy of the current data?

That's exactly the sort of solution I was wondering about.  Is there a
ready made module/library for handling this sort of thing?  Basically
it will just be a string of a few tens of characters that would be
kept up to date by one process and asked for by all the others.

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#196415 — Re: Best (simplest) way to share data (Posting On Python-List Prohibited)

On Mon, 8 Jul 2024 13:56:34 +0100, Chris Green wrote:

> Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>
>> If it’s a snapshot of current values, that does not persist when the
>> collector process is not running, then why not just keep the data in
>> the memory of the collector process, and have it concurrently listen on
>> a socket for connections from readers requesting a copy of the current
>> data?
> 
> That's exactly the sort of solution I was wondering about.  Is there a
> ready made module/library for handling this sort of thing?  Basically it
> will just be a string of a few tens of characters that would be kept up
> to date by one process and asked for by all the others.

You probably don’t need to define any nontrivial kind of protocol at all. 
Just accept a connection from a client, send it the current data in some 
convenient format (e.g. JSON), and disconnect.

[toc] | [prev] | [next] | [standalone]

#196418 — Re: Best (simplest) way to share data

Stefan Ram <ram@zedat.fu-berlin.de> wrote:
> Chris Green <cl@isbd.net> wrote or quoted:
> >That's exactly the sort of solution I was wondering about.  Is there a
> >ready made module/library for handling this sort of thing?  Basically
> >it will just be a string of a few tens of characters that would be
> >kept up to date by one process and asked for by all the others.
> 
>   I'm not an expert here, and just quickly tried to make it
>   run, so the code will still contain errors and not contain
>   something necessary, but might give you a starting point.
> 
>   A process doing something (here: printing an incrementing value
>   named "info") and also serving requests from other processes
>   for this "info" value:
> 
[snip]

Thanks, that should get me started!  :-)

-- 
Chris Green
·

[toc] | [prev] | [next] | [standalone]

#196399

> On 7 Jul 2024, at 22:13, Chris Green via Python-list <python-list@python.org> wrote:
> 
> a simple file lock can then
> be used to prevent simultaneous access (well, simultaneous access when
> the writing process is writing).

There is a simple pattern to make this robust.

Write new values to a tmp file.
Close the tmp file.
Then use os.rename(tmpfile, productionfile).

This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents,  never a mix of both.

Barry

[toc] | [prev] | [next] | [standalone]

#196400

On 2024-07-07 23:27, Barry via Python-list wrote:
> 
> 
>> On 7 Jul 2024, at 22:13, Chris Green via Python-list <python-list@python.org> wrote:
>> 
>> a simple file lock can then
>> be used to prevent simultaneous access (well, simultaneous access when
>> the writing process is writing).
> 
> There is a simple pattern to make this robust.
> 
> Write new values to a tmp file.
> Close the tmp file.
> Then use os.rename(tmpfile, productionfile).
> 
> This is guaranteed that any process that reads the file will only see all the old file contents or all the new file contents,  never a mix of both.
> 
For clarity I'd recommend os.replace instead. This is because on Windows 
os.rename it would complain if the target file already exists, but 
os.replace has the same behaviour on both Linux and Windows.

[toc] | [prev] | [next] | [standalone]

#196405

> On 7 Jul 2024, at 23:47, MRAB via Python-list <python-list@python.org> wrote:
> 
> For clarity I'd recommend os.replace instead. This is because on Windows os.rename it would complain if the target file already exists, but os.replace has the same behaviour on both Linux and Windows.

Agreed.

In this case the OP is on an RPi.

Barry

[toc] | [prev] | [next] | [standalone]

#196409

If resource usage isn't an issue, then the _easy_ thing to do, that
would also be easily correct is to have a server doing all the
h/w-related reading and clients talking to that server. Use for the
server the technology you feel most confident with. Eg. you may use
Python's http package. I believe that the server from this package
runs in a single thread, and thus processes all requests
synchronously. So, you'll get synchronization for free.

Then, the rest of the scripts that need to talk to h/w will instead be
talking to this server.

Again, this isn't an _efficient_ solution... but, sometimes you don't
need one. And this one is easy to make, easy to debug, easy to expand.
But, if instead you were looking for a more efficient solution, then,
the general idea that allows the http server to work in this case
would still apply: have a single synchronization program that takes
requests asynchronously, and orders them. So, a basic TCP server would
also work as well as a UNIX socket. Your idea with holding a lock on a
file would also work (in fact, plenty of Linux utilities work that
way, eg. apt-get or yum).

----

If you don't want to change the existing script, then instead of
running them directly, you could run them through batch:
https://man7.org/linux/man-pages/man1/batch.1p.html this is a very
simply queuing program that's available for Linux. It will take care
of synchronization by putting the scripts you want to run in a queue
and executing them one at a time.

On Sun, Jul 7, 2024 at 11:12 PM Chris Green via Python-list
<python-list@python.org> wrote:
>
> I have a Raspberry Pi in my boat that uses I2C to read a number of
> voltages and currents (using ADS1115 A2D) so I can monitor the battery
> condition etc.
>
> At present various different scripts (i.e. processes) just read the
> values using the I2C bus whenever they need to but I'm pretty sure
> this (quite rarely) results in false readings because two processes
> try to read at the same time.
>
> Thus I'm looking for ways to prevent simultaneous access.
>
> One fairly obvious way is to have single process/script which reads
> the A2D values continuously and writes them to a file.  All other
> scripts then read from the file as needed, a simple file lock can then
> be used to prevent simultaneous access (well, simultaneous access when
> the writing process is writing).
>
> Is this the simplest approach?  Are there better ways using
> multiprocess?  (They look more complicated though).
>
> The I2C bus itself has a mutex but I don't think this guarantees that
> (for example) an A2D reading is atomic because one reading takes more
> than one I2C bus access.
>
> Would a mutex of some sort around each I2C transaction (i.e. complete
> A2D reading) be a better way to go?
>
> --
> Chris Green
> ·
> --
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web