Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #24299 > unrolled thread

Embedded Linux: share data among different processes

Back to article view | Back to comp.arch.embedded

Contents

  Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-23 13:52 +0200
    Re: Embedded Linux: share data among different processes Jack <jack4747@gmail.com> - 2016-06-23 05:30 -0700
      Re: Embedded Linux: share data among different processes Reinhardt Behm <rbehm@hushmail.com> - 2016-06-23 21:05 +0800
        Re: Embedded Linux: share data among different processes Clifford Heath <no.spam@please.net> - 2016-06-24 09:45 +1000
          Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-23 17:19 -0700
            Re: Embedded Linux: share data among different processes Clifford Heath <no.spam@please.net> - 2016-06-24 13:25 +1000
              Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-23 21:27 -0700
                Re: Embedded Linux: share data among different processes Clifford Heath <no.spam@please.net> - 2016-06-24 14:40 +1000
                  Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-23 22:20 -0700
                    Re: Embedded Linux: share data among different processes Clifford Heath <no.spam@please.net> - 2016-06-24 17:19 +1000
                      Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 01:01 -0700
                  Re: Embedded Linux: share data among different processes Dimiter_Popoff <dp@tgi-sci.com> - 2016-06-24 16:38 +0300
                    Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 11:13 -0700
          Re: Embedded Linux: share data among different processes Reinhardt Behm <rbehm@hushmail.com> - 2016-06-24 12:39 +0800
            Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 01:53 -0700
              Re: Embedded Linux: share data among different processes Reinhardt Behm <rbehm@hushmail.com> - 2016-06-26 09:34 +0800
                Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-26 00:30 -0700
          Re: Embedded Linux: share data among different processes Les Cargill <lcargill99@comcast.com> - 2016-06-24 15:18 -0500
    Re: Embedded Linux: share data among different processes Mel Wilson <mwilson@the-wire.com> - 2016-06-23 13:47 +0000
      Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-24 13:36 +0200
        Re: Embedded Linux: share data among different processes Mel Wilson <mwilson@the-wire.com> - 2016-06-26 15:25 +0000
    Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-23 12:11 -0700
      Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-24 13:40 +0200
        Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 10:44 -0700
    Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-23 12:31 -0700
      Re: Embedded Linux: share data among different processes George Neuner <gneuner2@comcast.net> - 2016-06-23 17:27 -0400
      Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-24 13:43 +0200
        Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 10:48 -0700
          Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-27 14:46 +0200
            Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-27 13:41 -0700
        Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-24 13:52 -0700
          Re: Embedded Linux: share data among different processes Les Cargill <lcargill99@comcast.com> - 2016-06-24 16:42 -0500
          Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-27 16:00 +0200
            Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-27 20:06 -0700
              Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-28 08:41 +0200
                Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-28 00:27 -0700
                  Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-28 12:50 +0200
                    Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-28 10:01 -0700
                      Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-29 13:10 +0200
                        Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-29 10:47 -0700
                      Re: Embedded Linux: share data among different processes edward.ming.lee@gmail.com - 2016-06-29 14:01 -0700
                        Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-30 15:11 +0200
                          Re: Embedded Linux: share data among different processes edward.ming.lee@gmail.com - 2016-06-30 08:19 -0700
    Re: Embedded Linux: share data among different processes Les Cargill <lcargill99@comcast.com> - 2016-06-23 14:53 -0500
      Re: Embedded Linux: share data among different processes Paul Rubin <no.email@nospam.invalid> - 2016-06-23 13:05 -0700
      Re: Embedded Linux: share data among different processes pozz <pozzugno@gmail.com> - 2016-06-24 13:54 +0200
        Re: Embedded Linux: share data among different processes Les Cargill <lcargill99@comcast.com> - 2016-06-24 14:27 -0500
    Re: Embedded Linux: share data among different processes Les Cargill <lcargill99@comcast.com> - 2016-06-24 16:39 -0500
      Re: Embedded Linux: share data among different processes Don Y <blockedofcourse@foo.invalid> - 2016-06-24 15:11 -0700

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#24384

On Fri, 24 Jun 2016 13:36:36 +0200, pozz wrote:

> Il 23/06/2016 15:47, Mel Wilson ha scritto:
>> On Thu, 23 Jun 2016 13:52:21 +0200, pozz wrote:
>>
[ ... ]
>>> I think a good approach will be to develop a simple application, the
>>> poller, that communicates with the electronic device and implements
>>> the RS485 protocol. The poller continuously acquires the current
>>> status/settings of the device and store them in some "shared" way.
>>>
>>> The graphic application (QT-based) and the web server (CGI) should
>>> access to the data retrieved by the poller.
>>>
>>> What is the best method to share the data generated by an application
>>> (the poller) among two or more applications (QT and CGI)?
>>> In this scenario, I think it's important to lock the "shared data"
>>> before accessing them (reading or writing), in order to avoid reading
>>> incoerent data. Indeed, if the poller writes the data at the same time
>>> (Linux OS is multi-tasking) the web server reads them, they could be
>>> incoerent.
>>>
>>> I'm thinking to use SQLite database to store the data. The poller
>>> writes the database, HTTP and QT reads from it. It seems SQLite will
>>> take care the multi-thread/multi-process scenario.
>>>
>>> Any suggestions?
>>
>> I did that with System V IPC, using message passing for control
> 
> Yes, another problem to solve is the other way: QT and HTTP server could
> send some control commands to the electronic device (switch on, switch
> off, change this...).
> 
> 
>> and shared memory blocks for info common to all the processes.  I was
>> able to avoid any locking using the one-writer/many-readers trick.
> 
> It is exactly what I will have: one writer (the poller that will
> continuously fetch updated data from the device) and some readers (at
> least, HTTP server and QT graphic libraries).
> Doesn't it needed a syncronization mechanism (semaphore) in this
> scenario, using shared memory to share common data?

Just in case I should mention -- my system didn't contain just one Big 
Writer.  Rather, different processes were responsible for different 
sections of the application, and each of those processes was the sole 
writer for the shared data concerned with its section.

In this system, the Linux did the "cerebral" operations, and it 
communicated over a serial link with a "hind-brain" or "spinal column" 
run by a PIC32.  There was a polling process that communicated with the 
PIC32 and dispatched input via IPC messages to the responsible 
processes.  Some processes did quasi-real-time operations and, along with 
the polling process, ran at negative niceness for fastest response.  
Others could be given a twentieth of a second or so to respond and ran as 
normal processes.  But each task was the sole writer of shared data that 
it "owned" and ordered any other changes through IPC messages.

There were a bunch of Apache/CGI processes concerned with displaying 
diagnostic info remotely, and they never wrote shared data.  Some other 
CGI that implemented remote testing and control did their thing by 
sending IPC messages to the responsible processes.


	Mel.

[toc] | [prev] | [next] | [standalone]

#24304

On 6/23/2016 4:52 AM, pozz wrote:
> I'm new to embedded Linux so this question could be very simple for many of
> you. Most probably, it isn't directly related to embedded world, but to Linux
> OS generally. Anyway I think it is a common scenario in embedded applications.
>
> I'm going to develop a local/remote control of an electronic device. It
> communicates through a RS485 link.
> The local control will be a touch-screen display (I'm going to use QT graphic
> libraries).
> The remote control will be HTTP (web server).
>
> I think a good approach will be to develop a simple application, the poller,
> that communicates with the electronic device and implements the RS485 protocol.
> The poller continuously acquires the current status/settings of the device and
> store them in some "shared" way.
>
> The graphic application (QT-based) and the web server (CGI) should access to
> the data retrieved by the poller.
>
> What is the best method to share the data generated by an application (the
> poller) among two or more applications (QT and CGI)?
> In this scenario, I think it's important to lock the "shared data" before
> accessing them (reading or writing), in order to avoid reading incoerent data.
> Indeed, if the poller writes the data at the same time (Linux OS is
> multi-tasking) the web server reads them, they could be incoerent.
>
> I'm thinking to use SQLite database to store the data. The poller writes the
> database, HTTP and QT reads from it. It seems SQLite will take care the
> multi-thread/multi-process scenario.
>
> Any suggestions?

It depends on how "lightweight" you want the mechanism to be.  And,
how "elegant".

If you can implement the "Shared Memory" extensions (a kernel option
in the *BSD's -- not sure re: Linux) then this is the easiest way
forward.  Look for shmget(2), shmat(2), shmctl(2), et al.

Use a semaphore to control access (and discipline for thereaders/writers
to follow that convention) and keep atomic regions short and sweet.

If the data isn't changing all that often, you can push it to each
consumer through pipes and count on them to keep their own copies
intact.  (here, there be dragons)

The DBMS approach is how I am currently addressing all "persistent
data".  It's a fair bit heavier-handed (I've got lots of MIPS to spare)
but is more elegant; a producer can't inject data into the store
unless the data is appropriate for the tables/fields involved
(you can't store text somewhere that expects numerics!).  Likewise,
the consumers need not *check* the data because the checks can be
applied on admission to the DB.  I.e., if the data is *in* the
DB, then you know it satisfied the admission criteria!

However, I don't think SQLite has all of these hooks.

GNeuner is your go-to guy for the DB approach...

[toc] | [prev] | [next] | [standalone]

#24321

Il 23/06/2016 21:11, Don Y ha scritto:
> On 6/23/2016 4:52 AM, pozz wrote:
>> I'm new to embedded Linux so this question could be very simple for
>> many of
>> you. Most probably, it isn't directly related to embedded world, but
>> to Linux
>> OS generally. Anyway I think it is a common scenario in embedded
>> applications.
>>
>> I'm going to develop a local/remote control of an electronic device. It
>> communicates through a RS485 link.
>> The local control will be a touch-screen display (I'm going to use QT
>> graphic
>> libraries).
>> The remote control will be HTTP (web server).
>>
>> I think a good approach will be to develop a simple application, the
>> poller,
>> that communicates with the electronic device and implements the RS485
>> protocol.
>> The poller continuously acquires the current status/settings of the
>> device and
>> store them in some "shared" way.
>>
>> The graphic application (QT-based) and the web server (CGI) should
>> access to
>> the data retrieved by the poller.
>>
>> What is the best method to share the data generated by an application
>> (the
>> poller) among two or more applications (QT and CGI)?
>> In this scenario, I think it's important to lock the "shared data" before
>> accessing them (reading or writing), in order to avoid reading
>> incoerent data.
>> Indeed, if the poller writes the data at the same time (Linux OS is
>> multi-tasking) the web server reads them, they could be incoerent.
>>
>> I'm thinking to use SQLite database to store the data. The poller
>> writes the
>> database, HTTP and QT reads from it. It seems SQLite will take care the
>> multi-thread/multi-process scenario.
>>
>> Any suggestions?
>
> It depends on how "lightweight" you want the mechanism to be.  And,
> how "elegant".
>
> If you can implement the "Shared Memory" extensions (a kernel option
> in the *BSD's -- not sure re: Linux) then this is the easiest way
> forward.  Look for shmget(2), shmat(2), shmctl(2), et al.
>
> Use a semaphore to control access (and discipline for thereaders/writers
> to follow that convention) and keep atomic regions short and sweet.
>
> If the data isn't changing all that often, you can push it to each
> consumer through pipes and count on them to keep their own copies
> intact.  (here, there be dragons)

There are some analog signals that could change every second. The poller 
will fetch new updated data about every one second. I would avoid 
pushing new data to consumers (HTTP server and QT libraries) every second.


> The DBMS approach is how I am currently addressing all "persistent
> data".  It's a fair bit heavier-handed (I've got lots of MIPS to spare)
> but is more elegant;

I will have a lot of spare MIPS too.

I don't need to save data in a persistent way (after a reboot, the 
poller will start to retrieve new data).
I thought to create the SQLite database file in a RAM/temporary 
filesystem. This should speed up the access and should avoid Flash 
memory corruption.


> a producer can't inject data into the store
> unless the data is appropriate for the tables/fields involved
> (you can't store text somewhere that expects numerics!).  Likewise,
> the consumers need not *check* the data because the checks can be
> applied on admission to the DB.  I.e., if the data is *in* the
> DB, then you know it satisfied the admission criteria!
>
> However, I don't think SQLite has all of these hooks.
>
> GNeuner is your go-to guy for the DB approach...

[toc] | [prev] | [next] | [standalone]

#24325

On 6/24/2016 4:40 AM, pozz wrote:
> Il 23/06/2016 21:11, Don Y ha scritto:
>> It depends on how "lightweight" you want the mechanism to be.  And,
>> how "elegant".
>>
>> If you can implement the "Shared Memory" extensions (a kernel option
>> in the *BSD's -- not sure re: Linux) then this is the easiest way
>> forward.  Look for shmget(2), shmat(2), shmctl(2), et al.
>>
>> Use a semaphore to control access (and discipline for thereaders/writers
>> to follow that convention) and keep atomic regions short and sweet.
>>
>> If the data isn't changing all that often, you can push it to each
>> consumer through pipes and count on them to keep their own copies
>> intact.  (here, there be dragons)
>
> There are some analog signals that could change every second. The poller will
> fetch new updated data about every one second. I would avoid pushing new data
> to consumers (HTTP server and QT libraries) every second.

Ask yourself what the finest granularity you need/want in the data.
Is it one big block of data that only makes sense as a whole?  Or,
lots of smaller blocks that can stand on their own?

E.g., if you were reporting meteorological data (wind speed, air temp,
humidity, barometric pressure, etc.), allowing <someone> to see *one*
of these updated without the others being simultaneously updated
could lead to a faulty forecast (because forecasts rely on the
interpretation of ALL of the results, in concert).

>> The DBMS approach is how I am currently addressing all "persistent
>> data".  It's a fair bit heavier-handed (I've got lots of MIPS to spare)
>> but is more elegant;
>
> I will have a lot of spare MIPS too.

"A lot" can have different meanings to different people  :>

> I don't need to save data in a persistent way (after a reboot, the poller will
> start to retrieve new data).
> I thought to create the SQLite database file in a RAM/temporary filesystem.
> This should speed up the access and should avoid Flash memory corruption.

You typically would NOT want to be updating those <things> in FLASH as
you'll wear out the FLASH in short order.

Note that data in a DBMS typically is "larger" than you would achieve
if storing in a single, typed variable (before adding the overhead of
indexes, etc.).  And, you will also require resources for the WAL, if
supported.

I.e., before you make this DBMS decision, you might want to throw together
a simple schema and store your dataset to see just how big it ends up.

You should also think about how you will be interacting with the data.
E.g., returning to the weather report analogy, if you are planning on
cherry picking one value at a time from the *set* of values, then
you risk reporting data that "doesn't make sense" -- because the
"temperature" you reported a moment ago doesn't correspond to the
"barometric pressure" you are reporting now.

You may need to approach the data transactionally and, if you don't want
to lock up the DBMS while your client process chugs along at *its*
pace, this might require you to introduce some transaction buffering
*between* the two entities -- which might not have been in your
original design plans.

Lastly, if you settle on that approach, "unconstrain your thinking"
in considering what you *could* do with data thusly "organized".
I found that lots of code that I'd have normally placed in the
application (i.e., client) can be pushed back into the RDBMS
and the way I *think* about the data that it supplies.

>> a producer can't inject data into the store
>> unless the data is appropriate for the tables/fields involved
>> (you can't store text somewhere that expects numerics!).  Likewise,
>> the consumers need not *check* the data because the checks can be
>> applied on admission to the DB.  I.e., if the data is *in* the
>> DB, then you know it satisfied the admission criteria!
>>
>> However, I don't think SQLite has all of these hooks.
>>
>> GNeuner is your go-to guy for the DB approach...

[toc] | [prev] | [next] | [standalone]

#24305

pozz <pozzugno@gmail.com> writes:
> I'm thinking to use SQLite database to store the data. The poller
> writes the database, HTTP and QT reads from it. It seems SQLite will
> take care the multi-thread/multi-process scenario.

This is the classic and probably easiest approach, though not the most
economical in terms of machine use.  There are other databases besides
sqlite that you can also consider.  How much data are you talking about?
Do you need stuff like persistence across reboots?

[toc] | [prev] | [next] | [standalone]

#24309

On Thu, 23 Jun 2016 12:31:55 -0700, Paul Rubin
<no.email@nospam.invalid> wrote:

>pozz <pozzugno@gmail.com> writes:
>> I'm thinking to use SQLite database to store the data. The poller
>> writes the database, HTTP and QT reads from it. It seems SQLite will
>> take care the multi-thread/multi-process scenario.
>
>This is the classic and probably easiest approach, though not the most
>economical in terms of machine use.  

SQLite can be run completely in RAM (backed by swap if applicable).  

Although not possible out-of-the-box, wIth some hacking it can be made
to run from a memory mapped file (for persistence).

>There are other databases besides sqlite that you can also consider. 

If you have relational data, I believe SQLite is the lightest weight
file based RDBMS available.  There are a handful of RAM-only libraries
that accept SQL-like query languages.

If you need only something simple such as key-value pairs, then there
are many possible implementations to choose from.

>How much data are you talking about?
>Do you need stuff like persistence across reboots?

Excellent questions.

George

[toc] | [prev] | [next] | [standalone]

#24322

Il 23/06/2016 21:31, Paul Rubin ha scritto:
> pozz <pozzugno@gmail.com> writes:
>> I'm thinking to use SQLite database to store the data. The poller
>> writes the database, HTTP and QT reads from it. It seems SQLite will
>> take care the multi-thread/multi-process scenario.
>
> This is the classic and probably easiest approach, though not the most
> economical in terms of machine use.  There are other databases besides
> sqlite that you can also consider.  How much data are you talking about?

The overall configuration of the device could be maximum 1kB (I think it 
will be around 100 bytes). However it doesn't change frequently, so it 
can be retrieved only at startup and only when it really changes.

The status data that changes frequently will be maximum 100-200 bytes (I 
think it will be less than 100 bytes).

> Do you need stuff like persistence across reboots?

No.

[toc] | [prev] | [next] | [standalone]

#24326

On 6/24/2016 4:43 AM, pozz wrote:
> Il 23/06/2016 21:31, Paul Rubin ha scritto:
>> pozz <pozzugno@gmail.com> writes:
>>> I'm thinking to use SQLite database to store the data. The poller
>>> writes the database, HTTP and QT reads from it. It seems SQLite will
>>> take care the multi-thread/multi-process scenario.
>>
>> This is the classic and probably easiest approach, though not the most
>> economical in terms of machine use.  There are other databases besides
>> sqlite that you can also consider.  How much data are you talking about?
>
> The overall configuration of the device could be maximum 1kB (I think it will
> be around 100 bytes). However it doesn't change frequently, so it can be
> retrieved only at startup and only when it really changes.
>
> The status data that changes frequently will be maximum 100-200 bytes (I think
> it will be less than 100 bytes).

Then, why don't you just design an event-driven interface and move the data
across the protection barrier when these "update events" occur?  Or, better,
when they occur AND are "of interest"?

The "client" can keep a set of "most recently endorsed parameters"
and an "empty" buffer into which any *new* parameters that are in transit
from the "server" are accumulated.  Once they have "arrived", call
*that* buffer the authoritative reference and treat the previous
reference as a "disposable buffer" for the next update.

>> Do you need stuff like persistence across reboots?
>
> No.

[toc] | [prev] | [next] | [standalone]

#24394

Il 24/06/2016 19:48, Don Y ha scritto:
> On 6/24/2016 4:43 AM, pozz wrote:
>> Il 23/06/2016 21:31, Paul Rubin ha scritto:
>>> pozz <pozzugno@gmail.com> writes:
>>>> I'm thinking to use SQLite database to store the data. The poller
>>>> writes the database, HTTP and QT reads from it. It seems SQLite will
>>>> take care the multi-thread/multi-process scenario.
>>>
>>> This is the classic and probably easiest approach, though not the most
>>> economical in terms of machine use.  There are other databases besides
>>> sqlite that you can also consider.  How much data are you talking about?
>>
>> The overall configuration of the device could be maximum 1kB (I think
>> it will
>> be around 100 bytes). However it doesn't change frequently, so it can be
>> retrieved only at startup and only when it really changes.
>>
>> The status data that changes frequently will be maximum 100-200 bytes
>> (I think
>> it will be less than 100 bytes).
>
> Then, why don't you just design an event-driven interface and move the data
> across the protection barrier when these "update events" occur?  Or,
> better,
> when they occur AND are "of interest"?
>
> The "client" can keep a set of "most recently endorsed parameters"
> and an "empty" buffer into which any *new* parameters that are in transit
> from the "server" are accumulated.  Once they have "arrived", call
> *that* buffer the authoritative reference and treat the previous
> reference as a "disposable buffer" for the next update.

Of course this is another approach, the opposite way.
The "poller"/"server" pushes new data to the "clients" when they are 
available, about every 1 second.

However I don't know if this is the best approach. In this case, the 
"server" should know exactly how many "clients" are, open a different 
"communication channel" (a pipe) with them and continuously (even if at 
a low frequency) transmits data on that channel.
Today the clients are two (QT local graphical interface and remote HTTP 
interface), but tomorrow? If I will need to add a new "client" (local or 
remote) I should touch the "poller"/"server" process.

I think the opposite approach is better. The "poller"/"server" has only 
one goal: retrieve updated parameters and store them in a share-friendly 
way to whatever "client" needs them.
I only need to choose the best sharing method between "poller"/"server" 
and "clients".

Usually "clients" will access shared data for reading, but occasionally 
they will need to send new parameters (through graphica touch-screen 
display or remote web pages interface) to the device.
This is another problem to solve...


>>> Do you need stuff like persistence across reboots?
>>
>> No.
>
>

[toc] | [prev] | [next] | [standalone]

#24398

On 6/27/2016 5:46 AM, pozz wrote:
> Il 24/06/2016 19:48, Don Y ha scritto:
>> On 6/24/2016 4:43 AM, pozz wrote:
>>> Il 23/06/2016 21:31, Paul Rubin ha scritto:
>>>> pozz <pozzugno@gmail.com> writes:
>>>>> I'm thinking to use SQLite database to store the data. The poller
>>>>> writes the database, HTTP and QT reads from it. It seems SQLite will
>>>>> take care the multi-thread/multi-process scenario.
>>>>
>>>> This is the classic and probably easiest approach, though not the most
>>>> economical in terms of machine use.  There are other databases besides
>>>> sqlite that you can also consider.  How much data are you talking about?
>>>
>>> The overall configuration of the device could be maximum 1kB (I think
>>> it will
>>> be around 100 bytes). However it doesn't change frequently, so it can be
>>> retrieved only at startup and only when it really changes.
>>>
>>> The status data that changes frequently will be maximum 100-200 bytes
>>> (I think it will be less than 100 bytes).
>>
>> Then, why don't you just design an event-driven interface and move the data
>> across the protection barrier when these "update events" occur?  Or,
>> better,
>> when they occur AND are "of interest"?
>>
>> The "client" can keep a set of "most recently endorsed parameters"
>> and an "empty" buffer into which any *new* parameters that are in transit
>> from the "server" are accumulated.  Once they have "arrived", call
>> *that* buffer the authoritative reference and treat the previous
>> reference as a "disposable buffer" for the next update.
>
> Of course this is another approach, the opposite way.
> The "poller"/"server" pushes new data to the "clients" when they are available,
> about every 1 second.

That's what I was saying, above.  The client (i.e., a thread acting for the
client) opens a socket, blocks on a FIFO, etc. waiting for an "update" to
appear on his doorstep.

Meanwhile, the "real" client (thread) behaves normally, using the LAST set
of data (the COMPLETE SET) that it had received.  When a new set arrives,
it arrives in a coherent *block* -- an entire set of data (not individual
parameters).

WHEN IT IS CONVENIENT FOR THE CLIENT -- and, whomever the client serves (i.e.,
the user) -- the client swaps the new set of data to replace the old set.
In an indivisible/atomic act.  This allows you some better control over
how you implement that "sharing" on the client side.

E.g., you can "pause" the web server, grab the harvested data, update
the dataset that the web server consults, then resume the web server.
You just have to make sure you don't allow the "harvesting" thread to
update that dataset (with yet another!) while you are using it to
update the web server's copy!

[It is conceivable that the "harvester" thread may already be waiting
for yet another "update" -- as there's no guarantee that the "client" will
use this most recent data set before another set becomes available!]

Consider the (Qt) user's viewpoint of the data set.  If he is engaged
in an HTTP "session" with your device, he may (or may not) want to see
the most recent data exposed to him "asynchronously".  E.g., he may be
changing the previous set AS DISPLAYED TO HIM and arranging for
all of the paraameters to be consistent with each other; he wouldn't
want you to alter some PORTION of those -- even if that update (that
he hasn't yet seen) is "more current".

The "opposite way" is for clients to *pull* data across; to effectively
say "give me X" (and X might be "the entire data set").  A *true* "server"
implementation could be sitting with open socket(s) waiting for these
IPC requests and *immediately* responding with the most recent (or
"most consistent"?) version of the sought data -- which could be "all".

This allows the client to *pull* the data AT A TIME WHEN IT IS CONVENIENT
FOR IT!  But, it pushes the "sharing" issue into the server.

[If the server (poller) isn't ready and waiting, then you risk the
case where the server hangs waiting for NEW incoming "remote" data
and the web server (the poller's client) also ends up waiting!]

Note that you have to be careful in either implementation.  E.g.,
something like this set of threads:

Producer()
     ...
     acquireMutex(NAME)
     ...
     write(FIFO,...)
     ...
     releaseMutex(NAME)
     ...


Consumer()
     ...
     acquireMutex(NAME)
     ...
     read(FIFO)
     ...
     releaseMutex(NAME)
     ...


Other()
     ...
     acquireMutex(NAME)
     ...
     releaseMutex(NAME)
     ...

can get into deadlock -- regardless of the cleverness of the scheduling
algorithm!  (the FIFO operations being any sort of IPC; mutex a sort of
semaphore)

> However I don't know if this is the best approach. In this case, the "server"
> should know exactly how many "clients" are, open a different "communication
> channel" (a pipe) with them and continuously (even if at a low frequency)
> transmits data on that channel.

The easier way, IMO, is to let clients "express an interest" -- instead of
forcing the producer to know who all of the clients are at any given
moment in time.

Shared memory implicitly does this (only clients INTERESTED in the data will
poke around in the shared memory!  The producer doesn't care how many
of them there might be)

Likewise, clients opening connections (IPC's) to the "poller" explicitly
tell it who its consumers are.  A client uninterested in its data will
simply NOT open such a connection.

> Today the clients are two (QT local graphical interface and remote HTTP
> interface), but tomorrow? If I will need to add a new "client" (local or
> remote) I should touch the "poller"/"server" process.

IMO, you should *not* have to touch the poller!  It's job hasn't changed!

> I think the opposite approach is better. The "poller"/"server" has only one
> goal: retrieve updated parameters and store them in a share-friendly way to
> whatever "client" needs them.
> I only need to choose the best sharing method between "poller"/"server" and
> "clients".

You still have a sharing/locking problem in the client -- or in the server.
There are different versions of the data "live" in your system at any
given time:
- the data that the Qt GUI is using
- the data that the HTTPd interface is exporting
- the data that the poller is CURRENTLY acquiring (from the CAN? nodes)
- the most recent *set* of data that the poller had ALREADY acquired
- etc.

So, you have lots of opportunities for the system to be in an inconsistent
state.  E.g., imagine (for want of a better example as I have no idea of
the actual nature of the data that you are processing) that one of your
parameters is "today's date" (fetched from a remote node via your CAN?
interface).

The HTTPd client (web server) may have one notion of today's date -- because
it has deliberately NOT updated this in case the web user might be in the
process of updating the current *time* and would be disturbed to find the date
having changed while he was doing that.

The Qt GUI user can be in a similar situation -- maybe the Qt session began
before the HTTPd session and the date had a different value, at that time!
It was changed (by something) AFTER the GUI session started so the GUI is
technically showing stale data.  But, the HTTPd session, started after that
date-change, is showing a "more recent" version (which might *also* be stale!)

The poller has the "last date" successfully retrieved (from that remote
"clock" node).  But, happens to be retrieving an updated date, at this
very moment!

Plus, any "dates" stuck in pipes/IPC's awaiting delivery...

Plus, any changes in the date that are happening IN the "remote node"
(time marches on -- 23:59:00 becomes "tomorrow" in short order!)

> Usually "clients" will access shared data for reading, but occasionally they
> will need to send new parameters (through graphica touch-screen display or
> remote web pages interface) to the device.
> This is another problem to solve...

Another way of addressing this is with a token passing implementation.
The poller (?) builds a packet of data.  Then, sends it to client#1.
Client#1 extracts any values of interest and, possibly, modifies any
parameters that it considers as important.  Then, forwards the packet
to client#2 -- who does the same sort of things.  Eventually, the entire
"circuit" is traversed.  A second trip is made around the circuit in
which each "participant" compares the newly received packet to a
saved copy of the packet that *it* passed along previously.  It extracts
any parameters that differ to ensure its "copy" reflects this new packet
that it will be propagating.  When the second trip is complete, everyone
is known to have the same data and they can each use it.

[This is tricky to do and can be very brittle and nonperformant]

[toc] | [prev] | [next] | [standalone]

#24340

pozz <pozzugno@gmail.com> writes:
> The status data that changes frequently will be maximum 100-200 bytes
> (I think it will be less than 100 bytes).

If "frequently" means less than a few times per second and you have
enough hardware resources, you'll probably have an easier time writing
this as if it were a desktop or web app than using a traditional
embedded approach.  That can even including writing the host part in a
server-side scripting language instead of something like C.

[toc] | [prev] | [next] | [standalone]

#24343

Paul Rubin wrote:
> pozz <pozzugno@gmail.com> writes:
>> The status data that changes frequently will be maximum 100-200 bytes
>> (I think it will be less than 100 bytes).
>
> If "frequently" means less than a few times per second and you have
> enough hardware resources, you'll probably have an easier time writing
> this as if it were a desktop or web app than using a traditional
> embedded approach.  That can even including writing the host part in a
> server-side scripting language instead of something like C.
>


Yeah - I put up a link to a Go solution that looks great.

So many languages, so little time...


-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#24395

Il 24/06/2016 22:52, Paul Rubin ha scritto:
> pozz <pozzugno@gmail.com> writes:
>> The status data that changes frequently will be maximum 100-200 bytes
>> (I think it will be less than 100 bytes).
>
> If "frequently" means less than a few times per second and you have
> enough hardware resources, you'll probably have an easier time writing
> this as if it were a desktop or web app than using a traditional
> embedded approach.

I'm not an expert of desktop or web app... some starting points?

> That can even including writing the host part in a
> server-side scripting language instead of something like C.
>

[toc] | [prev] | [next] | [standalone]

#24401

pozz <pozzugno@gmail.com> writes:
> I'm not an expert of desktop or web app... some starting points?

https://wiki.python.org/moin/BeginnersGuide maybe.

Basically I mean: write in a scripting language, use high-level
libraries or services, and (within reason) don't worry about resource
consumption.  What hardware are you using?

[toc] | [prev] | [next] | [standalone]

#24404

Il 28/06/2016 05:06, Paul Rubin ha scritto:
> pozz <pozzugno@gmail.com> writes:
>> I'm not an expert of desktop or web app... some starting points?
>
> https://wiki.python.org/moin/BeginnersGuide maybe.
>
> Basically I mean: write in a scripting language, use high-level
> libraries or services, and (within reason) don't worry about resource
> consumption.  What hardware are you using?
>

A SOM module with i.MX 6SoloX, 256MB RAM, 512MB Flash.

[toc] | [prev] | [next] | [standalone]

#24405

pozz <pozzugno@gmail.com> writes:
> A SOM module with i.MX 6SoloX, 256MB RAM, 512MB Flash.

Oh you're fine, that's far more powerful than any embedded thing I've
ever used.  It would been a fairly powerful server back in the day.  You
can run a standard web server stack on something like that, with nginx,
python, some kind of web client talking to the sensors, etc.  In fact do
you really need the Qt interface instead of just using a browser?

[toc] | [prev] | [next] | [standalone]

#24406

Il 28/06/2016 09:27, Paul Rubin ha scritto:
> pozz <pozzugno@gmail.com> writes:
>> A SOM module with i.MX 6SoloX, 256MB RAM, 512MB Flash.
>
> Oh you're fine, that's far more powerful than any embedded thing I've
> ever used.

Me too :-)


> It would been a fairly powerful server back in the day.  You
> can run a standard web server stack on something like that, with nginx,

I'm going to use lighttpd.  Any reason to use nginx instead?


> python, some kind of web client talking to the sensors, etc.  In fact do
> you really need the Qt interface instead of just using a browser?

The web browser will be used from a remote location.

Do you mean to use the local display to show the windows of a web 
browser running on the embedded Linux?
This could be fine, because the interface will be the same (for the 
user... and for the developer).

Some questions:
- do I need a windows desktop environment (X.org, ...)?
- could I force the web browser window to be full-screen?
   I would avoid to see the Close Window button (the X button),
   the minimize button, the back button and so on...

[toc] | [prev] | [next] | [standalone]

#24413

pozz <pozzugno@gmail.com> writes:
> I'm going to use lighttpd.  Any reason to use nginx instead?

Lighttpd should be fine.

> Do you mean to use the local display to show the windows of a web
> browser running on the embedded Linux?

I meant have the local display run its own browser, viewing stuff served
by the embedded Linux.  I.e. the local display would be a small tablet.
Running a browser on the embedded Linux is another idea, hmm, might be
ok if the linux distro supports it.  Do you have an SD card slot on the
board?  A browser is pretty heavyweight.

> This could be fine, because the interface will be the same (for the
> user... and for the developer).

Yes :).

> - do I need a windows desktop environment (X.org, ...)?

No.  You'll have to run something like X on the local display if you
want to program it with Qt though.  Is this a consumer product where you
need a really slick interface, or is it something industrial where it's
more about functionality?

> - could I force the web browser window to be full-screen?
>   I would avoid to see the Close Window button (the X button),
>   the minimize button, the back button and so on...

Yes, that is called "kiosk mode" and there are usually ways to do it.

[toc] | [prev] | [next] | [standalone]

#24442

Il 28/06/2016 19:01, Paul Rubin ha scritto:
> pozz <pozzugno@gmail.com> writes:
>> I'm going to use lighttpd.  Any reason to use nginx instead?
>
> Lighttpd should be fine.
>
>> Do you mean to use the local display to show the windows of a web
>> browser running on the embedded Linux?
>
> I meant have the local display run its own browser, viewing stuff served
> by the embedded Linux.  I.e. the local display would be a small tablet

Hmmm... the cost will increase too much.


> Running a browser on the embedded Linux is another idea, hmm, might be
> ok if the linux distro supports it.  Do you have an SD card slot on the
> board?  A browser is pretty heavyweight.

No SD card, at the moment.


>> This could be fine, because the interface will be the same (for the
>> user... and for the developer).
>
> Yes :).
>
>> - do I need a windows desktop environment (X.org, ...)?
>
> No.  You'll have to run something like X on the local display if you
> want to program it with Qt though.

Are you sure? QT runs directly on the framebuffer too... I think.


> Is this a consumer product where you
> need a really slick interface, or is it something industrial where it's
> more about functionality?

Industrial.


>
>> - could I force the web browser window to be full-screen?
>>   I would avoid to see the Close Window button (the X button),
>>   the minimize button, the back button and so on...
>
> Yes, that is called "kiosk mode" and there are usually ways to do it.

I see, thank you.

[toc] | [prev] | [next] | [standalone]

#24455

pozz <pozzugno@gmail.com> writes:
>> the local display would be a small tablet
> Hmmm... the cost will increase too much.

If you've priced it out then ok.  Quantity must be pretty high for that.
For the thing I worked on, we did our own display unit but I'm sure it
would have been cheaper to use a tablet.

> Are you sure? QT runs directly on the framebuffer too... I think.

Hmm, ok.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web