Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25483 > unrolled thread

soft real-time on linux

Back to article view | Back to comp.arch.embedded

Contents

  soft real-time on linux Randy Yates <yates@digitalsignallabs.com> - 2016-11-12 08:53 -0500
    Re: soft real-time on linux Grant Edwards <invalid@invalid.invalid> - 2016-11-12 17:16 +0000
    Re: soft real-time on linux Mel Wilson <mwilson@the-wire.com> - 2016-11-12 17:33 +0000
    Re: soft real-time on linux upsidedown@downunder.com - 2016-11-12 20:34 +0200
    Re: soft real-time on linux Don Y <blockedofcourse@foo.invalid> - 2016-11-12 13:53 -0700
      Re: soft real-time on linux sbattazzo@gmail.com - 2017-02-09 11:06 -0800
    Re: soft real-time on linux Paul Rubin <no.email@nospam.invalid> - 2016-11-12 14:12 -0800
    Re: soft real-time on linux Phil Hobbs <pcdhSpamMeSenseless@electrooptical.net> - 2017-02-08 21:17 -0500
      Re: soft real-time on linux Jack <jack4747@gmail.com> - 2017-02-09 00:16 -0800
        Re: soft real-time on linux David Brown <david.brown@hesbynett.no> - 2017-02-09 10:00 +0100
        Re: soft real-time on linux Phil Hobbs <pcdhSpamMeSenseless@electrooptical.net> - 2017-02-10 11:12 -0500
      Re: soft real-time on linux George Neuner <gneuner2@comcast.net> - 2017-02-11 01:33 -0500
        Re: soft real-time on linux Phil Hobbs <pcdhSpamMeSenseless@electrooptical.net> - 2017-02-10 22:10 -0500

#25483 — soft real-time on linux

Hello All,

I have a requirement to port the MSTP physical layer (basically RS-485)
of the BACnet protocol stack to linux. The protocol requires certain
timing to be met, for example, the response to a "Poll For Master" must
be less than 20 milliseconds.

In order to develop this in as short an amount of time, I'd like to use
simple threads and read()/write() commands. However, the standard linux
kernel is not real-time, so any thread could be blocked for an unknown
amount of time. Thus for example a thread that is listening for a "Poll
For Master" packet might not run for more than 20 milliseconds after the
packet was received, thus missing the poll.

But missing a poll is not the end of the world. Eventually (and fairly
quickly) another PFM will be sent out and the device would have another
chance to respond.

Let us also presume such is the case for other response/time violations
throughout the protocol.

The question is, could such an application be written that was "close
enough to real-time" on a standard (non-real-time) kernel? 

I was told by someone on the ##kernel channel that if the kernel is
compiled with "preemptive" enabled, there is a way to set a thread's
priority to "real-time" and it will not only have priority over other
user-land threads but also will preempt much of the kernel's internal
threads. This sounds promising.

If this isn't "good enough" (which is not really well-defined at
this point), there are two alternatives that I can think of:

  1. Switch to the real-time kernel.

  2. Write our own interrupt handler/driver.

I'm not very confident as to which direction to go. Any input would be
appreciated.
-- 
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com

[toc] | [next] | [standalone]

#25484

On 2016-11-12, Randy Yates <yates@digitalsignallabs.com> wrote:

> I was told by someone on the ##kernel channel that if the kernel is
> compiled with "preemptive" enabled, there is a way to set a thread's
> priority to "real-time" and it will not only have priority over other
> user-land threads but also will preempt much of the kernel's internal
> threads. This sounds promising.

In my experience (on a 400MHz ARM9), if you build the kernel with all
the pre-emptable options, and run your task at the highest real-time
priority then latency between an external event that causes an
interrupt and your user task waking up and running is going to be
somwhere in the 100-300 microsecond range.  [Assuming the interrupt
routine and device driver require minimal processing.]

That, of course, depends on the exact set of drivers you have enabled.
All it takes is one long critical section in a poorly written driver
to run that into many hundreds of microseconds.

I would be shocked if it ever got up to 1ms.

Unless you disable the rx FIFO, I would guess that the vast majority
of the latency is going to be in the UART itself.  With the rxFIFO
enabled, there's often a delay of several tens of bit-times between
the last data byte and the receive interrupt.

-- 
Grant Edwards               grant.b.edwards        Yow! Zippy's brain cells
                                  at               are straining to bridge
                              gmail.com            synapses ...

[toc] | [prev] | [next] | [standalone]

#25485

On Sat, 12 Nov 2016 08:53:09 -0500, Randy Yates wrote:

> Hello All,
> 
> I have a requirement to port the MSTP physical layer (basically RS-485)
> of the BACnet protocol stack to linux. The protocol requires certain
> timing to be met, for example, the response to a "Poll For Master" must
> be less than 20 milliseconds.
> [ ... ]


I did something like that.  We  had a microcontroller sending packets of 
a couple of hundred bytes via a USB-serial link every 5ms containing 
state of the input hardware, and accepting similar packets meant to 
control output hardware.  We used straight-up Debian Linux on a VIA Epia 
board (chosen not for real-time ability but for low power consumption, 
reasonable speed, and lots of Ethernet and USB ports.)  Application was 
written in C with one process for each major sub-department, SysV IPC 
shared storage for common working data and configuration blocks, and SysV 
IPC messaging for coordination.  The most urgent processes (serial I/O 
and dispatching for one, and the core of the application for the other) 
ran with an assigned  niceness of -10 to get the dispatches they needed; 
what was left was enough for the other processes, given the processor we 
chose.  Everything marched to the 5ms beat of the input packets.  Testing 
showed all input being processed and responded to in the next output 
packet sent 5ms latter.

I've heard since that they've quickened the drum beat to 2ms, but I 
haven't heard whether they needed to change much code to get that done.

[toc] | [prev] | [next] | [standalone]

#25486

On Sat, 12 Nov 2016 08:53:09 -0500, Randy Yates
<yates@digitalsignallabs.com> wrote:

>
>I have a requirement to port the MSTP physical layer (basically RS-485)
>of the BACnet protocol stack to linux. The protocol requires certain
>timing to be met, for example, the response to a "Poll For Master" must
>be less than 20 milliseconds.

Not very hard requirements for any modern kernel (2.6 and later). 

The only thing that I would warn against is the RS-485 data direction
(Rx/Tx) control.  Do not try to use the RTS pin driven by a user mode
driver. Use proper hardware control for the Tx/Rx switching.

Of course with any half duplex protocols, a large pequest/response
latency will drop the effective throughput well below the theoretical
throughput.

[toc] | [prev] | [next] | [standalone]

#25487

On 11/12/2016 6:53 AM, Randy Yates wrote:
> Hello All,
>
> I have a requirement to port the MSTP physical layer (basically RS-485)
> of the BACnet protocol stack to linux. The protocol requires certain
> timing to be met, for example, the response to a "Poll For Master" must
> be less than 20 milliseconds.
>
> In order to develop this in as short an amount of time, I'd like to use
> simple threads and read()/write() commands. However, the standard linux
> kernel is not real-time, so any thread could be blocked for an unknown
> amount of time. Thus for example a thread that is listening for a "Poll
> For Master" packet might not run for more than 20 milliseconds after the
> packet was received, thus missing the poll.
>
> But missing a poll is not the end of the world. Eventually (and fairly
> quickly) another PFM will be sent out and the device would have another
> chance to respond.
>
> Let us also presume such is the case for other response/time violations
> throughout the protocol.
>
> The question is, could such an application be written that was "close
> enough to real-time" on a standard (non-real-time) kernel?
>
> I was told by someone on the ##kernel channel that if the kernel is
> compiled with "preemptive" enabled, there is a way to set a thread's
> priority to "real-time" and it will not only have priority over other
> user-land threads but also will preempt much of the kernel's internal
> threads. This sounds promising.
>
> If this isn't "good enough" (which is not really well-defined at
> this point), there are two alternatives that I can think of:
>
>   1. Switch to the real-time kernel.
>
>   2. Write our own interrupt handler/driver.
>
> I'm not very confident as to which direction to go. Any input would be
> appreciated.

If you don't want to rely on timeliness guarantees from the software,
then rely on those implicit in the hardware.  I.e., build a layer in
front of the existing RxISR that operates a trivial state machine that
does your PFM recognition.  You *know* that the Rx ISR will be serviced
before N characters have been received, at maximum bit rate (otherwise
characters are dropped and your application has bigger problems to address,
needs faster hardware, a re-tooling of the rest of the system tasks, etc.)
so that gives you an upper bound on the latency incurred by your modified
ISR.  This has the advantage of being somewhat portable (to different
OS's, different versions of the OS, different hardware capabilities, etc.).

And, as you're not doing ALL of your protocol handling/decoding in the
ISR, it's not as brittle as it would be, otherwise.

[It's akin to decoding XON/XOFF characters (a two bit FSM) in the Rx ISR
because the latency involved in doing it in userland would make it
impractical]

[toc] | [prev] | [next] | [standalone]

#25812

Xenomai does pretty much this, and it seems to work fairly well in my experience! It doesn't necessarily come with all the hardware drivers but in the worst case you can get away with doing some memmap'd I/O in userland if you don't want to spend the time to write formal device drivers.

On Saturday, November 12, 2016 at 12:53:31 PM UTC-8, Don Y wrote:

> If you don't want to rely on timeliness guarantees from the software,
> then rely on those implicit in the hardware.  I.e., build a layer in
> front of the existing RxISR that operates a trivial state machine ...

[toc] | [prev] | [next] | [standalone]

#25488

Randy Yates <yates@digitalsignallabs.com> writes:
> the response to a "Poll For Master" must be less than 20 milliseconds.
> ... I'd like to use simple threads and read()/write() commands.

Unless other processes are contending for the cpu, this shouldn't be too
much of a problem.  On a 300 mhz arm7 board a while back, I remember
timing a thread race and finding it could switch threads at around 20 khz.
It should be simple enough to set up such a test on your own system.

[toc] | [prev] | [next] | [standalone]

#25808

On 11/12/2016 08:53 AM, Randy Yates wrote:
> Hello All,
>
> I have a requirement to port the MSTP physical layer (basically RS-485)
> of the BACnet protocol stack to linux. The protocol requires certain
> timing to be met, for example, the response to a "Poll For Master" must
> be less than 20 milliseconds.
>
> In order to develop this in as short an amount of time, I'd like to use
> simple threads and read()/write() commands. However, the standard linux
> kernel is not real-time, so any thread could be blocked for an unknown
> amount of time. Thus for example a thread that is listening for a "Poll
> For Master" packet might not run for more than 20 milliseconds after the
> packet was received, thus missing the poll.
>
> But missing a poll is not the end of the world. Eventually (and fairly
> quickly) another PFM will be sent out and the device would have another
> chance to respond.
>
> Let us also presume such is the case for other response/time violations
> throughout the protocol.
>
> The question is, could such an application be written that was "close
> enough to real-time" on a standard (non-real-time) kernel?
>
> I was told by someone on the ##kernel channel that if the kernel is
> compiled with "preemptive" enabled, there is a way to set a thread's
> priority to "real-time" and it will not only have priority over other
> user-land threads but also will preempt much of the kernel's internal
> threads. This sounds promising.
>
> If this isn't "good enough" (which is not really well-defined at
> this point), there are two alternatives that I can think of:
>
>   1. Switch to the real-time kernel.
>
>   2. Write our own interrupt handler/driver.
>
> I'm not very confident as to which direction to go. Any input would be
> appreciated.
>

The Linux thread scheduler is horribly broken, or was last time I used 
it (a while ago, admittedly).  The info pages will tell you all about 
how to specify priorities, scheduling algorithms, and so forth, but 
leave out two vital items:

1. You can only hack those things if you're running as root, and
2. You can't have both real-time and normal threads in the same process.

Both (1) and (2) are easy in Windows (or the late lamented OS/2, which 
is where I learned multithreaded programming, 1992ish).

Note that for my purposes I'd have been totally happy if I could even 
_reduce_ the priority of some threads in my process to make clear which 
ones were less important than others.  Not trying to hog other users' 
gjhresources, no sir.  Still not doable as an ordinary user.

On the other hand, I think you can set processor affinities so as to 
reserve one or more cores for just your own code.  I hope so, because 
I'm looking at a similar requirement myself soon. ;)

Cheers

Phil Hobbs

[toc] | [prev] | [next] | [standalone]

#25810

Il giorno giovedì 9 febbraio 2017 03:17:31 UTC+1, Phil Hobbs ha scritto:
> 
> The Linux thread scheduler is horribly broken, or was last time I used 
> it (a while ago, admittedly).  The info pages will tell you all about 
> how to specify priorities, scheduling algorithms, and so forth, but 
> leave out two vital items:
> 
> 1. You can only hack those things if you're running as root, and

the priority of your processes can be modified with "nice".
If you want to change some kernel config of course you need to be root (same thing in any other OS)

> 2. You can't have both real-time and normal threads in the same process.

since threads are created/destroyed dinamically, and the OS can't know what the fuck is going on in the thread, it's pretty normal that the priority is process based and not thread based.

If you want to prioritize a particular thread in a process, you need to do it at programming time.

good old stackoverflow: http://stackoverflow.com/questions/3649281/how-to-increase-thread-priority-in-pthreads

> 
> Both (1) and (2) are easy in Windows (or the late lamented OS/2, which 

the (1) is "easy" in win because a normal user is automatically Administrator, if aou use a user without admin privileges, well you can change the priority of your processes, but not others.

Bye Jack

[toc] | [prev] | [next] | [standalone]

#25811

On 09/02/17 09:16, Jack wrote:
> Il giorno giovedì 9 febbraio 2017 03:17:31 UTC+1, Phil Hobbs ha
> scritto:
>> 
>> The Linux thread scheduler is horribly broken, or was last time I
>> used it (a while ago, admittedly).  The info pages will tell you
>> all about how to specify priorities, scheduling algorithms, and so
>> forth, but leave out two vital items:
>> 
>> 1. You can only hack those things if you're running as root, and
> 
> the priority of your processes can be modified with "nice". If you
> want to change some kernel config of course you need to be root (same
> thing in any other OS)
> 
>> 2. You can't have both real-time and normal threads in the same
>> process.
> 
> since threads are created/destroyed dinamically, and the OS can't
> know what the fuck is going on in the thread, it's pretty normal that
> the priority is process based and not thread based.
> 
> If you want to prioritize a particular thread in a process, you need
> to do it at programming time.
> 
> good old stackoverflow:
> http://stackoverflow.com/questions/3649281/how-to-increase-thread-priority-in-pthreads
>
> 
>> 
>> Both (1) and (2) are easy in Windows (or the late lamented OS/2,
>> which
> 
> the (1) is "easy" in win because a normal user is automatically
> Administrator, if aou use a user without admin privileges, well you
> can change the priority of your processes, but not others.
> 

And the (2) is "easy" in Windows because windows is poor at process
handling, and compensates by being quite good at thread handling.  Linux
(like other *nix) has always prioritised good process handling - it does
not /need/ such flexibility in thread prioritising, but that is simply
not the way you design programs in Linux.  If you want a program that
involves hard (or as hard as possible) real-time control of a motor,
soft real-time logging of the data, and non-real-time display on a gui,
you organise it differently on Windows and Linux.  With Windows, you put
everything in one process and have separate threads, because passing
data between processes is inefficient.  With Linux, you use three
separate processes with different priorities.

[toc] | [prev] | [next] | [standalone]

#25813

On 02/09/2017 03:16 AM, Jack wrote:
> Il giorno giovedì 9 febbraio 2017 03:17:31 UTC+1, Phil Hobbs ha
> scritto:
>>
>> The Linux thread scheduler is horribly broken, or was last time I
>> used it (a while ago, admittedly).  The info pages will tell you
>> all about how to specify priorities, scheduling algorithms, and so
>> forth, but leave out two vital items:
>>
>> 1. You can only hack those things if you're running as root, and
>
> the priority of your processes can be modified with "nice". If you
> want to change some kernel config of course you need to be root (same
> thing in any other OS)

Only process-by-process.
>
>> 2. You can't have both real-time and normal threads in the same
>> process.
>
> since threads are created/destroyed dinamically, and the OS can't
> know what the fuck is going on in the thread, it's pretty normal that
> the priority is process based and not thread based.

Not true in Windows, for instance.  It's a serious handicap in my case, 
where I have compute threads and comms threads in a clusterized 
simulator: another box can totally stall waiting for a comms thread on 
the next box to finish its job.  It trashes the scaling--Windows boxes 
scale nearly linearly up to 25 hosts or so, whereas Linux ones start 
getting less efficient at around 8 hosts.

On a single box, the thread scheduler has a global view of what's going 
on, so it isn't as big a deal.

>
> If you want to prioritize a particular thread in a process, you need
> to do it at programming time.
>
> good old stackoverflow:
> http://stackoverflow.com/questions/3649281/how-to-increase-thread-priority-in-pthreads
>
Only works as root, and if you want any thread to be real-time, you have 
to put the whole compute-bound process in the RT class, which brings the 
system to its knees.  What I need is a RT comms thread that is usually 
blocked on a semaphore and then runs furiously about 1% of the time.
>
>>
>> Both (1) and (2) are easy in Windows (or the late lamented OS/2,
>> which
>
> the (1) is "easy" in win because a normal user is automatically
> Administrator, if aou use a user without admin privileges, well you
> can change the priority of your processes, but not others.

But not the relative priority of threads within the process, whereas you 
can in Windows.  As I said originally, I'd be quite happy to be able 
just to _reduce_ the priority of the compute threads vs. the comms 
threads,  but _noooooooo_.  Broken as designed.

Cheers

Phil Hobbs


-- 
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC
Optics, Electro-optics, Photonics, Analog Electronics

160 North State Road #203
Briarcliff Manor NY 10510

hobbs at electrooptical dot net
http://electrooptical.net

[toc] | [prev] | [next] | [standalone]

#25823

On Wed, 8 Feb 2017 21:17:35 -0500, Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> wrote:


>The Linux thread scheduler is horribly broken, or was last time I used 
>it (a while ago, admittedly).  

It has improved quite a bit in the recent kernels.  
However ...


>The info pages will tell you all about 
>how to specify priorities, scheduling algorithms, and so forth, but 
>leave out two vital items:
>
>1. You can only hack those things if you're running as root, and
>2. You can't have both real-time and normal threads in the same process.

All of this still is true. 

The scheduler historically had problems dealing with overused and
underused cores, and it has been greatly improved of late.  However
control over scheduling has not changed significantly since 2.6.24
(circa 2008).


>Both (1) and (2) are easy in Windows (or the late lamented OS/2, which 
>is where I learned multithreaded programming, 1992ish).
>
>Note that for my purposes I'd have been totally happy if I could even 
>_reduce_ the priority of some threads in my process to make clear which 
>ones were less important than others.  Not trying to hog other users' 
>gjhresources, no sir.  Still not doable as an ordinary user.

Have you tried using clone(2)?
http://man7.org/linux/man-pages/man2/clone.2.html


Clone is a low level call that can create either new processes or new
(kernel) threads in the same process.  For new processes it gives
quite fine control over the execution environment.

You can, e.g., create a new process that *acts* as though it were a
thread: sharing memory (and/or other environment) with the parent, but
having a different priority, scheduling policy, process group, etc.

It is the best Linux can manage.


>On the other hand, I think you can set processor affinities so as to 
>reserve one or more cores for just your own code.  I hope so, because 
>I'm looking at a similar requirement myself soon. ;)

You can choose which cores will execute your code, but it is very
difficult to prevent other processes from also using your chosen
cores.

You can try to futz with group scheduling to achieve what you want,
but that is clumsy and difficult because it can involve manipulating
process *owners* as well as parent/child relationships and priorities.


I think the best you can do without inviting a whole lot of headache
is to try to refactor your (POSIX?) threaded application into a set of
clone processes.


>Cheers
>
>Phil Hobbs

George

[toc] | [prev] | [next] | [standalone]

#25825

On 02/11/2017 01:33 AM, George Neuner wrote:
> On Wed, 8 Feb 2017 21:17:35 -0500, Phil Hobbs
> <pcdhSpamMeSenseless@electrooptical.net> wrote:
>
>
>> The Linux thread scheduler is horribly broken, or was last time I used
>> it (a while ago, admittedly).
>
> It has improved quite a bit in the recent kernels.
> However ...
>
>
>> The info pages will tell you all about
>> how to specify priorities, scheduling algorithms, and so forth, but
>> leave out two vital items:
>>
>> 1. You can only hack those things if you're running as root, and
>> 2. You can't have both real-time and normal threads in the same process.
>
> All of this still is true.
>
> The scheduler historically had problems dealing with overused and
> underused cores, and it has been greatly improved of late.  However
> control over scheduling has not changed significantly since 2.6.24
> (circa 2008).
>
>
>> Both (1) and (2) are easy in Windows (or the late lamented OS/2, which
>> is where I learned multithreaded programming, 1992ish).
>>
>> Note that for my purposes I'd have been totally happy if I could even
>> _reduce_ the priority of some threads in my process to make clear which
>> ones were less important than others.  Not trying to hog other users'
>> gjhresources, no sir.  Still not doable as an ordinary user.
>
> Have you tried using clone(2)?
> http://man7.org/linux/man-pages/man2/clone.2.html
>
>
> Clone is a low level call that can create either new processes or new
> (kernel) threads in the same process.  For new processes it gives
> quite fine control over the execution environment.
>
> You can, e.g., create a new process that *acts* as though it were a
> thread: sharing memory (and/or other environment) with the parent, but
> having a different priority, scheduling policy, process group, etc.
>
> It is the best Linux can manage.
>
>
>> On the other hand, I think you can set processor affinities so as to
>> reserve one or more cores for just your own code.  I hope so, because
>> I'm looking at a similar requirement myself soon. ;)
>
> You can choose which cores will execute your code, but it is very
> difficult to prevent other processes from also using your chosen
> cores.
>
> You can try to futz with group scheduling to achieve what you want,
> but that is clumsy and difficult because it can involve manipulating
> process *owners* as well as parent/child relationships and priorities.
>
>
> I think the best you can do without inviting a whole lot of headache
> is to try to refactor your (POSIX?) threaded application into a set of
> clone processes.
=
> George
>
Interesting, thanks.  For most of my purposes nowadays I can run the 
simulator on one of my 24-core Opteron servers, running a single kernel 
instance, which greatly reduces the problem.  Back when I had a rack 
full of pizza-box servers, it was ugly.

Cheers

Phil Hobbs

-- 
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC
Optics, Electro-optics, Photonics, Analog Electronics

160 North State Road #203
Briarcliff Manor NY 10510

hobbs at electrooptical dot net
http://electrooptical.net

[toc] | [prev] | [standalone]

Back to top | Article view | comp.arch.embedded

csiph-web