Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #13665 > unrolled thread

a kind of a web-shell-api for managing long running processes

Back to article view | Back to comp.unix.programmer

Contents

  a kind of a web-shell-api for managing long running processes Meredith Montgomery <mmontgomery@levado.to> - 2021-12-30 17:16 -0300
    Re: a kind of a web-shell-api for managing long running processes scott@slp53.sl.home (Scott Lurndal) - 2021-12-30 21:55 +0000
      Re: a kind of a web-shell-api for managing long running processes Meredith Montgomery <mmontgomery@levado.to> - 2021-12-31 08:06 -0300
        Re: a kind of a web-shell-api for managing long running processes scott@slp53.sl.home (Scott Lurndal) - 2021-12-31 15:28 +0000
    Re: a kind of a web-shell-api for managing long running processes Jorgen Grahn <grahn+nntp@snipabacken.se> - 2022-01-01 00:46 +0000
    Re: a kind of a web-shell-api for managing long running processes Jorgen Grahn <grahn+nntp@snipabacken.se> - 2022-01-01 01:03 +0000

#13665 — a kind of a web-shell-api for managing long running processes

I'm helping someone to improve a CMS-type of web system.  This web
system sometimes needs to run some background processes --- such as send
a batch of e-mail or other process that takes a while to finish.

I see the challenge here as just writing something that will look like a
very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on.
(``Web'' because it will be used by a web system through some API.)

This thing must run really well.  It has to be flawless.  I'm looking
for design principles and advice.

(*) Where will it run

It will run on GNU systems running the Linux kernel.

(*) My own thoughts

The interface to shell will be through HTTP requests, so this shell will
likely be a web server of some sort.  But before I get involved in the
web at all, I need the shell working flawlessly.

So I need a laboratory first.  I could write a program that reads some
named pipe on disk to get commands such as ``run this or that'' while I
work.  (Later I can write a web server replacing this named-pipe
interface.)

Just like a UNIX shell, this web-api-shell must know all every process
it runs.  I suppose the work is essentially fork(), execve() followed by
waitpid().

One concern I have is the following.  Is it possible for a process to
simply ``get out of'' the shell?  What do I mean by that?  A process
that does fork() and puts itself into background would leave the
web-api-shell's control, wouldn't it?

I think I must avoid that.  Perhaps I can't let just any process run.
Perhaps the web-api-shell must only offer a few processes carefully
written by myself --- so that I know they won't put themselves in
background.  (For instance, I can't let them change PIDs, otherwise I
won't have any idea who they are and that's a mess.  I'd love to somehow
restrict system calls such as fork().)

(*) What is my level of training?

In the past I've studied many parts of

  Advanced Programming in the UNIX Environment
  W. Richard Stevens

I will definitely have to read it again to get work on this project.
Can you mention any UNIX concepts that are of great relevance for this
project?  I don't think I ever got my mind wrapped around things like
sessions, session leaders and so on.  Are these concepts relevant to
this application?

Thank you very much.

[toc] | [next] | [standalone]

#13666

Meredith Montgomery <mmontgomery@levado.to> writes:
>I'm helping someone to improve a CMS-type of web system.  This web
>system sometimes needs to run some background processes --- such as send
>a batch of e-mail or other process that takes a while to finish.
>
>I see the challenge here as just writing something that will look like a
>very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on.
>(``Web'' because it will be used by a web system through some API.)
>
>This thing must run really well.  It has to be flawless.  I'm looking
>for design principles and advice.
>
>(*) Where will it run
>
>It will run on GNU systems running the Linux kernel.
>
>(*) My own thoughts
>
>The interface to shell will be through HTTP requests, so this shell will
>likely be a web server of some sort.  But before I get involved in the
>web at all, I need the shell working flawlessly.
>
>So I need a laboratory first.  I could write a program that reads some
>named pipe on disk to get commands such as ``run this or that'' while I
>work.  (Later I can write a web server replacing this named-pipe
>interface.)
>
>Just like a UNIX shell, this web-api-shell must know all every process
>it runs.  I suppose the work is essentially fork(), execve() followed by
>waitpid().
>
>One concern I have is the following.  Is it possible for a process to
>simply ``get out of'' the shell?  What do I mean by that?  A process
>that does fork() and puts itself into background would leave the
>web-api-shell's control, wouldn't it?

Define 'get out of' in this context.   Any process forked by a
shell is by definition a child of the shell; the only way it can
become a child of some other process (pid=1 (init)) is for the
shell to "orphan" the child by exiting before the child exits.

The shell itself is known as a 'session leader'.  A session consists
of one or more 'process groups' (a concept developed originally in BSD4).
Each process group contains one or more processes.    One, and only
one, of the process groups within a session is considered the foreground
process group and is allowed access to the "controlling" terminal of the
session.  Background process groups run in the background until they need
to access the terminal, in which case a signal is thrown (e.g. SIGTTIN) and
the process will be suspended until the shell makes the process group the
foreground process group.

You'll find a great deal of useful information, if you can find a
copy, in:

 Session Management in System V Release 4
 Usenix Conference Proceedings Winter 1989 pp. 365-375.
 Tim Williams  (Tim was an engineer at AT&T/USL).

See setsid(2), getsid(2), setpgid(2), setpgrp(2), tcgetsid(3)
and other man pages for more information. 

https://pubs.opengroup.org/onlinepubs/9699919799/functions/setsid.html#
https://pubs.opengroup.org/onlinepubs/9699919799/functions/setpgid.html#

Note that the full source is available on-line for a dozen
or more open source shells (Korn shell, Bourne Again Shell, Bourne Shell,
C Shell, and many others);  useful to see how they handle
sessions.

[toc] | [prev] | [next] | [standalone]

#13667

scott@slp53.sl.home (Scott Lurndal) writes:

> Meredith Montgomery <mmontgomery@levado.to> writes:
>>I'm helping someone to improve a CMS-type of web system.  This web
>>system sometimes needs to run some background processes --- such as send
>>a batch of e-mail or other process that takes a while to finish.
>>
>>I see the challenge here as just writing something that will look like a
>>very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on.
>>(``Web'' because it will be used by a web system through some API.)
>>
>>This thing must run really well.  It has to be flawless.  I'm looking
>>for design principles and advice.
>>
>>(*) Where will it run
>>
>>It will run on GNU systems running the Linux kernel.
>>
>>(*) My own thoughts
>>
>>The interface to shell will be through HTTP requests, so this shell will
>>likely be a web server of some sort.  But before I get involved in the
>>web at all, I need the shell working flawlessly.
>>
>>So I need a laboratory first.  I could write a program that reads some
>>named pipe on disk to get commands such as ``run this or that'' while I
>>work.  (Later I can write a web server replacing this named-pipe
>>interface.)
>>
>>Just like a UNIX shell, this web-api-shell must know all every process
>>it runs.  I suppose the work is essentially fork(), execve() followed by
>>waitpid().
>>
>>One concern I have is the following.  Is it possible for a process to
>>simply ``get out of'' the shell?  What do I mean by that?  A process
>>that does fork() and puts itself into background would leave the
>>web-api-shell's control, wouldn't it?
>
> Define 'get out of' in this context.

Say the shell runs a process P and suppose P forks() --- creating Q ---
then P exits, leaving Q orphaned.  I believe Q's parent is now process
1.  I wonder if this creates difficulties to my objectives.  It's not
clear to me.  (Perhaps this project should use a PID namespace provided
by the Linux kernel, say.  Perhaps I'm worried about irrelevant things.
Maybe I should just run every process under a certain uid and search the
process table by that uid and these are the processes I should manage:
on the web, I will list out all current background processes and let the
sysadmin send signals to them --- kill them, for example.  I can't let
the sysadmin send signals to other [unrelated] processes and I should
always display every process that directly or indirectly was born by way
of this background-process-manager, so I must book-keep every process
spawned.)

> Any process forked by a shell is by definition a child of the shell;
> the only way it can become a child of some other process (pid=1
> (init)) is for the shell to "orphan" the child by exiting before the
> child exits.

Written down.  Thanks.

> The shell itself is known as a 'session leader'.  A session consists
> of one or more 'process groups' (a concept developed originally in BSD4).
> Each process group contains one or more processes.    One, and only
> one, of the process groups within a session is considered the foreground
> process group and is allowed access to the "controlling" terminal of the
> session.  Background process groups run in the background until they need
> to access the terminal, in which case a signal is thrown (e.g. SIGTTIN) and
> the process will be suspended until the shell makes the process group the
> foreground process group.
>
> You'll find a great deal of useful information, if you can find a
> copy, in:
>
>  Session Management in System V Release 4
>  Usenix Conference Proceedings Winter 1989 pp. 365-375.
>  Tim Williams  (Tim was an engineer at AT&T/USL).
>
> See setsid(2), getsid(2), setpgid(2), setpgrp(2), tcgetsid(3)
> and other man pages for more information. 
>
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/setsid.html#
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/setpgid.html#
>
> Note that the full source is available on-line for a dozen
> or more open source shells (Korn shell, Bourne Again Shell, Bourne Shell,
> C Shell, and many others);  useful to see how they handle
> sessions.

Thank you so much for info.  Very appreciated.

[toc] | [prev] | [next] | [standalone]

#13668

Meredith Montgomery <mmontgomery@levado.to> writes:
>scott@slp53.sl.home (Scott Lurndal) writes:
>

>>>One concern I have is the following.  Is it possible for a process to
>>>simply ``get out of'' the shell?  What do I mean by that?  A process
>>>that does fork() and puts itself into background would leave the
>>>web-api-shell's control, wouldn't it?
>>
>> Define 'get out of' in this context.
>
>Say the shell runs a process P and suppose P forks() --- creating Q ---
>then P exits, leaving Q orphaned.  I believe Q's parent is now process
>1.  I wonder if this creates difficulties to my objectives. 

Process Q is still part of the shell "session" until it explicitly
starts a new session (setsid).   Consider the case where process Q
is another shell.

What you probably want is to run your application in a container.

https://en.wikipedia.org/wiki/LXC

For the case you enumerated (background tasks) you may want
to consider a cooperative tasking setup where multiple independent
processes communicate (e.g. the web system communicates through
some form of inter process communication (pipes, unix domain sockets,
network sockets, FIFO's or posix message queues) with a mailer or other
process rather than forking and exec'ing each time you need to perform
the task).

[toc] | [prev] | [next] | [standalone]

#13669

On Thu, 2021-12-30, Meredith Montgomery wrote:
> I'm helping someone to improve a CMS-type of web system.  This web
> system sometimes needs to run some background processes --- such as send
> a batch of e-mail or other process that takes a while to finish.

The right way to send mail is to pipe it into sendmail(8); I don't
think that takes very long.  I'm mostly bringing this up because it
could perhaps serve as a pattern for some of the other tasks. And
because I hate mail done badly.

Using sendmail(8) means it's fire-and-forget.  The local MTA takes
responsibility for delivery, does a really good job, and most failures
end up as bounced mails.

[snip]

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [next] | [standalone]

#13670

On Thu, 2021-12-30, Meredith Montgomery wrote:
> I'm helping someone to improve a CMS-type of web system.  This web
> system sometimes needs to run some background processes --- such as send
> a batch of e-mail or other process that takes a while to finish.
>
> I see the challenge here as just writing something that will look like a
> very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on.
> (``Web'' because it will be used by a web system through some API.)
>
> This thing must run really well.  It has to be flawless.  I'm looking
> for design principles and advice.
>
> (*) Where will it run
>
> It will run on GNU systems running the Linux kernel.
>
> (*) My own thoughts
>
> The interface to shell will be through HTTP requests, so this shell will
> likely be a web server of some sort.  But before I get involved in the
> web at all, I need the shell working flawlessly.
...

I don't know CMS-type web systems at all, but haven't web servers done
this since CGI was invented, in the early 1990s?  There's always been
a need to run Unix processes as part of serving HTTP requests, so
there must be a wealth of patterns and frameworks already.  Even
though I suspect a lot of the frameworks want you to reimplement that
Unix command you want to run in Node.js, or whatever's fashionable
this week.

I'm tired, and haven't read very carefully, but I don't think you've
explained your requirements very well.  You're saying this thing is
like the Unix shell.  Then why not let it be the Unix shell?  I don't
see how writing your own with a HTTP interface instead of pipes would
help[0].  You're just stuck with a socket to manage instead of a pipe.

If you want to run it on some other host, there's always ssh.

/Jorgen

[0] Except it would be more fashionable.

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

[toc] | [prev] | [standalone]

Back to top | Article view | comp.unix.programmer

csiph-web