Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.os.development > #8201 > unrolled thread

Re: How to bind HLLs to OS calls

Back to article view | Back to alt.os.development

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: How to bind HLLs to OS calls "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-06-18 02:10 -0400
    Re: How to bind HLLs to OS calls "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-06-19 06:19 -0400
      Re: How to bind HLLs to OS calls "James Harris" <james.harris.1@gmail.com> - 2015-06-22 21:22 +0100
        Re: How to bind HLLs to OS calls "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-06-23 19:24 -0400
          Re: How to bind HLLs to OS calls "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-06-24 22:15 -0400
      Re: How to bind HLLs to OS calls "James Harris" <james.harris.1@gmail.com> - 2015-06-24 07:55 +0100
        Re: How to bind HLLs to OS calls "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-06-24 21:00 -0400

#8201 — Re: How to bind HLLs to OS calls

On Wed, 17 Jun 2015 11:05:30 -0400, James Harris  
<james.harris.1@gmail.com> wrote:

> OK. Potentially big issue. Anyone up for it...?
>
> I'll start slow! ;-)
>
> I am not sure this is right so confirmation or correction would be  
> welcome but I recently discovered that C calls to OS functions don't  
> work as I expected. If a C program calls such as fopen as follows
>
>   FILE *f = fopen(....);
>
> I expected the fopen function code (which I'll refer to as the immediate  
> callee) to be in the C library. I still think that's right. But what  
> about OS functions such as plain open? For example, if C code has
>
>   int f = open(....);
>
> Where is the immediate callee code for that open function?

It's not part of the C compiler proper, so it's in a library somewhere.

There are usually only a handful of functions which are part of the
C library that are used within a C compiler  *without*  including the
C library.  They usually call these internal versions of the function.

> It is not a C function but an OS one. (I don't mean the OS code which  
> finally gets called but the immediate target of the "CALL open"  
> instruction.) I now think it is probably *also* in CLIB. Am I right?  
> That seems weird.

Are the include files for open() with or near the include files for  
fopen()?
I would expect them to be.  The actual C code to implement open() may
be located elsewhere from the code to implement fopen().

> If such OS functions are in CLIB then is that the right place for them?

It's probably the "OS library" integrated into the "CLIB".

You need at least 18 to 20 OS functions to implement a C library,
including the half dozen low-level Unix file I/O functions.

> Should CLIB contain functions that are specific to a given OS?

I believe GCC's CLIB does so.

> Or should OS functions be defined in something which is OS-specific
> rather than language-specific?

As far as I'm concerned, the compiler just needs to find them.
The headers should be all in one place.

> What if we want to call an OS function from another language, say Pascal?

There would need to be code to convert the Pascal call to the OS call.

> Would there be a Pascal library which *also* has to match
> the target OS?

I would assume so.

> If we make a new OS which has its own API would we have to get our  
> function callees also into CLIB and Pascal's library [...]

Yes.  The C library is build on the OS API.  At minimum, eighteen
to twenty OS functions are needed.  I posted these to a.o.d. in 2008:

K&R C, e.g., PDP-7 and/or PDP-11 UNIX C libraries:
open, close, read, write, creat, link, fork, exec, wait, exit, lseek, pipe

PJ Plauger's "Standard C Library" book:
clock, close, environ, execl, exit, fork, getpid, kill, link, lseek, open,  
read, sbrk, signal, time, unlink, wait, write

Redhat's newlib:
close, environ, _exit, execve, fork, fstat, getpid, isatty, kill, link,  
lseek, open, read, sbrk, stat, times, unlink, wait, write

You might also want to add "unlink" for the K&R C set ...

> or where should the new OS's function immediate callee's reside?
...

> What if we want to call an OS from assembly language code?

OMG!  Heaven forbid!  You shouldn't do that just because that would
... excite wolfgang!  I'm just joking.  :-)

> What if we want to call an OS from assembly language code?

You make the OS interface work for assembly, e.g., stack and interrupt  
call.
Then, you wrap C, and possibly some assembly, around the OS assembly call.

Didn't you do this already with the code I was calling a "wrapper"
which you said isn't a "wrapper" ...

> Should we have yet another set of functions or, perhaps, invoke  
> subroutines
> in CLIB?

?

> (I don't want to call my OS functions directly as that would limit  
> application source code to one specific means of getting to the kernel,

Why is that an issue?

> and force apps to know which OS calls needed kernel involvement.

I'm not sure what you mean by this.

> Implementing all OS services as functions allows some to be handled in  
> user mode without forcing all to transition to privileged mode.)

Yes, but I think you would need a library of each language, except
the primary language used for the OS API, e.g., except assembly.

IIRC, this is were some of those advanced C preprocessor macro's come
in to play.  I.e., they have a set of files which construct the code to
call the OS for every interrupt number from the macro's and code snippets.

> Would it be desirable and would it even be possible to have a single  
> library of functions that can be called from any language, perhaps with  
> different calling conventions?

So, functions overloaded with multi-language support? ...

I guess that depends on the similarity of the implementation of the
underlying languages.  Similar would mean that it's possible.  Different
would mean that it's not, or not easily.

> By the way, I know that such OS-call subroutines (the immediate callees)  
> will mostly be small shims which vector into the kernel or they might be  
> a single routine which, when called, patches in the real routine and  
> thus provides dynamic linking. But even then they will be OS-specific  
> functions in a language-specific library. Is there a better way to go?

... a better way to go, e.g., like:

comp.lang.misc
comp.arch


The code must all interconnect or be linked together either statically
or dynamically at some point, yes?  I guess I'm saying that you can
only abstract things so far ...  Of course, someone may know a solution.


Rod Pemberton

-- 
It's time to put an end to gun violence!
Use a hammer ...

[toc] | [next] | [standalone]

#8206

On Thu, 18 Jun 2015 13:24:52 -0400, James Harris  
<james.harris.1@gmail.com> wrote:
> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
> news:op.x0ewr50gyfako5@localhost...
>> On Wed, 17 Jun 2015 11:05:30 -0400, James Harris  
>> <james.harris.1@gmail.com> wrote:

>>> If such OS functions are in CLIB then is that the right place for them?
>>
>> It's probably the "OS library" integrated into the "CLIB".
>>
>> You need at least 18 to 20 OS functions to implement a C library,
>> including the half dozen low-level Unix file I/O functions.
>
> You may need just 18 to 20 to implement the C library but it seems that  
> libc has to include over 100 (possibly over 200) wrapper functions just  
> for Unix so that calls to a Unix OS are available to C programs. See the  
> list at
>
>   http://docs.cs.up.ac.za/programming/asm/derick_tut/syscalls.html
>

Yes.

Linux 2.6.17 kernal has 290 syscalls, whereas Linux v0.01 had 40.
Well, v0.01 had 67 actually.  13 were unused.  14 were minimally
implemented.  Only 40 were complete.  You can look at DrAcOnUx's
update of v0.01 here:

http://draconux.free.fr/os_dev/linux0.01.html

> ISTM now that just about all of those need to have a wrapper in libc.

I'm not as familiar with Linux as DJGPP.  With DJGPP, the C function
has assembly wrapped around the OS call, i.e., DPMI through to DOS.

> Essentially the C compiler has to recognise and provide subroutines for  
> every single OS call that a program might make, making that version of  
> libc OS-specific.

s/C compiler/either the CLIB or LIBC

> I had assumed that libc had code just for the C library functions so it  
> was a C library not a Linux C library or a Windows C library etc.

AFAIK, there is no reason they have to be merged together.
It's convenient to have the headers in the same directory.
The C code for routines are usually compiled into libraries.
Does it matter if you have one or two libraries?  I.e., merged
or separate?  IDK ...

> Windows has many more OS functions, doesn't it? Would a C compiler for  
> Windows have to provide handlers for every possible Windows OS call?

I don't have any information on Windows.

The Java virtual machine had 206 bytecodes at one point in time
and Single UNIX V3 specification has 1,742 functions.

>>> What if we want to call an OS function from another language, say  
>>> Pascal?
>>
>> There would need to be code to convert the Pascal call to the OS call.
>>
>>> Would there be a Pascal library which *also* has to match
>>> the target OS?
>>
>> I would assume so.
>
> And if there are Cobol or Fortran or Haskell or etc compilers which have  
> to provide OS access they all need to have subroutines for each OS call?
>
> That seems like a lot of work and a lot of duplication of effort. And it  
> seems to be a maintenance headache in that when a new version of the OS  
> is released with a new API all of the compilers need to adjust their  
> libraries to match.

Why did M$ once support the Pascal calling convention with C? ...

> And, of course, such compiler providers need to have bindings for each  
> OS that the compiler will run on.
>
> And if someone comes out with a new OS all the compilers (that will run  
> on that OS) need to be adjusted again to bind to the subroutines for  
> that new OS.
>
> And if a hobbyist comes out with a new OS he has to persuade compiler  
> writers to add functions to support his OS?
>
> Surely that cannot all be right. As you can probably gather I cannot see  
> the sense in doing things as described.

...

>>> If we make a new OS which has its own API would we have to get our  
>>> function callees also into CLIB and Pascal's library [...]
>>
>> Yes.  The C library is build on the OS API.  At minimum, eighteen
>> to twenty OS functions are needed.  I posted these to a.o.d. in 2008:
>>
>> K&R C, e.g., PDP-7 and/or PDP-11 UNIX C libraries:
>> open, close, read, write, creat, link, fork, exec, wait, exit, lseek,  
>> pipe
>>
>> PJ Plauger's "Standard C Library" book:
>> clock, close, environ, execl, exit, fork, getpid, kill, link, lseek,  
>> open,  read, sbrk, signal, time, unlink, wait, write
>>
>> Redhat's newlib:
>> close, environ, _exit, execve, fork, fstat, getpid, isatty, kill,  
>> link,  lseek, open, read, sbrk, stat, times, unlink, wait, write
>>
>> You might also want to add "unlink" for the K&R C set ...
>
> I am not sure if you are talking about calls which are needed in order  
> for C library functions to complete their work, or the little shims  
> (wrappers, if you like) that are also needed so that C source can call  
> OS functions. A C library can be written in terms of limited functions  
> that a certain OS provides but there's no control over which OS calls a  
> C programmer might want to make.

The way this works is the C library is written entirely in pure C code
without using any other C functions, except for the few listed above,
which are present as C functions to the C library.  These minimal
functions required to implement the library may have shims or wrappers
depending on how they're implemented.  I'd suspect that generally
they're in C but also using inline assembly to call the host OS function,
i.e, shim or wrap etc.

Of course, a C library like GLIBC is going to call from 40 to 290 or more
C functions which wrap/shim etc the OS API call(s).

>>> (I don't want to call my OS functions directly as that would limit  
>>> application source code to one specific means of getting to the kernel,
>>
>> Why is that an issue?
>
> Two reasons:
>
> 1. A given version of the kernel may be invoked by a software interrupt  
> or SYSENTER or SYSCALL or a far call or something else. They are all  
> hard or impossible to write in C or another HLL, and the best approach  
> may change.
>
> 2. Some OS calls may be satisfiable without taking the trip to privleged  
> mode. In that case it makes sense to have the subroutine which is  
> immediately called by the app just do the work necessary and return  
> without invoking the kernel.

By "call my OS functions directly," I thought you meant something like:

    call_my_OS(0xNN);

I.e., it was presented as a C function for C.

Each of those minimal 18 or 20 functions would be calling the OS directly
too, but from a C named function.

> 1. A given version of the kernel may be invoked by a software interrupt  
> or SYSENTER or SYSCALL or a far call or something else. They are all  
> hard or impossible to write in C or another HLL, and the best approach  
> may change.

So, they should be easily called from the host language without special
code or special language attributes or #pragma's etc, yes?  I.e.,
call_my_OS() for C, or jmp far call_my_OS for assembly, etc.

> 2. Some OS calls may be satisfiable without taking the trip to privleged  
> mode. In that case it makes sense to have the subroutine which is  
> immediately called by the app just do the work necessary and return  
> without invoking the kernel.

If you don't have to invoke the kernel, do these constitute an OS function?

>>> Implementing all OS services as functions allows some to be handled  
>>> in  user mode without forcing all to transition to privileged mode.)
>>
>> Yes, but I think you would need a library of each language, except
>> the primary language used for the OS API, e.g., except assembly.
>
> Sorry, I don't understand that sentence.

s/of/for

Essentially, I was saying you need to do what you responded to above as:
JH> "That seems like a lot of work and a lot of duplication of effort."
Every compiler would "need to have subroutines for each OS call" in order
"to provide OS access."

>>> [...]
>>
>> So, functions overloaded with multi-language support? ...
>
> Yes. One interface per calling convention rather than one per language.

How do you unify that across multiple languages which have different
methods for calling a function and different calling conventions?

To sufficiently abstract this in the way you're wanting it to be,
I'd think it'd have to be via a common feature of all OSes.  Perhaps,
such as file I/O for platforms where "everything is a file" philosphy
is implemented.  E.g., you write out a text file the parameters, or
a IPC message as text, a function is called without parameters for
the host language, the function being called reads the text file and
parses it.  IIRC, you were discussing something like this in the past,
e.g., text formatted data, such as XML, and binary endian conversions
between modules, or across the network, etc.

>> I guess that depends on the similarity of the implementation of the
>> underlying languages.  Similar would mean that it's possible. Different
>> would mean that it's not, or not easily.
>

Also, see above.  Text files might be a round-about solution.

>> I guess that depends on the similarity of the implementation of the
>> underlying languages.  Similar would mean that it's possible. Different
>> would mean that it's not, or not easily.
>
> The benefit would be to have the library of OS-interface functions  
> provided with the OS, regardless of how many languages call that OS,  
> rather than one library per language per OS.

IIUC, you can have a common "library of OS-interface functions," but each
language would need to have a wrap/shim etc to call the OS.  E.g., if the
OS API is in assembly, say like Int 0x80 for Linux, C functions must wrap
or shim Int 0x80 to call the OS, Pascal too, Fortran too.  If the OS API
is in C, then assembly, Pascal, Fortran would need to wrap or shim the C
function.  It's easier to wrap/shim etc than it is to support more than
one interface language for the OS, especially if you have a large number
of functions to support.  If the set of OS functions is small, then the
OS could support each set in a multitude of languages directly, but then
you've just shifted the shim/wrap issue from the C, Pascal, Fortran
library domain to the OS API domain.

> I am thinking that the library would be provided by the OS developer
> and all languages could use its subroutines.
>
> Is that feasible?

I'm not real sure what you're asking here ...

If you're specifying the OS API for your OS, then all languages implemented
on your platform have to use it.

> I know that there would need to be some interface info for each language  
> (for C that would be header files) but the interface info for all  
> languages could be generated from a single source so would only have to  
> be defined once, and the implementing code itself (that applications  
> call) would only have to be written once.

Ok.  That usually requires some sort of automatic tool to build such files.
E.g., M4, Gnu Make or Auto-conf, perhaps Grep or AWK or C preprocessor,  
etc.
The C preprocessor has been used to preprocess or create code for other
situations.  Personally, I'd probably avoid that.


Rod Pemberton

-- 
It's time to put an end to gun violence!
Use a hammer ...

[toc] | [prev] | [next] | [standalone]

#8237

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x0g2zlo8yfako5@localhost...
> On Thu, 18 Jun 2015 13:24:52 -0400, James Harris 
> <james.harris.1@gmail.com> wrote:
>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
>> news:op.x0ewr50gyfako5@localhost...
>>> On Wed, 17 Jun 2015 11:05:30 -0400, James Harris 
>>> <james.harris.1@gmail.com> wrote:

...

>> Essentially the C compiler has to recognise and provide subroutines 
>> for  every single OS call that a program might make, making that 
>> version of  libc OS-specific.
>
> s/C compiler/either the CLIB or LIBC

s#LIBC#LIBC/#

;-)

>> I had assumed that libc had code just for the C library functions so 
>> it  was a C library not a Linux C library or a Windows C library etc.
>
> AFAIK, there is no reason they have to be merged together.
> It's convenient to have the headers in the same directory.

As long as the headers are in the path it should be OK.

...

>>>> (I don't want to call my OS functions directly as that would limit 
>>>> application source code to one specific means of getting to the 
>>>> kernel,
>>>
>>> Why is that an issue?
>>
>> Two reasons:
>>
>> 1. A given version of the kernel may be invoked by a software 
>> interrupt  or SYSENTER or SYSCALL or a far call or something else. 
>> They are all  hard or impossible to write in C or another HLL, and 
>> the best approach  may change.
>>
>> 2. Some OS calls may be satisfiable without taking the trip to 
>> privleged  mode. In that case it makes sense to have the subroutine 
>> which is  immediately called by the app just do the work necessary 
>> and return  without invoking the kernel.
>
> By "call my OS functions directly," I thought you meant something 
> like:
>
>    call_my_OS(0xNN);

Not quite but I can now give you an example of the intention. Here is a 
way I have worked out to call a couple of OS functions (from C) that 
certainly heads a good way in the direction I want to go. It is not a 
complete solution but it illustrates the kind of thing that I am trying 
to do. I only worked this out last night and so far ISTM a really 
exciting step forward!!!.

Can you see how it is supposed to work and what I am trying to achieve? 
I will put some explanatory comments below the code but basically there 
is to be a struct (declared in the header) where each member is a 
pointer to an OS function, and that structure ends up being called "os" 
in the body of the code. See the part "Issue some OS calls" for how I am 
thinking apps will call the OS.

  #define os oscalls_v0dev
  #include "oscalls_v0dev.h"

  int apptest2(void) {

    /* Issue some OS calls */
    os.qret(); /* Quick return */
    os.kret(); /* Kernel return */

    /* Successful completion */
    return 0;
  }

I like this approach because:

* it makes OS calls easy to use without mixing them into the language's 
library

* the compiler does not need to know anything about the OS. All it needs 
to know is what it sees in the OS-supplied header

* OS calls will be fast and, despite the indirection, can be branch 
predicted (more on that below)

* the call table can be created dynamically, if appropriate (I think! - 
work to do here)

* it gives OS calls their own namespace.

It was the quest for a namespace which led me to the idea of the struct. 
The namespace/struct ensures OS function names will not conflict with 
any others that appear in the program.

The idea is that this function, apptest2, wants to make a couple of 
calls to an OS and expects the OS to provide a certain call interface 
version or API. In this case I have called it v0dev for Version 0, 
Development branch, but the name can be longer or shorter. The idea of 
versioning is that apps written for an older version of the OS API can 
still be run on a newer OS and that the OS can be upgraded without 
breaking old apps, as long as it retains the ability to service older 
API versions.

Because the specific version name can be long I used the #define to give 
it a shorter name. The short name can then be used later in the code 
when making the calls.

The #include brings in the appropriate headers for the version the code 
expects to use. Within the included file is this line:

  extern struct oscalls_v0dev oscalls_v0dev;

That declares oscalls_v0dev (which later gets called just os) as the 
table of function pointers where each pointer points to the appropriate 
routine. There will be one pointer for each OS service that the app can 
call. Here is an example struct declaration with just three calls in it. 
The real one would have many calls to the OS.

  struct oscalls_v0dev {
    void (*qret)(void); /* Quick return with no effect */
    void (*kret)(void); /* Kernel return with no effect */
    unsigned (*build_id)(void); /* Return the OS build version */
  };

The key intention here is that this is implemented as an array of 
pointers (where each pointer has the same size), but from the C 
perspective each pointer has a particuar type, and the compiler will 
know what that type is. It can therefore use the type to ensure that 
calls to each routine have the correct parameters.

Being a struct, too, each member has its own name. Despite being 
implented as an array, C programs can therefore call each OS function by 
name and the compiler will match the types.

...

>> 1. A given version of the kernel may be invoked by a software 
>> interrupt  or SYSENTER or SYSCALL or a far call or something else. 
>> They are all  hard or impossible to write in C or another HLL, and 
>> the best approach  may change.
>
> So, they should be easily called from the host language without 
> special
> code or special language attributes or #pragma's etc, yes?  I.e.,
> call_my_OS() for C, or jmp far call_my_OS for assembly, etc.

Yes. In the sample above, because the function pointers are in a struct 
the app has to use the struct name to call them. I have this in the code 
as these two calls

    os.qret(); /* Quick return */
    os.kret(); /* Kernel return */

I haven't yet worked out how to make the table and the OS routines 
available to the app. Ideas welcome, especially if they allow OS 
routines to be dynamically bound (if that would be faster overall).

>> 2. Some OS calls may be satisfiable without taking the trip to 
>> privleged  mode. In that case it makes sense to have the subroutine 
>> which is  immediately called by the app just do the work necessary 
>> and return  without invoking the kernel.
>
> If you don't have to invoke the kernel, do these constitute an OS 
> function?

I would say so, yes. IMO any service routine that the OS provides is an 
OS function. How it provides it is not so important.

Some functions may be able to do what they have to do without privilege 
such as functions to read a certain time clock.

Other functions may *sometimes* be able to complete without going into 
kernel mode and other times need to go into kernel mode. For example, if 
told to read the next 2 bytes from a file, if there are 2 bytes buffered 
then that request can be satisfied without a kernel call. Otherwise the 
routine has to invoke the kernel to ask for more data.

I'll have to get back to you separately about the rest of the things we 
were discussing but I said I would comment on the performance of calls 
to the OS and branch predictability. Code speed is always exciting, 
right! I understand that x86_32 indirect near calls can be branch 
predicted. The apptest2 subroutine above compiles on x86_32 with -O2 to 
the following (gcc AT&T format with destinations on the right).

  push   %ebp
  mov    %esp,%ebp
  sub    $0x8,%esp
  call   *0x0
  call   *0x4
  xor    %eax,%eax
  leave
  ret

The thing to note is the two calls to the OS. Both are simple (indirect) 
calls and need no special registers or addresses or parameters to be set 
up. They have apparently been compiled to be indirect via the struct 
mentioned above. Being near they should both be branch predictable and 
just as fast as ordinary calls.

In short I am VERY happy with the progress so far BUT there is more to 
do and I won't really be able to say I have any solution until all of it 
is addressed. But hopefully the above gives you an idea of where I am 
'coming from' in trying to achieve efficiency, convenience, separation, 
etc.

If you have read through to this point, many thanks for staying with it. 
You never know with these things. Some may find it interesting, some 
not.

James

[toc] | [prev] | [next] | [standalone]

#8240

On Mon, 22 Jun 2015 16:22:19 -0400, James Harris
<james.harris.1@gmail.com> wrote:

> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
> news:op.x0g2zlo8yfako5@localhost...
>> On Thu, 18 Jun 2015 13:24:52 -0400, James Harris  
>> <james.harris.1@gmail.com> wrote:
>>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
>>> news:op.x0ewr50gyfako5@localhost...
>>>> On Wed, 17 Jun 2015 11:05:30 -0400, James Harris  
>>>> <james.harris.1@gmail.com> wrote:

>>>>> (I don't want to call my OS functions directly as that would limit  
>>>>> application source code to one specific means of getting to the  
>>>>> kernel,
>>>>
>>>> Why is that an issue?
>>>
>>> Two reasons:
>>>
>>> 1. A given version of the kernel may be invoked by a software  
>>> interrupt  or SYSENTER or SYSCALL or a far call or something else.  
>>> They are all  hard or impossible to write in C or another HLL, and the  
>>> best approach  may change.
>>>
>>> 2. Some OS calls may be satisfiable without taking the trip to  
>>> privleged  mode. In that case it makes sense to have the subroutine  
>>> which is  immediately called by the app just do the work necessary and  
>>> return  without invoking the kernel.
>>
>> By "call my OS functions directly," I thought you meant something like:
>>
>>    call_my_OS(0xNN);
>
> Not quite but I can now give you an example of the intention

Ok.

> Here is a way I have worked out to call a couple of OS functions (from  
> C) that certainly heads a good way in the direction I want to go. It is  
> not a complete solution but it illustrates the kind of thing that I am  
> trying to do. I only worked this out last night and so far ISTM a really  
> exciting step forward!!!.
>
> Can you see how it is supposed to work and what I am trying to achieve?

Not to be pedantic, but you have yet to explain it here.  It's below.
So, I'll copy the question down below.  (Ok. I answered it too.)

> I will put some explanatory comments below the code but basically there  
> is to be a struct (declared in the header) where each member is a  
> pointer to an OS function, and that structure ends up being called "os"  
> in the body of the code. See the part "Issue some OS calls" for how I am  
> thinking apps will call the OS.

The idea seems clean to me.  It's easier than a library.  You can
always add more to the end of the struct and keep #defines for the
number of OS calls available or the struct version.  I.e., to allow
for backward compatibility with future changes.

Are these functions to just be extern's?

>   #define os oscalls_v0dev
>   #include "oscalls_v0dev.h"
>
>   int apptest2(void) {
>
>     /* Issue some OS calls */
>     os.qret(); /* Quick return */
>     os.kret(); /* Kernel return */
>
>     /* Successful completion */
>     return 0;
>   }
>

Are there to be just the two calls with kernel following quick,
i.e., is there a specific order or implementation here?  Or,
is this example just supposed to be a mix of whatever calls are
required to implement an app?

(I'm assuming that the latter question is true, but I'm being
a bit pedantic.)

> I like this approach because:
>
> * it makes OS calls easy to use without mixing them into the language's  
> library
>
> * the compiler does not need to know anything about the OS. All it needs  
> to know is what it sees in the OS-supplied header
>
> * OS calls will be fast and, despite the indirection, can be branch  
> predicted (more on that below)
>
> * the call table can be created dynamically, if appropriate (I think! -  
> work to do here)

...

> * it gives OS calls their own namespace.

That could be done with underscores or dollar sign symbol too, but
three characters for "os." is short and sweet.

> It was the quest for a namespace which led me to the idea of the struct.  
> The namespace/struct ensures OS function names will not conflict with  
> any others that appear in the program.

Well, as long as no one happens to use "os", but the odds of that
IMO are low and it's easy for them to adjust their code.

> The idea is that this function, apptest2, wants to make a couple of  
> calls to an OS and expects the OS to provide a certain call interface  
> version or API. In this case I have called it v0dev for Version 0,  
> Development branch, but the name can be longer or shorter. The idea of  
> versioning is that apps written for an older version of the OS API can  
> still be run on a newer OS and that the OS can be upgraded without  
> breaking old apps, as long as it retains the ability to service older  
> API versions.

So, the next version might be v1dev, then v2dev, etc.?

Have you considered an integer passed as the first parameter to
indicate the expected version of the function?  Then, it can
always be named "vdev.h".  E.g.,

      #define os oscalls_vdev
      #include "oscalls_vdev.h"

/* in oscalls_vdev.h */
// #define VERSION 0

      int apptest2(void) {

        /* Issue some OS calls */
        os.qret(VERSION, ...); /* Quick return */
        os.kret(VERSION, ...); /* Kernel return */

        /* Successful completion */
        return 0;
      }

qret and kret won't be C style "void func(void)"
anymore, which isn't a problem for C, but may
complicate your assembly, Pascal, Fortran, etc goals.

> Because the specific version name can be long I used the #define to give  
> it a shorter name. The short name can then be used later in the code  
> when making the calls.
>
> The #include brings in the appropriate headers for the version the code  
> expects to use. Within the included file is this line:
>
>   extern struct oscalls_v0dev oscalls_v0dev;
>
> That declares oscalls_v0dev (which later gets called just os) as the  
> table of function pointers where each pointer points to the appropriate  
> routine. There will be one pointer for each OS service that the app can  
> call. Here is an example struct declaration with just three calls in it.  
> The real one would have many calls to the OS.
>
>   struct oscalls_v0dev {
>     void (*qret)(void); /* Quick return with no effect */
>     void (*kret)(void); /* Kernel return with no effect */
>     unsigned (*build_id)(void); /* Return the OS build version */
>   };

The only question I have is whether or not the extern struct
and function pointers, which are intended for C here, will
also be compatible with assembly, Pascal, Fortran, etc in
terms of layout and format.  Hopefully, that is a: "Yes."

> The key intention here is that this is implemented as an array of  
> pointers (where each pointer has the same size), but from the C  
> perspective each pointer has a particuar type, and the compiler will  
> know what that type is. It can therefore use the type to ensure that  
> calls to each routine have the correct parameters.
>
> Being a struct, too, each member has its own name. Despite being
> [implemented] as an array, C programs can therefore call each OS
> function by name and the compiler will match the types.

Well, an array would be indexable by an integer.  If you used a
union, you could possibly use both names and an index to each call.

Alternately, if you used an array of function pointers, you
could do something like the following.  Then, you could access
the function pointers by index or by name.

// tested code.
// Well, GCC 4.7.2 on Linux likes it ...

#include <stdio.h>

#define OS(n) os[ ## n ## ]
typedef void (*osfn)(void);

// would be extern
void hello()
{
   printf("Hello ");
}

// would be extern
void world()
{
   printf("World!\n");
}


// array of pointers to functions
// would be extern in your OS header
osfn os_table[]=
{
   hello,
   world,
   NULL
};

#define os_qret() (*(os_table[0]))()
#define os_kret() (*(os_table[1]))()

int main(void)
{

   (*(os_table[0]))();
   (*(os_table[1]))();
   os_qret();
   os_kret();

return(0);
}

Unfortunately, you can't use a '.' dot, in the #defines
for "os.qret" etc., AFAIK, but you could ensure a more
exclusive "namespace" with underscores, e.g., os_qret__();

> Can you see how it is supposed to work and what I am trying to achieve?

Yes and I hope so. ;-)

>>> 2. Some OS calls may be satisfiable without taking the trip to  
>>> privleged  mode. In that case it makes sense to have the subroutine  
>>> which is  immediately called by the app just do the work necessary and  
>>> return  without invoking the kernel.
>>
>> If you don't have to invoke the kernel, do these constitute an OS  
>> function?
>
> I would say so, yes. IMO any service routine that the OS provides is an  
> OS function. How it provides it is not so important.

If these are an OS function, should you provide them to the user?

You just answered that it's acceptable to do so, but I'm thinking
about portability and safety, which seem to be of importance to you.

For portibility, using the functions would likely make the users
code non-portable since they are host specific functions.  Whereas,
if the user only had access to kernel functions, they would have to
code their own equivalent to the OS function in portable code.

As for the safety issue, if a function isn't a kernel function,
how does the OS ensure that the function hasn't been replaced
by malicious code?  ...  I.e., that function is essentialy a
built-in man-in-the-middle attack waiting to happen.  Isn't it?

> Some functions may be able to do what they have to do without privilege  
> such as functions to read a certain time clock.

I can't immediately "see" how that would allow for software hacking,
but I immediately realize that could be used for hardware hacking.

This issue may be way outside the scope of your needs at this point.

E.g., an electrical engineer or someone familiar with both electronics
and programming could write code to read and write the hardware clock.
However, they've also installed hardware, logic circuitry, that
redirects the I/O to port address to the addresses they desire to hack.

> Other functions may *sometimes* be able to complete without going into  
> kernel mode and other times need to go into kernel mode. For example, if  
> told to read the next 2 bytes from a file, if there are 2 bytes buffered  
> then that request can be satisfied without a kernel call. Otherwise the  
> routine has to invoke the kernel to ask for more data.

Ok.

> I'll have to get back to you separately about the rest of the things we  
> were discussing but I said I would comment on the performance of calls  
> to the OS and branch predictability. Code speed is always exciting,  
> right! I understand that x86_32 indirect near calls can be branch  
> predicted. The apptest2 subroutine above compiles on x86_32 with -O2 to  
> the following (gcc AT&T format with destinations on the right).
>
>   push   %ebp
>   mov    %esp,%ebp
>   sub    $0x8,%esp
>   call   *0x0
>   call   *0x4
>   xor    %eax,%eax
>   leave
>   ret

LEAVE? ...

Was there an ENTER somewhere?  Or, is there some missing prologue (UK) or
prolog (US) code which would require LEAVE?

Also, IIRC, LEAVE is "slow."  Avoid.

> The thing to note is the two calls to the OS. Both are simple (indirect)  
> calls and need no special registers or addresses or parameters to be set  
> up. They have apparently been compiled to be indirect via the struct  
> mentioned above.

...

> Being near they should both be branch predictable and just as fast as  
> ordinary calls.

The branch address is stored in memory, which can be modified.
I.e., this shouldn't be predictable on older processors since
it's not a fixed value that's part of the actual instruction
byte stream, but as long as the address doesn't change, it
should be predictable on the newer processors with a BTB, AISI,
since that location should generally remain fixed and be cached.

Like I said once before, I haven't followed processor design
since the mid 1990's and so Wikipedia and Google and AMD/Intel
manual are my main methods of slowly keeping up to date when
neded.

Sigh, apparently, this topic was from one of the other a.o.d.
or c.l.m. threads I haven't gotten too, and simply may not
due to life ...


Rod Pemberton

-- 
It's time to put an end to gun violence!
Use a hammer ...

[toc] | [prev] | [next] | [standalone]

#8248

On Wed, 24 Jun 2015 14:49:42 -0400, James Harris  
<james.harris.1@gmail.com> wrote:
> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
> news:op.x0phzqyjyfako5@localhost...
>> On Mon, 22 Jun 2015 16:22:19 -0400, James Harris
>> <james.harris.1@gmail.com> wrote:
>>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
>>> news:op.x0g2zlo8yfako5@localhost...
>>>> On Thu, 18 Jun 2015 13:24:52 -0400, James Harris  
>>>> <james.harris.1@gmail.com> wrote:
>>>>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
>>>>> news:op.x0ewr50gyfako5@localhost...
>>>>>> On Wed, 17 Jun 2015 11:05:30 -0400, James Harris  
>>>>>> <james.harris.1@gmail.com> wrote:

>>> I will put some explanatory comments below the code but basically  
>>> there  is to be a struct (declared in the header) where each member is  
>>> a  pointer to an OS function, and that structure ends up being called  
>>> "os"  in the body of the code. See the part "Issue some OS calls" for  
>>> how I am  thinking apps will call the OS.
>>
>> The idea seems clean to me.
>
> That's good to hear.
>
>> It's easier than a library.
>
> I have been trying to work out what you mean there but I am not sure.

A struct of functions is easier to do than compiling a library, yes?

>> You can
>> always add more to the end of the struct and keep #defines for the
>> number of OS calls available or the struct version.  I.e., to allow
>> for backward compatibility with future changes.
>
> I was thinking that the name of the struct would fully identify the set  
> of calls. I am not sure whether that is enough but nor am I sure that  
> adding a separate #define struct version would help.
>
> To illustrate, the real struct name would probably be determined by the  
> name of the OS (which I have no idea about yet). Say the os were to be  
> called JHOS. In that case the structure could be called something like
>
>   struct jhos_rv1_45
>
> where rv1 means release version 1, and 45 is the number of calls therein  
> or the highest call number or something like that where larger numbers  
> imply later revisions and the number can only increase with each  
> addition to rv1.

Can this info be obtained when/if needed without parsing "jhos_rv1_45"?

> Thus the struct type name would fully identify the version of the call  
> table, and jhos_rv1_45 would be exactly the same as the first elements  
> of jhos_rv1_49, which would be exactly the same as the first elements of  
> jhos_rv1_53, etc. That would mean that the _53 call table could be used  
> even for the _45 interface because an app compiled to use the _45  
> interface would need just the lower 45 or so entries from the _53 table.

Ok, so, no duplication, but overlay.

How do you stop a _53 app from calling a function a _45 OS doesn't have?

Any plans to prevent a _45 app from calling a function for a _53 OS?
I.e., hacking.

>>> The key intention here is that this is implemented as an array of  
>>> pointers (where each pointer has the same size), but from the C  
>>> perspective each pointer has a particuar type, and the compiler will  
>>> know what that type is. It can therefore use the type to ensure that  
>>> calls to each routine have the correct parameters.
>>>
>>> Being a struct, too, each member has its own name. Despite being
>>> [implemented] as an array, C programs can therefore call each OS
>>> function by name and the compiler will match the types.
>>
>> Well, an array would be indexable by an integer.  If you used a
>> union, you could possibly use both names and an index to each call.
>>
>> Alternately, if you used an array of function pointers, you
>> could do something like the following.  Then, you could access
>> the function pointers by index or by name.
>>
>> // tested code.
>> // Well, GCC 4.7.2 on Linux likes it ...
>>
>> #include <stdio.h>
>>
>> #define OS(n) os[ ## n ## ]
>> typedef void (*osfn)(void);
>>
>> // would be extern
>> void hello()
>> {
>>   printf("Hello ");
>> }
>>
>> // would be extern
>> void world()
>> {
>>   printf("World!\n");
>> }
>>
>>
>> // array of pointers to functions
>> // would be extern in your OS header
>> osfn os_table[]=
>> {
>>   hello,
>>   world,
>>   NULL
>> };
>>
>> #define os_qret() (*(os_table[0]))()
>> #define os_kret() (*(os_table[1]))()
>>
>> int main(void)
>> {
>>
>>   (*(os_table[0]))();
>>   (*(os_table[1]))();
>>   os_qret();
>>   os_kret();
>>
>> return(0);

>> [code snip]
>
> I see your point (and it's good to see use of ## which I never really  
> got to understand) but the trouble with an array is that a C compiler  
> will believe that every element has the same signature. With a struct,  
> on the other hand, each element has its own signature. That allows the  
> compiler to check that it is being called correctly.

That's the only time I've ever used ## .  I wasn't sure it was legal
for two to be used on the same line, but GCC's preprocessor didn't choke
on it.

## is the merge operator for macro's
# is the string operator for macro's, i.e., places quotes around value

>> Unfortunately, you can't use a '.' dot, in the #defines
>> for "os.qret" etc., AFAIK, but you could ensure a more
>> exclusive "namespace" with underscores, e.g., os_qret__();
>
> As a matter of personal taste do you prefer the form with underscores  
> over that with the "os." prefix?

No.

> I should say that in your version you have apparently chosen to fix the  
> "os_" part into the definition. In mine, "os." is something that the  
> programmer has chosen.

You could put in another #define.

> Also, with my approach, it would be possible to use a pointer to the  
> struct which might be handy of the version of the os calls table were to  
> be passed to a subroutine or if the programmer simply preferred to refer  
> to it by dereferencing a pointer. If the pointer was declared as
>
>   struct oscalls_v0dev *osp;
>
> Then, once the pointer had been initialised, OS calls could be made  
> using the pointer with such as
>
>   osp->kret();
>
> In this case, too, the programmer has chosen the "osp->" prefix rather  
> than the OS developer fixing it, putting the name in the hands of the  
> programmer and allowing the OS calls table version to change without the  
> programmer having to go through and update any call statements in his  
> source.

Sure.

>>>>> 2. Some OS calls may be satisfiable without taking the trip to  
>>>>> privleged  mode. In that case it makes sense to have the subroutine  
>>>>> which is  immediately called by the app just do the work necessary  
>>>>> and  return  without invoking the kernel.
>>>>
>>>> If you don't have to invoke the kernel, do these constitute an OS  
>>>> function?
>>>
>>> I would say so, yes. IMO any service routine that the OS provides is  
>>> an  OS function. How it provides it is not so important.
>>
>> If these are an OS function, should you provide them to the user?
>>
>> You just answered that it's acceptable to do so, but I'm thinking
>> about portability and safety, which seem to be of importance to you.
>
> [snip]
>
>> For [portability], using the functions would likely make the users
>> code non-portable since they are host specific functions.  Whereas,
>> if the user only had access to kernel functions, they would have to
>> code their own equivalent to the OS function in portable code.
>
> I am not sure I understand this.

If the kernel has functions which the app can call directly,
that set of functions or API is the core interface for the OS, yes?
(Well, I think so.)  So, is there any point in an additional layer
of functions between the the app and the kernel?  I.e., either these
functions should not be provided, or possibly, they should be moved
into the kernel.  Historical OSes, like Unix and Windows, have a set
of basic minimal functions, mostly file I/O, that all code can build
from.  So, let's say the kernel file open is "open" and the OS file
open is "xopen".  Is there a point to writing apps for "xopen", when
only OSes compliant with your design specifications can use that code?
I.e., Linux and Windows won't have "xopen", but will have Unix "open"
and C "fopen".  So, code written for your platform specifications
using "xopen" becomes non-portable.  In which case, isn't it better
to eliminate use of "xopen"? ...

> I was thinking that the OS would present to the apps a set of functions,  
> and that the same set would be made available regardless of platform.

Ok.  I think that answers the question.

IIUC, there is a requirement somewhere by you as to the set of
functions which is required to be implemented that apps can call.
These functions are used by the app, and the app won't directly
call the functions exported by the kernel.

So, app code for your platform will only be portable to platforms
compliant with your set of functions.

>>> Some functions may be able to do what they have to do without  
>>> privilege  such as functions to read a certain time clock.
>>
>> I can't immediately "see" how that would allow for software hacking,
>> but I immediately realize that could be used for hardware hacking.
>>
>> This issue may be way outside the scope of your needs at this point.
>>
>> E.g., an electrical engineer or someone familiar with both electronics
>> and programming could write code to read and write the hardware clock.
>> However, they've also installed hardware, logic circuitry, that
>> redirects the I/O to port address to the addresses they desire to hack.
>
> I am not sure an OS can be proof against custom hardware but in normal  
> cases only the kernel would allow port IO. Nothing in user mode would be  
> able to do port IO directly.
>

Even without custom hardware, it allows for information leakage via
"port knocking."  E.g., in the future, your OS is now used for a high
security platform.  A hacker is having problems getting restricted
information out of the computer.  The hacker programs the reading
of the ports to do Morse code or other patterns that can be recorded,
e.g., a resistor and high-speed led on bit 0 of the port address when
flashing can be recorded as cell phone video.

>>> I'll have to get back to you separately about the rest of the things  
>>> we  were discussing but I said I would comment on the performance of  
>>> calls  to the OS and branch predictability. Code speed is always  
>>> exciting,  right! I understand that x86_32 indirect near calls can be  
>>> branch  predicted. The apptest2 subroutine above compiles on x86_32  
>>> with -O2 to  the following (gcc AT&T format with destinations on the  
>>> right).
>>>
>>>   push   %ebp
>>>   mov    %esp,%ebp
>>>   sub    $0x8,%esp
>>>   call   *0x0
>>>   call   *0x4
>>>   xor    %eax,%eax
>>>   leave
>>>   ret
>>
>> LEAVE? ...
>>
>> Was there an ENTER somewhere?  Or, is there some missing prologue (UK)  
>> or prolog (US) code which would require LEAVE?

I must've been losing it at this point.  The PUSH and MOV are the ENTER.

>> Also, IIRC, LEAVE is "slow."  Avoid.
>
> I don't know about that.

Were you referring to "slowness" comment or the missing prolog comment?

> The use of LEAVE was a GCC choice, not mine. Is it possible that ENTER  
> can be slow but not LEAVE? Maybe LEAVE is faster than the equivalent. I  
> have seen it as compiler output before.

ENTER may be even slower.  ;-)  Yeah, I probably mixed them up.

> That said, I am puzzled by GCC's optimisation. Notice it has subtracted  
> 8 from ESP. At the moment I cannot see why.

Is this standard 32-bit x86 code?  That would seem to indicate that
it has pushed two 32-bit values.

My guesses would be:

1) the two "void func(void)" functions are popping one 32-bit value each
2) the posted code was changed, e.g., was "void func(int)" originally
3) the code places a barrier/fence to protect the return address
4) the stack direction is reversed and sizes are 64-bit ... (?)
5) something wrong with the code generator, e.g., off-by-one, or optimizer

If #1, you'll need to look into why they have a hidden parameter.

If #4, IDK WTF that's about, but then the "sub 8" compensates for
the **64-bit** push of 32-bit ebp perhaps zero-/sign-extended to 64-bits  
...


Rod Pemberton

-- 
It's time to put an end to gun violence!
Use a hammer ...

[toc] | [prev] | [next] | [standalone]

#8241

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x0g2zlo8yfako5@localhost...
> On Thu, 18 Jun 2015 13:24:52 -0400, James Harris 
> <james.harris.1@gmail.com> wrote:
>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
>> news:op.x0ewr50gyfako5@localhost...
>>> On Wed, 17 Jun 2015 11:05:30 -0400, James Harris 
>>> <james.harris.1@gmail.com> wrote:

...

>>>> Implementing all OS services as functions allows some to be handled 
>>>> in  user mode without forcing all to transition to privileged 
>>>> mode.)
>>>
>>> Yes, but I think you would need a library of each language, except
>>> the primary language used for the OS API, e.g., except assembly.
>>
>> Sorry, I don't understand that sentence.
>
> s/of/for
>
> Essentially, I was saying you need to do what you responded to above 
> as:
> JH> "That seems like a lot of work and a lot of duplication of 
> effort."
> Every compiler would "need to have subroutines for each OS call" in 
> order
> "to provide OS access."

If I understand, you are saying that I might need a library for each 
language. I don't think so for two reasons:

1. There are only so many calling conventions. There is a good list of 
x86 ones at

https://en.wikipedia.org/wiki/X86_calling_conventions#List_of_x86_calling_conventions

and a wider picture at

https://en.wikipedia.org/wiki/Calling_convention

I think that rather than a library for each language it would be more 
accurate to say that there would need to be a library (or at least a set 
of entry points) for each calling convention. That is still quite a lot 
but the idea is that the OS developer would provide such 'libraries' to 
go with each new release of the OS.

2. Using the naming of components

  app --> interface code --> kernel

rather than link the interface code (i.e. the OS code that the app calls 
directly) in with the app I would rather find a way of providing that 
code at load time. To explain why:

If an app goes through the traditional process of being compiled and 
linked to form a load module then the interface code (supplied by the OS 
developer) would have to be available when linking happens and would end 
up being part of the load module. That's fine in a sense but it is 
inflexible for at least two reasons.

i) If it turns out there is a bug in the interface code then that bug 
would end up being built in to the load module.

ii) Most interface code functions would include an instruction to enter 
into the kernel. There are multiple ways of getting to kernel mode and 
some ways are faster than others. The way that is best will depend on 
the CPU the OS is running on. For example, on a 486 the best way to 
enter the kernel may be via a software interrupt or a far call whereas 
on a Core 2 the best way may be via SYSENTER or SYSCALL. Application 
programmers shouldn't have to concern themselves with which one to use. 
IMO the OS should choose that for them.

For both those reasons it would be better for the interface code to be 
made available by the running OS, rather than built into apps 
statically.

The upshot of the above, I think, is that app calls to the OS should be 
resolved when the app is loaded. That suggests that there needs to be 
some form of dynamic linking. The specific interface code used should be 
made available and linked when the app is loaded (or even lazy linked 
when it is running). Such code can avoid the above two problems: it can 
have had any known bugs corrected and can use the best method of getting 
to kernel mode, all without the app having to be recompiled.

Coming back to the point you made about interface code being available 
at compile time, for the above reasons I would rather make it available 
at load time. Although the compiler *would* need to know the interface 
to OS calls (such as would be supplied in a C header) it would *not* 
need to have the real interface code available. (However, the linker 
might need some dummy code or stub code so that the link process could 
happen. That code would get replaced at load time or run time.)

If this sounds like I am making life hard for myself that is probably 
true. But it just seems the right thing to do for the reasons mentioned 
above.

I am currently playing with ideas on *how* to dynamically link the 
interface code with the app but don't have a good answer yet.

>>> So, functions overloaded with multi-language support? ...
>>
>> Yes. One interface per calling convention rather than one per 
>> language.
>
> How do you unify that across multiple languages which have different
> methods for calling a function and different calling conventions?

Wouldn't having a separate entry point or separate function for each 
calling convention be enough?

For example, say there was an OS function which took an integer and a 
string and returned a character. Pascal pushes parameters in the 
opposite order to C. That could be dealt with easily enough. One entry 
point could expect parms left-to-right and the other could expect them 
right-to-left. Pascal strings have a different form. That could be more 
of a problem. Perhaps there could be two subroutines, one of which 
handled C strings and the other of which handled Pascal strings. Where 
appropriate they could call common functions but otherwise could remain 
separate pieces of code. The return value would also be returned in a 
form suitable to the calling convention.

If another language or compiler used the exact same calling convention 
as C or Pascal then it too could call the same library routine. Thus 
there would not be one entry point per language but one per calling 
convention (taking data types as part of the convention).

One problem is getting the calls from C and Pascal etc to the 
appropriate entry points. If the called OS routine was named M then apps 
should be able to just call M by name and leave the computer to convert 
M into M_cdecl or M_stdcall or whatever, according to the convention the 
compiler uses. I haven't yet worked out a way to do this.

> To sufficiently abstract this in the way you're wanting it to be,
> I'd think it'd have to be via a common feature of all OSes.  Perhaps,
> such as file I/O for platforms where "everything is a file" philosphy
> is implemented.  E.g., you write out a text file the parameters, or
> a IPC message as text, a function is called without parameters for
> the host language, the function being called reads the text file and
> parses it.  IIRC, you were discussing something like this in the past,
> e.g., text formatted data, such as XML, and binary endian conversions
> between modules, or across the network, etc.

That is flexible but unfortunately would probably be unnecessarily slow 
for most OS calls.

>>> I guess that depends on the similarity of the implementation of the
>>> underlying languages.  Similar would mean that it's possible. 
>>> Different
>>> would mean that it's not, or not easily.
>>
>
> Also, see above.  Text files might be a round-about solution.
>
>>> I guess that depends on the similarity of the implementation of the
>>> underlying languages.  Similar would mean that it's possible. 
>>> Different
>>> would mean that it's not, or not easily.
>>
>> The benefit would be to have the library of OS-interface functions 
>> provided with the OS, regardless of how many languages call that OS, 
>> rather than one library per language per OS.
>
> IIUC, you can have a common "library of OS-interface functions," but 
> each
> language would need to have a wrap/shim etc to call the OS.  E.g., if 
> the
> OS API is in assembly, say like Int 0x80 for Linux, C functions must 
> wrap
> or shim Int 0x80 to call the OS, Pascal too, Fortran too.  If the OS 
> API
> is in C, then assembly, Pascal, Fortran would need to wrap or shim the 
> C
> function.  It's easier to wrap/shim etc than it is to support more 
> than
> one interface language for the OS, especially if you have a large 
> number
> of functions to support.  If the set of OS functions is small, then 
> the
> OS could support each set in a multitude of languages directly, but 
> then
> you've just shifted the shim/wrap issue from the C, Pascal, Fortran
> library domain to the OS API domain.

By API I am not sure if you are thinking about how to get to kernel mode 
(int 0x80 or far call etc), or are thinking of the defined set of OS 
syscalls. In either case I agree that the calls to interface code (i.e. 
before any transtion to kernel mode) could be wrapped/shimmed. The 
interface code fragments - which are to be supplied by the OS - could 
invoke the kernel so the app wouldn't have to worry about how that was 
done.)

>> I am thinking that the library would be provided by the OS developer
>> and all languages could use its subroutines.
>>
>> Is that feasible?
>
> I'm not real sure what you're asking here ...
>
> If you're specifying the OS API for your OS, then all languages 
> implemented
> on your platform have to use it.

In the sequence

  app --> interface code --> kernel

I was saying that the library of interface code would be provided by the 
OS developer and asking if it was feasible for different languages to 
use the functions within it. Same issue as above, and one I don't have 
the answer to yet.

>> I know that there would need to be some interface info for each 
>> language  (for C that would be header files) but the interface info 
>> for all  languages could be generated from a single source so would 
>> only have to  be defined once, and the implementing code itself (that 
>> applications  call) would only have to be written once.
>
> Ok.  That usually requires some sort of automatic tool to build such 
> files.
> E.g., M4, Gnu Make or Auto-conf, perhaps Grep or AWK or C 
> preprocessor,  etc.
> The C preprocessor has been used to preprocess or create code for 
> other
> situations.  Personally, I'd probably avoid that.

The tool used to build such files may be easiest to write in Python 
which is great for text processing. Or it could be written in C but 
speed would not be important as it would only be run rarely, i.e. when 
there was a new OS interface definition.

James

[toc] | [prev] | [next] | [standalone]

#8247

On Wed, 24 Jun 2015 02:55:49 -0400, James Harris  
<james.harris.1@gmail.com> wrote:

> I think that rather than a library for each language it would be more  
> accurate to say that there would need to be a library (or at least a set  
> of entry points) for each calling convention. That is still quite a lot  
> but the idea is that the OS developer would provide such 'libraries' to  
> go with each new release of the OS.

Sure.

If you don't want a separate library for each, you could do a single
"library" with multiple supported calling conventions.  Wasn't that
what you indicated you wanted to do somewhere else?  I.e., overloaded
multi-language functions or somesuch.

> 2. Using the naming of components
>
>   app --> interface code --> kernel
>
> rather than link the interface code (i.e. the OS code that the app calls  
> directly) in with the app I would rather find a way of providing that  
> code at load time.

Ok.

> To explain why:
>
> If an app goes through the traditional process of being compiled and  
> linked to form a load module then the interface code (supplied by the OS  
> developer) would have to be available when linking happens and would end  
> up being part of the load module. That's fine in a sense but it is  
> inflexible for at least two reasons.

AIUI, the "way of providing that code at load time," is called a DLL
(dynamic-linked library) or shared library.  Yes?

> i) If it turns out there is a bug in the interface code then that bug  
> would end up being built in to the load module.

Are the load module and interface code separate, or merged but separatable?

If so, then the bug is only built in to the load module,
if the load module uses the code with the bug.

> ii) Most interface code functions would include an instruction to enter  
> into the kernel. There are multiple ways of getting to kernel mode and  
> some ways are faster than others. The way that is best will depend on  
> the CPU the OS is running on. For example, on a 486 the best way to  
> enter the kernel may be via a software interrupt or a far call whereas  
> on a Core 2 the best way may be via SYSENTER or SYSCALL. Application  
> programmers shouldn't have to concern themselves with which one to use.

FYI, too much user choice.

> IMO the OS should choose that for them.

Yeah, aren't you the one designing the OS?  So, shouldn't you be the
one deciding which one of them to use? ...

> For both those reasons it would be better for the interface code to be  
> made available by the running OS, rather than built into apps statically.

Ok.  This seems to be another answer to my question elsewhere in the thread
about the man-in-the-middle attack, i.e., "interface code" here.

> The upshot of the above, I think, is that app calls to the OS should be  
> resolved when the app is loaded. That suggests that there needs to be  
> some form of dynamic linking.

Yes.  Why engage in code bloat?  The app is only likely to execute
correctly on one OS anyway ...

> The specific interface code used should be made available and linked  
> when the app is loaded (or even lazy linked when it is running). Such  
> code can avoid the above two problems: it can have had any known bugs  
> corrected and can use the best method of getting to kernel mode, all  
> without the app having to be recompiled.

...

> Coming back to the point you made about interface code being available  
> at compile time, for the above reasons I would rather make it available  
> at load time.

Hm?  I'm replying to this post before the other.  At this point, I'm not
quite sure which comment I made on compile time versus load time.  I don't
recall making such a distinction.

> Although the compiler *would* need to know the interface to OS calls  
> (such as would be supplied in a C header) it would *not* need to have  
> the real interface code available. (However, the linker might need some  
> dummy code or stub code so that the link process could happen. That code  
> would get replaced at load time or run time.)

Ok.

> If this sounds like I am making life hard for myself that is probably  
> true. But it just seems the right thing to do for the reasons mentioned  
> above.

No pain.  No pain.  And, perhaps, no gain.  Sometimes things work out.
Sometimes they don't.  "Almond Joy's got nuts, Mounds don't ..."
It's not a matter of easy or hard.  Both can succeed.  Both can fail.

> I am currently playing with ideas on *how* to dynamically link the  
> interface code with the app but don't have a good answer yet.

At the rudimentary level, indirect pointers, just like the indirect
branch in your compiled sample.  I.e., the instruction specifies
the location of a memory address.  That address is filled in later.

>>>> So, functions overloaded with multi-language support? ...
>>>
>>> Yes. One interface per calling convention rather than one per language.
>>
>> How do you unify that across multiple languages which have different
>> methods for calling a function and different calling conventions?
>
> Wouldn't having a separate entry point or separate function for each  
> calling convention be enough?

Ok.  So, you're thinking about one function per calling convention,
which can call many OS or kernel functions.  Then, you only have a
few generic calling functions total.  This is like the DPMI or interrupt
calling method used in DOS, but you've got more of them.  And, as
long as each language correctly complies with the designated calling
convention for the language, then multiple languages could use the same
calling convention.  The methods for calling the function are handled
by the respective compiler.

> One problem is getting the calls from C and Pascal etc to the  
> appropriate entry points. If the called OS routine was named M then apps  
> should be able to just call M by name and leave the computer to convert  
> M into M_cdecl or M_stdcall or whatever, according to the convention the  
> compiler uses. I haven't yet worked out a way to do this.

a) Different namespaces?  E.g., compiler prepends some name
for C and a different name for Pascal.

b) Different access?  I.e., C compiler can only find M_cdecl
version of M, Pascal can only find M_stdcall, etc.

>> To sufficiently abstract this in the way you're wanting it to be,
>> I'd think it'd have to be via a common feature of all OSes.  Perhaps,
>> such as file I/O for platforms where "everything is a file" philosphy
>> is implemented.  E.g., you write out a text file the parameters, or
>> a IPC message as text, a function is called without parameters for
>> the host language, the function being called reads the text file and
>> parses it.  IIRC, you were discussing something like this in the past,
>> e.g., text formatted data, such as XML, and binary endian conversions
>> between modules, or across the network, etc.
>
> That is flexible but unfortunately would probably be unnecessarily slow  
> for most OS calls.

True.

>>>> I guess that depends on the similarity of the implementation of the
>>>> underlying languages.  Similar would mean that it's possible.  
>>>> Different
>>>> would mean that it's not, or not easily.
>>>
>>
>> Also, see above.  Text files might be a round-about solution.
>>
>>>> I guess that depends on the similarity of the implementation of the
>>>> underlying languages.  Similar would mean that it's possible.  
>>>> Different
>>>> would mean that it's not, or not easily.
>>>
>>> The benefit would be to have the library of OS-interface functions  
>>> provided with the OS, regardless of how many languages call that OS,  
>>> rather than one library per language per OS.
>>
>> IIUC, you can have a common "library of OS-interface functions," but  
>> each language would need to have a wrap/shim etc to call the OS.
>> E.g., if the OS API is in assembly, say like Int 0x80 for Linux,
>> C functions must wrap or shim Int 0x80 to call the OS, Pascal too,
>> Fortran too.  If the OS API is in C, then assembly, Pascal, Fortran
>> would need to wrap or shim the C function.  It's easier to wrap/shim
>> etc than it is to support more than one interface language for the OS,
>> especially if you have a large number of functions to support.  If
>> the set of OS functions is small, then the OS could support each set
>> in a multitude of languages directly, but then you've just shifted
>> the shim/wrap issue from the C, Pascal, Fortran library domain to
>> the OS API domain.
>
> By API I am not sure if you are thinking about how to get to kernel mode  
> (int 0x80 or far call etc), or are thinking of the defined set of OS  
> syscalls.

Frequently, those are equivalent.

> In either case I agree that the calls to interface code (i.e. before any  
> [transition] to kernel mode) could be wrapped/shimmed. The interface  
> code fragments - which are to be supplied by the OS - could invoke the  
> kernel so the app wouldn't have to worry about how that was done.)

It's possible different languages can use the same calling convention.
So, while I expressed that in terms of the language, i.e., where each
would use it's own unique calling convention, you're correct in that
it's really the calling convention that should be wrapped/shimmed, and
not the language.

>>> I am thinking that the library would be provided by the OS developer
>>> and all languages could use its subroutines.
>>>
>>> Is that feasible?
>>
>> I'm not real sure what you're asking here ...
>>
>> If you're specifying the OS API for your OS, then all languages  
>> implemented on your platform have to use it.
>
> In the sequence
>
>   app --> interface code --> kernel
>
> I was saying that the library of interface code would be provided by the  
> OS developer and asking if it was feasible for different languages to  
> use the functions within it. Same issue as above, and one I don't have  
> the answer to yet.

Is "the OS developer" you? ...  You keep referring to the OS developer
in the abstract or as an independent third person.

As long as each language can correctly call the "interface code" API,
then sure it's feasible.  I.e., I'm assuming a single entry-point here,
not multiple to support different calling conventions or languages, e.g.,
Int 0x80 for Linux, Int 0x21 for DOS.  Each of those is a single
entry-point with an interface method.  In those cases, they're intended
for assembly, and other languages like C need support code.

> The tool used to build such files may be easiest to write in Python  
> which is great for text processing.

EX via VI is good for line-oriented text processing, and I use it.
C is excellent for text processing, and I still use it.
AWK is good for text processing, but I don't use it much.
BASIC is excellent for text processing, but I don't use it anymore.
SNOBOL is good for text processing, but I only have a dusty
   book on it packed away somewhere ...

I.e., just because something is good for something doesn't
mean it will be used for it or even should be.

For a low use situation, your skill level with a language
is probably the most determinant factor for you.  For others,
I'd recommend going the C route.  C is going to be available
to those attempting to bootstrap an OS.  I'm not sure whether
someone in that situation will have access to Python.  I do
now because I now have Linux.


Rod Pemberton

-- 
It's time to put an end to gun violence!
Use a hammer ...

[toc] | [prev] | [standalone]

Back to top | Article view | alt.os.development

csiph-web