Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.os.development > #8326 > unrolled thread

64-bit detection

Back to article view | Back to alt.os.development

Contents

  64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-09 16:47 -0700
    Re: 64-bit detection "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-07-09 23:41 -0400
      Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-10 08:30 +0100
        Re: 64-bit detection "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-07-10 05:17 -0400
          Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-10 13:41 +0100
            Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-12 19:04 +0100
              Re: 64-bit detection "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-07-13 02:30 -0400
                Re: 64-bit detection "Alexei A. Frounze" <alexfrunews@gmail.com> - 2015-07-13 02:07 -0700
                  Re: 64-bit detection "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-07-13 23:10 -0400
                    Re: 64-bit detection "Alexei A. Frounze" <alexfrunews@gmail.com> - 2015-07-14 01:23 -0700
                Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-13 12:34 +0100
                  Re: 64-bit detection "Rod Pemberton" <boo@fasdfrewar.cdm> - 2015-07-13 22:50 -0400
                    Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-14 08:37 +0100
                      Re: 64-bit detection "Alexei A. Frounze" <alexfrunews@gmail.com> - 2015-07-14 01:41 -0700
                        Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-14 10:04 +0100
        Re: 64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-10 08:44 -0700
        Re: 64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-10 10:52 -0700
      Re: 64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-10 08:38 -0700
        Re: 64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-10 10:58 -0700
    Re: 64-bit detection CN <qmbmnp3799@pacbell.net> - 2015-07-09 23:00 -0700
      Re: 64-bit detection "Benjamin David Lunt" <zfysz@fysnet.net> - 2015-07-10 08:40 -0700
        Re: 64-bit detection "James Harris" <james.harris.1@gmail.com> - 2015-07-10 21:29 +0100

Page 1 of 2  [1] 2  Next page →

#8326 — 64-bit detection

Hi guys,

Sorry I have been away for a little while.  I have been
trying to follow some of the threads here, but haven't
had much time to read them all.

However, I had a thought a few days ago.  Will my 32-bit
OS run on a 64-bit machine?

Well, after a few tweaks and fixes, yes it does.  As far
as I can tell, it runs as if it was a 32-bit machine.

I haven't looked into 64-bit machines too much, nor at the
moment have any interest in doing so.

However, I am interested in simply detecting the 64-bit
machine and displaying a message stating so.

I thought I would dig out the .pdf's for the 64-bit and
see if there is a flag in the eflags register (while in
32-bit (non-long) mode) that might (inadvertently) indicate
this.

I haven't yet, but thought I would look into this and
was interested in your opinions.

Have you done anything for 64-bit?  Will you?  And if
so, how do you detect if 64-bit is available?

Anyway, just stopped by for a few thoughts.

Thanks,
Ben

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Forever Young Software
http://www.fysnet.net/index.htm
http://www.fysnet.net/osdesign_book_series.htm
To reply by email, please remove the zzzzzz's

Batteries not included, some Assembly required.

[toc] | [next] | [standalone]

#8327

On Thu, 09 Jul 2015 19:47:49 -0400, Benjamin David Lunt <zfysz@fysnet.net>  
wrote:

> Hi guys,

Hi Ben.

> Sorry I have been away for a little while.  I have been
> trying to follow some of the threads here, but haven't
> had much time to read them all.

...

> However, I had a thought a few days ago.  Will my 32-bit
> OS run on a 64-bit machine?

Why wouldn't it?  AFAIK, they didn't remove 32-bit mode.

[some edited changes/re-organization to Ben's comments follow]

> As far as I can tell, it runs as if it was a 32-bit machine.

...

> Well, after a few tweaks and fixes, yes it does.

Why did it need any tweaks?

What were the tweaks?

(I.e., useful for any of us to know?)

> I haven't looked into 64-bit machines too much, nor at the
> moment have any interest in doing so.

Ditto.

> However, I am interested in simply detecting the 64-bit
> machine and displaying a message stating so.
> [...]
> how do you detect if 64-bit is available?

Ok.

I have no idea how to do that.

My guess about how to detect 64-bit mode would be:

1) attempt to switch into 64-bit mode and set a flag on success
2) check one of the bits in model specific registers (MSRs)
3) see if CPUID returns something useful

Otherwise, it's a polite RTFM from me, since I need
to do so as well ...


Of course, as would be expected, at least by me anyway,
Sandpile.org has three webpages on those topics directly
off of their main webpage.  Imagine that.

http://www.sandpile.org/x86/cpuid.htm
http://www.sandpile.org/x86/msr.htm
http://www.sandpile.org/x86/flags.htm

Sandpile.org
http://www.sandpile.org

> I thought I would dig out the .pdf's for the 64-bit and
> see if there is a flag in the eflags register (while in
> 32-bit (non-long) mode) that might (inadvertently) indicate
> this.

I don't recall anything in flags.

> I haven't yet, but thought I would look into this and
> was interested in your opinions.
>
> Have you done anything for 64-bit?

No.

I did try some assembly from the 'net for 64-bit
startup for DOS.  However, it didn't work.  The guy
made some shortcuts to the mode switch which he claimed
wasn't necessary and so the code worked for his processor,
but not for mine ...

> Will you?

I'd say probably not any time soon.

I have a few problems with using 64-bit:

1) Most of my code is in C for DOS and the DOS C compilers
I use don't support 64-bit mode.  They don't seem like they're
going to be able to support 64-bit mode any time soon either.

2) Most of my assembly is for DOS.  DOS can only get into
32-bit mode with DPMI or VCPI etc.  I.e., I'd need to code
my own 64-bit startup for DOS just to use 64-bit assembly,
but I already have my 32-bit startup code and DPMI, VCPI,
etc.

3) I simply don't _need_ 64-bit, AFAICT.  64-bit Linux from
2010 updated to 2015 with many speed improvements running on
multiple cores is simply no faster than 32-bit Windows SE on
one core.

4) Linux supports 64-bit assembly and C, but I'd need to
"port" a bunch of code, which may be trivial in some cases.


Rod Pemberton

-- 
Tolerance and socialism attracts intolerance and terrorism.  See:
France, United Kingdom, Germany, Denmark, Belgium, Netherlands, ...
UK: Strong encryption, non-issue. GR: Eurozone currency, priority.

[toc] | [prev] | [next] | [standalone]

#8329

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1jgivqzyfako5@localhost...
> On Thu, 09 Jul 2015 19:47:49 -0400, Benjamin David Lunt 
> <zfysz@fysnet.net>  wrote:

...

>> However, I had a thought a few days ago.  Will my 32-bit
>> OS run on a 64-bit machine?

...

>> Well, after a few tweaks and fixes, yes it does.
>
> Why did it need any tweaks?
>
> What were the tweaks?
>
> (I.e., useful for any of us to know?)

Yes, that would be useful to know.

>> I haven't looked into 64-bit machines too much, nor at the
>> moment have any interest in doing so.
>
> Ditto.

Ditto too.

>> However, I am interested in simply detecting the 64-bit
>> machine and displaying a message stating so.
>> [...]
>> how do you detect if 64-bit is available?

I didn't know either but found that there is a CPUID bit for it.

  CPUID[0x8000_0001].EDX[29]

Almost by chance I found the following snippet in an AMD manual 
(24593-Rev. 3.23-May 2013).

  ; Use CPUID to determine if the processor supports long mode. ;
  mov eax, 80000000h ; Extended-function 8000000h.
  cpuid ; Is largest extended function
  cmp eax, 80000000h ; any function > 80000000h?
  jbe no_long_mode ; If not, no long mode.

  mov eax, 80000001h ; Extended-function 8000001h.
  cpuid ; Now EDX = extended-features flags.
  bt edx, 29 ; Test if long mode is supported.
  jnc no_long_mode ; Exit if not supported.

...

>> Have you done anything for 64-bit?
>
> No.

Nor me. Though much of the code I am writing is intended to work in 
64-bit mode as well as other modes.

...

>> Will you?
>
> I'd say probably not any time soon.

You would have to add paging if you did - something you are or were 
reluctant to do. x86-64 won't run without it.

James

[toc] | [prev] | [next] | [standalone]

#8332

On Fri, 10 Jul 2015 03:30:05 -0400, James Harris  
<james.harris.1@gmail.com> wrote:
> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
> news:op.x1jgivqzyfako5@localhost...
>> On Thu, 09 Jul 2015 19:47:49 -0400, Benjamin David Lunt  
>> <zfysz@fysnet.net>  wrote:

>>> Have you done anything for 64-bit?
>>> Will you?
>>
>> I'd say probably not any time soon.
>
> You would have to add paging if you did - something you
> are or were reluctant to do. x86-64 won't run without it.
>

Sigh, yes, I've not done that.  I don't think it would detract
 from my OS project, but it seems to just be extra work.  It
might allow it to work better with one C compiler, but I was
planning on dropping support for it.

Although, I understand that I can identity map or physical
map memory, perhaps or hopefully all of it.  I'm not sure
how 64-bit memory addressing might affect that.  IIRC, 64-bit
did something special with offsets, like sign-extension from
32-bits to 64-bit (?).  Wouldn't that affect the address range?


Rod Pemberton

-- 
Tolerance and socialism attracts intolerance and terrorism.  See:
France, United Kingdom, Germany, Denmark, Belgium, Netherlands, ...
UK: Strong encryption, non-issue. GR: Eurozone currency, priority.

[toc] | [prev] | [next] | [standalone]

#8334

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1jv3mgoyfako5@localhost...
> On Fri, 10 Jul 2015 03:30:05 -0400, James Harris 
> <james.harris.1@gmail.com> wrote:
>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
>> news:op.x1jgivqzyfako5@localhost...
>>> On Thu, 09 Jul 2015 19:47:49 -0400, Benjamin David Lunt 
>>> <zfysz@fysnet.net>  wrote:
>
>>>> Have you done anything for 64-bit?
>>>> Will you?
>>>
>>> I'd say probably not any time soon.
>>
>> You would have to add paging if you did - something you
>> are or were reluctant to do. x86-64 won't run without it.
>>
>
> Sigh, yes, I've not done that.  I don't think it would detract
> from my OS project, but it seems to just be extra work.  It
> might allow it to work better with one C compiler, but I was
> planning on dropping support for it.

As we have discussed before, your source is far too dependent on 
particular compilers, IMO, but that's your choice.

If you ever get to work on your OS again I wonder if you would remove 
some or all of that dependency...?

> Although, I understand that I can identity map or physical
> map memory, perhaps or hopefully all of it.  I'm not sure
> how 64-bit memory addressing might affect that.  IIRC, 64-bit
> did something special with offsets, like sign-extension from
> 32-bits to 64-bit (?).  Wouldn't that affect the address range?

You may be thinking of canonical addressing wherein the top N bits of an 
address have to be all the same, either all zeroes or all ones. AIUI 
these are the bits of an address prior to translation via the page 
tables, i.e. the rule applies to virtual addresses rather than physical 
ones.

That being the case you could choose to use addresses where the top bits 
were zeroes. If you identity mapped those then you would have paging 
operating (thus allowing your OS to run in the poorly named "Long Mode") 
while still working with what appear to be physical addresses. That 
should make the transition to 64-bit working invisible to much of your 
OS.

Of course, you would need to migrate to a compiler which could produce 
Long Mode object code. Just one more reason to get away from compiler 
dependence in your source.

James

[toc] | [prev] | [next] | [standalone]

#8347

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1m0bkpvyfako5@localhost...
> On Fri, 10 Jul 2015 08:41:43 -0400, James Harris
> <james.harris.1@gmail.com> wrote:
>
>> You may be thinking of canonical addressing wherein the top N bits of 
>> an  address have to be all the same, either all zeroes or all ones.
>
> How do they become set (ones) or cleared (zeroes)?
>
> Is that from sign-extension or done by user/assembler?

I don't think that the upper bits normally get *explicitly* set or 
cleared. It is just that the values given out by the memory allocator 
would be canonically valid. For example, these statements might appear 
in an app.

  xp = &x;
  y = malloc(100);

The first of those would get a canonical address because x will have 
been placed at a suitable memory address when the app was loaded.

The second would get a canonical address because malloc would have been 
set up to work with valid addresses. For example, if you wanted to, you 
could choose to have malloc use low addresses and a kmalloc to give out 
high addresses for kernel memory. Or you could have both malloc and 
kmalloc give out addresses from the low part of memory (where all 
addresses have leading zeroes) as you would if you were running with 
identity paging.

>> You may be thinking of canonical addressing wherein the top N bits of 
>> an  address have to be all the same, either all zeroes or all ones. 
>> AIUI  these are the bits of an address prior to translation via the 
>> page  tables, i.e. the rule applies to virtual addresses rather than 
>> physical  ones.
>
> How many bits are in the address for long mode? 64? 70? 80? 36? ...

Virtual addresses have 64 bits, AIUI.

> What value is N?

According to Wikipedia https://en.wikipedia.org/wiki/X86-64 on AMD's 
initial processors N would be 16 (because they require the top 16 bits 
to match) but the number for a given CPU will depend on that CPU.

> I.e., at this point in time, ISTM, that the canonical addressing
> would limit size of the total address range, if some bits must
> always be set or cleared.  E.g., if N=32, and physical addresses
> are 64-bits, the actual virtual and physical address range would
> only be 32-bits.

It does limit the range but not by much. There can be many address 
spaces and each address space will always have far more addresses than 
the machine has physical RAM.

If you want an example it seems that a number of processors allow 48 
bits of virtual address and 40 bits of physical address. They would 
thereby have 256 times as much address space as the maximum possible 
amount of RAM that the machine could have, and probably many more times 
the amount of actual RAM.

...

>> As we have discussed before, your source is far too dependent on 
>> particular compilers, IMO, but that's your choice.
>>
>> [...]
>>
>> Of course, you would need to migrate to a compiler which could 
>> produce  Long Mode object code. Just one more reason to get away from 
>> compiler  dependence in your source.
>
> I love how you always mention that in regards to _my_ code.

Sorry ... no persecution intended. :-)

Actally, when I mentioned it I was thinking that, like myself, you have 
considered getting parts of the same OS source code to compile to 32-bit 
and 64-bit x86 so I thought it may be of interest.

> My OS wasn't intended to be an OS or to be independent of those
> compilers.  An early goal was to execute executables from both
> compilers.

OK.

> So, yes, while there is a tremendous amount of
> independence, there is also much dependence too.  If I decide
> to no longer use either compiler, then a portion of the design
> of my OS will need to be reworked.

OK.

> E.g., if I decided on a Linux
> compiler, part of the OS would need to be reworked to fit Linux'
> syscall interface.  Or, if I decide to convert to my own toolchain,
> I might be free of such constraints.

I wonder if the code could be adjusted so that is not dependent on 
either DPMI or Linux and did not need to be converted to any particular 
toolchain. Perhaps such code needs an OS-specific linker and loader but 
not an OS-specific compiler. Maybe even the linker could be a generic 
one.

> Hosted C compilers emit code for a specific host OS, specific
> processor mode, specific memory set up such as paging or
> segmented or flat, interrupt method, position independent code
> or not, static or dynamic linked code, C run-time, etc.

You raise some very interesting points. I haven't, until recently, had 
to consider such things and I have been deciding on a few application 
memory layouts (and checking that there are ways to get an *existing* 
build chain to use them). That has delayed my responses to you in the 
"How to bind HLLs to OS calls" thread. I have two of your posts there to 
get back to once I have worked something out. I think I am nearly there. 
I can, at least, make a comment or two here on the points you mentioned 
above.

RP: Hosted C compilers emitting code for:

RP: specific host OS.

If you mean the hosting of the generated code then IMO the compiler 
should not generate *any* OS-dependent code. It should be able to do 
that as long as the source code doesn't make any calls specific to a 
certain OS, shouldn't it?

RP: specific processor mode.

If you mean 16-bit or 64-bit etc then that's OK. Emitting code for a 
particular CPU mode is what the compiler *should* do.

RP: a specific memory setup such as paging or segmented or flat.

So far I intend all apps just to use a flat memory. Perhaps it is odd 
that anything else was ever considered. Exposing the underlying 
different segmentation models so directly to apps was always a bad idea, 
IMO. I can explain that further if you wish but don't want to bore you. 
Suffice to say that when all apps use flat memory (or think they are 
using flat memory) the issues don't arise.

RP: interrupt method.

I am not sure what you mean there but you may be referring to writing 
interrupt-handling code in C. I do that but use assembly to set up a 
suitable environment first. (To be clear, the assembly you saw before 
had two parts: 1. Receive the interrupt and save CPU state which was the 
bulk of the code, 2. set up the environment ready for HLL calls, which 
was small.)

RP: position-independent code or not.

This is a hot topic in that it is something I have been looking at in 
the past couple of days. I have looked at how compilers emit code for 
x86 and ARM for both stand alone apps (not PIC) and shared libraries 
(PIC). I am not sure whether or not I like the existing approach which 
makes PIC slow and bulky but I can at least see why it is done that way: 
shared libraries need to access their static and global data, if they 
have any. I may have an answer to this that is not compiler-depdendent 
but it would make this post overlong so I won't include it just now.

RP: statically or dynamically linked code.

I would want to allow programmers to use the OS as they wish but my 
preference is to have apps distributed as folders of object modules. As 
such, the OS itself would link them so all such linking would appear to 
be dynamic but could use static mechanisms. I am not sure whether that 
covers what you meant, though.

RP: C run time.

Possibly a particular compiler's C run time can be avoided by 1) not 
having a C main function and 2) not invoking any features from the C 
run-time code...?

> IIRC, you're using multiple 16-bit and 32-bit C compilers too
> which aren't part of your own toolchain.  So, I think you're
> going to run into a bunch of dependencies too.  I don't see
> how you can avoid it entirely while using a hosted C compiler.

Do the above points cover the things you had in mind? I should say that 
I am not convinced yet. I am just feeling my way forward on this.

> AIUI, both the OS and it's C compiler must be bootstrapped
> and working together to not have such issues.  That, of course,
> comes after the fact, unless you decide to use C compilers from
> another host as-is for your OS and after your OS is complete,
> like I was doing with DOS compilers.
>
> hosted C compiler -> OS in C -> C compiler ported/written for new OS

ISTM OK for a compiler to generate code for a particular CPU mode. That 
is a fundamental job of a compiler. But I am not sure that a compiler 
should be tied to a particular OS. Maybe it should. Maybe it shouldn't. 
I can see that many are. I am just not sure yet that that is a good or 
necessary approach.

In the other thread we spoke about OS calls. The mechanism used in that 
to make OS calls should work on any compiler and *not* use any compiler 
dependencies so maybe that is an example of avoiding the OS-dependencies 
built in to a compiler.

> For my original design, the initial and final C compilers above are
> the same.  I don't have to bootstrap, port, or code a C compiler
> for my OS.  I use the same C compiler I used to write the OS.

OK.

> For
> your design, they're likely going to be different.  You need a
> hosted C compiler that can produce host independent code, and once
> the initial OS is complete, you're going to need another C compiler
> designed to specifically work with your OS, i.e., hosted, that must
> be bootstrapped, ported, or written from scratch.  So, the need for
> compiler independence for your design is greater, IMO IIUC.

What would it take for a compiler to be independent of the hosting OS? 
Some suggestions:

* The compiler, when compiling code, should not make calls that are 
dependent on a particular OS. It should handle IO and the like by having 
a library to interface the compiler's calls to those of the underlying 
OS.

* The compiler should not generate code which is dependent on a 
particular OS. Perhaps it should handle this by having the compiler do 
only what it is told to do...!

James

[toc] | [prev] | [next] | [standalone]

#8350

On Sun, 12 Jul 2015 14:04:51 -0400, James Harris  
<james.harris.1@gmail.com> wrote:

>> Hosted C compilers emit code for a specific host OS, specific
>> processor mode, specific memory set up such as paging or
>> segmented or flat, interrupt method, position independent code
>> or not, static or dynamic linked code, C run-time, etc.
>
> You raise some very interesting points. I haven't, until recently, had  
> to consider such things and I have been deciding on a few application  
> memory layouts (and checking that there are ways to get an *existing*  
> build chain to use them). That has delayed my responses to you in the  
> "How to bind HLLs to OS calls" thread. I have two of your posts there to  
> get back to once I have worked something out. I think I am nearly there.  
> I can, at least, make a comment or two here on the points you mentioned  
> above.

I've over a dozen of yours, plus most of yours over the past week, and
basically all of Alexei's ...

> RP: Hosted C compilers emitting code for:
>
> RP: specific host OS.
>
> If you mean the hosting of the generated code then IMO the compiler  
> should not generate *any* OS-dependent code. It should be able to do  
> that as long as the source code doesn't make any calls specific to a  
> certain OS, shouldn't it?

It depends on the host OS.

AIR, C compilers for certain OSes call a host OS function in the prolog
of each function.  E.g., perhaps a C compiler calls a stack size check,
a privilege level check, an OS stack allocation routine, or a call trace
routine in the prolog.  If there is a stack allocation routine in the
prolog, there might be a free routine in the epilog too.  It's possible
that a C compiler for DOS which is independent of DOS calls BIOS in
compiled code ...  If it does, it'd be OS-independent.  AIR, C++ compilers
call the "new" and "delete" operators behind the scenes in the compiled
C++ code, i.e., you'd need to implement those memory allocation functions.
You can't say that the expected state of DFflag is not OS-dependent.
E.g., if your OS expects DF to be set, but the host compiler clears DF.
Obviously, host specific code is needed for main() and C-runtime.

> RP: specific processor mode.
>
> If you mean 16-bit or 64-bit etc then that's OK. Emitting code for a  
> particular CPU mode is what the compiler *should* do.

Yes, but you don't get to decide how that mode is set up.  You're stuck
with whatever the C compiler needs for generating code for the host.

E.g., 32-bit, ring-3, code/data segments limited to 1MB in size ...
Obviously, you'd prefer a 32-bit compiler that emitted code for ring-0,
code/data segments 4GB, flat memory, etc.

> RP: a specific memory setup such as paging or segmented or flat.
>
> So far I intend all apps just to use a flat memory. Perhaps it is odd  
> that anything else was ever considered. Exposing the underlying  
> different segmentation models so directly to apps was always a bad idea,  
> IMO. I can explain that further if you wish but don't want to bore you.  
> Suffice to say that when all apps use flat memory (or think they are  
> using flat memory) the issues don't arise.

Yes, but, again, if you use a hosted C compiler, you don't get to decide
how that is set up.  E.g., if you use a Windows C compiler that used
segmented memory setup, your OS has to implement that same model to use
that code.

> RP: interrupt method.
>
> I am not sure what you mean there but you may be referring to writing  
> interrupt-handling code in C. I do that but use assembly to set up a  
> suitable environment first. (To be clear, the assembly you saw before  
> had two parts: 1. Receive the interrupt and save CPU state which was the  
> bulk of the code, 2. set up the environment ready for HLL calls, which  
> was small.)

The C compiler shouldn't generate code for the C language which calls
interrupts as you're not calling the C library or the OS, but it might.
If the C code when compiled generates an interrupt, then it'll emit
instructions specific to that OS, e.g., Int 21h for DOS vs. Int 80h
for Linux or perhaps a BIOS call.  Some C compilers for just the
C language, not C libraries, aren't 100% clean.

> RP: position-independent code or not.
>
> This is a hot topic in that it is something I have been looking at in  
> the past couple of days. I have looked at how compilers emit code for  
> x86 and ARM for both stand alone apps (not PIC) and shared libraries  
> (PIC). I am not sure whether or not I like the existing approach which  
> makes PIC slow and bulky but I can at least see why it is done that way:  
> shared libraries need to access their static and global data, if they  
> have any. I may have an answer to this that is not compiler-depdendent  
> but it would make this post overlong so I won't include it just now.

I mentioned this one because of DJGPP apps versus OpenWatcom apps.
DJGPP uses a split memory model.  The app's virtual starting address
is zero, no matter where it's loaded, but memory below 1MB is hard
mapped and accessed via special functions.  So, you can load a DJGPP
app to physical address 1MB or 2MB or whatever and you can execute it
there, and you can load multiple apps.  OpenWatcom apps aren't relative
like this.  They're coded for a specific address.  It has to be loaded
to that address to be executed.  So, if you want multiple OW apps to
load and execute, you're out of luck.  You can adjust the base address
of the segment to allow app to execute where loaded, but if it attempts
to access low memory, e.g., screen, the addresses will be wrong.
Basically, you'd need a way to use paging to map all loaded OW apps to
the same address.  I don't believe that can be done.

> RP: statically or dynamically linked code.
>
> I would want to allow programmers to use the OS as they wish but my  
> preference is to have apps distributed as folders of object modules. As  
> such, the OS itself would link them so all such linking would appear to  
> be dynamic but could use static mechanisms. I am not sure whether that  
> covers what you meant, though.

If you intend to use dynamically linked apps, but the compiler uses
statically linked, you'll need a compiler option to select.  If that's
not available, you're out of luck.  You're stuck with whatever model
the compiler uses.

> RP: C run time.
>
> Possibly a particular compiler's C run time can be avoided by 1) not  
> having a C main function and 2) not invoking any features from the C  
> run-time code...?

...

> In the other thread we spoke about OS calls. The mechanism used in that  
> to make OS calls should work on any compiler and *not* use any compiler  
> dependencies so maybe that is an example of avoiding the OS-dependencies  
> built in to a compiler.

Ok.  But, does any existing hosted compiler already implement that?
(IIRC, no ...)

> What would it take for a compiler to be independent of the hosting OS?

Did you mean "compiler proper" or "compiler emitted code" here? ...

The compiler itself is definately going to be OS dependent just to read
and write files or do stdio/stdin/stdout or to allocate memory, when
compiling code.  Yes?

So, I'm thinking you meant emitted code:

0) be able to compile code for multiple processors
1) be able to emit code for different memory models than the OS uses
2) be able to emit code for processor modes other than what the OS uses
3) be able to emit code which doesn't use any OS data structures
4) be able to remove all calls to the OS especially prolog/epilog and C  
language
5) be able to remove all calls to the C libraries  (much can be  
independent ...)
6) not be C++ due to "new" and "delete" memory allocation
7) not use an OS call or memory allocation call to set up a parameter stack
8) not accept parameters or provide a return from main()
9) be able to select register or stack calling convention
10) be able to change parameter convention, e.g., stdcall, cdecl, pascal,  
etc.
...

Basically, it would need to be able to do two things to be independent
*enough* for you to use, not necessarily 100% independent:

1) do what the host OS needs
2) do what your OS needs
3) you don't need to care about anything else or any other OS

ISTM, that you really need a full featured C compiler, perhaps GCC.


Rod Pemberton

-- 
Tolerance and socialism attracts intolerance and terrorism.  See:
France, United Kingdom, Germany, Denmark, Belgium, Netherlands, ...
UK: Strong encryption, non-issue. GR: Eurozone currency, priority.

[toc] | [prev] | [next] | [standalone]

#8353

On Sunday, July 12, 2015 at 11:31:10 PM UTC-7, Rod Pemberton wrote:
...
> I mentioned this one because of DJGPP apps versus OpenWatcom apps.
> DJGPP uses a split memory model.  The app's virtual starting address
> is zero, no matter where it's loaded, but memory below 1MB is hard
> mapped and accessed via special functions.  So, you can load a DJGPP
> app to physical address 1MB or 2MB or whatever and you can execute it
> there, and you can load multiple apps. 

I'm familiar with how DJGPP manages memory.

> OpenWatcom apps aren't relative
> like this.  They're coded for a specific address.  It has to be loaded
> to that address to be executed.  So, if you want multiple OW apps to
> load and execute, you're out of luck. 

This I didn't know.

I recall, however, that support for nested DPMI programs (or for
starting any programs from a DPMI program?) has been somehow broken/
unreliable in Windows since Win2K or WinXP or something like that.
I think it's the one described in __maybe_fix_w2k_ntvdm_bug()
(see the DJGPP sources; I haven't dug deeper).
Which is one of the reasons I'm still unsure if I should support
DPMI in Smaller C, which should be a fairly simple and easy thing to
do since most required things are already supported (32-bit code
generation; the existing routines that invoke DOS services can be
used as a foundation for DPMI versions).
I'd really like system()/exec*()/spawn*()/whatever to work under
DPMI, if it's supported.

> You can adjust the base address
> of the segment to allow app to execute where loaded, but if it attempts
> to access low memory, e.g., screen, the addresses will be wrong.
> Basically, you'd need a way to use paging to map all loaded OW apps to
> the same address.  I don't believe that can be done.

I'm not exactly sure about OW/4G w.r.t. memory management, but
it looks like a simple relocation table would solve the problem
in both compilers without the need to use segmentation or page
translation to simulate relocation. One thing though, if you
use zero base in segment descriptors and no page translation
(or without read-only pages), you can't effectively catch or
contain NULL pointer dereferences. Sometimes you wish the IVT
was elsewhere. :) And you can actually move it (via IDTR), but
then what does the realmode code do when its IVT has been
towed away? :)

There's one thing I really liked in Watcom/DOS4GW over DJGPP/GO32.
It's that you can access the first meg without using any special
segment selectors or any additional pointer arithmetic. Want to
access the video buffer? Just do it. It's still there at 0xA0000.
Less code and it's cleaner.

> ISTM, that you really need a full featured C compiler, perhaps GCC.

And that should probably be a custom build of gcc. As I remember,
these days gcc on Linux skips allocation of small stack frames
(128 bytes or less) because Linux guarantees not to overwrite certain
amount of the user stack below ESP. This is called a red zone.
There's a way (option) to disable this feature, but I don't think
cluttering makefiles with everchanging options is a good idea.
Another thing is stack overflow checks. gcc normally emits those,
where it thinks there's a chance of overflowing, and they refer
to something in glibc. There's an option to disable this as well.
So, one really needs a custom build of gcc (cross compiler?) that
would not generate things like that. There might be yet another
problem. It looks like gcc now uses a great deal of special
sections and relies a lot on linker scripts and startup and
other special code (object constructors and exception handlers(?))
to work with them. It looks like with gcc you're going to constantly
chase changes in it and if you only work on your OS code
alone and not so often, you're probably going to upgrade gcc
rarely in order to avoid such distractions.
Maybe for custom one-man OS projects one really wants a simple
compiler that doesn't change much and doesn't break code often?

Alex

[toc] | [prev] | [next] | [standalone]

#8361

On Mon, 13 Jul 2015 05:07:44 -0400, Alexei A. Frounze  
<alexfrunews@gmail.com> wrote:

[...]

> I recall, however, that support for nested DPMI programs (or for
> starting any programs from a DPMI program?)[...]
> I'd really like system()/exec*()/spawn*()/whatever to work under
> DPMI, if it's supported.

I'm not sure what you mean ...


AFAIK, system() will start any program from a DJGPP DPMI program
including other DJGPP DPMI programs both stubbed (16-bit CRT/DPMI)
and unstubbed (raw COFF), other non-DJGPP DPMI programs such as
 from OpenWatcom or PMODEDJ for DJGPP etc, regular 16-bit DOS apps,
or internal DOS commands.

You may need to call _check_v2_prog() to correctly set or clear the
flag __system_call_cmdproc in __system_flags prior to calling system().

CWSDPMI is a re-entrant/nestable DPMI host.   Only a single instance
of CWSDPMI will be started.  Nesting won't cause additional instances
of CWSDPMI to be loaded into memory, unlike many other DPMI hosts.
Other DPMI hosts will start another instance of the host, i.e.,
consuming additional memory for each DPMI instance and nesting.


Rod Pemberton

-- 
Tolerance and socialism attracts intolerance and terrorism.  See:
France, United Kingdom, Germany, Denmark, Belgium, Netherlands, ...
UK: Strong encryption, non-issue. GR: Eurozone currency, priority.

[toc] | [prev] | [next] | [standalone]

#8367

On Monday, July 13, 2015 at 8:10:21 PM UTC-7, Rod Pemberton wrote:
> On Mon, 13 Jul 2015 05:07:44 -0400, Alexei A. Frounze  
> <...@gmail.com> wrote:
> 
> [...]
> 
> > I recall, however, that support for nested DPMI programs (or for
> > starting any programs from a DPMI program?)[...]
> > I'd really like system()/exec*()/spawn*()/whatever to work under
> > DPMI, if it's supported.
> 
> I'm not sure what you mean ...

I mean the DPMI-related NTVDM bug and the workaround as written
in DJGPP's library source code.

Alex

[toc] | [prev] | [next] | [standalone]

#8355

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1o8dle4yfako5@localhost...
> On Sun, 12 Jul 2015 14:04:51 -0400, James Harris 
> <james.harris.1@gmail.com> wrote:

...

>> I have two of your posts there to  get back to once I have worked 
>> something out. I think I am nearly there.  I can, at least, make a 
>> comment or two here on the points you mentioned  above.
>
> I've over a dozen of yours, plus most of yours over the past week, and
> basically all of Alexei's ...

Sounds like stress city! Not that we should all approach these things in 
the same way but you may have noticed that I often try to *reduce* some 
of these discussions by closing down points and not asking questions. 
That way, the correspondent isn't prompted to respond if he doesn't want 
to and that subthread can just die a natural death. If you are 
overloaded by posts to respond to perhaps you could reply but also offer 
to reduce the content. When you ask lots of questions and bring up new 
subpoints the threads can increase in size.

>> RP: Hosted C compilers emitting code for:
>>
>> RP: specific host OS.
>>
>> If you mean the hosting of the generated code then IMO the compiler 
>> should not generate *any* OS-dependent code. It should be able to do 
>> that as long as the source code doesn't make any calls specific to a 
>> certain OS, shouldn't it?
>
> It depends on the host OS.
>
> AIR, C compilers for certain OSes call a host OS function in the 
> prolog
> of each function.  E.g., perhaps a C compiler calls a stack size 
> check,

A stack size check is a good example. That is something that I may come 
back to one day in a separate thread when I've got a bit more time or 
really need it.

In the context of this discussion perhaps such a check could be included 
only as and when needed, by use of a compiler switch. Then the 
compiler's output does not have to be environment specific.

> a privilege level check, an OS stack allocation routine,

I don't agree about those as I don't think it's the compiler's job to 
automatically insert them. If the programmer wants such things (and they 
are permitted!) then he or she can code them. In the quest for portable 
apps I don't think I would even want apps to do either of those things.

> or a call trace routine in the prolog.

Again, maybe a compiler switch would insert this if the user requires it 
so that the default code would be independent of the intended execution 
environment.

> If there is a stack allocation routine in the
> prolog, there might be a free routine in the epilog too.  It's 
> possible
> that a C compiler for DOS which is independent of DOS calls BIOS in
> compiled code ...  If it does, it'd be OS-independent.

IMO compilers should not call anything other than what they are told to. 
I don't yet know how feasible that is but it seems a good maxim.

> AIR, C++ compilers
> call the "new" and "delete" operators behind the scenes in the 
> compiled
> C++ code, i.e., you'd need to implement those memory allocation 
> functions.

Memory allocation functions can be in a standard library rather than in 
the compiler.

> You can't say that the expected state of DFflag is not OS-dependent.
> E.g., if your OS expects DF to be set, but the host compiler clears 
> DF.

I think x86 flag DF is sometimes defined by the calling convention. A 
bad idea, IMO, but it would be good if compilers allowed the calling 
convention to be specified, at least for calls to external functions 
which might have been written in another language.

> Obviously, host specific code is needed for main() and C-runtime.

Yes, but neither of those is emitted by the compiler so the compiler 
should be able to remain independent of the host-specific requirements, 
IMO.

>> RP: specific processor mode.
>>
>> If you mean 16-bit or 64-bit etc then that's OK. Emitting code for a 
>> particular CPU mode is what the compiler *should* do.
>
> Yes, but you don't get to decide how that mode is set up.  You're 
> stuck
> with whatever the C compiler needs for generating code for the host.
>
> E.g., 32-bit, ring-3, code/data segments limited to 1MB in size ...

I don't understand this. IMO the compiler should know nothing about what 
ring the code will run in and the memory requirements of the app are 
what they are.

> Obviously, you'd prefer a 32-bit compiler that emitted code for 
> ring-0,
> code/data segments 4GB, flat memory, etc.

Yes. At least the emitted code shouldn't be dependent on the things you 
mention. It should just be executable code. I suppose the linker can 
carry out various gyrations to get the code to fit in the available 
memory but not the compiler.

>> RP: a specific memory setup such as paging or segmented or flat.
>>
>> So far I intend all apps just to use a flat memory. Perhaps it is odd 
>> that anything else was ever considered. Exposing the underlying 
>> different segmentation models so directly to apps was always a bad 
>> idea,  IMO. I can explain that further if you wish but don't want to 
>> bore you.  Suffice to say that when all apps use flat memory (or 
>> think they are  using flat memory) the issues don't arise.
>
> Yes, but, again, if you use a hosted C compiler, you don't get to 
> decide
> how that is set up.  E.g., if you use a Windows C compiler that used
> segmented memory setup, your OS has to implement that same model to 
> use
> that code.

Sure. Things may be done that way. I am just saying that that seems to 
me a bad idea and unnecessary.

>> RP: interrupt method.
>>
>> I am not sure what you mean there but you may be referring to writing 
>> interrupt-handling code in C. I do that but use assembly to set up a 
>> suitable environment first. (To be clear, the assembly you saw before 
>> had two parts: 1. Receive the interrupt and save CPU state which was 
>> the  bulk of the code, 2. set up the environment ready for HLL calls, 
>> which  was small.)
>
> The C compiler shouldn't generate code for the C language which calls
> interrupts as you're not calling the C library or the OS, but it 
> might.

Oh, you mean the generated code including software interrupt 
instructions? I didn't guess that. I agree that the compiler should not 
emit such instructions. If it needs services that are invoked by such 
instructions on certain OSes then the emitted code should call a library 
function which issues the software interrupt. That keeps the compiler's 
output clean and maybe allows the library function to be replaced with 
one for a different OS.

Further, the compiler's output probably should not call any library 
functions which the programmer has not asked it to.

> If the C code when compiled generates an interrupt, then it'll emit
> instructions specific to that OS, e.g., Int 21h for DOS vs. Int 80h
> for Linux or perhaps a BIOS call.  Some C compilers for just the
> C language, not C libraries, aren't 100% clean.

Yuck! Bad compiler! ..... :-(

>> RP: position-independent code or not.
>>
>> This is a hot topic in that it is something I have been looking at in 
>> the past couple of days. I have looked at how compilers emit code for 
>> x86 and ARM for both stand alone apps (not PIC) and shared libraries 
>> (PIC). I am not sure whether or not I like the existing approach 
>> which  makes PIC slow and bulky but I can at least see why it is done 
>> that way:  shared libraries need to access their static and global 
>> data, if they  have any. I may have an answer to this that is not 
>> compiler-depdendent  but it would make this post overlong so I won't 
>> include it just now.
>
> I mentioned this one because of DJGPP apps versus OpenWatcom apps.
> DJGPP uses a split memory model.  The app's virtual starting address
> is zero, no matter where it's loaded,

But can be relocated, I hope, without having to set segment registers to 
point to the start of the image...?

> but memory below 1MB is hard
> mapped and accessed via special functions.

Can you not simply ask for a pointer to the memory you want, as in the 
following?

  x = malloc(....)  /* Point to some generally-usable space */
  y = vgamem_base()  /* Point to some special memory */

I don't know if I would use exactly that last name but you get the idea. 
After each call the pointer would be what was needed and the compiler 
would not have to emit any special code which was dependent on the 
architecture of the machine it was running on.

> So, you can load a DJGPP
> app to physical address 1MB or 2MB or whatever and you can execute it
> there, and you can load multiple apps.

OK.

> OpenWatcom apps aren't relative
> like this.  They're coded for a specific address.  It has to be loaded
> to that address to be executed.  So, if you want multiple OW apps to
> load and execute, you're out of luck.  You can adjust the base address
> of the segment to allow app to execute where loaded, but if it 
> attempts
> to access low memory, e.g., screen, the addresses will be wrong.

OK.

> Basically, you'd need a way to use paging to map all loaded OW apps to
> the same address.  I don't believe that can be done.

OK.

From some of the things you have descibed over the years ISTM that the 
compilers you chose to use were a big benefit - at least initially - in 
that they did a lot to help, but that after a while their helpfulness 
has become a burden in that they had quirks and limitations, and 
adjusting to those caused a big lock-in to those specific compilers. I 
guess it would be hard now to convert the code you have to assembly and 
a portable pure-C form that could work on, say, gcc or any another 
standard C compiler.

>> RP: statically or dynamically linked code.
>>
>> I would want to allow programmers to use the OS as they wish but my 
>> preference is to have apps distributed as folders of object modules. 
>> As  such, the OS itself would link them so all such linking would 
>> appear to  be dynamic but could use static mechanisms. I am not sure 
>> whether that  covers what you meant, though.
>
> If you intend to use dynamically linked apps, but the compiler uses
> statically linked, you'll need a compiler option to select.  If that's
> not available, you're out of luck.  You're stuck with whatever model
> the compiler uses.

Yes, that may be true. I don't know for sure, yet. Best guess at this 
time is that C's flexibility may rescue me from what a compiler expects. 
I don't mean to be cryptic but this is something to come back to.

...

>> In the other thread we spoke about OS calls. The mechanism used in 
>> that  to make OS calls should work on any compiler and *not* use any 
>> compiler  dependencies so maybe that is an example of avoiding the 
>> OS-dependencies  built in to a compiler.
>
> Ok.  But, does any existing hosted compiler already implement that?
> (IIRC, no ...)

No offense intended but I could see that I hadn't explained it well 
enough and you had some misunderstandings over what I proposed. I need 
to revisit that thread but the key point here is that the mechanism I 
proposed did not require anything special of the compiler and should 
thus work on any C compiler. It only used pointers, structs and function 
calls. Nothing more. All standard C stuff, completely portable, not 
dependent on a particular C compiler, and not requiring the compiler to 
emit anything that the programmer had not written.

>> What would it take for a compiler to be independent of the hosting 
>> OS?
>
> Did you mean "compiler proper" or "compiler emitted code" here? ...

Both. The next two points in my post (which I think you have snipped) 
covered one each.

> The compiler itself is definately going to be OS dependent just to 
> read
> and write files or do stdio/stdin/stdout or to allocate memory, when
> compiling code.  Yes?

No. I was thinking that the compiler would call its standard library and 
that the standard library would call the OS that the compiler was 
running on. The library would thus interface the compiler to the OS. The 
library would be OS-specific but the compiler would not. There would be 
a separate version of the library for each hosting OS.

> So, I'm thinking you meant emitted code:
>
> 0) be able to compile code for multiple processors
> 1) be able to emit code for different memory models than the OS uses
> 2) be able to emit code for processor modes other than what the OS 
> uses
> 3) be able to emit code which doesn't use any OS data structures
> 4) be able to remove all calls to the OS especially prolog/epilog and 
> C  language
> 5) be able to remove all calls to the C libraries  (much can be 
> independent ...)
> 6) not be C++ due to "new" and "delete" memory allocation
> 7) not use an OS call or memory allocation call to set up a parameter 
> stack
> 8) not accept parameters or provide a return from main()
> 9) be able to select register or stack calling convention
> 10) be able to change parameter convention, e.g., stdcall, cdecl, 
> pascal,  etc.
> ...
>
> Basically, it would need to be able to do two things to be independent
>*enough* for you to use, not necessarily 100% independent:
>
> 1) do what the host OS needs
> 2) do what your OS needs
> 3) you don't need to care about anything else or any other OS
>
> ISTM, that you really need a full featured C compiler, perhaps GCC.

That seems to be a good list but as I am unsure what you mean it is 
probably best that I don't guess. I may have covered some of your points 
in what I wrote above.

James

[toc] | [prev] | [next] | [standalone]

#8360

On Mon, 13 Jul 2015 07:34:46 -0400, James Harris  
<james.harris.1@gmail.com> wrote:

> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message  
> news:op.x1o8dle4yfako5@localhost...
>> On Sun, 12 Jul 2015 14:04:51 -0400, James Harris  
>> <james.harris.1@gmail.com> wrote:


[...]

> IMO compilers should not call anything other than what they are told to.

I agree, but I accept reality as-is.

>>> RP: specific processor mode.
>>>
>>> If you mean 16-bit or 64-bit etc then that's OK. Emitting code for a  
>>> particular CPU mode is what the compiler *should* do.
>>
>> Yes, but you don't get to decide how that mode is set up.  You're stuck
>> with whatever the C compiler needs for generating code for the host.
>>
>> E.g., 32-bit, ring-3, code/data segments limited to 1MB in size ...
>
> I don't understand this. IMO the compiler should know nothing about what  
> ring the code will run in and the memory requirements of the app are  
> what they are.

The compiler has to generate addresses for the code and select
instructions to emit.  Those are dependent on the mode and processor
set up for that mode.  E.g., addresses are dependent on base address
of the segment in the descriptor.  Various instructions are dependent
on privilege levels.  I.e., the compiler may need to emit them for
ring-0 kernel code, but not emit them for ring-3 user code, if the OS
is set up in that manner.

>>> RP: position-independent code or not.
>>>
>>> This is a hot topic in that it is something I have been looking at in  
>>> the past couple of days. I have looked at how compilers emit code for  
>>> x86 and ARM for both stand alone apps (not PIC) and shared libraries  
>>> (PIC). I am not sure whether or not I like the existing approach  
>>> which  makes PIC slow and bulky but I can at least see why it is done  
>>> that way:  shared libraries need to access their static and global  
>>> data, if they  have any. I may have an answer to this that is not  
>>> compiler-depdendent  but it would make this post overlong so I won't  
>>> include it just now.
>>
>> I mentioned this one because of DJGPP apps versus OpenWatcom apps.
>> DJGPP uses a split memory model.  The app's virtual starting address
>> is zero, no matter where it's loaded,
>
> But can be relocated, I hope, without having to set segment registers to  
> point to the start of the image...?

Relocated, yes.

Not having to set a descriptor base address, no, AIUI.  The addresses in
the image start at zero.

I think PIC code would be needed to not have to adjust descriptors.

Perhaps paging could solve this, but I'm not sure how at this point.

>> but memory below 1MB is hard
>> mapped and accessed via special functions.
>
> Can you not simply ask for a pointer to the memory you want, as in the  
> following?
>
>   x = malloc(....)  /* Point to some generally-usable space */
>   y = vgamem_base()  /* Point to some special memory */
>
> I don't know if I would use exactly that last name but you get the idea.  
> After each call the pointer would be what was needed and the compiler  
> would not have to emit any special code which was dependent on the  
> architecture of the machine it was running on.

DJGPP has a constant or a macro that you can either add or subtract,
appropriately, to a pointer to obtain the actual address.  However,
their use requires that zero page protection be disabled, i.e., due
to address wrap.

> From some of the things you have descibed over the years ISTM that the  
> compilers you chose to use were a big benefit - at least initially - in  
> that they did a lot to help, but that after a while their helpfulness  
> has become a burden in that they had quirks and limitations, and  
> adjusting to those caused a big lock-in to those specific compilers. I  
> guess it would be hard now to convert the code you have to assembly and  
> a portable pure-C form that could work on, say, gcc or any another  
> standard C compiler.

I always already locked-in, wanting to execute apps for DJGPP.
But, yes, working around code for an OS other than my own is not
advantageous.  It works on GCC for DOS (DJGPP).  It's not "portable
pure-C" form OS.  Issues with bugs in OW's C library was what prompted
me to halt my OS project and begin working on my own tools.  In hindsight,
I probably could've just cut OW code out, continuing with DJGPP's GCC,
but OW warns on a few things GCC misses.

>>> In the other thread we spoke about OS calls. The mechanism used in  
>>> that  to make OS calls should work on any compiler and *not* use any  
>>> compiler  dependencies so maybe that is an example of avoiding the  
>>> OS-dependencies  built in to a compiler.
>>
>> Ok.  But, does any existing hosted compiler already implement that?
>> (IIRC, no ...)
>
> No offense intended but I could see that I hadn't explained it well  
> enough and you had some misunderstandings over what I proposed. I need  
> to revisit that thread but the key point here is that the mechanism I  
> proposed did not require anything special of the compiler and should  
> thus work on any C compiler. It only used pointers, structs and function  
> calls. Nothing more. All standard C stuff, completely portable, not  
> dependent on a particular C compiler, and not requiring the compiler to  
> emit anything that the programmer had not written.

Oh, sorry, that mechanism, ok, yes.

>> The compiler itself is definately going to be OS dependent just to read
>> and write files or do stdio/stdin/stdout or to allocate memory, when
>> compiling code.  Yes?
>
> No. I was thinking that the compiler would call its standard library and  
> that the standard library would call the OS that the compiler was  
> running on. The library would thus interface the compiler to the OS. The  
> library would be OS-specific but the compiler would not. There would be  
> a separate version of the library for each hosting OS.

We clearly have different definitions of, or meanings, or scope for, or
boundaries for use of "dependent" here.

You're just looking at the compiler's C code, which can be made independent
of the OS by simply not using the OS or C library, or calling a function  
not
named in the OS or C library, but the compiler still needs to call the OS  
to
do OS level work.  The compiler itself, once compiled into a binary, must
call something in the OS to allow for file I/O and memory allocation, which
makes the compiler dependent on the OS.

E.g., if the compiler must call jh_open() in the C compiler to open a file,
which calls fopen() in the C library, which calls open() in the OS, then  
the
compiler C code is technically independent of the OS.  It doesn't call any  
OS
functions or C library functions directly, but does so indirectly.  The  
compiler
itself is actually dependent on the OS for file I/O.  If the C compiler  
doesn't
call jh_open(), fopen(), open() directly or indirectly, then it's  
independent
in the sense that I meant.  That's impossible, AFAIK.


Rod Pemberton

-- 
Tolerance and socialism attracts intolerance and terrorism.  See:
France, United Kingdom, Germany, Denmark, Belgium, Netherlands, ...
UK: Strong encryption, non-issue. GR: Eurozone currency, priority.

[toc] | [prev] | [next] | [standalone]

#8365

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1qstw1myfako5@localhost...
> On Mon, 13 Jul 2015 07:34:46 -0400, James Harris 
> <james.harris.1@gmail.com> wrote:
>> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
>> news:op.x1o8dle4yfako5@localhost...
>>> On Sun, 12 Jul 2015 14:04:51 -0400, James Harris 
>>> <james.harris.1@gmail.com> wrote:

>> IMO compilers should not call anything other than what they are told 
>> to.
>
> I agree, but I accept reality as-is.

So says the man who is developing his own toolchain!

Besides, IME so far a good compiler will emit only what it is told to. 
At least that's with the exception of calling a stack checker on entry 
to a function. As mentioned before, that topic is something I may need 
to come back to.

DJGPP and OpenWatcom may call lots of things in the background. I don't 
know. But bcc and gcc seem, so far, to do just what they are told. DJGPP 
is a version of gcc, isn't it? So where did the DPMI stuff come in? Is 
DPMI a non-core add-on to DJGPP, and thus optional, or is it intrinsic 
and impossible to get away from if you use DJGPP?

>>>> RP: specific processor mode.
>>>>
>>>> If you mean 16-bit or 64-bit etc then that's OK. Emitting code for 
>>>> a  particular CPU mode is what the compiler *should* do.
>>>
>>> Yes, but you don't get to decide how that mode is set up.  You're 
>>> stuck
>>> with whatever the C compiler needs for generating code for the host.
>>>
>>> E.g., 32-bit, ring-3, code/data segments limited to 1MB in size ...
>>
>> I don't understand this. IMO the compiler should know nothing about 
>> what  ring the code will run in and the memory requirements of the 
>> app are  what they are.
>
> The compiler has to generate addresses for the code and select
> instructions to emit. Those are dependent on the mode and processor
> set up for that mode.

OK.

> E.g., addresses are dependent on base address
> of the segment in the descriptor.

Wouldn't relocation fix that?

Of course, an OS image might be loaded in bare form and therefore need 
to relocate itself so needing to begin with some relocation code.

> Various instructions are dependent
> on privilege levels.  I.e., the compiler may need to emit them for
> ring-0 kernel code, but not emit them for ring-3 user code, if the OS
> is set up in that manner.

You can always call an assembly routine if you want to run privileged 
instructions. The compiler can just do the universal stuff and not need 
to concern itself with ring0-only instructions.

>>>> RP: position-independent code or not.
>>>>
>>>> This is a hot topic in that it is something I have been looking at 
>>>> in  the past couple of days. I have looked at how compilers emit 
>>>> code for  x86 and ARM for both stand alone apps (not PIC) and 
>>>> shared libraries  (PIC). I am not sure whether or not I like the 
>>>> existing approach  which  makes PIC slow and bulky but I can at 
>>>> least see why it is done  that way:  shared libraries need to 
>>>> access their static and global  data, if they  have any. I may have 
>>>> an answer to this that is not  compiler-depdendent  but it would 
>>>> make this post overlong so I won't  include it just now.
>>>
>>> I mentioned this one because of DJGPP apps versus OpenWatcom apps.
>>> DJGPP uses a split memory model.  The app's virtual starting address
>>> is zero, no matter where it's loaded,
>>
>> But can be relocated, I hope, without having to set segment registers 
>> to  point to the start of the image...?
>
> Relocated, yes.
>
> Not having to set a descriptor base address, no, AIUI.  The addresses 
> in
> the image start at zero.
>
> I think PIC code would be needed to not have to adjust descriptors.
>
> Perhaps paging could solve this, but I'm not sure how at this point.

I don't know enough about DGJPP or OW to comment and I suspect you are 
happy with what you have. That said, if you want to try to convert to 
portable/gcc source I would be interested to try and help. The whole 
DPMI stuff has always been a bit of a mystery and it's something I would 
genuinely be interested in - at least in learning how to convert away 
from! Otherwise I'll leave the subthread there.

...

>>> The compiler itself is definately going to be OS dependent just to 
>>> read
>>> and write files or do stdio/stdin/stdout or to allocate memory, when
>>> compiling code.  Yes?
>>
>> No. I was thinking that the compiler would call its standard library 
>> and  that the standard library would call the OS that the compiler 
>> was  running on. The library would thus interface the compiler to the 
>> OS. The  library would be OS-specific but the compiler would not. 
>> There would be  a separate version of the library for each hosting 
>> OS.
>
> We clearly have different definitions of, or meanings, or scope for, 
> or
> boundaries for use of "dependent" here.
>
> You're just looking at the compiler's C code, which can be made 
> independent
> of the OS by simply not using the OS or C library, or calling a 
> function  not
> named in the OS or C library, but the compiler still needs to call the 
> OS  to
> do OS level work.  The compiler itself, once compiled into a binary, 
> must
> call something in the OS to allow for file I/O and memory allocation, 
> which
> makes the compiler dependent on the OS.

I am not sure what you mean by "the compiler's C code" and I may have 
misunderstood more of that paragraph (incidentally, the line endings 
seem to have come in odd places. Is that your Usenet interface or mine?) 
but I think of there being these units

  1. A compiler
  2. A standard library for Unix
  3. A standard library for DOS
  4. A standard library for MacOS
  5. A standard library for HomeBrewOS
etc

The point being that each library would provide the same functions. It's 
just that their implementations would differ. The single compiler should 
be able to interact with different operating systems by being linked 
with the appropriate library.

> E.g., if the compiler must call jh_open() in the C compiler to open a 
> file,
> which calls fopen() in the C library, which calls open() in the OS, 
> then  the
> compiler C code is technically independent of the OS.  It doesn't call 
> any  OS
> functions or C library functions directly, but does so indirectly. 
> The  compiler
> itself is actually dependent on the OS for file I/O.  If the C 
> compiler  doesn't
> call jh_open(), fopen(), open() directly or indirectly, then it's 
> independent
> in the sense that I meant.  That's impossible, AFAIK.

Yes, the compiler would ultimately have to get the OS to interact with 
the outside world. I was just saying that the compiler wouldn't itself 
have to make any calls which were specific to a certain OS. If could 
just focus on its input (source code and headers) and its output 
(assembly or machine code). I thought the standard libraries were there 
to interface the compiler or any other app to the OS. As we discussed 
elsewhere, though, it seems that the standard libraries include OS 
interfacing.

James

[toc] | [prev] | [next] | [standalone]

#8368

On Tuesday, July 14, 2015 at 12:37:15 AM UTC-7, James Harris wrote:
...
> DJGPP and OpenWatcom may call lots of things in the background. I don't 
> know. But bcc and gcc seem, so far, to do just what they are told.

Like I said elsewhere in the (other?) thread, gcc has a habit of
generating calls and references to the guts of the standard library.
Pre-main() initialization, stack overflow checks and so on.

> DJGPP 
> is a version of gcc, isn't it?

It is, but gcc is highly configurable and varies across different
host OSes even on the same x86 architecture.

> So where did the DPMI stuff come in? Is 
> DPMI a non-core add-on to DJGPP, and thus optional, or is it intrinsic 
> and impossible to get away from if you use DJGPP?

gcc does not generate 16-bit x86 code (if it did, you would've probably
already used it for your boot loader:). So, in DOS there's no way to
run 32-bit other than in protected mode. And for that very thing DPMI
was invented. So, there's some start up code that queries DPMI
availability, switches to protected mode, sets up segments and page
translation using DPMI services as needed for the program and then
goes on to main() at some point. malloc() and such end up invoking
DPMI memory allocation functions. DPMI is a software layer to
bridge 32-bit protected mode code with 16-bit code of DOS and BIOS.

...
> The point being that each library would provide the same functions. It's 
> just that their implementations would differ. The single compiler should 
> be able to interact with different operating systems by being linked 
> with the appropriate library.

Not necessarily. Again, as it has been mentioned before, there are some
OS-specific things that the compiler may choose to use or is actually
forced to use:
1. 128-byte "red-zone" below ESP/RSP on Linux is guaranteed to not be
overwritten by the OS (ISRs, signal handlers) and gcc may not
explicitly subtract the size of the local variables from ESP/RSP and
thus save an instruction in many places.
2. More then 4KB worth of local variables on the stack on Windows
require special handling due to the way Windows automatically grows
the stack as it's being used. Windows checks that it doesn't grow by
more than 4KB at a time (more is considered a programming error). So,
if you allocate an 8KB buffer and write to its first byte, you get a
crash induced by the OS. To avoid it, you need to touch a byte of
the stack in every 4KB page, from ESP/RSP downwards.

So, in practice it's not as simple.

Alex

[toc] | [prev] | [next] | [standalone]

#8369

"Alexei A. Frounze" <alexfrunews@gmail.com> wrote in message 
news:aea7adff-b96a-492c-a592-14ffbf5bb026@googlegroups.com...
> On Tuesday, July 14, 2015 at 12:37:15 AM UTC-7, James Harris wrote:
> ...
>> DJGPP and OpenWatcom may call lots of things in the background. I 
>> don't
>> know. But bcc and gcc seem, so far, to do just what they are told.
>
> Like I said elsewhere in the (other?) thread, gcc has a habit of
> generating calls and references to the guts of the standard library.
> Pre-main() initialization, stack overflow checks and so on.

Main does not need to be used for an OS (thus avoiding linking the C 
runtime) and if main *is* used (as it would be for an app) a custom C 
runtime can be provided.

Even with main being present the C runtime calls main, not the other way 
around, AIUI, so the pre-main initialisation you mention does not need 
to affect the compiler's output. IME gcc will compile a function, even 
main, I believe, to do just what the programmer has coded (with the 
possible exception of stack checking, mentioned below).

I mentioned stack overflow checks in the paragraph preceding the one you 
quoted above as something I'll have to come back to.

...

Thanks for the explanation of DPMI (snipped).

>> The point being that each library would provide the same functions. 
>> It's
>> just that their implementations would differ. The single compiler 
>> should
>> be able to interact with different operating systems by being linked
>> with the appropriate library.
>
> Not necessarily. Again, as it has been mentioned before, there are 
> some
> OS-specific things that the compiler may choose to use or is actually
> forced to use:
> 1. 128-byte "red-zone" below ESP/RSP on Linux is guaranteed to not be
> overwritten by the OS (ISRs, signal handlers) and gcc may not
> explicitly subtract the size of the local variables from ESP/RSP and
> thus save an instruction in many places.
> 2. More then 4KB worth of local variables on the stack on Windows
> require special handling due to the way Windows automatically grows
> the stack as it's being used. Windows checks that it doesn't grow by
> more than 4KB at a time (more is considered a programming error). So,
> if you allocate an 8KB buffer and write to its first byte, you get a
> crash induced by the OS. To avoid it, you need to touch a byte of
> the stack in every 4KB page, from ESP/RSP downwards.

I am aware of that, and that Windows provides a function (called _stkchk 
or _chkstk or something similar) to do such stack extension.

Paging OSes can handle stack extension and checking differently from 
non-paging ones. x86 and ARM have different mechanisms to do such 
checks. Leaf functions can work differently to ones which call others, 
etc, and I have some (incomplete) ideas on how to approach such checks 
portably but I'll have to come back to the topic.

In this context, though, are you saying that gcc, say, has to know what 
OS it is compiling for, or at least what stack checking mechanism its 
host OS uses? Can the compiler not be told what stack checking mechanism 
to use by some command-line switch?

If a compiler inserted, say, Windows-style stack checks in every 
function then the generated code could not be expected to work elsewhere 
and the compiler would not be suitable for writing OS code so I would be 
surprised if the checking mechanism was fixed.

James

[toc] | [prev] | [next] | [standalone]

#8338

"James Harris" <james.harris.1@gmail.com> wrote in message 
news:mnns78$v0t$1@dont-email.me...
> "Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
> news:op.x1jgivqzyfako5@localhost...

>> (I.e., useful for any of us to know?)
>
> Yes, that would be useful to know.

Please see my reply to Rod.

> I didn't know either but found that there is a CPUID bit for it.
>
>  CPUID[0x8000_0001].EDX[29]
>
> Almost by chance I found the following snippet in an AMD manual 
> (24593-Rev. 3.23-May 2013).
>
>  ; Use CPUID to determine if the processor supports long mode. ;
>  mov eax, 80000000h ; Extended-function 8000000h.
>  cpuid ; Is largest extended function
>  cmp eax, 80000000h ; any function > 80000000h?
>  jbe no_long_mode ; If not, no long mode.
>
>  mov eax, 80000001h ; Extended-function 8000001h.
>  cpuid ; Now EDX = extended-features flags.
>  bt edx, 29 ; Test if long mode is supported.
>  jnc no_long_mode ; Exit if not supported.

This was my conclusion too, though I hadn't found this reference
yet.

Thanks,
Ben

[toc] | [prev] | [next] | [standalone]

#8339

"James Harris" <james.harris.1@gmail.com> wrote in message 
news:mnns78$v0t$1@dont-email.me...
>
> I didn't know either but found that there is a CPUID bit for it.
>
>  CPUID[0x8000_0001].EDX[29]
>
> Almost by chance I found the following snippet in an AMD manual 
> (24593-Rev. 3.23-May 2013).
>

For future reference and for those who might be looking for the
same answer, here is my code to check for a 64-bit machine.

// Use CPUID to determine if the processor supports long mode:
// reference:
//  - AMD manual (24593-Rev. 3.25-June 2015)
//    http://support.amd.com/TechDocs/24593.pdf
//  - Intel manual Vol 3C, page 34-6, section 34.4.1.1
//           (325384-055US, June 2015)
//    "Support for Intel 64 architecture is reported by:
//           CPUID.80000001:EDX[29] = 1"
//  - Intel manual Vol 2A, page 3-186, Table 3-17
//           (253666-055US, June 2015)
_naked bool is_64bit_cpu() {
  _asm (
    "  mov  eax,80000000h       ; Get highest extended function\n"
    "  cpuid                    ; \n"
    "  cmp  eax,80000001h       ; is at least 80000001h allowed?\n"
    "  jb   short @f            ; If not, long mode not support\n"
    "  mov  eax,80000001h       ; Get extended function 8000001h flags\n"
    "  cpuid                    ; EDX = extended-features flags\n"
    "  bt   edx,29              ; If bit 29 is set, long mode is 
supported\n"
    "  jnc  short @f            ; \n"
    "  mov  eax,1               ; return TRUE\n"
    "  ret                      ; we're done\n"
    "@@:                        ; \n"
    "  xor  eax,eax             ; return FALSE\n"
    "  ret                      ; we're done\n"
  );
}

Notes:
 - I use Alex's SmallerC to compile this to .asm
     https://github.com/alexfru/SmallerC
 - Then I use my NBASM to assemble to binary
     http://www.fysnet.net/newbasic.htm
 - The _naked keyword tells SmallerC not to include
    the epilog and prolog (an addition I made to SmallerC)

Thanks guys for your comments.

Ben

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Forever Young Software
http://www.fysnet.net/index.htm
http://www.fysnet.net/osdesign_book_series.htm
To reply by email, please remove the zzzzzz's

Batteries not included, some Assembly required.

[toc] | [prev] | [next] | [standalone]

#8336

"Rod Pemberton" <boo@fasdfrewar.cdm> wrote in message 
news:op.x1jgivqzyfako5@localhost...
> On Thu, 09 Jul 2015 19:47:49 -0400, Benjamin David Lunt <zfysz@fysnet.net> 
> wrote:
>
>> Hi guys,
>
> Hi Ben.
>
>> Sorry I have been away for a little while.  I have been
>> trying to follow some of the threads here, but haven't
>> had much time to read them all.
>
> ...
>
>> However, I had a thought a few days ago.  Will my 32-bit
>> OS run on a 64-bit machine?
>
> Why wouldn't it?  AFAIK, they didn't remove 32-bit mode.

As you probably can tell, I haven't read about it much at all.
My first experience was trying to install WinXP SP3 (32-bit)
on a 64-bit machine.  It didn't even get past the initialization.
Don't know why, but it wouldn't install.

I later tried to run my OS on that same machine and it crashed
with a tripple fault.

I was just curious so at that time, I didn't persue the
tripple fault.

>> Well, after a few tweaks and fixes, yes it does.
>
> Why did it need any tweaks?
> What were the tweaks?

The modifications that I made may not have anything to do with
64-bit versus 32-bit cpus, it just so happened that I had to
make them to get it to work on a 64-bit machine.

For example, the first change had nothing to do with the CPU
and everything to do with my code.  As you probably remember,
we recently had a discussion on the BIOS' memory reporting.
On this particular machine, it was reporting just over 6 gig
of RAM.  Since that value uses at least 33 bits and my code
used a 32-bit value to store the total RAM, I was getting far
less than the 6-gig I expected.

A simple change from:

  bit32u  max_memory_in_bytes;

to

  bit64u  max_memory_in_bytes;

solved the issue.  My memory handler uses 64-bit integers, but
I had completely forgotten about the limit check which still used
a 32-bit value. :-)

As for the triple fault, this was a little more complicated.
My loader file runs in unreal mode and has a 32-bit stack located
above the 1 meg mark  (at 4 meg if I remember correctly, but the
location is arbitrary, can be anywhere in the 32-bit range
and of any size).[1]

However, the loader file still calls the BIOS which expects
the SS:SP value to be below the 1 meg limit.  Therefore, I
call a BIOS wrapper that sets up a 16-bit stack, copies the
relevant stack contents from the 32-bit stack, calls the BIOS
and upon return, copies the relevant stack contents from the
16-bit stack to the 32-bit stack, then returning.

It just so happened that one of my functions was calling the
BIOS, bypassing the wrapper, and working just fine on a
32-bit machine.  However, just as soon as I moved to the
64-bit machine, it faulted.

It was by chance that the high part of ESP was either not
being modified or was being preserved on return from the
call on the 32-bit machine, but on the 64-bit machine, the
high 32-bits was being modified and not preserved.  Faulty
BIOS for not preserving all of ESP?  Maybe, but why would it
have to save it knowing that in general, a call to the BIOS
would be from a 16-bit environment?

Anyway, I wrapped that BIOS call, changing

  _asm {
     mov  eax,service number
     ...
     int  interrupt_num
  }

to

  regs.eax = service number
  ...
  intx(interrupt_num, &regs);

and we were in business.

Anyway, with these modifications, you are absolutely right,
as far as I can tell, my OS runs on the 64-bit CPU as if it
was a 32-bit CPU, on top of that, paging is enabled.

My purpose for this thread is simply to indicate to the user,
especially myself in the debug logs, that the CPU is a 64-bit
processor.  With the other posts in this thread, I believe
I will be able to do that just fine.

Thanks guys,
Ben

[1] Just an interesting note, this 32-bit stack which is
at least 4 meg in size is used just fine in C code produced
by Alex's SmallerC.  All I did was include some assembly
code to set up the stack and his code generation uses
this 32-bit stack just fine.  Which reminds me, I need
to finish those promised modifications so that I can release
my modified version of his compiler.  So much to do, so little
time to do it in...

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Forever Young Software
http://www.fysnet.net/index.htm
http://www.fysnet.net/osdesign_book_series.htm
To reply by email, please remove the zzzzzz's

Batteries not included, some Assembly required. 

[toc] | [prev] | [next] | [standalone]

#8340

"Benjamin David Lunt" <zfysz@fysnet.net> wrote in message 
news:mnp0o7$c64$1@speranza.aioe.org...

correction:

> It was by chance that the high part of ESP was either not
> being modified or was being preserved on return from the
> call on the 32-bit machine, but on the 64-bit machine, the
> high 32-bits was being modified and not preserved.  Faulty

[ Should have been "high 16 bits" ]

> BIOS for not preserving all of ESP?  Maybe, but why would it
> have to save it knowing that in general, a call to the BIOS
> would be from a 16-bit environment?

Ben

[toc] | [prev] | [next] | [standalone]

#8328

On 7/9/2015 4:47 PM, Benjamin David Lunt wrote:
> Hi guys,
>
> Sorry I have been away for a little while.  I have been
> trying to follow some of the threads here, but haven't
> had much time to read them all.
>
> However, I had a thought a few days ago.  Will my 32-bit
> OS run on a 64-bit machine?

Depends on the platform. But since you mentioned eflags below, you are 
obviously talking about x86. The answer for the x86 PC architecture is yes.

> Well, after a few tweaks and fixes, yes it does.  As far
> as I can tell, it runs as if it was a 32-bit machine.

It is very strange that it required any fixes. 64-bit machines are 100% 
backward compatible with 32-bit machines. Could you share with us what 
the fixes were?

> I haven't looked into 64-bit machines too much, nor at the
> moment have any interest in doing so.
>
> However, I am interested in simply detecting the 64-bit
> machine and displaying a message stating so.
>
> I thought I would dig out the .pdf's for the 64-bit and
> see if there is a flag in the eflags register (while in
> 32-bit (non-long) mode) that might (inadvertently) indicate
> this.
>
> I haven't yet, but thought I would look into this and
> was interested in your opinions.

Opinions? This is not an undeterministic subject like the meaning of 
life where people can have different opinions, each person with their 
own truth. It's not even an East vs West thing. There is exactly one 
correct answer to this question.

> Have you done anything for 64-bit?  Will you?  And if
> so, how do you detect if 64-bit is available?

Yes. Like the FM says: by checking the LM (long mode) bit in CPUID. 
Here's a link for you: 
http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | alt.os.development

csiph-web