Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #381741 > unrolled thread

Build system proposal

Back to article view | Back to comp.lang.c

Contents

  Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-04 15:39 +0000
    Re: Build system proposal bart <bc@freeuk.com> - 2024-02-04 15:58 +0000
      Re: Build system proposal scott@slp53.sl.home (Scott Lurndal) - 2024-02-04 16:58 +0000
        Re: Build system proposal bart <bc@freeuk.com> - 2024-02-04 17:15 +0000
          Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-04 17:40 +0000
        Re: Build system proposal Ben Bacarisse <ben.usenet@bsb.me.uk> - 2024-02-04 23:32 +0000
          Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-04 23:58 +0000
            Re: Build system proposal Ben Bacarisse <ben.usenet@bsb.me.uk> - 2024-02-05 14:42 +0000
              Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-05 15:17 +0000
                Re: Build system proposal bart <bc@freeuk.com> - 2024-02-05 16:00 +0000
                  Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-05 16:51 +0000
                    Re: Build system proposal bart <bc@freeuk.com> - 2024-02-05 17:39 +0000
                      Re: Build system proposal "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2024-02-05 11:13 -0800
                      Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-05 23:41 +0000
                  Re: Build system proposal Michael S <already5chosen@yahoo.com> - 2024-02-05 19:14 +0200
                Re: Build system proposal David Brown <david.brown@hesbynett.no> - 2024-02-05 20:47 +0100
                  Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-05 21:19 +0000
                    Re: Build system proposal Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2024-02-05 13:38 -0800
                    Re: Build system proposal David Brown <david.brown@hesbynett.no> - 2024-02-06 11:10 +0100
                  Re: Build system proposal Michael S <already5chosen@yahoo.com> - 2024-02-05 23:26 +0200
                  Re: Build system proposal bart <bc@freeuk.com> - 2024-02-05 21:28 +0000
                    Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-05 23:38 +0000
                      Re: Build system proposal David Brown <david.brown@hesbynett.no> - 2024-02-06 12:56 +0100
                    Re: Build system proposal David Brown <david.brown@hesbynett.no> - 2024-02-06 12:47 +0100
                Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-05 23:34 +0000
                  Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-06 01:41 +0000
                    Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-06 02:27 +0000
                      Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-06 02:59 +0000
                        Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-06 03:09 +0000
                          Re: Build system proposal "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2024-02-05 22:33 -0800
                      Re: Build system proposal "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2024-02-05 22:32 -0800
                        Re: Build system proposal Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-02-06 07:25 +0000
                          Re: Build system proposal "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> - 2024-02-05 23:33 -0800
                            Re: Build system proposal Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2024-02-06 08:20 +0000
              Re: Build system proposal Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2024-02-05 08:19 -0800
          Re: Build system proposal Tim Rentsch <tr.17687@z991.linuxsc.com> - 2024-02-04 23:24 -0800
    Re: Build system proposal scott@slp53.sl.home (Scott Lurndal) - 2024-02-04 16:32 +0000

Page 2 of 2 — ← Prev page 1 [2]

#381853

On 05/02/2024 19:47, David Brown wrote:
> On 05/02/2024 16:17, Malcolm McLean wrote:

>> The library would have facilities for working with lists of strings, 
>> of course. But C can do that perfectly well, and I don't see it as big 
>> problem.
>>
> 
> Have you ever tried /any/ other programming language?  Apart from 
> assembly and Forth, I have not seen a language that is more cumbersome 
> for string handling than C.  Yes, you /can/ do string handling in C, but 
> no one would /choose/ C for that.

See the build.c demo elsewhere in the thread.


> I have a directory "src".  I want to find all the .c files in "src".  I 
> want to make a list of all these files, and a list of matching object 
> files to make in the "build/obj/src" directory.  For each file, I want 
> to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to do so in 
> parallel, up to 8 commands at a time.  (Ignore any possible runtime 
> errors.)

> With Python, you could have (untested) :
> 
>      import glob
>      import os
>      import multiprocessing
>      import subprocess
> 
>      srcdir = "src"
>      objdir = "build/obj"
> 
>      srcfiles = glob.glob(srcdir + "/*.c")
>      objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>      objfiles = [objdir + "/" + fn for fn in srcfiles]
> 
>      os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
> 
>      with multiprocessing.Pool(5) as pool :
>          pool.map(lambda src, obj : subprocess.call("gcc -c " +
>                   src + " -o " + obj, shell = True),
>                  zip(srcfiles, objfiles))
> 
> Obviously much of this could be put in a reusable library, so that end 
> users don't have to know about process pools.  (And there are many other 
> ways to structure such code.)

I had a go at expressing the Python version in my scripting language:

     srcdir := "src/"
     objdir := "build/obj/"+srcdir

     srcfiles := dirlist(srcdir + "*.c")

     objfiles_src := mapvs(changeext, srcfiles, "o")
     objfiles := mapsv(+, objdir, objfiles_src)

     createdir(objdir)

     for i, sfile in srcfiles do
         execcmd(sfprint("gcc -c # -o #", srcdir+sfile, objfiles[i]))
     od

A couple of issues: 'createdir' can only create one directory at a time; 
for a chain of them like a/b/c, I'd need to split it up and do it one by 
one. (To test this, I created it manually.)

Also, I don't have any features for parallel executions. However 
'execcmd' starts a process but then doesn't wait for it to complete. If 
I instead use 'system' (or my 'execwait'), then compiling all Lua .c 
files in ./src takes 6.5 seconds instead of 2.5 seconds.

But this isn't the point of posting this. My interpreter can be 
expressed as a single C file. That allows you to use this kind of 
scripting, without adding any extra dependencies. You still only need a 
C compiler.

(I tried your Python, but os.mkdir(dir) didn't work, even after fixing 
the name typo and removing the named argument. Bypassing that, there 
were all sorts of errors to do with pickle.py and 'unbounded' methods.

I believe your glob.glob routine returns filenames with path prepended 
(my dirlist doesn't). I suspect that's why you chose a dest path as 
build/obj/src rather than build/obj. Anyway, I wasn't able to compare 
peformance.)

BTW this scripting is still hard work. For building you want to express 
the requires as data, not code. If you can do that (see my build.c 
example), then C may be adequate.

[toc] | [prev] | [next] | [standalone]

#381870

On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:

> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
> the name typo and removing the named argument.

“mkdirs” should have been “makedirs”
<https://docs.python.org/3/library/os.html#os.makedirs>.

> Bypassing that, there
> were all sorts of errors to do with pickle.py and 'unbounded' methods.

Trouble with your Python install? In other words, Windows trouble?

[toc] | [prev] | [next] | [standalone]

#381913

On 06/02/2024 00:38, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:
> 
>> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
>> the name typo and removing the named argument.
> 
> “mkdirs” should have been “makedirs”
> <https://docs.python.org/3/library/os.html#os.makedirs>.
> 

Oh, so it was a typo - the "s" was right, but not the "mk" :-(

The code was just for reference, not tested or complete.

[toc] | [prev] | [next] | [standalone]

#381912

On 05/02/2024 22:28, bart wrote:
> On 05/02/2024 19:47, David Brown wrote:
>> On 05/02/2024 16:17, Malcolm McLean wrote:
> 
>>> The library would have facilities for working with lists of strings, 
>>> of course. But C can do that perfectly well, and I don't see it as 
>>> big problem.
>>>
>>
>> Have you ever tried /any/ other programming language?  Apart from 
>> assembly and Forth, I have not seen a language that is more cumbersome 
>> for string handling than C.  Yes, you /can/ do string handling in C, 
>> but no one would /choose/ C for that.
> 
> See the build.c demo elsewhere in the thread.
> 
> 
>> I have a directory "src".  I want to find all the .c files in "src".  
>> I want to make a list of all these files, and a list of matching 
>> object files to make in the "build/obj/src" directory.  For each file, 
>> I want to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to 
>> do so in parallel, up to 8 commands at a time.  (Ignore any possible 
>> runtime errors.)
> 
>> With Python, you could have (untested) :
>>
>>      import glob
>>      import os
>>      import multiprocessing
>>      import subprocess
>>
>>      srcdir = "src"
>>      objdir = "build/obj"
>>
>>      srcfiles = glob.glob(srcdir + "/*.c")
>>      objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>>      objfiles = [objdir + "/" + fn for fn in srcfiles]
>>
>>      os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>>
>>      with multiprocessing.Pool(5) as pool :
>>          pool.map(lambda src, obj : subprocess.call("gcc -c " +
>>                   src + " -o " + obj, shell = True),
>>                  zip(srcfiles, objfiles))
>>
>> Obviously much of this could be put in a reusable library, so that end 
>> users don't have to know about process pools.  (And there are many 
>> other ways to structure such code.)
> 
> I had a go at expressing the Python version in my scripting language:
> 
>      srcdir := "src/"
>      objdir := "build/obj/"+srcdir
> 
>      srcfiles := dirlist(srcdir + "*.c")
> 
>      objfiles_src := mapvs(changeext, srcfiles, "o")
>      objfiles := mapsv(+, objdir, objfiles_src)
> 
>      createdir(objdir)
> 
>      for i, sfile in srcfiles do
>          execcmd(sfprint("gcc -c # -o #", srcdir+sfile, objfiles[i]))
>      od
> 
> A couple of issues: 'createdir' can only create one directory at a time; 
> for a chain of them like a/b/c, I'd need to split it up and do it one by 
> one. (To test this, I created it manually.)

Your scripting language is definitely an improvement over C for this 
kind of thing.  That is as I would expect - scripting languages tend to 
be much better at working with strings and external commands.

> 
> Also, I don't have any features for parallel executions. However 
> 'execcmd' starts a process but then doesn't wait for it to complete. If 
> I instead use 'system' (or my 'execwait'), then compiling all Lua .c 
> files in ./src takes 6.5 seconds instead of 2.5 seconds.
> 
> But this isn't the point of posting this. My interpreter can be 
> expressed as a single C file. That allows you to use this kind of 
> scripting, without adding any extra dependencies. You still only need a 
> C compiler.
> 
> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing 
> the name typo and removing the named argument. Bypassing that, there 
> were all sorts of errors to do with pickle.py and 'unbounded' methods.
> 

"os.mkdirs()" was not a typo.  "mkdirs()" is a different function from 
"mkdir()" - it can make multiple directories as needed.  But I may have 
had other errors.

> I believe your glob.glob routine returns filenames with path prepended 
> (my dirlist doesn't).

Yes.

> I suspect that's why you chose a dest path as 
> build/obj/src rather than build/obj. 

No.  I used that because it is the structure I use for bigger projects. 
I generally have multiple source directories, and in my build 
directories I have "obj", "deps" (for dependency files), perhaps also 
directories for listings files and other bits and pieces, and inside 
these the directory structure mirrors the source tree.  (I actually 
often have another layer between "build" and "obj" because there can be 
multiple different builds for the same project.)

> Anyway, I wasn't able to compare 
> peformance.)

There was no intention of that.  I merely wanted to show that something 
that is key to software building can be done relatively easily in a 
higher-level language suitable for scripting, in a way that is simpler 
and easier than in C.

It was not a full build in any sense, and did not cover dependency graphs.

> 
> BTW this scripting is still hard work. For building you want to express 
> the requires as data, not code. If you can do that (see my build.c 
> example), then C may be adequate.
> 

Eventually, when you add all the features and flexibility people want, 
you end up with a domain-specific language as you data, interpreted by 
code written in some other language (C or whatever - though again I 
expect other languages to be more suitable).  You've just re-invented 
"make".

Of course it is possible to re-invent make in a better way than the 
original (at least for some people's preferences and needs - "better" is 
always subjective).  If I were doing so, I'd want an embedded high-level 
scripting language of some sort that was not a variant of Lisp.  (Some 
people like Lisp, I don't.)  Possible options from existing languages 
would be Lua, TCL and Python.  And I'd want it neatly integrated, not 
just a way to make extensions, as that would simplify the main DSL. 
(There's plenty of other things I'd change, but I don't want to go 
through a list here.)

[toc] | [prev] | [next] | [standalone]

#381869

On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:

> It would be easier if the CMake scripts were in
> C. It might be bit less convenient.

Aren’t “easier” and “less convenient” kind of ... opposites?

[toc] | [prev] | [next] | [standalone]

#381884

On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
> 
>> It would be easier if the CMake scripts were in
>> C. It might be bit less convenient.
> 
> Aren’t “easier” and “less convenient” kind of ... opposites?

Less convenient for the whizzy suoer skilled build system programmeer to 
write. Easier for the humble C programmer who has it fall over on him to 
fix.

-- 
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

[toc] | [prev] | [next] | [standalone]

#381888

On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:

> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>
>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>> 
>>> It would be easier if the CMake scripts were in C. It might be bit
>>> less convenient.
>> 
>> Aren’t “easier” and “less convenient” kind of ... opposites?
> 
> Less convenient for the whizzy suoer skilled build system programmeer to
> write. Easier for the humble C programmer who has it fall over on him to
> fix.

But the C code will be anything up to an order of magnitude larger than 
the build rules written in the domain-specific language. So more work 
required overall.

[toc] | [prev] | [next] | [standalone]

#381892

On 06/02/2024 02:27, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:
> 
>> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>>
>>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>>
>>>> It would be easier if the CMake scripts were in C. It might be bit
>>>> less convenient.
>>>
>>> Aren’t “easier” and “less convenient” kind of ... opposites?
>>
>> Less convenient for the whizzy suoer skilled build system programmeer to
>> write. Easier for the humble C programmer who has it fall over on him to
>> fix.
> 
> But the C code will be anything up to an order of magnitude larger than
> the build rules written in the domain-specific language. So more work
> required overall.

Well yes. If the program to build the program has to be more elaborate 
than the porgram itself, something is going wrong. But whilst the CMake 
scripts are quite short, they aren't trivial, and there's a lot of 
fussing about needed to pull in the sources, set the IDE uo in the right 
way, and then make sure that the non-IDE build is exactly the same. But 
if it falls over I can't work, and you don't like to call in someone 
else unless it's established that the script itself actually needs 
modification. So you tend to trouble shoot. But whilst I do write my own 
CMkae scripts, I don't write them often enough to really know the 
language, and I just forget all the details. So troubleshooting a CMake 
scriot is quite hard work. Not as hard as debugging a complex C 
algorithm. But that's my core job, whilst the build system is not.

-- 
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

[toc] | [prev] | [next] | [standalone]

#381895

On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:

> But whilst the CMake scripts are quite short, they aren't trivial ...

I know, I’ve fiddled with CMake scripts myself. The power is needed 
because builds can be complex.

> But whilst I do write my own CMkae scripts, I don't write them often
> enough to really know the language ...

Same here. That’s why, every time I need to touch a CMake script, I make 
sure to have the docs open in a browser window
<https://cmake.org/documentation/>. I don’t fiddle at random: I try to 
understand what I am doing. And sometimes I find the person who fiddled 
the script before me didn’t understand that there was a simpler way to do 
something.

[toc] | [prev] | [next] | [standalone]

#381898

On 2/5/2024 7:09 PM, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:
> 
>> But whilst the CMake scripts are quite short, they aren't trivial ...
> 
> I know, I’ve fiddled with CMake scripts myself. The power is needed
> because builds can be complex.
> 
>> But whilst I do write my own CMkae scripts, I don't write them often
>> enough to really know the language ...
> 
> Same here. That’s why, every time I need to touch a CMake script, I make
> sure to have the docs open in a browser window
> <https://cmake.org/documentation/>.

Nothing wrong with that! :^)


  I don’t fiddle at random: I try to
> understand what I am doing. And sometimes I find the person who fiddled
> the script before me didn’t understand that there was a simpler way to do
> something.

[toc] | [prev] | [next] | [standalone]

#381897

On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:
> 
>> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>>
>>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>>
>>>> It would be easier if the CMake scripts were in C. It might be bit
>>>> less convenient.
>>>
>>> Aren’t “easier” and “less convenient” kind of ... opposites?
>>
>> Less convenient for the whizzy suoer skilled build system programmeer to
>> write. Easier for the humble C programmer who has it fall over on him to
>> fix.
> 
> But the C code will be anything up to an order of magnitude larger than
> the build rules written in the domain-specific language. So more work
> required overall.

Putting the build rules in source code, is a bad idea, TM?

[toc] | [prev] | [next] | [standalone]

#381900

On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:

> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>
>> But the C code will be anything up to an order of magnitude larger than
>> the build rules written in the domain-specific language. So more work
>> required overall.
> 
> Putting the build rules in source code, is a bad idea, TM?

Build rules *are* source code.

[toc] | [prev] | [next] | [standalone]

#381901

On 2/5/2024 11:25 PM, Lawrence D'Oliveiro wrote:
> On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:
> 
>> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>>
>>> But the C code will be anything up to an order of magnitude larger than
>>> the build rules written in the domain-specific language. So more work
>>> required overall.
>>
>> Putting the build rules in source code, is a bad idea, TM?
> 
> Build rules *are* source code.

Well, trying to put something that is akin to a makefile into source 
code is a bad idea? Any better?

[toc] | [prev] | [next] | [standalone]

#381902

On 06/02/2024 07:33, Chris M. Thomasson wrote:
> On 2/5/2024 11:25 PM, Lawrence D'Oliveiro wrote:
>> On Mon, 5 Feb 2024 22:32:40 -0800, Chris M. Thomasson wrote:
>>
>>> On 2/5/2024 6:27 PM, Lawrence D'Oliveiro wrote:
>>>
>>>> But the C code will be anything up to an order of magnitude larger than
>>>> the build rules written in the domain-specific language. So more work
>>>> required overall.
>>>
>>> Putting the build rules in source code, is a bad idea, TM?
>>
>> Build rules *are* source code.
> 
> Well, trying to put something that is akin to a makefile into source 
> code is a bad idea? Any better?

CMake is a programming language in its own right and there's no 
ambiguity about that, though it is special purpose one unsuitable to 
write anything except build scripts. make is syntax for instructions to 
a program which builds programs. There might be something in it which is 
technically Turing compatible and thus can be used to create arbitrary 
behaviour, but if true it will be difficult to do and abuse of the 
intended purpose.

C is a general purpose programming language. However if you provide a 
library of functions designed to help with building programs then it 
becomes effectively a special purpose programming language, but still 
with the general purpose grammar. It's still suitable for general 
purpose programming and so less suitable for wriitng build scripts than 
make, but the language is familar.

CMake and make both gained wide traction. Whilst apparently build 
scripts using C as the scripting language have been tried before and 
it's not a totally new idea, it's been tried out rather than seriously 
tried.
-- 
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

[toc] | [prev] | [next] | [standalone]

#381823

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>> On 04/02/2024 23:32, Ben Bacarisse wrote:
[...]
>> No. But the concept of a C interpreter isn't new.
>
> Indeed.  But C is not a good fit for interpretation, so there are often
> compromises.
>
>> There's one called pico C
>
> Yes, and there's tcc's -run option.

From the tcc man page:

       "tcc -run a.c"
           Compile a.c and execute it directly

So it's not exactly an interpreter, though it might be useful in some of
the same contexts where an interpreter would be useful.  It would be
straightforward to implement a wrapper for any C compiler that does the
same thing.

-- 
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

[toc] | [prev] | [next] | [standalone]

#381799

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:

>> [...]
>
> csh isn't a C interpreter by any stretch of the imagination!

IT ISN'T???  No wonder I've been having trouble writing csh
scripts...

[toc] | [prev] | [next] | [standalone]

#381747

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:
>How shall we write a better build system?
>
>Current systems use scripts with ad hoc, weird syntax.

Fact not in evidence.   Your opinion thereof does not make it a fact.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web