Groups | Search | Server Info | Login | Register

Groups > comp.arch.embedded > #32422

Re: Improving build system

Show all headers | View raw

Il 14/05/2025 11:03, David Brown ha scritto:
> On 13/05/2025 17:57, pozz wrote:
[...]
> 
> You are asking a lot of questions here.  They are good questions, but it 
> would be a very long post if I tried to answer them all fully.  So 
> instead, I will try to give a few hints and suggestions that you can 
> take further.  I'll put numbers on them in case you want to reference 
> them in replies.

Ok, thank you very much for your time.


> 1.
> 
> Windows path naming is insane.  Fortunately, you can almost always 
> override it.  Whenever you install any serious program in Windows, 
> especially if you ever want to refer to it from the command line, batch 
> files, makefiles, etc., avoid names with spaces or "awkward" characters. 
>   I recommend making top-level directories like "progs" or "compilers" 
> and putting the tools in there as appropriate.  This also makes it 
> vastly easier to copy tools to other machines.  And since you should 
> never upgrade your toolchains - merely add new versions to your 
> collection, in separate directories - it is easier if they are better 
> organised.

I know, but not all software installers work well if you change their 
default installation path.  When it comes to stupid and big IDEs (such 
as Atmel/Microchip Studio), I prefer to avoid changing the default 
installation path (C:\Program Files (x86)\Atmel...) to avoid other 
obscure issues.
It is already a miracle if that software runs without problems with the 
default installation path.  I don't want to imagine what happens if I 
changed it.

Anyway until now I didn't find issues with spaces.  Even in msys2 shell 
I can use "/c/Program\ Files\ (x86)/...".

The other IDE I use is MCUXpresso.  It is Eclipse based so I installed 
it in c:\ without any temptations.


> 2.
> 
> You don't need to use bash or other *nix shells for makefile or other 
> tools if you don't want to.  When I do builds on Windows, I run "make" 
> from a normal command line (or from an editor / IDE).  It is helpful to 
> have msys2's usr/bin on your path so that make can use *nix command-line 
> utilities like cp, mv, sed, etc.  But if you want to make a minimal 
> build system, you don't need a full msys2 installation - you only need 
> the utilities you want to use, and they can be copied directly (unlike 
> with Cygwin or WSL).
> 
> Of course you /can/ use fuller shells if you want.  But don't make your 
> makefiles depend on that, as it will be harder to use them from IDEs, 
> editors, or any other automation.

In the beginning (some years ago) I started installing GNU Make for 
Windows, putting it in c:\tools\make.  Then I created a simple Makefile 
and tried to process it on a standard Windows command line.  It was a 
mess!  I remember there were many issues regarding: slash/backslash on 
file paths, lack of Unix commands (rm, mv, ...) and so on.  Native 
Windows tools need backslash in the paths, but some unix tools need 
slash.  It was a mess to transform the paths between the two forms.

After this attempt, I gave up.  I thought it was much better to use the 
IDE and build system suggested by the MCU manufacturer.

Now I'm trying a Unix shell in Windows (msys, WSL or even the bash 
installed with git) and it seems many issues I had are disappearing.


> And of course you will want an msys2/mingw64 (/not/ old mingw32) for 
> native gcc compilation.  

The goal of the simulator is to detect problems on the software that 
runs directly on Windows, without flashing, debug probes and so on.  I 
increased my productivity a lot when I started this approach.

Obviously, the software running on Windows (the simulator) should be 
very similar to the sofware running on the embedded target.  Cortex-M 
MCUs are 32-bits so I thought it should be better to use a 32-bits 
compiler even for the simulator.

Moreover, I think many issues aries on a 64-bits compilation, for 
example static allocated buffers that would be too small on a 64-bits 
platforms.  Or some issues on serializers.


> Don't bother with WSL unless you actually need 
> a fuller Linux system - and if you /do/ need that, dump the wasteful 
> crap that is Windows and use Linux for your development.  Build speeds 
> will double on the same hardware.  (In my testing, done a good while 
> back, I did some comparisons of a larger build on different setups on 
> the same PC, using native Windows build as the baseline.  Native Linux 
> builds were twice the speed.  Running VirtualBox on Windows host, with a 
> Linux virtual machine, or running VirtualBox on Linux with a Windows 
> virtual machine, both beat native Windows solidly.)

I completely agree with you.  At the moment msys2 seems ok.


> 3.
> 
> Makefiles can be split up.  Use "include" - and remember that you can do 
> so using macros.  In my makefile setups, I have a file "host.mk" that is 
> used to identify the build host, then pull in a file that is specific to 
> the host:
> 
> # This is is for identifying host computer to get the paths right
> 
> ifeq ($(OS),Windows_NT)
>    # We are on a Windows machine
>    host_os := windows
>    host := $(COMPUTERNAME)
> else
>    # Linux machine
>    host_os := linux
>    host := $(shell hostname)
> endif
> 
> ifeq "$(call file-exists,makes/host_$(host).mk)" "1"
>    include makes/host_$(host).mk
> else
>    $(error No host makefile host_$(host).mk found)
> endif
> 
> Then I have files like "host_xxx.mk" for a computer named "xxx", 
> containing things like :
> 
> toolchain_path := /opt/gcc-arm-none-eabi-10-2020-q4-major/bin/
> 
> or
> 
> toolchain_path := c:/micros/gcc-arm-none-eabi-10_2020-q4-major/bin/
> 
> 
> All paths to compilers and other build-related programs are specified in 
> these files.  The only things that are taken from the OS path are 
> standard and common programs that do not affect the resulting binary files.

It is an interesting and uncommon (at least for me) approach.

What happens if multiple developers work on the same repository?  Are 
they forced to create a host_xxx.mk for all their development machines? 
Should the host_xxx.mk files be added to the repository?

I guess the only goal of host_xxx.mk is to avoid changing PATH before 
make.  Why don't you like setting the PATH according to the project 
you're working on?


> Then I have a "commands.mk" file with things like :
> 
> ATDEP := @
> 
> toolchain_prefix := arm-none-eabi-
> 
> CCDEP := $(ATDEP)$(toolchain_path)$(toolchain_prefix)gcc
> CC := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)gcc
> LD := $(AT)$(toolchain_path)$(toolchain_prefix)gcc
> OBJCOPY := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)objcopy
> OBJDUMP := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)dump
> SIZE := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)size
> 
> 
> Put CONFIG dependent stuff in "config_full.mk" and similar files.  Put 
> TARGET specific stuff in "target_simulator.mk".  And so on.  It makes it 
> much easier to keep track of things, and you only need a few high-level 
> "ifeq".
> 
> 
> Keep your various makefiles in a separate directory.  Your project 
> makefile is then clear and simple - much of it will be comments about 
> usage (parameters like CONFIG).

Yes, splitting makefiles is a good suggestion.


> 4.
> 
> Generate dependency files, using the same compiler and the same include 
> flags and -D flags as you have for the normal compilation, but with 
> flags like -MM -MP -MT and -MF to make .d dependency files.  Include 
> them all in the makefile, using "-include" so that your makefile does 
> not stop before they are generated.

I have to admit that ChatGPT helped me to create the Makefile.  The 
CFLAGS include -MMD and -MP and at the end I have

   -include $(DEP_FILES)

Of course, DEP_FILES are:

   DEP_FILES := $(OBJ_FILES:.o=.d)

Sincerely I don't know if it is good, but I tried to change an include 
file and related C files are compiled again as expected (so I think the 
dependency are correctly managed).

There's a thing that doesn't work.  If I change the Makefile itsel, for 
example changing CFLAGS adding a new compiler option, I need to manually 
invoke a clean.


> 5.
> 
> Keep your build directories neat, separate from all source directories, 
> and mirroring the tree structure of the source files.  So if you have a 
> file "src/gui/main_window.c", and you are building with CONFIG=FULL 
> TARGET=embedded, the object file generated should go in something akin 
> to "builds/FULL/embedded/obj/src/gui/main_window.o".  I like to have 
> separate parts for obj (.o files), dep (.d files), and bin (linked 
> binaries, map files, etc.).  You could also mix .d and .o files in the 
> same directory if you prefer.
> 
> This means you can happily do incremental builds for all your 
> configurations and targets, and don't risk mixing object files from 
> different setups.

Yes, perfectly agreed.


> 6.
> 
> Learn to use submakes.  When you use plain "make" (or, more 
> realistically, "make -j") to build multiple configurations, have each 
> configuration spawned off in a separate submake.  Then you don't need to 
> track multiple copies of your "TARGET" macro in the same build - each 
> submake has just one target, and one config.

I don't think I got the point.  Now I invoke the build of a single build 
configuration.  Are you talking about running make to build multiple 
configurations at the same time?

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Improving build system pozz <pozzugno@gmail.com> - 2025-05-13 17:57 +0200
  Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-13 22:48 +0200
  Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-14 11:03 +0200
    Re: Improving build system George Neuner <gneuner2@comcast.net> - 2025-05-14 15:21 -0400
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 09:48 +0200
      Re: Improving build system Nioclás Pól Caileán de Ghloucester <Spamassassin@irrt.De> - 2025-07-04 18:38 +0200
    Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-14 23:51 +0200
      Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-15 01:00 +0200
        Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 11:17 +0200
          Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-16 12:21 +0200
            Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 14:42 +0200
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 11:03 +0200
        Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-15 23:25 +0200
          Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 11:12 +0200
            Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 12:46 +0200
              Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 15:30 +0200
                Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 16:17 +0200
    Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 15:45 +0200
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 17:20 +0200
        Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-17 10:56 +0200
  Re: Improving build system Stefan Reuther <stefan.news@arcor.de> - 2025-05-14 18:06 +0200

csiph-web