Groups | Search | Server Info | Login | Register

Groups > comp.arch.embedded > #32425

Re: Improving build system

Show all headers | View raw

On 14/05/2025 23:51, pozz wrote:
> Il 14/05/2025 11:03, David Brown ha scritto:
>> On 13/05/2025 17:57, pozz wrote:
> [...]
>>
>> You are asking a lot of questions here.  They are good questions, but 
>> it would be a very long post if I tried to answer them all fully.  So 
>> instead, I will try to give a few hints and suggestions that you can 
>> take further.  I'll put numbers on them in case you want to reference 
>> them in replies.
> 
> Ok, thank you very much for your time.
> 
> 
>> 1.
>>
>> Windows path naming is insane.  Fortunately, you can almost always 
>> override it.  Whenever you install any serious program in Windows, 
>> especially if you ever want to refer to it from the command line, 
>> batch files, makefiles, etc., avoid names with spaces or "awkward" 
>> characters.   I recommend making top-level directories like "progs" or 
>> "compilers" and putting the tools in there as appropriate.  This also 
>> makes it vastly easier to copy tools to other machines.  And since you 
>> should never upgrade your toolchains - merely add new versions to your 
>> collection, in separate directories - it is easier if they are better 
>> organised.
> 
> I know, but not all software installers work well if you change their 
> default installation path.  When it comes to stupid and big IDEs (such 
> as Atmel/Microchip Studio), I prefer to avoid changing the default 
> installation path (C:\Program Files (x86)\Atmel...) to avoid other 
> obscure issues.

Almost all of them work fine in different directories (and indeed 
different drives).  First assume they work - only fall back on the 
crappy defaults if there is no other option.

Of course, the best answer is to avoid any tool made by Microchip - they 
are the worst of the bunch.  (Atmel was always a bit behind in second 
place for the title of worst toolchain supplier, but Microchip has 
gradually integrated them.)  I have many fine things to say about 
Microchip and Atmel as hardware suppliers, but I make a point of 
avoiding their microcontrollers because of their tools.

My strong preference - regardless of the manufacturer - is to use the 
ARM-supplied gcc toolchains (for ARM microcontrollers, obviously) rather 
than the usually older tools supplied by manufacturers.

> It is already a miracle if that software runs without problems with the 
> default installation path.  I don't want to imagine what happens if I 
> changed it.
> 
> Anyway until now I didn't find issues with spaces.  Even in msys2 shell 
> I can use "/c/Program\ Files\ (x86)/...".

It usually works - but that does not stop it being a PITA and an insane 
choice of pathnames.

Still, you have to find what works best for you - I am giving 
recommendations and suggestions, not a unique solution or single 
"correct" answer.

> 
> The other IDE I use is MCUXpresso.  It is Eclipse based so I installed 
> it in c:\ without any temptations.
> 

Yes, that has always worked for me.

(Of course I normally have it on Linux, rather than Windows, and most 
manufacturer-supplied software uses a sensible default path - in /opt or 
in /usr/local.)


> 
>> 2.
>>
>> You don't need to use bash or other *nix shells for makefile or other 
>> tools if you don't want to.  When I do builds on Windows, I run "make" 
>> from a normal command line (or from an editor / IDE).  It is helpful 
>> to have msys2's usr/bin on your path so that make can use *nix 
>> command-line utilities like cp, mv, sed, etc.  But if you want to make 
>> a minimal build system, you don't need a full msys2 installation - you 
>> only need the utilities you want to use, and they can be copied 
>> directly (unlike with Cygwin or WSL).
>>
>> Of course you /can/ use fuller shells if you want.  But don't make 
>> your makefiles depend on that, as it will be harder to use them from 
>> IDEs, editors, or any other automation.
> 
> In the beginning (some years ago) I started installing GNU Make for 
> Windows, putting it in c:\tools\make.  Then I created a simple Makefile 
> and tried to process it on a standard Windows command line.  It was a 
> mess!  I remember there were many issues regarding: slash/backslash on 
> file paths, lack of Unix commands (rm, mv, ...) and so on.  Native 
> Windows tools need backslash in the paths, but some unix tools need 
> slash.  It was a mess to transform the paths between the two forms.
> 

Most tools on Windows are happy with forward slash for path separators 
as well.  Certainly everything that is originally a *nix tool will be 
fine with that.

Of course if you have a makefile that uses commands like "rm" and you 
don't have them on your path, and don't specify the path in the 
makefile, then it won't work.  This is why the norm in advanced 
makefiles is to use macros for these things :

# Put this in the host-specific file, with blank for no path needed
bin_path :=

# Use this instead of "rm".
RM := $(bin_path) rm


> After this attempt, I gave up.  I thought it was much better to use the 
> IDE and build system suggested by the MCU manufacturer.
> 

For most IDEs, the build system is "make".  But the IDE generates the 
makefiles - slowly for big projects, and usually overly simplistic with 
far too limited options.

But IDE's are certainly much easier for getting started.  On new 
projects, or new devices, I will often use the IDE to get going and then 
move it over to an independent makefile.  (And I'll often continue to 
use the IDE after that as a solid editor and debugger - IDE's are 
generally happy with external makefiles.)

> Now I'm trying a Unix shell in Windows (msys, WSL or even the bash 
> installed with git) and it seems many issues I had are disappearing.
> 
> 
>> And of course you will want an msys2/mingw64 (/not/ old mingw32) for 
>> native gcc compilation. 
> 
> The goal of the simulator is to detect problems on the software that 
> runs directly on Windows, without flashing, debug probes and so on.  I 
> increased my productivity a lot when I started this approach.
> 
> Obviously, the software running on Windows (the simulator) should be 
> very similar to the sofware running on the embedded target.  Cortex-M 
> MCUs are 32-bits so I thought it should be better to use a 32-bits 
> compiler even for the simulator.
> 

mingw-w64 can happily generate 32-bit Windows executables.  IIRC you 
just use the "-m32" flag.  It is significantly better than old mingw in 
a number of ways - in particular it has vastly better standard C library 
support.

> Moreover, I think many issues aries on a 64-bits compilation, for 
> example static allocated buffers that would be too small on a 64-bits 
> platforms.  Or some issues on serializers.
> 
> 
>> Don't bother with WSL unless you actually need a fuller Linux system - 
>> and if you /do/ need that, dump the wasteful crap that is Windows and 
>> use Linux for your development.  Build speeds will double on the same 
>> hardware.  (In my testing, done a good while back, I did some 
>> comparisons of a larger build on different setups on the same PC, 
>> using native Windows build as the baseline.  Native Linux builds were 
>> twice the speed.  Running VirtualBox on Windows host, with a Linux 
>> virtual machine, or running VirtualBox on Linux with a Windows virtual 
>> machine, both beat native Windows solidly.)
> 
> I completely agree with you.  At the moment msys2 seems ok.
> 
> 
>> 3.
>>
>> Makefiles can be split up.  Use "include" - and remember that you can 
>> do so using macros.  In my makefile setups, I have a file "host.mk" 
>> that is used to identify the build host, then pull in a file that is 
>> specific to the host:
>>
>> # This is is for identifying host computer to get the paths right
>>
>> ifeq ($(OS),Windows_NT)
>>    # We are on a Windows machine
>>    host_os := windows
>>    host := $(COMPUTERNAME)
>> else
>>    # Linux machine
>>    host_os := linux
>>    host := $(shell hostname)
>> endif
>>
>> ifeq "$(call file-exists,makes/host_$(host).mk)" "1"
>>    include makes/host_$(host).mk
>> else
>>    $(error No host makefile host_$(host).mk found)
>> endif
>>
>> Then I have files like "host_xxx.mk" for a computer named "xxx", 
>> containing things like :
>>
>> toolchain_path := /opt/gcc-arm-none-eabi-10-2020-q4-major/bin/
>>
>> or
>>
>> toolchain_path := c:/micros/gcc-arm-none-eabi-10_2020-q4-major/bin/
>>
>>
>> All paths to compilers and other build-related programs are specified 
>> in these files.  The only things that are taken from the OS path are 
>> standard and common programs that do not affect the resulting binary 
>> files.
> 
> It is an interesting and uncommon (at least for me) approach.
> 
> What happens if multiple developers work on the same repository?  Are 
> they forced to create a host_xxx.mk for all their development machines? 
> Should the host_xxx.mk files be added to the repository?

Yes, that is /exactly/ what you do.  It also applies to a single 
developer using multiple different machines.  For any long-term project, 
you want to be sure you can check out the repository, do a clean build, 
and get an identical binary from more than one machine.  Having 
individual "host_XXX.mk" files means that the build adapts automatically 
to the machine.  What you don't want is each developer making changes to 
a single makefile so that it works on their machine - and then either 
not checking in the changes, or checking them in and messing things up 
for someone else.

> 
> I guess the only goal of host_xxx.mk is to avoid changing PATH before 
> make.  Why don't you like setting the PATH according to the project 
> you're working on?
> 

No, that is not the only goal - there can be many differences between 
machines.  For example, I usually have ccache on my Linux systems but it 
is rare to have it on (native) Windows systems - thus that can be 
enabled or disabled in a host_xxx.mk file.  Some machines might also 
support building the documentation, or running a simulator, or signing 
binaries.

Setting the path would be an extra complication of no benefit, but a 
significant source of risk or error.  How do you make sure your IDE is 
using the right PATH settings before it runs "make"?  How do you deal 
with multiple projects - do you keep swapping PATHs?  (I usually have a 
half-dozen projects "open" at a time, in different workspaces on my 
Linux machine.)  Do you now have a makefile and a separate path-setting 
batch file or shell script that you need to run before doing a project 
build?  How do you handle things when you install some new Windows 
program that messes with your path?

It is /vastly/ simpler and safer to put the paths to the binaries in a 
couple of macros in your makefile(s).  It also gives clear and 
unequivocal documentation of the tools you need -  if your makefile has 
this line :

toolchain_path := c:/micros/gcc-arm-none-eabi-10_2020-q4-major/bin/

then there is never any doubt as to exactly which toolchain is used for 
the project.

> 
>> Then I have a "commands.mk" file with things like :
>>
>> ATDEP := @
>>
>> toolchain_prefix := arm-none-eabi-
>>
>> CCDEP := $(ATDEP)$(toolchain_path)$(toolchain_prefix)gcc
>> CC := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)gcc
>> LD := $(AT)$(toolchain_path)$(toolchain_prefix)gcc
>> OBJCOPY := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)objcopy
>> OBJDUMP := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)dump
>> SIZE := $(AT)$(CCACHE) $(toolchain_path)$(toolchain_prefix)size
>>
>>
>> Put CONFIG dependent stuff in "config_full.mk" and similar files.  Put 
>> TARGET specific stuff in "target_simulator.mk".  And so on.  It makes 
>> it much easier to keep track of things, and you only need a few 
>> high-level "ifeq".
>>
>>
>> Keep your various makefiles in a separate directory.  Your project 
>> makefile is then clear and simple - much of it will be comments about 
>> usage (parameters like CONFIG).
> 
> Yes, splitting makefiles is a good suggestion.
> 
> 
>> 4.
>>
>> Generate dependency files, using the same compiler and the same 
>> include flags and -D flags as you have for the normal compilation, but 
>> with flags like -MM -MP -MT and -MF to make .d dependency files.  
>> Include them all in the makefile, using "-include" so that your 
>> makefile does not stop before they are generated.
> 
> I have to admit that ChatGPT helped me to create the Makefile.  The 
> CFLAGS include -MMD and -MP and at the end I have
> 
>    -include $(DEP_FILES)
> 
> Of course, DEP_FILES are:
> 
>    DEP_FILES := $(OBJ_FILES:.o=.d)
> 

That's a good start.  There are quite a few articles and blog posts 
about automatic generation of makefile dependencies that can be worth 
reading.

> Sincerely I don't know if it is good, but I tried to change an include 
> file and related C files are compiled again as expected (so I think the 
> dependency are correctly managed).
> 
> There's a thing that doesn't work.  If I change the Makefile itsel, for 
> example changing CFLAGS adding a new compiler option, I need to manually 
> invoke a clean.
> 

depfiles_src := $(cfiles:.c=.d) $(cppfiles:.cpp=.d)
depfiles := $(addprefix $(dep_dir),$(patsubst ../%,%,$(depfiles_src)))

-include $(depfiles)

alldepends := makefile $(wildcard makes/*.mk)
all : $(alldepends) $(depfiles)
depends : $(alldepends)

# "depends" target just makes dep files
depends : $(depfiles)
         @echo Updated dependencies


Vary according to your needs.  But basically, if something has 
$(alldepends) in its dependency list, it will be rebuild if one of your 
makefiles changes.

> 
>> 5.
>>
>> Keep your build directories neat, separate from all source 
>> directories, and mirroring the tree structure of the source files.  So 
>> if you have a file "src/gui/main_window.c", and you are building with 
>> CONFIG=FULL TARGET=embedded, the object file generated should go in 
>> something akin to "builds/FULL/embedded/obj/src/gui/main_window.o".  I 
>> like to have separate parts for obj (.o files), dep (.d files), and 
>> bin (linked binaries, map files, etc.).  You could also mix .d and .o 
>> files in the same directory if you prefer.
>>
>> This means you can happily do incremental builds for all your 
>> configurations and targets, and don't risk mixing object files from 
>> different setups.
> 
> Yes, perfectly agreed.
> 
> 
>> 6.
>>
>> Learn to use submakes.  When you use plain "make" (or, more 
>> realistically, "make -j") to build multiple configurations, have each 
>> configuration spawned off in a separate submake.  Then you don't need 
>> to track multiple copies of your "TARGET" macro in the same build - 
>> each submake has just one target, and one config.
> 
> I don't think I got the point.  Now I invoke the build of a single build 
> configuration.  Are you talking about running make to build multiple 
> configurations at the same time?

Yes.

Obviously it depends on the stage you are in development and the kind of 
project - much of the time, you will want to build just one 
configuration.  But sometimes you will also want to make multiple builds 
to check that a small change has not caused trouble elsewhere, or for 
different kinds of testing?  Why run multiple "make" commands when you 
can do a full project build from one "make" ?

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Improving build system pozz <pozzugno@gmail.com> - 2025-05-13 17:57 +0200
  Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-13 22:48 +0200
  Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-14 11:03 +0200
    Re: Improving build system George Neuner <gneuner2@comcast.net> - 2025-05-14 15:21 -0400
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 09:48 +0200
      Re: Improving build system Nioclás Pól Caileán de Ghloucester <Spamassassin@irrt.De> - 2025-07-04 18:38 +0200
    Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-14 23:51 +0200
      Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-15 01:00 +0200
        Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 11:17 +0200
          Re: Improving build system Nicolas Paul Colin de Glocester <Spamassassin@irrt.De> - 2025-05-16 12:21 +0200
            Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 14:42 +0200
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-15 11:03 +0200
        Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-15 23:25 +0200
          Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 11:12 +0200
            Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 12:46 +0200
              Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 15:30 +0200
                Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 16:17 +0200
    Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-16 15:45 +0200
      Re: Improving build system David Brown <david.brown@hesbynett.no> - 2025-05-16 17:20 +0200
        Re: Improving build system pozz <pozzugno@gmail.com> - 2025-05-17 10:56 +0200
  Re: Improving build system Stefan Reuther <stefan.news@arcor.de> - 2025-05-14 18:06 +0200

csiph-web