Musl Based LFS pt. 1

If you followed along with my previous blog post you know we built a mini distro for our user land. Today we’re going to revisit the idea but use GNU user land built against the musl libc.

The idea for this project came from trying to build simple c programs statically. Using glibc is a pain in the ass! That’s when I stumbled upon musl. It’s a strict implementation of the c standard library. It’s easy to link statically or dynamically, and as demonstrated by Chimera Linux It can provide a decent improvement to performance and memory. – Though it at times can be a pain when software is written with GNUism in mind.

In this blog post I doubt we’ll get to a booting system. Unless we use busybox as our init system. However, I would rather avoid this in the long term at least in favor in dinit, but this means we’re going to have to get a c++ compiler up and running. As an added bonus the goal is to not use GCC as well. Though we’ll see how long that last.

I assume you have a working build environment. While I’m going to use clang as my system compiler you can swap this out with GCC. Otherwise compare with

Setup working environment

mkdir -pv ~/mlfs && cd ~/mlfs/
mkdir -pv packages
mkdir -pv rootfs && cd rootfs

export MLFS=$HOME/mlfs
export ROOTFS=$HOME/mlfs/rootfs
export MLFS_TGT=x86_64-unknown-linux-musl

# Core top-level directories
mkdir -vp "$ROOTFS"/{boot,dev,proc,sys,run,tmp,home,mnt,etc,opt}

# Our init dir
mkdir -vp "$ROOTFS"/etc/init.d

# /usr — merged hierarchy, single-lib
mkdir -vp "$ROOTFS/usr"/{bin,sbin,lib,share}

# /var — variable data
mkdir -vp "$ROOTFS/var"/{log,run,cache,tmp,lib}

# Permissions
chmod 0755 "$ROOTFS"
chmod 1777 "$ROOTFS/tmp" "$ROOTFS/var/tmp"

# symlink our rootlevel dirs for backwards compatibility
ln -sv usr/bin "$ROOTFS/bin"
ln -sv usr/sbin "$ROOTFS/sbin"
ln -sv usr/lib "$ROOTFS/lib"
ln -sv usr/lib "$ROOTFS/lib64"
ln -sv lib "$ROOTFS/usr/lib64"

Get our source

This is going to be just enough packages to let us chroot in and have some room to play. As you’ll be able to glean from this exercise it the program you want to build is written in c and doesn’t too heavily rely on GNUism it should build just fine either statically or dynamically allowing you to customize your bare bones linux system to your hearts content.

cd $MLFS/packages
wget --no-clobber https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.19.tar.xz
wget --no-clobber https://musl.libc.org/releases/musl-1.2.5.tar.gz
wget --no-clobber https://ftp.gnu.org/gnu/coreutils/coreutils-9.6.tar.xz
wget --no-clobber http://gondor.apana.org.au/~herbert/dash/files/dash-0.5.12.tar.gz
wget --no-clobber https://ftp.gnu.org/gnu/diffutils/diffutils-3.11.tar.xz
wget --no-clobber https://ftp.gnu.org/gnu/findutils/findutils-4.10.0.tar.xz
wget --no-clobber https://ftp.gnu.org/gnu/grep/grep-3.11.tar.xz
wget --no-clobber https://ftp.gnu.org/gnu/sed/sed-4.9.tar.xz
wget --no-clobber https://invisible-mirror.net/archives/ncurses/ncurses-6.5.tar.gz
wget --no-clobber https://ftp.gnu.org/gnu/bash/bash-5.2.37.tar.gz
wget --no-clobber https://busybox.net/downloads/busybox-1.37.0.tar.bz2
wget --no-clobber https://www.nano-editor.org/dist/v8/nano-8.7.1.tar.gz
wget --no-clobber https://ftp.gnu.org/gnu/gawk/gawk-5.3.1.tar.xz
wget --no-clobber https://ftp.gnu.org/gnu/readline/readline-8.2.13.tar.gz

Build environment

I’m using clang as my compiler, you do not have to do the same. Though when building against musl I have found through trial and error that the cflags are not needed in every build however, they’re needed in some do to musl header files/level of strictness. Or so says the random sites and blogs I read when initially trying to get stuff to build.

Sometimes these flags need to be modified. A good one is the -std=gnu99. Some packages won’t build unless it’s 89 for example.

libgcc is a runtime library that is usually dynamically linked in. If you build with the -static flag, -static-libgcc is redundant. However, there are a few spots where we don’t build static. We cannot have them try and load a runtime library that may not exist.

If you see “clang: warning: argument unused during compilation: ‘-static-libgcc’ [-Wunused-command-line-argument]” don’t worry. It’s better to not need it than need it and not have it.

 export CC="clang"
 export CXX="clang++"
 export CFLAGS="-std=gnu99 --sysroot=$ROOTFS -Wno-old-style-definition -static -I$ROOTFS/usr/include -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wno-error=implicit-function-declaration -static-libgcc"
 export LDFLAGS="--sysroot=$ROOTFS -Wl,--dynamic-linker=/lib/ld-musl-x86_64.so.1"
 export CXXFLAGS="-static-libgcc"
 export PKG_CONFIG_SYSROOT_DIR="$ROOTFS"

Building the packages

As a high level overview the first thing we need are linux header files and musl installed. After that because were just cross compiling it opens the door up to be able to build lots of stuff.

Kernel header files.

# prep source
cd $MLFS/packages
tar -xf linux-6.19.tar.xz
cd linux-6.19


# build headers
make mrproper
make headers


# find and copy them in place
find usr/include -type f ! -name '*.h' -delete
cp -rv usr/include $ROOTFS/usr

Musl libc

#prep source
cd $MLFS/packages
tar -xf musl-1.2.5.tar.gz
cd musl-1.2.5

# Musl is one of the exception that breaks with the add flags.
export CFLAGS="-std=gnu99 --sysroot=$ROOTFS"

./configure --prefix=$ROOTFS/usr --syslibdir=$ROOTFS/lib 
make
make install

# verify our install
ls -l $ROOTFS/lib/ld-musl-x86_64.so.1
# lrwxrwxrwx 1 dakota dakota 40 Feb 28 23:04 /home/dakota/mlfs/rootfs/lib/ld-musl-x86_64.so.1 -> /home/dakota/mlfs/rootfs/usr/lib/libc.so
# This will cause breakages later see the section on chroot for details.

# restore our long cflags.
export CFLAGS="-std=gnu99 --sysroot=$ROOTFS -Wno-old-style-definition -static -I$ROOTFS/usr/include -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wno-error=implicit-function-declaration -static-libgcc"

Coreutils

#prep source
cd $MLFS/packages
tar -xf coreutils-9.6.tar.xz
cd coreutils-9.6

./configure --prefix=$ROOTFS/usr \
           --build=$(build-aux/config.guess) \
           --enable-install-program=hostname \
           --enable-no-install-program=kill,uptime \
           --host=$MLFS_TGT


 make
 make install

Diffutils

#prep source
cd $MLFS/packages
tar -xf diffutils-3.11.tar.xz
cd diffutils-3.11

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT


make
make install

Findutils

#prep source
cd $MLFS/packages
tar -xf findutils-4.10.0.tar.xz
cd findutils-4.10.0

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT


make
make  install

Sed

#prep source
cd $MLFS/packages
tar -xf sed-4.9.tar.xz
cd sed-4.9

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT

make
make  install

Grep

#prep source
cd $MLFS/packages
tar -xf grep-3.11.tar.xz
cd grep-3.11

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT


make
make  install

Dash

#prep source
cd $MLFS/packages
tar -xf dash-0.5.12.tar.gz
cd dash-0.5.12

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT

make
make  install

# set dash as /bin/sh
ln -svf dash $ROOTFS/bin/sh

GAWK

#prep source
cd $MLFS/packages
tar -xf gawk-5.3.1.tar.xz
cd gawk-5.3.1

sed -i 's/extras//' Makefile.in

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT

make
make  install

Readline

#prep source
cd $MLFS/packages
tar -xf readline-8.2.13.tar.gz
cd readline-8.2.13

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT


make
make  install

Ncurses

We need to build ncurses without c++ support for now. Ncurses does not like being built with -static. Even though I build both a static and non-static version, I have yet to figure out how to link bash statically against ncurses… This is why I have dash statically built.

#prep source
cd $MLFS/packages
tar -xf ncurses-6.5.tar.gz
cd ncurses-6.5

alias ldconfig=true
# remove the -static and add -fPIC
export CFLAGS="-std=gnu99 -fPIC --sysroot=$ROOTFS -Wno-old-style-definition -I$ROOTFS/usr/include -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wno-error=implicit-function-declaration -static-libgcc"

mkdir -p build
pushd build
   ../configure --prefix=$ROOTFS/usr \
       --without-gpm \
       --host=$MLFS_TGT
   make -C include
   make -C progs/../include
   make -C progs tic
popd


./configure --prefix=$ROOTFS/usr      \
   --build=$(./config.guess)     \
   --with-shared                 \
   --with-normal              \
   --without-cxx             \
   --without-debug               \
   --without-ada                 \
   --without-tests \
   --disable-stripping           \
   --without-gpm   \
   --host=$MLFS_TGT  \
   AWK=gawk


make && make install

# Now build a static version -- 
# This is optional, The reason I want a static ncurses is 
# to be able to build a static bash as my system shell.
export CFLAGS="-std=gnu99 --sysroot=$ROOTFS -Wno-old-style-definition -static -I$ROOTFS/usr/include -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wno-error=implicit-function-declaration -static-libgcc"export LDFLAGS="--sysroot=$ROOTFS -Wl,--dynamic-linker=/lib/ld-musl-x86_64.so.1 -static -static-libgcc"
./configure --prefix=$ROOTFS/usr      \
   --build=$(./config.guess)     \
   --with-normal              \
   --without-cxx             \
   --without-debug               \
   --without-ada                 \
   --without-tests \
   --disable-stripping           \
   --without-gpm   \
   --host=$MLFS_TGT  \
   AWK=gawk

make && make install

# This is for backwards compatibility
ln -svf libncursesw.a $ROOTFS/usr/lib/libncurses.a
ln -svf libncursesw.so $ROOTFS/usr/lib/libncurses.so

# These are here for nano
ln -svf ncursesw/curses.h $ROOTFS/usr/include/curses.h
ln -svf ncursesw/ncurses.h $ROOTFS/usr/include/ncurses.h
ln -svf ncursesw/term.h $ROOTFS/usr/include/term.h

ln -svf libncursesw.a $ROOTFS/usr/lib/libtinfo.a
ln -svf libncursesw.so $ROOTFS/usr/lib/libtinfo.so

# restore our general CFLAGS
export CFLAGS="-std=gnu99 --sysroot=$ROOTFS -Wno-old-style-definition -static -I$ROOTFS/usr/include -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wno-error=implicit-function-declaration -static-libgcc"

bash

#prep source
cd $MLFS/packages
tar -xf bash-5.2.37.tar.gz
cd bash-5.2.37

./configure --prefix=$ROOTFS/usr   \
           --without-bash-malloc \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT

make
make  install

Nano

#prep source
cd $MLFS/packages
tar -xf nano-8.7.1.tar.gz
cd nano-8.7.1

./configure --prefix=$ROOTFS/usr   \
           --build=$(./build-aux/config.guess) \
           --host=$MLFS_TGT

make
make  install

Busybox

Here we can limit busybox to just being an init system. Or you can build it with everything. The key point is were not installing it with it’s symlinks and once we get a working c++ compiler it will be removed for dinit.

If you use my sed commands this is all that will be built Currently defined functions: ash, echo, halt, init, mount, poweroff, reboot, sh, umount

#prep source
cd $MLFS/packages
tar -xf busybox-1.37.0.tar.bz2
cd busybox-1.37.0

# This creates a default config where everything is disable by default
make allnoconfig

# Enable the bare minimum for init
sed -i 's/# CONFIG_INIT is not set/CONFIG_INIT=y/' .config
sed -i 's/# CONFIG_ASH is not set/CONFIG_ASH=y/' .config
sed -i 's/# CONFIG_MOUNT is not set/CONFIG_MOUNT=y/' .config
sed -i 's/# CONFIG_UMOUNT is not set/CONFIG_UMOUNT=y/' .config
sed -i 's/# CONFIG_ECHO is not set/CONFIG_ECHO=y/' .config
sed -i 's/# CONFIG_SH_IS_ASH is not set/CONFIG_SH_IS_ASH=y/' .config
sed -i 's/# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
sed -i 's/# CONFIG_FEATURE_USE_INITTAB is not set/CONFIG_FEATURE_USE_INITTAB=y/' .config
sed -i 's/# CONFIG_HALT is not set/CONFIG_HALT=y/' .config
sed -i 's/# CONFIG_REBOOT is not set/CONFIG_REBOOT=y/' .config
sed -i 's/# CONFIG_POWEROFF is not set/CONFIG_POWEROFF=y/' .config
sed -i 's/# CONFIG_BUSYBOX is not set/CONFIG_BUSYBOX=y/' .config

# Will pick up anything these depend on
make oldconfig

#############################################
# This is a very minimal build of busybox to act as an init system
# If you would like to add other functionality be my guest.
# I must caution against installing the applets since we built gnu utilities!
# remember all applets can be ran by busybox <appletname> eg. busybox ls
# and these can be wrapped in shell scripts!
#
# OPTIONAL: If you want to add more applets, run menuconfig.
# There is a weird bug where you must let it fail first, patch it, then rerun.
# Skip this block entirely if you're happy with the sed config above.
#############################################
# A weird bug in the code where I had to run make menuconfig and let it fail then
# patch it with sed and then rerun make menuconfig
make menuconfig

# To get busybox make menuconfig to work
sed -i 's/^\(always\s*:=[^#]*\)$/#\1/' scripts/kconfig/lxdialog/Makefile

# Configure busybox. You must disable under Networking Utilities -> tc for static builds to work
# Static build option is Settings -> Build static binary (no shared libs)
make menuconfig
###########################################
# Build and install
make -j$(nproc)

# If you choose to add other applets this will let you run them.
# e.g. busybox ls
cp -apvr busybox $ROOTFS/usr/bin
cp -apvr busybox $ROOTFS/sbin/init
ln -svf /usr/bin/busybox $ROOTFS/sbin/halt
ln -svf /usr/bin/busybox $ROOTFS/sbin/poweroff
ln -svf /usr/bin/busybox $ROOTFS/sbin/reboot
ln -svf /usr/bin/busybox $ROOTFS/bin/mount
ln -svf /usr/bin/busybox $ROOTFS/bin/umount

Init setup

This is the most bare bones init setup. We mount are essential temp file systems and drop you into a shell. While this will work today for our init, to see a running system. These will be replaced with dinit when the time is right

cat > "$ROOTFS/etc/inittab" <<'EOF'
::sysinit:/usr/bin/mount -t proc proc /proc
::sysinit:/usr/bin/mount -t sysfs sys /sys
::sysinit:/usr/bin/mount -t devtmpfs dev /dev
::sysinit:/usr/bin/mkdir -p /dev/pts
::sysinit:/usr/bin/mount -t devpts devpts /dev/pts
::respawn:/usr/bin/sh
EOF

Linux

If you’re running this inside of an environment like qemu the default config will work just fine.

LFS people will probably not like this, but we’re not going to be building our current kernel against our sysroot. In a proper LFS build the kernel would not be built this early in the process anyway. This is just a hack to get the system to boot inside of qemu. – If you don’t plan on booting it inside of qemu yet, you can skip this until we get to a more appropriate LFS stage for it.

#prep source
cd $MLFS/packages
tar -xf linux-6.19.tar.xz
cd linux-6.19

unset CFLAGS
unset LDFLAGS
unset PKG_CONFIG_SYSROOT_DIR

# USE LLVM for building the kernel (clang)
export LLVM=1

make defconfig
make -j$(nproc) bzImage
cp -vr arch/x86/boot/bzImage $ROOTFS/boot

Setup the root account

While the system will run a shell that will in practice have root power. It will not be recognized as any user. To overcome this we need to create /etc/{passwd,shadow,group} along with the root user directory.

mkdir -pv $ROOTFS/root
chmod 0700 $ROOTFS/root

cat > $ROOTFS/etc/passwd << 'EOF'
root:x:0:0:root:/root:/bin/bash
EOF

cat > $ROOTFS/etc/shadow << 'EOF'
root:::0:99999:7:::
EOF

cat > $ROOTFS/etc/group << 'EOF'
root:x:0:
EOF

chmod 600 $ROOTFS/etc/shadow

If you are use to using busybox ash setting up a default environment, you’ll be shocked to find bash and dash do not do this. We’re going to create two files. /etc/profile which is a global config that any user can source and a local /root/.bashrc

cat > $ROOTFS/etc/profile << 'EOF'

export SHELL=/bin/bash
export PATH=/usr/bin/:/usr/sbin
export TERM=linux
export PS1='\[\e[1;36m\]\u\[\e[1;35m\]@\[\e[1;33m\]\h\[\e[1;34m\] \w \$\[\e[0m\] '
alias ls="ls --color=always"
EOF

cat > $ROOTFS/root/.bash_profile << 'EOF'
export HOME=/root
export USER=root
export LOGNAME=root
source /etc/profile
source ~/.bashrc
EOF

cat > $ROOTFS/root/.bashrc << 'EOF'
export HOME=/root
export USER=root
export LOGNAME=root
source /etc/profile
EOF

Chrooting

Before we can chroot into our system if we dynamically linked stuff we must update the symlink “/lib/ld-musl-x86_64.so.1”. Currently it points to a file path that would not exist inside the chroot. /home/dakota/mlfs/rootfs/lib/ld-musl-x86_64.so.1 -> /home/dakota/mlfs/rootfs/usr/lib/libc.so

What we need to do is point it to /lib/libc.so

ln -svf libc.so /home/dakota/mlfs/rootfs/lib/ld-musl-x86_64.so.1

After spending many hours banging my head against the wall with this to figure it out. I decided to build the tools statically. Cause while the above did work, I’m of the opinion that your system shell, coreutils, findutils, grep, sed, command line editor (vim,nano,flow,etc) and init should all be statically built anyway to minimize on potential issues and give you a basic system that should work even if everything else is broken.

enter the new system

sudo chroot $ROOTFS
source /root/.bashrc

Assembling the Disk Image

cd $MLFS

# Create a 10GB image to give room to play around
dd if=/dev/zero of=disk.img bs=1M count=10240
sudo mkfs.ext4 disk.img

mkdir -pv mntimg
sudo mount disk.img mntimg

sudo rsync -av --delete $ROOTFS/ mntimg/
sudo mkdir -vp mntimg/boot/extlinux

# Copy Syslinux modules - Arch Linux
sudo cp -vr /usr/lib/syslinux/bios/*.c32 mntimg/boot/extlinux/

# Copy Syslinux modules - Slackware
sudo cp -vr /usr/share/syslinux/*.c32 mntimg/boot/extlinux/

# If unsure, find them
find / -name "*.c32" 2>/dev/null

cat >> extlinux.conf << 'EOF'
DEFAULT linux
LABEL linux
   KERNEL /boot/bzImage
   APPEND root=/dev/sda rw init=/usr/sbin/init console=ttyS0
   TEXT HELP
       Boot the minimal playground kernel with your static rootfs.
   ENDTEXT
EOF

sudo cp -apvr extlinux.conf mntimg/boot/extlinux/extlinux.conf

sudo extlinux -i mntimg/boot/extlinux
sync
sudo umount mntimg

Booting in QEMU

qemu-system-x86_64 -cpu host -enable-kvm -m 2G -drive file=disk.img,format=raw -nographic -serial mon:stdio
bash
source /root/.bashrc

Bootsystem output

   [    1.281226] EXT4-fs (sda): mounted filesystem 25090fc9-d5e6-4e5b-bd32-c3d4ee949336 r/w with ordered data mode. Quota mode: none.
   [    1.282897] VFS: Mounted root (ext4 filesystem) on device 8:0.
   [    1.284150] devtmpfs: mounted
   [    1.284809] Freeing unused kernel image (initmem) memory: 2844K
   [    1.285700] Write protecting the kernel read-only data: 28672k
   [    1.286702] Freeing unused kernel image (text/rodata gap) memory: 968K
   [    1.287716] Freeing unused kernel image (rodata/data gap) memory: 776K
   [    1.316277] x86/mm: Checked W+X mappings: passed, no W+X pages found.
   [    1.317217] Run /usr/sbin/init as init process
   init started: BusyBox v1.37.0 (2026-03-01 00:36:30 EST)
   starting pid 57, tty '': '/usr/bin/mount -t proc proc /proc'
   [    1.324655] mount (57) used greatest stack depth: 13056 bytes left
   starting pid 58, tty '': '/usr/bin/mount -t sysfs sys /sys'
   starting pid 59, tty '': '/usr/bin/mount -t devtmpfs dev /dev'
   mount: mounting dev on /dev failed: Device or resource busy
   starting pid 60, tty '': '/usr/bin/mkdir -p /dev/pts'
   starting pid 61, tty '': '/usr/bin/mount -t devpts devpts /dev/pts'
   starting pid 62, tty '': '/usr/bin/sh'
   /usr/bin/sh: 0: can't access tty; job control turned off
   #
   # whoami
   root
   # bash
   bash: cannot set terminal process group (62): Not a tty
   bash: no job control in this shell
   # bash
   bash: cannot set terminal process group (62): Not a tty
   bash: no job control in this shell
   bash-5.2# source .bashrc
   root@(none) ~ #

Conclusion

Now that everything is said and done. We have a simple musl based Linux system that we can chroot into and boot inside of qemu. This is no small feat and many purpose driven systems could be built using this method. (Robotics, firewall, router, etc) Though it lacks a proper development environment. According to the du command my entire build is only 93M! Talk about minimalism.

Going forward I have two choices. Use GCC like is described in the LFS book or try and bootstrap llvm/clang. Naturally I’m going to try the later, and probably settle for the former. – You’ll have to stay in touch for future blog posts to know which path I take.

Sincerely, a concrete worker. May the peace and grace of our Lord be with you.