Kernel panic on a Debian NAS: recovering a truncated libc without rebooting into the void

The symptom

NAS unreachable over ping and SSH. On the physical console, GRUB shows Boot Option Restored then a lonely grub>. An exit brings back the normal menu, but with a single kernel entry: no older kernel, no recovery mode. On boot:

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

panic → do_exit.cold → do_group_exit → __x64_sys_exit_group
     → do_syscall_64 → handle_mm_fault → do_user_addr_fault

Segfault in PID 1. Systemd dies on the very first userspace exec, and the kernel has no choice. Same result after manually stripping the hardening options via grub edit: the problem isn't in the kernel command line, it's right after it.

The hardware context: a Terramaster F6-424 whose stock TOS was replaced by Debian 13 (Trixie) installed on the internal 4 GB USB stick. The 6 bays host old Synology disks with RAID5 + LVM + Btrfs structures mounted on /mnt/data. The system is hardened against a CIS / ANSSI baseline: AppArmor in enforce, auditd active, kernel options oops=panic slab_nomerge init_on_alloc=1 …, and GRUB_DISABLE_RECOVERY="true".

Nearly every one of those choices is about to backfire.

Diagnosis: live USB boot and debugfs

After booting a Debian 13.5 XFCE live image, lsblk -f confirms the system lives on /dev/sdf (3.75 GiB USB stick): sdf1 as FAT32 for /boot/efi, sdf2 as ext4 for /.

First reflex: fsck -n -f /dev/sdf2. Clean ext4 structure, no filesystem errors. So the corruption isn't structural — it's in file contents.

debugfs reads an ext4 FS without mounting it, so strictly read-only:

debugfs -R 'cat /etc/debian_version' /dev/sdf2
debugfs -R 'ls -l /boot' /dev/sdf2
debugfs -R 'stat /lib/systemd/systemd' /dev/sdf2
debugfs -R 'cat /var/log/apt/history.log' /dev/sdf2

Surprise in /var/log/apt/: history.log and term.log are 0 bytes. Truncated. The archived .gz still holds the earlier history.

Mount the partition to chroot:

mount /dev/sdf2 /mnt/sysroot
mount /dev/sdf1 /mnt/sysroot/boot/efi
chroot /mnt/sysroot /lib/systemd/systemd

error while loading shared libraries:
/lib/x86_64-linux-gnu/libc.so.6: invalid ELF header

And file delivers the verdict:

$ file /mnt/sysroot/usr/lib/x86_64-linux-gnu/libc.so.6
ISO-8859 text, with very long lines

libc is no longer an ELF. It's still 1,999,312 bytes — almost the right size — but its blocks are random text. The kernel loads, execs init (= systemd), the loader tries to map libc, segfault, init dies, panic.

The reconstructed timeline

history.log.1.gz tells the whole story:

• February 21, 12:45: first disk-full event during an apt purge. Error: mandb: can't write to /var/cache/man/...: No space left on device.
• February 27, 06:45: unattended-upgrade upgrades libnss3, triggering the libc-bin triggers.
• February 28, 23:45: history.log and term.log truncated to zero.

The second disk-full event happened during the libc-bin trigger. dpkg wrote libc.so.6 partially: the file keeps its apparent size, but blocks are unwritten or overwritten with something else. Invalid ELF.

Why the crash was inevitable

Six reasonable decisions in isolation, catastrophic combined:

1. A 3.5 GB system disk for an OS downloading hundreds of MB of updates per month.
2. unattended-upgrade running unsupervised, with no mail alert, no pre-transaction space check.
3. GRUB_DISABLE_RECOVERY="true": no way to boot single-user to repair.
4. A single installed kernel (reflexive apt autoremove). No fallback.
5. oops=panic: every kernel oops becomes an immediate full panic.
6. Aggressive hardening: optimising for the attacker, not for the operator panicking at 7am.

None of these is bad on its own. Combined, they build a system that cannot catch itself.

The fix (the chicken-and-egg trap)

The whole difficulty fits in one sentence: apt and dpkg depend on libc. While libc is broken, you can't chroot and run apt --reinstall — everything segfaults. The solution is to extract the .deb files by hand from the live USB environment, which has its own intact libc.

1. Prepare the bind mounts

R=/mnt/sysroot
mount --bind /dev      "$R/dev"
mount --bind /dev/pts  "$R/dev/pts"
mount -t proc proc     "$R/proc"
mount -t sysfs sys     "$R/sys"
mount --bind /run      "$R/run"
cp -L /etc/resolv.conf "$R/etc/resolv.conf"

2. Free up space

On a hardened system, plenty of large logs can go with no functional risk. Target: ≥ 500 MB free so the reinstalls can write without filling the disk again.

R=/mnt/sysroot
find "$R/var/cache/apt/archives" -name '*.deb' -delete
rm -rf "$R/var/cache/swcatalog/"*
rm -rf "$R/var/log/installer"
rm -f  "$R/var/log/"*.gz  "$R/var/log/"*.old
rm -rf "$R/var/log/journal/"*
find "$R/var/log/sudo-io" -mindepth 1 -delete
find "$R/var/log/pcp" -type f -delete
rm -f  "$R/var/log/audit/audit.log."*
truncate -s 0 "$R/var/log/audit/audit.log"

3. Download the right .deb files in the live env, not in the chroot

cd /tmp
apt-get update
apt-get download libc6 libc-bin libc-l10n

# Back up the corrupted libc, useful for forensics
cp /mnt/sysroot/usr/lib/x86_64-linux-gnu/libc.so.6 /tmp/libc.so.6.corrupted.bak

4. Extract the .deb files straight onto the target

This is the key step. dpkg-deb -x just overwrites files, without touching the dpkg database:

dpkg-deb -x /tmp/libc6_*.deb     /mnt/sysroot
dpkg-deb -x /tmp/libc-bin_*.deb  /mnt/sysroot
dpkg-deb -x /tmp/libc-l10n_*.deb /mnt/sysroot

# Check libc.so.6 is executable again
$ /mnt/sysroot/usr/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.41-12+deb13u3) stable release version 2.41.

5. Chroot, and reinstall cleanly through apt

chroot "$R" /bin/bash
/bin/true && echo "userspace OK"

apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install --reinstall -y \
    libc6 libc-bin libc-l10n

apt sees the files are already there, but this time it records the version in the dpkg database and runs the postinst scripts. Consistent state restored.

6. Full audit with debsums

debsums verifies the MD5 checksums shipped by each .deb for every installed file — far more thorough than dpkg --verify, which only covers conffiles:

apt-get install -y debsums
debsums -ac 2>&1 | tee /tmp/debsums.out

Interpretation:

• Modified /etc/* files → normal, these are admin-modified conffiles. Don't restore them.
• Failing or missing /usr/lib/modules/* files → critical, reinstall the linux-image-X package.
• Failing files in /usr/bin, /usr/lib, /usr/sbin → critical, reinstall the owning package.

In this case: 7 corrupted kernel modules + a MISSING btrfs.ko.xz. The disk-full event had hit the kernel modules too. Rebooting after fixing only libc would have produced a fresh panic on the Btrfs mount.

apt-get install --reinstall -y linux-image-6.12.73+deb13-amd64

This regenerates the initramfs automatically too.

7. Regenerate GRUB, unmount cleanly, reboot

update-initramfs -u -k all
update-grub
exit

sync
umount -R /mnt/sysroot/sys /mnt/sysroot/proc /mnt/sysroot/dev
umount /mnt/sysroot/run /mnt/sysroot/boot/efi /mnt/sysroot
fsck.ext4 -f /dev/sdf2
fsck.vfat -a /dev/sdf1
reboot

No need for grub-install if the EFI partition is intact: the /EFI/BOOT/BOOTX64.EFI fallback plus the fbx64.efi shim recreate the NVRAM entry on next boot.

The twist: the real root cause

Three months later, the same scenario starts building. An SSH check shows:

/dev/sde2  3.4G  3.1G  107M  97% /    ← root at 97%
load average: 8.54

PID 4595   root  /usr/bin/python3 /usr/bin/unattended-upgrade
PID 17266  root  /usr/bin/dpkg --unpack ... /tmp/apt-dpkg-install-dv8QKx
PID 343    md2_raid5     38.9% CPU
PID 820    md2_resync    16.7% CPU

unattended-upgrade is running, dpkg is unpacking, the RAID resync is hammering I/O, and plenty of processes sit in D state. Thirty seconds later SSH refuses connections, then no TCP port answers at all. ping still works: the kernel is alive, userspace is frozen. On the console:

EXT4-fs (sde2): failed to convert unwritten extents to written extents
                -- potential data loss! (inode XXXX, error -5)
EXT4-fs error (device sde2) in ext4_reserve_inode_write:5980: IO failure
[FAILED] Failed unmounting boot-efi.mount
[FAILED] Failed to start docker.service (×6)

error -5 = EIO, a hardware I/O error. This wasn't the disk filling up: the USB stick was starting to rot. Flash wear was February's root cause; the full disk had only been the last straw.

Hard power-off at the button, with the risk of reproducing the libc corruption. On reboot:

$ file /usr/lib/x86_64-linux-gnu/libc.so.6
ELF 64-bit LSB shared object [...]

libc OK, FS clean. The dpkg transaction in flight at crash time was gpg / gpg-agent / dirmngr / gnupg-utils — nothing critical. Had it been libc or systemd, back to the live USB.

The guardrails to put in place

These measures are free, take ten minutes, and reproduce what every sysadmin used to do before unattended-upgrades became transparent enough to forget about.

1. Disable unattended-upgrades (the trigger)

sudo systemctl stop unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service
sudo systemctl mask apt-daily.timer apt-daily-upgrade.timer

On a fragile system (small disk, RAID resyncing, aggressive hardening), unattended-upgrades is a risk, not a protection. Manual updates, supervised, when the machine has headroom.

2. apt-mark hold on critical packages

sudo apt-mark hold linux-image-6.12.73+deb13-amd64 linux-image-amd64 \
    libc6 libc-bin libc-l10n systemd systemd-sysv grub-efi-amd64

These packages won't move until an explicit apt-mark unhold. For a kernel, that means testing the new version elsewhere first and keeping the old one as fallback.

3. APT pre-invoke disk space check

A hook that refuses any transaction if the headroom isn't there:

# /usr/local/bin/apt-disk-check.sh
#!/bin/bash
set -e
MIN_ROOT_MB=300
MIN_VAR_MB=300
MIN_CACHE_MB=500
PHASE="${1:-dpkg}"

free_mb() { df -BM --output=avail "$1" 2>/dev/null | tail -1 | tr -d ' M'; }

if [ "$PHASE" = "update" ]; then
  c=$(free_mb /var/cache/apt)
  [ -n "$c" ] && [ "$c" -lt "$MIN_CACHE_MB" ] && {
    echo "REFUSED APT update: ${c}M free on /var/cache/apt" >&2; exit 1; }
else
  r=$(free_mb /); v=$(free_mb /var)
  [ -n "$r" ] && [ "$r" -lt "$MIN_ROOT_MB" ] && {
    echo "REFUSED dpkg: ${r}M free on /" >&2; exit 1; }
  [ -n "$v" ] && [ "$v" -lt "$MIN_VAR_MB" ] && {
    echo "REFUSED dpkg: ${v}M free on /var" >&2; exit 1; }
fi

// /etc/apt/apt.conf.d/99-disk-space-check
APT::Update::Pre-Invoke   { "/usr/local/bin/apt-disk-check.sh update"; };
DPkg::Pre-Invoke          { "/usr/local/bin/apt-disk-check.sh dpkg"; };

Note: inlining the awk check directly in the .conf file with nested \" doesn't survive the APT parser. An external script is testable, debuggable, and keeps esoteric syntax out of a config file.

4. Disk space alert below 25% free

# /usr/local/bin/disk-alert.sh
#!/bin/bash
THRESHOLD="${THRESHOLD:-75}"
MAIL_TO="${MAIL_TO:-root}"
HOST=$(hostname)
df --output=source,pcent,target -x tmpfs -x devtmpfs -x squashfs -x overlay -x ecryptfs \
  | tail -n +2 \
  | while read fs use mnt; do
      pct="${use%\%}"
      [ -z "$pct" ] && continue
      case "$pct" in (*[!0-9]*) continue;; esac
      if [ "$pct" -ge "$THRESHOLD" ]; then
        msg="[disk-alert] ${HOST}: ${fs} at ${pct}% (mount ${mnt})"
        logger -t disk-alert -p user.warning "$msg"
        command -v mail >/dev/null 2>&1 && echo "$msg" | mail -s "$msg" "$MAIL_TO" || true
      fi
    done

# /etc/cron.d/disk-alert
*/15 * * * * root THRESHOLD=75 MAIL_TO=admin@example.com /usr/local/bin/disk-alert.sh

Always push locally via logger at minimum: even without a configured MTA, the information lands in journald.

5. Re-enable recovery mode in GRUB

sudo sed -i 's/^GRUB_DISABLE_RECOVERY=.*/GRUB_DISABLE_RECOVERY="false"/' /etc/default/grub

Recovery mode gives a single-user root shell without starting any service. That's the difference between a 5-minute repair and an evening with a live USB.

6. oops=panic → panic=30

On a critical datacenter server, oops=panic forces an immediate reboot on the slightest kernel warning to limit a potential compromise. On a personal NAS, every benign oops (a flaky USB driver) becomes a full panic.

# /etc/default/grub
# BEFORE: ...debugfs=off oops=panic audit_backlog_limit=8192 panic=10
# AFTER:  ...debugfs=off audit_backlog_limit=8192 panic=30
sudo update-grub

panic=30 keeps the essential benefit: auto-reboot after a real panic, with 30 seconds to read the context on the console.

7. Protect kernels from apt autoremove

// /etc/apt/apt.conf.d/01autoremove-kernels
APT::NeverAutoRemove {
  "^linux-image-.*";
  "^linux-headers-.*";
  "^linux-modules-.*";
};

apt autoremove stays useful for orphaned packages, but it can never take a kernel again. You remove those by hand, knowingly.

The real fix: drop the USB stick

Every measure above is a band-aid. A 4 GB USB stick as a system disk means flash wear, no usable SMART data, a cheap controller and zero headroom for updates.

• Migrate to an M.2 or SATA SSD ≥ 64 GB (a 250 GB drive at €30 does the job, ×100 endurance).
• Before migrating: a full dd of the current stick (dd if=/dev/sdX of=/mnt/data/backup/nas-usb-$(date +%F).img bs=64M conv=fsync + gzip --best).
• Btrfs Snapper snapshots on /mnt/data: free space-wise thanks to CoW, and btrfs send --proto 3 (new in Debian 13) makes incremental sends ×3 faster.
• Real 3-2-1 backups: a RAID is not a backup. Borgmatic + offsite, monthly borg check --verify-data.
• SMART monitoring on the HDDs (smartd + mail). A RAID5 with a disk dying during a resync means total loss — hence preferring RAID6 / mirror+stripe on disks over 5 years old.

What to take away

A corrupted libc.so.6 means an immediate kernel panic with no clear signature in the panic message. The stack trace shows do_user_addr_fault → segfault in PID 1, but never says which library is at fault. Manual inspection required.

CIS hardening protects against attackers, but reduces fault tolerance. On a personal NAS, a softer compromise (recovery mode enabled, older kernels kept, oops=panic removed) is wiser than the strict Level 2 baseline.

APT pre-check hooks for disk space are trivial to set up and badly underused. If you take one thing from this article, set one up tonight.

debsums is the reference tool for post-incident auditing on Debian, worth installing by default on every server.

And above all: document your incidents. What saved time in May was having a written trace of February's diagnosis, instead of having to reinvent it under pressure.

References

• Debian 13 Trixie Release Notes (2026-05-10)
• Crash recovery using debsums - Arthur de Jong
• Recovering from a broken libc - mythopoeic.org
• APT Hooks reference
• Kernel.org - panic, oops and crash dumps