Making AOSP’s nsjail Work with AppArmor (Ubuntu 24.04)

TL;DR

On Ubuntu 24.04 (kernel 6.8) with kernel.apparmor_restrict_unprivileged_userns=1, our AOSP 16 (also applicable to 15 and similar versions) builds showed:

• Build sandboxing disabled due to nsjail error.
• mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied
• Unable to connect socket: No Access

Fix: Keep the system hardening knob enabled (kernel.apparmor_restrict_unprivileged_userns=1). Put AOSP’s nsjail under an AppArmor profile, start in complain mode, read kernel logs, and allow only what’s necessary (user namespaces, UNIX sockets, netlink, minimal loader/lib/tool paths). Flip to enforce once clean. No need to disable the system restriction.

Environment

• OS: Ubuntu 24.04.3 LTS (kernel 6.8)
• AOSP: Platform 16
• System knob: kernel.apparmor_restrict_unprivileged_userns=1
• Goal: Keep the restriction, make AOSP’s nsjail happy.

Symptoms We Saw

Frequent failures/logs included:

• Build sandboxing disabled due to nsjail error.
• mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied
• Unable to connect socket: No Access

Important nuance: In our case, the failures (e.g., “Unable to connect socket,” netlink errors) were primarily caused by an incomplete nsjail configuration and environment—missing bind mounts for a minimal rootfs (e.g., /bin, /lib, /lib64, /usr, /dev) and missing AppArmor allowances like unix, and network netlink raw. These gaps caused the child process spawned by nsjail to fail during initialization when it needed sockets/netlink, the dynamic linker, and tool paths. This is not the same as complain mode “blocking”—complain mode itself does not block.

1. Start with the tiniest profile (complain)

Create /etc/apparmor.d/nsjail-aosp:

#include <tunables/global>

profile nsjail-aosp /**/prebuilts/build-tools/linux-x86/bin/nsjail flags=(attach_disconnected) {
  userns,
}

Load it in complain mode:

sudo apparmor_parser -r -C /etc/apparmor.d/nsjail-aosp

This guarantees the profile attaches to the right nsjail binary under any AOSP source tree (thanks to the /**/ wildcard) and that user namespaces are allowed—without blocking.

2. Build once and read the logs

Kick off a build (or a small nsjail test) to collect logs, then inspect:

journalctl -k -g apparmor --since "300 minutes ago"

In complain mode, you’ll see “DENIED” entries as would-deny (they don’t block). Use them to learn what to allow.

3. Minimal working profile (small and reusable)

After a quick log pass, this was enough to make nsjail behave for AOSP 16:

include <tunables/global>

profile nsjail-aosp /**/prebuilts/build-tools/linux-x86/bin/nsjail flags=(attach_disconnected) {
  signal receive set=exists peer=nsjail-aosp,
  signal receive set=term peer=nsjail-aosp,
  signal receive set=urg peer=nsjail-aosp,
  signal receive set=usr1 peer=nsjail-aosp,
  signal receive set=usr2 peer=nsjail-aosp,
  signal send set=exists peer=nsjail-aosp,
  signal send set=term peer=nsjail-aosp,
  signal send set=urg peer=nsjail-aosp,
  signal send set=usr1 peer=nsjail-aosp,
  signal send set=usr2 peer=nsjail-aosp,

  capability dac_override,
  capability sys_admin,
  capability setgid,
  capability setpcap,
  capability net_admin,

  userns,
  mount,
  umount,
  pivot_root,

  network unix,
  network netlink raw,

  / mr,
  /** mr,
  /** ix,

  /dev/tty rw,
  /dev/null rw,
  /dev/shm/** mrwklm,

  /proc/[0-9]*/setgroups w,
  /proc/[0-9]*/gid_map w,
  /proc/[0-9]*/uid_map w,
  /proc/[0-9]*/coredump_filter w,

  /run/user/*/nsjail/** mrwklm,
  /run/user/nsjail.*.root/** mrwklm,
  /run/user/nsjail.*.tmp/** mrwklm,

  /tmp/** mrwklm,
  /mnt/data2/aosp-16/** mrwklm,

  /nsjail_build_sandbox/** mrwklm,
}

Reload (keep complain while iterating):

sudo apparmor_parser -r -C /etc/apparmor.d/nsjail-aosp

Optional: ccache (AppArmor-only)

If you use ccache, add only these AppArmor rules as requested:

  @{HOME}/.cache/ccache/ mrwklm,
  @{HOME}/.cache/ccache/** mrwklm,

For full ccache setup/tuning (env vars, sizing, compression), see my separate guide:

Quick sanity test (standalone)

Bind a tiny rootfs into the jail and confirm the label:

NSJ="/path/to/aosp/prebuilts/build-tools/linux-x86/bin/nsjail"

"$NSJ" -R /bin -R /lib -R /lib64 -R /usr -R /dev -- \
  /bin/sh -c 'cat /proc/self/attr/current; sleep 1'
# Expect: nsjail-aosp (complain)

Build again & what to watch

Review fresh logs during a failed run:

journalctl -k -g apparmor --since "300 minutes ago"

Accept only what you really need; keep the profile small.

Flip to enforce (when stable)

Once logs are quiet and builds are reliable, switch:

sudo apparmor_parser -r /etc/apparmor.d/nsjail-aosp   # No -C means enforce

Safe revert

Remove from kernel (keeps the file on disk):

sudo apparmor_parser -R /etc/apparmor.d/nsjail-aosp

Reload later with -r (add -C for complain).