m0x41-podman Licensed under Apache 2.0.
This post describes a verification-first methodology for Systems Engineering, one where the effort to prove an artifact works exceeds the effort to create it by an order of magnitude. The methodology is human-defined but AI-agent-executed, combining Subject Matter Expertise (SME) with agentic implementation bandwidth to achieve what neither could accomplish alone within a reasonable timeframe. The illustrative example is an unofficial Podman [1] v5.8.1 snap package, a Linux packaging format that bundles an application and all its dependencies into a single installable file.
The core packaging is approximately 700 lines across the snapcraft.yaml, patches, wrappers, hooks, and configuration. Proving it works across five Linux distributions required a seven-phase verification process that produced over 7,100 lines of test automation and documentation across 44 files. The iterative process spanned 76 commits and an estimated 10,000+ test executions, generating over 18,000 lines of shell at its peak before stabilising into the current repository. The package now runs in two production deployments in rootless mode. This is the first in a series of open source tools from Curio Data Pro designed to support agentic workflows, and a demonstration of how we apply Artificial Intelligence tools in practice.
Introduction
AI-assisted software development is typically characterised as code generation: an agent writes a function, a test, a configuration file. This framing understates the harder problem. In Systems Engineering, creating an artifact is rarely the bottleneck; verifying that it works across environments, under real-world conditions, and against edge cases is where the significant effort is required. This asymmetry between creation and verification is a recurring theme in consulting engagements, and one that is not naturally addressed in agentic workflows.
To illustrate this, the post walks through the development of an unofficial Podman v5.8.1 snap package, a packaging challenge that has remained unsolved for over six years [2] despite multiple upstream attempts. The package itself is a useful tool: it provides the container runtime that powers the pipeline described in a previous post, Ingesting a Nation’s Railway. But the development process is the real subject. It demonstrates how a system thinker who understands the architecture but is not intimately familiar with the underlying C, Go, and Rust code can direct AI agents to execute a rigorous, multi-phase verification methodology that would otherwise be impractical.
Why Podman
Podman is an Open Container Initiative [3] (OCI) compliant container runtime developed by Red Hat and the Containers [4] open source community. It provides a Command-Line Interface (CLI) largely compatible with Docker [5], meaning existing Dockerfile workflows and OCI images work without modification. The architectural differences, however, are significant.
| Podman | Docker | |
|---|---|---|
| Architecture | Daemonless (fork-exec) | Central daemon (dockerd) |
| Root requirement | Rootless by default | Daemon runs as root |
| Systemd integration | Native (Quadlets [6]) | External (Docker Compose) |
| Pod support | Built-in (podman pod) |
No native equivalent |
| Process model | Each container is a child process | All containers managed by daemon |
The fork-exec model is the standard Unix process creation pattern: the calling process creates a copy of itself (fork), and the child replaces itself with the target program (exec). When you run podman run, Podman directly forks and execs the container runtime as a child process of your user session, with no intermediary. With Docker, docker run sends a request over a socket to the dockerd daemon, a long-running root process that manages all containers centrally. The distinction matters: fork-exec means each container inherits the invoking user’s privilege boundary, there is no single daemon to crash and take all containers with it, and child processes are directly manageable by Systemd, which is what makes Quadlets work cleanly.
Docker’s daemon-centric design concentrates both privilege and risk. If the daemon is compromised, every container it manages is exposed. Podman’s daemonless architecture eliminates this single point of failure. Each container runs as a direct child process of the invoking user, inheriting their privilege boundary.
The Agent Sandbox
For AI agent workflows, this architectural distinction matters. An agent that needs to build, test, and deploy containerised services requires a container runtime, but granting that agent root access defeats the purpose of sandboxing. Podman’s rootless mode resolves this directly: the container runtime operates entirely within the agent’s user namespace, requiring no elevated privileges.
Coupled with standard Linux primitives such as a dedicated user account with no login shell, restricted filesystem permissions, and user-scoped Systemd services, rootless Podman creates a practical sandbox. The agent can pull images, build containers, create multi-container pods, and deploy services, all without sudo and with clear boundaries around what it can affect on the host.
Podman v5 and the Deployment Workflow
Podman v5 substantially improved two capabilities that transform what an agent can achieve.
Most real-world applications consist of multiple containers that need to run together: a web server, a database, a background worker. Podman can group these into a pod, a single deployable unit where the containers share a network namespace, mirroring Kubernetes [7] pod semantics. Before v5, pods served primarily as a local development primitive, allowing developers to compose and test multi-container applications before deploying them to Kubernetes. Production deployment still required an orchestrator. Podman v5 closes this gap through expanded Quadlet [6] support, making Podman itself a simpler, cheaper, and in many circumstances equally effective substitute for Kubernetes on single-node deployments.
Quadlets are simple text files (.pod, .container, .volume, .network) that declare how containers should run. Systemd generators convert these files into native service units, so the same pod definition tested locally becomes a production service that starts on boot, restarts on failure, and is managed by the host’s service manager.
Together, these enable an end-to-end workflow:
- The agent writes application code and
Dockerfiledefinitions - It builds OCI images with
podman build - It groups containers into
podsand validates them withpodman pod - It writes Quadlet unit files defining the deployment
- A
systemctl --user daemon-reloadactivates the services - Systemd manages the lifecycle: automatic restarts, dependency ordering, boot persistence
This is not a development convenience. It is a production deployment path: rootless, daemonless, and fully integrated with the host’s service manager. A previous post on this blog, Ingesting a Nation’s Railway, described a real-time data pipeline running as rootless Podman containers orchestrated by Systemd Quadlets, the very capability this snap package provides on distributions where it would otherwise be unavailable.
The Packaging Gap
Ubuntu 24.04 LTS, the current long-term support release and a common choice for servers, ships Podman v4.9.3. This version lacks the pod and Quadlet improvements described above. No official snap package exists. The upstream Podman maintainers have indicated that snap packaging is a community responsibility rather than a project-maintained effort. Several community attempts between 2018 and 2024 have not reached completion, and a non-functional Snap Store listing from 2020 remains the only visible artifact.
The difficulty is not in compiling Podman itself. It is in bundling the ecosystem: the OCI runtime (crun), the networking backend (netavark and aardvark-dns), the container monitor (conmon), rootless networking (slirp4netns), rootless storage (fuse-overlayfs), and the configuration files that bind them together. A snap must be self-contained: every binary, library, and configuration path must resolve within the snap’s filesystem. Classic confinement is required because the setuid binaries (newuidmap, newgidmap) that enable rootless user namespaces cannot be bundled; they must be present on the host.
setuid
snap packages normally run inside a security sandbox that restricts their access to the host system. Classic confinement removes this sandbox, allowing the snap to behave like a traditionally installed application with full access to the host filesystem and system calls.
This is necessary for Podman because rootless containers depend on setuid binaries, programs that temporarily escalate privileges to perform a specific operation. The newuidmap and newgidmap utilities (from the uidmap package) use this mechanism to map user and group IDs into a container’s namespace, which is how unprivileged users can run containers without root access. Because setuid is a property set on the file by the host operating system, these binaries cannot be bundled inside a snap. They must already exist on the host with the correct permissions.
This had been a personal ambition for six years. The complexity of the dependency chain and the breadth of integration testing required had kept it perpetually on the backlog until AI agents made the verification effort tractable.
Verification-First Methodology
Creating the snapcraft.yaml from Podman’s upstream build instructions was straightforward, a task well-suited to an experienced engineer or an AI code generation tool. The file defines seven build parts (Go toolchain, crun, conmon, netavark, Podman, configuration files, and wrapper scripts) totalling 199 lines.
But a snap that builds is not a snap that works. This distinction between an artifact that exists and one that is verified is a pattern encountered repeatedly in client engagements. The verification effort for this package exceeded the creation effort by an order of magnitude, driven by an iterative methodology where each phase of testing revealed issues that shaped the next.
The following table summarises the seven phases. Each row’s Discovered column motivated the subsequent phase.
| Phase | Approach | Verified | Discovered |
|---|---|---|---|
| 1 | Manual compilation of v5.8.1 on Ubuntu 24.04 | Source compiles; components link correctly | Complex dependency chain across 6 components |
| 2 | snap package with core24 base |
snapcraft.yaml produces installable artifact |
Installation alone proves nothing about functionality |
| 3 | LXD test environment; upstream smoke tests migrated | Core rootless and rootful operations | glibc >= 2.34 required; core22 base broadens compatibility |
| 4 | Multi-distro testing (5 distributions) | Cross-distribution compatibility | Distro-specific host dependencies (uidmap, iptables, libgpg-error) |
| 5 | LXD Virtual Machine (VM) validation | Results reproducible outside containers | LXD containers can mask kernel-level behaviours |
| 6 | Real-world deployment with Quadlet services | Day-to-day usability | Systemd generators and socket units missing from snap hooks |
| 7 | Full upstream BATS test suite (785 tests) | Comprehensive functional coverage | ≈96% pass rate (root), 84% (rootless); residual failures categorised |
This was not a linear plan executed to specification. It was an empirical process where each phase generated evidence that redirected effort. The decision to switch from core24 to core22 (phase 3) was driven by test failures on CentOS 9 Stream, which uses glibc 2.34, below core24’s minimum. The expansion to the full upstream test suite (phase 7) was prompted by real-world deployment revealing Systemd integration gaps that the initial smoke tests had not covered. At its peak, the iterative process generated over 18,000 lines of shell across build, test, and orchestration scripts; the current repository represents the stabilised result after refactoring, with some automation moved to a separate repository.
Engineering Challenges
The iterative methodology surfaced four significant engineering challenges, each requiring investigation and a Root Cause and Corrective Action (RCCA) analysis. Two required patches to the upstream Podman source, totalling 72 lines, each with a dedicated security review confirming no new privilege escalation vectors. The RCCA documents linked in the table below are published in the project repository.
The snap bundles its own shared libraries and initially set LD_LIBRARY_PATH globally to locate them. Without scoping, these paths leak into child processes launched by Systemd. A Quadlet service started by Systemd on behalf of the snap could poison the host’s library resolution, potentially affecting unrelated services.
| Problem | LD_LIBRARY_PATH leaks into host Systemd services via child processes, potentially poisoning unrelated library resolution |
| Resolution | Replaced global export with scoped wrappers for conmon and crun that set the variable only for the specific binaries that need it |
| RCCA | RCCA-LIBRARY-POISONING |
Podman healthchecks spawn transient Systemd units via systemd-run, and these units do not inherit the invoking process’s environment. When Systemd executes the healthcheck timer, conmon and crun cannot find the snap-bundled libraries, causing healthchecks to fail silently.
| Problem | systemd-run transient units do not inherit the invoking process’s environment; conmon and crun cannot find snap-bundled libraries |
| Resolution | 36-line upstream patch propagating LD_LIBRARY_PATH into the transient unit’s environment |
| RCCA | HEALTHCHECK_ISSUES |
podman generate systemd (and podman container runlabel) hardcode the path to the podman binary as discovered at runtime. Inside the snap, this resolves to a deeply nested path like /snap/m0x41-podman/x1/usr/bin/podman. Generated units referencing this path would break after snap upgrades change the revision directory.
| Problem | podman generate systemd hardcodes the snap-internal path, which breaks after revision upgrades |
| Resolution | 36-line upstream patch introducing PODMAN_BINARY environment variable override |
| RCCA | RCCA-GENERATE-SYSTEMD |
Snap strict confinement replaces /usr/bin with the base snap’s copy, making the host’s setuid newuidmap and newgidmap binaries invisible. snapcraft strips setuid bits from bundled binaries and squashfs mounts with nosuid, so these cannot be replicated inside the snap. Classic confinement also permits the install hook to register Systemd generators for Quadlet and place the podman shim on PATH, operations that strict confinement does not allow.
| Problem | Strict confinement hides host setuid binaries (newuidmap, newgidmap) required for rootless operation |
| Resolution | Classic confinement adopted; technical justification documented |
| Reference | CLASSIC_CONFINEMENT |
The Testing Infrastructure
The multi-distro testing infrastructure deserves particular attention because it was the enabler for the iterative methodology.
LXD [8] is a system container and virtual machine manager developed by Canonical. Unlike application containers (which run a single process), LXD system containers run a full Linux operating system complete with Systemd, package managers, and user accounts, while sharing the host kernel. This makes them significantly faster to launch and lighter on resources than traditional virtual machines, whilst still providing the isolation needed to test software on different distributions. LXD can also manage full virtual machines when kernel-level isolation is required, using the same command-line interface.
Built on LXD containers and VMs, the test infrastructure provided:
- Parallel execution across Ubuntu 22.04, Ubuntu 24.04, Debian [9] 12, CentOS [10] 9 Stream, and Fedora [11] 43
- Clean-room environments where each test run started from a fresh container, eliminating state leakage between runs
- Automated distro-specific dependency installation,
snapsideloading, and user configuration handled by scripts - Seven tiers of progressively deeper validation, from
snapintegrity checks through full upstream BATS suite execution
Once this infrastructure existed, testing a different snap base (e.g. switching from core24 to core22) required changing a single variable. Reaffirming results on a full VM rather than an LXD container was a configuration flag. The upfront investment in test infrastructure paid for itself repeatedly as the methodology iterated.
Test Coverage
The final test suite comprises seven tiers. Tiers 1-5 form the core regression suite and run across all five target distributions. Tier 6 requires a Virtual Machine. Tier 7 runs the full upstream BATS suite on demand.
| Tier | Scope | Tests | Coverage |
|---|---|---|---|
| 1 | snap integrity |
7 | Binary discovery, component versions, configuration paths |
| 2 | Rootless operations | 8 | Pull, run, build, pod, volume, Domain Name System (DNS) resolution |
| 3 | Rootful operations | 6 | Run, build, pod, volume (as root) |
| 4 | Upstream BATS smoke | 31 | Migrated from Podman upstream repository |
| 5 | Integration | 20+ | Install hooks, Quadlet dry-run, live services, socket activation, healthcheck validation |
| 6 | VM validation | 25+ | Network integrity, library path isolation, Systemd health, reboot survival, snap removal cleanup (VM only) |
| 7 | Full upstream BATS | 785 | Complete upstream test suite in root and rootless modes (on-demand) |
snap integrity (7 tests). Binary discovery, component versions, and configuration paths. Confirms the snap is structurally sound before any functional testing.
Rootless operations (8 tests). Pull, run, build, pod, volume, and Domain Name System (DNS) resolution in rootless mode.
Rootful operations (6 tests). Run, build, pod, and volume operations as root.
Upstream BATS smoke (31 tests). Migrated from the Podman upstream repository. These are the same smoke tests used to validate releases of Podman itself.
Integration (20+ tests). Install hooks, Quadlet dry-run, live rootful and rootless Quadlet services, socket activation, and healthcheck transient unit validation.
VM validation (25+ tests). Requires a Virtual Machine rather than an LXD container, as it validates behaviours that containers can mask, such as reboot survival, network integrity, library path isolation under real Systemd, and snap removal cleanup.
Full upstream BATS (785 tests). The complete upstream suite: 78 files, run in both root and rootless modes on demand.
Of the 785 upstream tests, 180 are skipped by the test harness. These cover pasta networking (the snap bundles slirp4netns instead), SELinux, checkpoint/restore, and SSH/remote, none of which the snap ships. Of the 605 applicable tests in root mode, 559 pass (92%) unmodified. An adapted pass, where the shim respects pre-existing configuration environment variables, recovers 5 more, and the PODMAN_BINARY patch recovers a further 16, bringing the combined total to approximately 580 of 605 (≈96%). In rootless mode, excluding pasta tests, 511 of 611 applicable tests pass (84%). All residual failures are categorised and documented with root cause analyses.
The Role of AI Agents
The methodology described above could not have been executed manually within any reasonable timeframe. The AI agents’ contribution extended far beyond the initial snapcraft.yaml:
- Investigating and patching two upstream Podman issues (healthcheck
LD_LIBRARY_PATHpropagation and binary path resolution), including security reviews for each patch - Designing and implementing scoped wrappers to replace the initial
LD_LIBRARY_PATHapproach after the library path poisoning investigation revealed host-side risks - Migrating and adapting upstream BATS tests to the
snapenvironment, then building a two-pass test harness (upstream followed by adapted shim) - Writing distro-specific dependency detection in the wrapper script
- Debugging cross-distribution failures involving library paths, capability flags, and namespace configurations
- Building the multi-distro test orchestration scripts, which exceeded 18,000 lines of shell at peak before being refactored into the current ≈3,700-line core, with some automation moved to a separate repository
- Iterating on install and remove hooks as each phase revealed new requirements
- Categorising and documenting all residual upstream test failures with root cause analyses
- Implementing supply-chain integrity verification (SHA256, cosign, SLSA provenance) in the GitHub Actions release workflow
The human contribution was architectural: defining the verification methodology, deciding when to switch snap bases, choosing which upstream tests to prioritise, directing the library path poisoning investigation, and determining what constituted working at each phase.
The AI agents provided the implementation bandwidth to execute that methodology across multiple environments simultaneously. This division of human architecture and agentic execution is the model that Curio Data Pro applies in client engagements and the reason this package exists as a demonstration of that approach.
Components
The snap bundles the following runtime dependencies into a single installable artifact:
| Component | Version | Source | Purpose |
|---|---|---|---|
| Podman | 5.8.1 | Built from source (with 2 patches) | Container runtime and CLI |
| crun | 1.19.1 | Built from source | OCI runtime |
| conmon | 2.0.26 | Built from source | Container monitor |
| netavark | 1.14.1 | Pre-built binary | Container networking |
| aardvark-dns | 1.14.0 | Pre-built binary | Container DNS resolution |
| slirp4netns | 1.0.1 | Ubuntu 22.04 package | Rootless networking |
| fuse-overlayfs | 1.7.1 | Ubuntu 22.04 package | Rootless storage driver |
| catatonit | 0.1.7 | Ubuntu 22.04 package | Minimal init process |
Three of the eight components are compiled from source: Podman, crun, and conmon. conmon was upgraded from the Ubuntu 22.04 package (v2.0.25) to a source build of v2.0.26 after root cause analysis identified a stderr handling bug in v2.0.25 that caused spurious test failures. Two patches totalling 72 lines are applied to the Podman source at build time: one to propagate LD_LIBRARY_PATH into healthcheck transient units, and one to allow the binary path in generated Systemd units to be overridden via a PODMAN_BINARY environment variable.
The snap uses core22 (Ubuntu 22.04, glibc 2.35) as its base, providing compatibility with any distribution running glibc 2.34 or later. Classic confinement is required for access to host setuid binaries. A wrapper script detects missing host dependencies (uidmap, dbus-user-session) on first run and provides distro-specific installation instructions.
Limitations
The verification-first methodology is effective at surfacing what works and what does not. The same rigour that achieved a ≈96% pass rate against 605 applicable upstream tests also documented the boundaries of the current approach:
- Rootless networking uses
slirp4netnsrather thanpasta(part of thepasstproject). Native Podman v5.x defaults topastafor faster rootless networking with features like port ranges in--publishand improved IPv6 handling. Thesnapusesslirp4netnsbecausepastais not available on thecore22(Ubuntu 22.04) base. Rootless networking works but may be slower than a native install, and--network pastawill fail. This accounts for the majority of the 180 skipped upstream tests. - Component versions are constrained by the base. The three Ubuntu 22.04 packages (
slirp4netns1.0.1,fuse-overlayfs1.7.1,catatonit0.1.7) are older than current upstream releases. This is inherent to using a stable base for broad compatibility, a trade-off the verification methodology made visible.conmonwas moved from a staged package to a source build (v2.0.26) to resolve a specific bug, demonstrating how the iterative methodology drives component decisions. - The
setuidbinariesnewuidmapandnewgidmap(from theuidmappackage) anddbus-user-sessioncannot be bundled in asnapand must be installed on the host for rootless operation. - The current package targets x86_64 only. Advanced RISC Machine (ARM) builds are not yet available.
Distribution
The snap is not available on the Snap Store. A classic confinement request [12] was submitted and denied. The decision was not made on technical grounds but because the Snap Store’s publisher vetting process requires classic confinement packages to come from the official upstream project or a trusted group such as the Snapcrafters community. The reviewer acknowledged the engineering work but explained that publisher credentials take precedence over technical merit in classic confinement decisions.
An adoption request [13] has been submitted to the Snapcrafters community, and a separate feedback thread [14] has been posted seeking community input on the package’s engineering and test methodology. Both remain open at the time of writing.
In the interim, the snap is distributed via GitHub Releases and must be sideloaded with --dangerous --classic. Each release includes supply-chain integrity verification: SHA256 checksums, cosign keyless signatures using GitHub’s OIDC identity, and SLSA provenance attestations. Together, these allow users to verify that the artifact they download was built by the repository’s GitHub Actions workflow, from a specific commit, without tampering.
This outcome illustrates a recurring pattern in open source distribution: the engineering problem and the governance problem operate on different axes. The verification methodology described in this post demonstrates that the package works; the Snap Store’s vetting process requires that the publisher is trusted. Neither substitutes for the other.
Conclusion
The core packaging that defines the Podman v5.8.1 snap is approximately 700 lines. The test automation and documentation that prove it works across five Linux distributions exceed 7,100 lines. The iterative process that produced them generated over 18,000 lines of shell before stabilising. That ratio of creation to verification reflects a reality of Systems Engineering that the current narrative around AI-assisted development tends to overlook.
The verification-first methodology described in this post is not specific to snap packaging. It applies wherever the gap between building an artifact and proving it works is large: infrastructure deployments, cross-platform libraries, compliance-driven systems, migration projects. The pattern is consistent: a Subject Matter Expert defines the methodology and makes architectural decisions, while AI agents provide the implementation bandwidth to execute that methodology across environments, test tiers, and edge cases that would be impractical to cover manually.
The distribution outcome adds a further dimension. The engineering rigour is demonstrable: 76 commits, an estimated 10,000+ test executions, a ≈96% upstream test pass rate, two production deployments, supply-chain integrity verification, and documented root cause analyses for every residual failure. But the Snap Store’s publisher vetting process operates on a different axis entirely. Technical verification and institutional trust are complementary requirements, not substitutes.
The m0x41-podman package is one outcome of this approach, a useful tool that brings rootless pod orchestration and Quadlet deployment to distributions limited to Podman v4. But it is also evidence that at Curio Data Pro we apply agentic workflows in practice, not (just) as code generation, but as a force multiplier for engineering rigour.
Version History
2026-04-10- Initial release.
Attribution
Podman is developed by Red Hat and the Containers open source community. crun is developed by Giuseppe Scrivano. netavark and aardvark-dns are part of the Containers project. We gratefully acknowledge these projects and their contributors.
Images used in this post have been generated using multiple Machine Learning (or Artificial Intelligence) models and subsequently modified by the author.
Where ever possible these have been identified with the following symbol:
The text has been reviewed using Large Language Models for spelling, grammar, and word choice; however, the content, analysis, and conclusions are entirely the author’s own.
References
Citation
@online{miah2026,
author = {Miah, Ashraf},
title = {Building {Trust,} {Not} {Just} {Builds:} {Verification-First}
{Engineering} with {AI} {Agents}},
date = {2026-04-10},
url = {https://blog.curiodata.pro/posts/21-podman-package/},
langid = {en}
}