# netlinux-ai: Package Build & Deployment Infrastructure

This document describes how packages.netlinux.co.uk works — how packages are
built, signed, deployed, and served. It is intended for both humans and agentic
systems that need to add new packages or maintain the existing pipeline.

## Overview

packages.netlinux.co.uk is a Debian APT repository serving custom-built amd64
packages. Source code lives in GitHub repos under the `netlinux-ai` organisation.
Packages are built automatically by GitHub Actions on every push to `main`,
published as GitHub Releases, then pulled into the server's reprepro repository
via a webhook.

### Suites

The repository serves two suites with distinct target distributions:

| Codename | Target | apt line |
|----------|--------|----------|
| `stable` | Debian 12 (bookworm) — used by NetLinux Desktop/Server | `deb https://packages.netlinux.co.uk/debian stable main` |
| `resolute` | Ubuntu 26.04 LTS ("Resolute Raccoon") | `deb https://packages.netlinux.co.uk/debian resolute main` |

Packages reach these suites by two complementary pipelines:

- **GitHub Actions (per-repo, push-triggered)** — documented below. Publishes
  to `stable` only.
- **Nightly dual-build (dev2, 03:00 UTC)** — clones every repo, runs
  `brain-code-agent` inside a `debian:bookworm` container and an `ubuntu:26.04`
  container, publishes each resulting `.deb` to the matching suite. This is
  the only path that populates `resolute`. See the [Nightly dual-build
  pipeline](#nightly-dual-build-pipeline) section below.

```
Developer pushes to main
        |
        v
GitHub Actions (ubuntu-24.04)
  1. Build from source
  2. checkinstall → .deb
  3. Repack zstd → xz
  4. Create GitHub Release with .deb asset
  5. POST webhook with HMAC-SHA256 signature
        |
        v
packages.netlinux.co.uk
  Apache → ProxyPass /webhook/ → listener.py (port 9090)
        |
        v
update-repo.sh
  1. Download .deb from GitHub Release
  2. Repack if zstd (safety net)
  3. reprepro includedeb → signs and indexes
  4. Update HTML index pages
        |
        v
Available via: sudo apt install <package>
```

---

## Server

- **Host:** packages.netlinux.co.uk (DigitalOcean)
- **SSH:** `ssh root@packages.netlinux.co.uk`
- **OS:** Debian (amd64)
- **Web server:** Apache2 with Let's Encrypt TLS
- **Repository tool:** reprepro

### Apache vhost (relevant section)

```apache
<VirtualHost *:80 *:443>
    ServerName packages.netlinux.co.uk
    ServerAlias packages.netlinux.org.uk
    DocumentRoot /Sites/netlinux/packages

    <Directory "/Sites/netlinux/packages">
        Options +Indexes +FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>

    ProxyPass /webhook/ http://127.0.0.1:9090/
    ProxyPassReverse /webhook/ http://127.0.0.1:9090/

    SSLCertificateFile /etc/letsencrypt/live/packages.netlinux.co.uk/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/packages.netlinux.co.uk/privkey.pem
</VirtualHost>
```

### Directory layout

```
/Sites/netlinux/packages/
├── index.html                          # Landing page with package table
├── netlinux-ai.md                      # This document
├── webhook/
│   ├── listener.py                     # Python webhook HTTP server
│   └── update-repo.sh                  # Downloads .deb, adds to reprepro
└── debian/                             # reprepro repository root
    ├── conf/
    │   ├── distributions              # reprepro config
    │   └── options
    ├── repo-key.gpg                    # Exported GPG public key
    ├── db/                             # reprepro database
    ├── dists/
    │   ├── stable/                    # Debian bookworm-targeted indexes
    │   │   ├── Release / Release.gpg / InRelease
    │   │   └── main/binary-amd64/
    │   │       └── Packages / Packages.gz
    │   └── resolute/                  # Ubuntu 26.04-targeted indexes
    │       ├── Release / Release.gpg / InRelease
    │       └── main/binary-amd64/
    │           └── Packages / Packages.gz
    └── pool/                          # shared by both suites
        └── main/
            ├── a/applesmc-next-dkms/
            ├── b/battery-tray/
            ├── b/blender/
            ├── c/chromium-browser-stable/
            ├── c/claude-code/
            ├── c/curaengine/
            ├── d/dovecot/
            ├── f/fstl/
            ├── g/gimp/
            ├── k/kdenlive/
            ├── k/kio-extras/
            ├── k/kio-rsync/
            ├── k/konqueror/
            ├── l/linux-upstream/
            ├── o/openbox/
            ├── p/pavucontrol/
            ├── q/qpwgraph/
            ├── r/rsync/
            ├── s/simplescreenrecorder/
            ├── s/snappymail/
            ├── t/tightvnc/
            ├── x/x11vnc/
            ├── x/xfce4-panel/
            ├── x/xfce4-power-manager/
            ├── x/xfwm4/
            └── x/xterm/
```

---

## Reprepro configuration

### `/Sites/netlinux/packages/debian/conf/distributions`

```
Origin: NetLinux
Label: NetLinux
Codename: stable
Architectures: amd64
Components: main
Description: NetLinux custom packages repository
SignWith: default

Origin: NetLinux
Label: NetLinux
Codename: resolute
Suite: resolute
Architectures: amd64
Components: main
Description: NetLinux packages for Ubuntu 26.04 LTS (Resolute Raccoon)
SignWith: default
```

Both suites share the same `pool/` directory on disk. Because bookworm and
resolute builds use different Debian version suffixes (`~bookworm1` /
`~resolute1`) the `.deb` filenames differ, so two targeted builds of the same
upstream can coexist in the pool without collision.

### `/Sites/netlinux/packages/debian/conf/options`

```
verbose
basedir /Sites/netlinux/packages/debian
```

### GPG signing key

```
pub   rsa4096/3F9A57A88A3D96CA 2026-02-17 [SCEA]
      EA339EE150E13D30D568F9353F9A57A88A3D96CA
uid                 [ultimate] NetLinux Packages <packages@netlinux.org.uk>
```

The public key is exported to `/Sites/netlinux/packages/debian/repo-key.gpg` and
served at `https://packages.netlinux.co.uk/debian/repo-key.gpg`. reprepro
automatically signs Release files on every `includedeb` using `SignWith: default`.

---

## NetLinux Desktop

NetLinux Desktop is a Debian bookworm-based live ISO that bundles all NetLinux
packages into a ready-to-use desktop distribution.

- **Source:** [netlinux-ai/netlinux-desktop](https://github.com/netlinux-ai/netlinux-desktop)
- **Release page:** [packages.netlinux.co.uk/netlinux.html](https://packages.netlinux.co.uk/netlinux.html)
- **Current release:** v0.1 (25 February 2026)
- **Base:** Debian 12 (bookworm), Xfce 4, custom 6.19 kernel
- **Installer:** Calamares graphical installer
- **Build tool:** Debian live-build
- **ISO output:** `netlinux-desktop-amd64.hybrid.iso` (~1.3 GB)
- **Meta-package:** `netlinux-desktop` — installs the full desktop stack via apt

The ISO is built with `live-build` using the scripts in the `netlinux-desktop`
repo. It pulls packages from both the Debian bookworm repos and the NetLinux
APT repository (`packages.netlinux.co.uk/debian`). APT pinning ensures
NetLinux packages take priority where available.

Build: `sudo ./build.sh` (requires Debian/Ubuntu host, ~20 GB free space).

---

## NetLinux Server

NetLinux Server is a minimal headless Debian bookworm-based live ISO with
Docker, firewall, monitoring, and SSH pre-configured for server deployments.

- **Source:** [netlinux-ai/netlinux-server](https://github.com/netlinux-ai/netlinux-server)
- **Release page:** [packages.netlinux.co.uk/netlinux.html](https://packages.netlinux.co.uk/netlinux.html)
- **Base:** Debian 12 (bookworm), headless (no desktop environment), custom 6.19 kernel
- **Services:** OpenSSH, Docker, UFW firewall, fail2ban, unattended-upgrades
- **Build tool:** Debian live-build
- **ISO output:** `netlinux-server-amd64.hybrid.iso` (~532 MB)
- **Meta-package:** `netlinux-server` — installs the full server stack via apt

The ISO is built with `live-build` using the scripts in the `netlinux-server`
repo. No desktop environment, no GUI — just a hardened server baseline with
SSH, Docker, and firewall enabled out of the box.

Build: `sudo ./build.sh` (requires Debian/Ubuntu host, ~15 GB free space).

---

## Current packages

| Package | GitHub Repo | Pool Dir | Description |
|---------|-------------|----------|-------------|
| applesmc-next-dkms | netlinux-ai/applesmc-next | a/applesmc-next-dkms | DKMS kernel module for Apple SMC battery charge threshold control |
| netlinux-desktop | netlinux-ai/netlinux-desktop | n/netlinux-desktop | Meta-package: installs the full NetLinux desktop stack (Xfce, Chromium, GIMP, Blender, Claude Code, etc.) |
| netlinux-server | netlinux-ai/netlinux-server | n/netlinux-server | Meta-package: installs the full NetLinux server stack (SSH, Docker, UFW, fail2ban, etc.) |
| battery-tray | netlinux-ai/battery-tray | b/battery-tray | GTK3 systray application for battery status monitoring |
| blender | netlinux-ai/blender | b/blender | Blender 5.0 3D creation suite |
| chromium-browser-stable | netlinux-ai/chromium-browser | c/chromium-browser-stable | Chromium web browser with full media codec support |
| claude-code | netlinux-ai/claude-code | c/claude-code | Anthropic Claude Code CLI (repackaged with source-built components) |
| dovecot | netlinux-ai/dovecot | d/dovecot | High-performance IMAP server |
| curaengine | netlinux-ai/curaengine | c/curaengine | CuraEngine 3D printing slicer (STL/3MF to G-code) |
| fstl | netlinux-ai/fstl | f/fstl | Fast STL file viewer for 3D model inspection |
| gimp | netlinux-ai/gimp | g/gimp | GIMP 3.0 image editor (bundled babl/GEGL) |
| kdenlive | netlinux-ai/kdenlive | k/kdenlive | Kdenlive video editor with bundled MLT |
| kio-extras | netlinux-ai/kio-extras | k/kio-extras | KDE KIO workers (sftp, smb, mtp, rsync, etc.) |
| kio-rsync | netlinux-ai/kio-rsync | k/kio-rsync | KIO worker for rsync:// in KDE |
| konqueror | netlinux-ai/konqueror | k/konqueror | KDE web browser and file manager |
| linux-image-* | netlinux-ai/linux | l/linux-upstream | Custom upstream kernel |
| linux-headers-* | netlinux-ai/linux | l/linux-upstream | Kernel headers |
| openbox | netlinux-ai/openbox | o/openbox | Lightweight standards-compliant window manager |
| pavucontrol | netlinux-ai/pavucontrol | p/pavucontrol | PulseAudio Volume Control |
| qpwgraph | netlinux-ai/qpwgraph | q/qpwgraph | PipeWire graph manager |
| rsync | netlinux-ai/rsync | r/rsync | Fast file synchronisation utility (3.4.x) |
| snappymail | netlinux-ai/snappymail | s/snappymail | Lightweight webmail client with PGP support |
| simplescreenrecorder | netlinux-ai/ssr | s/simplescreenrecorder | Screen recorder |
| tightvnc | netlinux-ai/tightvnc | t/tightvnc | TightVNC server and viewer |
| x11vnc | netlinux-ai/x11vnc | x/x11vnc | VNC server for real X displays (robustness patches) |
| xfce4-panel | netlinux-ai/xfce4-panel | x/xfce4-panel | Xfce4 desktop panel |
| xfce4-power-manager | netlinux-ai/xfce4-power-manager | x/xfce4-power-manager | Xfce power manager with battery charge threshold control |
| xfwm4 | netlinux-ai/xfwm4 | x/xfwm4 | XFCE window manager |
| xterm | netlinux-ai/xterm | x/xterm | Standard X11 terminal emulator |
---

## GitHub Actions workflow

Every repo under `netlinux-ai` that produces a `.deb` has a workflow at
`.github/workflows/release-deb.yml`. The structure is the same across all repos,
with build steps varying per project.

### Template workflow

```yaml
name: Build .deb package

on:
  push:
    branches: [main]

permissions:
  contents: write

jobs:
  build:
    runs-on: ubuntu-24.04

    steps:
      - uses: actions/checkout@v4

      - name: Install build dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y <project-specific-deps>

      - name: Build
        run: |
          # Project-specific build commands
          # e.g. cmake -B build -DCMAKE_INSTALL_PREFIX=/usr && cmake --build build

      - name: Determine version
        id: version
        run: |
          rev=${{ github.run_number }}
          echo "version=<base>-${rev}netlinux1" >> "$GITHUB_OUTPUT"
          echo "tag=v<base>-${rev}netlinux1" >> "$GITHUB_OUTPUT"

      - name: Package with checkinstall
        run: |
          sudo apt-get install -y checkinstall
          sudo checkinstall --default \
            --pkgname=<package-name> \
            --pkgversion="${{ steps.version.outputs.version }}" \
            --maintainer="graham@netlinux.co.uk" \
            --requires="<runtime-deps>" \
            --pakdir=. \
            --install=no \
            --fstrans=no \
            <install-command>

      - name: Repack .deb for reprepro compatibility
        run: |
          DEB=$(ls <package-name>_*.deb)
          sudo chmod 666 "$DEB"
          mkdir repack && cd repack
          ar x "../$DEB"
          for f in *.zst; do
            [ -f "$f" ] || continue
            zstd -d "$f"
            xz "${f%.zst}"
            rm "$f"
          done
          rm "../$DEB"
          ar rcs "../$DEB" debian-binary control.tar.xz data.tar.xz

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          tag_name: ${{ steps.version.outputs.tag }}
          name: <package-name> ${{ steps.version.outputs.version }}
          body: |
            Automated build from commit ${{ github.sha }}
            Install: `sudo dpkg -i <package-name>_*.deb`
          files: <package-name>_*.deb
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Notify package repository
        run: |
          sleep 5
          SIGNATURE=$(echo -n '{"repo":"netlinux-ai/<repo>","tag":"${{ steps.version.outputs.tag }}"}' \
            | openssl dgst -sha256 -hmac "${{ secrets.WEBHOOK_SECRET }}" -binary \
            | xxd -p -c 256)
          curl -sf -X POST \
            -H "Content-Type: application/json" \
            -H "X-Webhook-Signature: sha256=${SIGNATURE}" \
            -d '{"repo":"netlinux-ai/<repo>","tag":"${{ steps.version.outputs.tag }}"}' \
            https://packages.netlinux.co.uk/webhook/update-repo
```

### Key details

- **Trigger:** push to `main` branch only
- **Runner:** `ubuntu-24.04` (provides standard build toolchain)
- **Packaging:** `checkinstall` wraps any `make install`-style command into a `.deb`
- **Repack step:** Ubuntu 24.04's checkinstall produces zstd-compressed `.deb` files, but the server's reprepro only supports xz. The repack step converts `control.tar.zst` and `data.tar.zst` to `.tar.xz`.
- **Version scheme:** `<upstream-version>-<run_number>netlinux1`. checkinstall appends `-1` as a release suffix, so the final filename becomes e.g. `kio-rsync_1.0-2netlinux1-1_amd64.deb`.
- **GitHub Release:** created by `softprops/action-gh-release@v2` with the `.deb` as an asset
- **Webhook:** HMAC-SHA256 signed POST to `https://packages.netlinux.co.uk/webhook/update-repo`

### Required GitHub secrets (per repo)

| Secret | Purpose |
|--------|---------|
| `GITHUB_TOKEN` | Automatic — used by `action-gh-release` to create releases |
| `WEBHOOK_SECRET` | Shared HMAC secret for authenticating webhook calls |

Set the webhook secret on each repo:
```bash
gh secret set WEBHOOK_SECRET --repo netlinux-ai/<repo>
```

The same shared secret is used across all repos. Its value is configured in the
systemd service on the server (see below).

---

## Webhook system

### Systemd service

```ini
# /etc/systemd/system/repo-webhook.service
[Unit]
Description=Package repository webhook listener
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /Sites/netlinux/packages/webhook/listener.py
Environment=WEBHOOK_SECRET=<shared-secret>
Environment=PORT=9090
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Manage with: `systemctl {start,stop,restart,status} repo-webhook`

### listener.py

A minimal Python HTTP server on `127.0.0.1:9090` (not exposed directly —
Apache proxies `/webhook/` to it). Handles:

- **POST /update-repo** — validates HMAC-SHA256 signature, then runs
  `update-repo.sh <tag> <repo>` as a subprocess with 600s timeout
- **GET /health** — returns 200 OK (for monitoring)

The signature is verified by computing `HMAC-SHA256(request_body, WEBHOOK_SECRET)`
and comparing against the `X-Webhook-Signature: sha256=<hex>` header.

Logs to `/var/log/repo-webhook.log`.

### update-repo.sh

The main deployment script. Called as: `update-repo.sh <tag> <github-repo>`

**What it does:**

1. Maps `github-repo` to `PKG_NAME` and `POOL_DIR` via a case statement
2. Fetches the release metadata from GitHub API to find the `.deb` asset URL
3. Downloads the `.deb`
4. If the `.deb` contains zstd-compressed members, repacks to xz (safety net
   in case the GitHub Actions repack step was skipped)
5. Removes any old version: `reprepro remove stable <pkg>`
6. Adds the new version: `reprepro includedeb stable <deb>`
7. Looks up the actual filename in the pool (handles checkinstall's `-1` suffix)
8. Updates the pool's `index.html` with the new version/filename
9. Updates the main `/Sites/netlinux/packages/index.html`

**The repo whitelist** (case statement) must be updated when adding a new package:

```bash
case "$GITHUB_REPO" in
    netlinux-ai/dovecot)      PKG_NAME="dovecot";             POOL_DIR="d/dovecot" ;;
    netlinux-ai/snappymail)   PKG_NAME="snappymail";          POOL_DIR="s/snappymail" ;;
    netlinux-ai/ssr|"")        PKG_NAME="simplescreenrecorder"; POOL_DIR="s/simplescreenrecorder" ;;
    netlinux-ai/qpwgraph)      PKG_NAME="qpwgraph";             POOL_DIR="q/qpwgraph" ;;
    netlinux-ai/tightvnc)      PKG_NAME="tightvnc";             POOL_DIR="t/tightvnc" ;;
    netlinux-ai/pavucontrol)   PKG_NAME="pavucontrol";          POOL_DIR="p/pavucontrol" ;;
    netlinux-ai/xfce4-panel)   PKG_NAME="xfce4-panel";          POOL_DIR="x/xfce4-panel" ;;
    netlinux-ai/xfce4-power-manager) PKG_NAME="xfce4-power-manager"; POOL_DIR="x/xfce4-power-manager" ;;
    netlinux-ai/kio-rsync)     PKG_NAME="kio-rsync";            POOL_DIR="k/kio-rsync" ;;
    netlinux-ai/rsync)         PKG_NAME="rsync";                POOL_DIR="r/rsync" ;;
    netlinux-ai/kio-extras)    PKG_NAME="kio-extras";           POOL_DIR="k/kio-extras" ;;
    netlinux-ai/konqueror)     PKG_NAME="konqueror";            POOL_DIR="k/konqueror" ;;
    netlinux-ai/curaengine)    PKG_NAME="curaengine";           POOL_DIR="c/curaengine" ;;
    netlinux-ai/gimp)          PKG_NAME="gimp";                 POOL_DIR="g/gimp" ;;
    netlinux-ai/chromium-browser) PKG_NAME="chromium-browser-stable"; POOL_DIR="c/chromium-browser-stable" ;;
    netlinux-ai/xfwm4)        PKG_NAME="xfwm4";               POOL_DIR="x/xfwm4" ;;
    netlinux-ai/openbox)       PKG_NAME="openbox";              POOL_DIR="o/openbox" ;;
    netlinux-ai/fstl)          PKG_NAME="fstl";                 POOL_DIR="f/fstl" ;;
    netlinux-ai/xterm)         PKG_NAME="xterm";                POOL_DIR="x/xterm" ;;
    netlinux-ai/x11vnc)        PKG_NAME="x11vnc";               POOL_DIR="x/x11vnc" ;;
    netlinux-ai/battery-tray)  PKG_NAME="battery-tray";         POOL_DIR="b/battery-tray" ;;
    netlinux-ai/blender)       PKG_NAME="blender";              POOL_DIR="b/blender" ;;
    netlinux-ai/kdenlive)      PKG_NAME="kdenlive";             POOL_DIR="k/kdenlive" ;;
    netlinux-ai/claude-code)   PKG_NAME="claude-code";          POOL_DIR="c/claude-code" ;;
    netlinux-ai/applesmc-next) PKG_NAME="applesmc-next-dkms";   POOL_DIR="a/applesmc-next-dkms" ;;
    netlinux-ai/linux)         IS_KERNEL=true;                   POOL_DIR="l/linux-upstream" ;;
    *)                         log "ERROR: Unknown repo"; exit 1 ;;
esac
```

Kernel packages (`netlinux-ai/linux`) are handled specially because they produce
multiple `.deb` files (linux-image, linux-headers, linux-libc-dev) with version
numbers embedded in the package name.

> **Note on suites:** `update-repo.sh` currently publishes every webhook-triggered
> build into the `stable` suite only. `resolute` is populated by the nightly
> pipeline (see below). If you need a just-pushed package in `resolute` before
> the next nightly, either re-run the nightly manually on dev2 or run
> `reprepro -b /Sites/netlinux/packages/debian copy resolute stable <pkg>`
> on the server to copy the existing bookworm-compat .deb into `resolute` as a
> temporary stand-in.

---

## Nightly dual-build pipeline

A second pipeline runs on dev2 (147.182.205.211) at 03:00 UTC via cron and
is the only source of `resolute` suite builds. It lives at
`/home/graham/nightly/` (see `README.md` there for the full runbook).

### Flow

```
Developer pushes to main
         |
         ├──────> GitHub Actions (as documented above) → stable suite
         |
         |
dev2 cron @ 03:00 UTC
         |
         v
nightly-packages.sh
         |
         ├──> build-images.sh      (refresh netlinux-build:{bookworm,resolute}
         |                           Docker images if >14 days old)
         |
         v
  For each repo in repos.conf (if GitHub has new commits):
         |
         ├──> Clone once into build/<repo>/src/
         |
         ├──> docker run --rm --network host \
         |        -v src:/src -v output:/output \
         |        netlinux-build:bookworm \
         |        brain-code-agent --prompt "..."
         |                → .deb tagged ~bookworm1
         |                → publish_deb to suite=stable
         |
         └──> docker run --rm --network host \
                  -v src:/src -v output:/output \
                  netlinux-build:resolute \
                  brain-code-agent --prompt "..."
                          → .deb tagged ~resolute1
                          → publish_deb to suite=resolute
```

### Key properties

- **Build-env isolation:** each target builds inside its own Docker container,
  so `checkinstall`/`apt` link against the correct distro's libraries. The
  container base images are `debian:bookworm` and `ubuntu:26.04`.
- **Version tagging:** the nightly agent is instructed to emit versions
  suffixed `-<build_num>netlinux1~<target>1` so both `.deb`s can coexist in
  the shared pool and clients on each suite resolve the correct artefact.
- **Last-commit advancement:** a repo's stored SHA is only updated when at
  least one target publishes successfully. A both-targets failure keeps the
  repo in the retry set for the next nightly run.
- **Agent:** `brain-code-agent` is a bash-based LLM agent backed by a
  locally hosted Qwen2.5-Coder-14B via llama-server (SSH-tunnelled to
  `localhost:8090`). The agent reads each repo's
  `.github/workflows/release-deb.yml` for build instructions, runs them
  inside the container, then repacks zstd→xz.
- **Smoke tests:** after all builds, `test-packages.sh` boots three VMs
  (NetLinux Server ISO, Debian bookworm, Ubuntu Resolute) and runs per-package
  smoke tests. The Resolute VM points at the `resolute` suite; the other two
  point at `stable`.

### Configuring the server side (one-time)

On dev2 (in `/home/graham/nightly`):
```bash
./add-resolute-suite.sh      # ssh's to packages server, appends resolute
                             # codename to conf/distributions, reprepro export
./build-images.sh            # docker builds netlinux-build:{bookworm,resolute}
./prepare-resolute-vm.sh     # downloads + customises the test VM image
```

After these, `./nightly-packages.sh` dual-builds and dual-publishes.

---

## Adding a new package

### Step-by-step checklist

1. **Create the GitHub repo** under `netlinux-ai`:
   ```bash
   gh repo create netlinux-ai/<name> --public --description "<description>"
   ```

2. **Add the source code** and a `.github/workflows/release-deb.yml` following
   the template above. Customise:
   - Build dependencies (`apt-get install`)
   - Build commands
   - `--pkgname`, `--requires`, and install command in the checkinstall step
   - Version base string
   - Repo name in the webhook payload

3. **Set the webhook secret** on the repo:
   ```bash
   gh secret set WEBHOOK_SECRET --repo netlinux-ai/<name>
   ```
   Use the same shared secret as other repos.

4. **On the server**, add the new package to `update-repo.sh`:
   ```bash
   ssh root@packages.netlinux.co.uk
   vi /Sites/netlinux/packages/webhook/update-repo.sh
   ```
   Add a new case entry:
   ```bash
   netlinux-ai/<name>)
       PKG_NAME="<package-name>"
       POOL_DIR="<first-letter>/<package-name>"
       ;;
   ```

5. **Create the pool directory and index page** on the server:
   ```bash
   mkdir -p /Sites/netlinux/packages/debian/pool/main/<first-letter>/<package-name>/
   ```
   Create an `index.html` in that directory. Every pool index page **must**
   include the following sections:

   - **Breadcrumb** — link back to the main packages page
   - **Package name and description** — what this package is
   - **"Why this build?" box** — a green highlighted box explaining why someone
     should use the NetLinux version instead of the distro package. This is the
     most important section. It should include:
     - What version the distro ships and what version NetLinux provides
     - Specific improvements: security fixes, new features, bug fixes, or
       patches unique to the NetLinux build
     - For entirely new packages (no distro equivalent), explain what gap this
       fills and what capabilities it provides
   - **Comparison table** (where applicable) — a side-by-side feature comparison
     between the distro version and the NetLinux version
   - **Package details table** — version, architecture, component, download link,
     upstream/source links
   - **Install instructions** — `sudo apt install` command and link to the main
     page for repository setup

   Use the template below. See any existing pool page (e.g.
   [rsync](https://packages.netlinux.co.uk/debian/pool/main/r/rsync/)) for a
   complete example.

   ```html
   <!DOCTYPE html>
   <html lang="en">
   <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
   <title><package-name> - NetLinux Packages</title>
   <style>
     body {
       font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto,
         "Helvetica Neue", Arial, sans-serif;
       max-width: 720px; margin: 2rem auto; padding: 0 1rem;
       line-height: 1.6; color: #222; background: #fafafa;
     }
     h1 { margin-bottom: 0.25rem; }
     h2 { margin-top: 2rem; border-bottom: 1px solid #ddd;
       padding-bottom: 0.25rem; }
     table { border-collapse: collapse; width: 100%; margin: 1rem 0; }
     th, td { text-align: left; padding: 0.5rem 0.75rem;
       border: 1px solid #ddd; }
     th { background: #f0f0f0; width: 10rem; }
     code, pre { background: #f0f0f0; border-radius: 3px; }
     code { padding: 0.15em 0.3em; font-size: 0.9em; }
     pre { padding: 1rem; overflow-x: auto; }
     pre code { padding: 0; background: none; }
     a { color: #0366d6; }
     .breadcrumb { font-size: 0.9em; color: #666; margin-bottom: 1rem; }
     .breadcrumb a { color: #666; }
     .why { background: #e8f5e9; border: 1px solid #a5d6a7;
       border-radius: 6px; padding: 1rem 1.25rem; margin: 1.5rem 0; }
     .why h3 { margin: 0 0 0.5rem 0; color: #2e7d32; font-size: 1rem; }
     .why ul { margin: 0.5rem 0 0 1.25rem; padding: 0; }
     .why li { margin: 0.3rem 0; }
     .compare { font-size: 0.9em; }
     .compare td:first-child { font-weight: 600; }
   </style>
   </head>
   <body>
   <div class="breadcrumb">
     <a href="https://packages.netlinux.co.uk/">NetLinux Packages</a>
       / <package-name>
   </div>
   <h1><package-name></h1>
   <p>Short description of the package.</p>

   <div class="why">
   <h3>Why this build?</h3>
   <p>Ubuntu 24.04 ships <distro-version>. This build provides:</p>
   <ul>
     <li><strong>Feature or fix</strong> — description of what it does
       and why it matters</li>
     <li><strong>Another improvement</strong> — be specific: version
       numbers, CVE IDs, measurable differences</li>
   </ul>
   </div>

   <!-- Optional: include when there are clear feature differences -->
   <h2>Compared to the distro version</h2>
   <table class="compare">
     <tr><td>Feature</td><td>Ubuntu 24.04</td><td>NetLinux</td></tr>
     <tr><td>Example feature</td><td>No</td><td>Yes</td></tr>
   </table>

   <h2>Package details</h2>
   <table>
     <tr><th>Version</th><td>VERSION</td></tr>
     <tr><th>Architecture</th><td>amd64</td></tr>
     <tr><th>Component</th><td>main</td></tr>
     <tr><th>Download</th><td><a href="<package-name>_VERSION_amd64.deb">
       <package-name>_VERSION_amd64.deb</a></td></tr>
     <tr><th>Upstream</th><td><a href="https://github.com/netlinux-ai/<repo>">
       github.com/netlinux-ai/<repo></a></td></tr>
   </table>
   <h2>Install</h2>
   <pre><code>sudo apt install <package-name></code></pre>
   <p>See <a href="https://packages.netlinux.co.uk/">the main page</a>
     for repository setup instructions.</p>
   </body>
   </html>
   ```
   The version and download filename will be updated automatically by
   `update-repo.sh` on the first successful build.

6. **Add a row to the main index.html** at `/Sites/netlinux/packages/index.html`.

   The main index page has the following structure (in order):
   - **Title and description** — "NetLinux Packages" heading with tagline
   - **"Why NetLinux?" box** — green `.why` box explaining the value of the
     repository: upstream tracking, security-first approach, restored packages,
     and the AI-assisted development model with security/stability guardrails
   - **Quick setup** — `curl` + `apt` commands to add the repo and install
   - **Available packages table** — one row per package with columns: Package
     (linking to pool index page), Version (linking to `.deb`), Description,
     Source (linking to GitHub repo)
   - **Repository details** — URL, architectures, signing key, browse links
   - **Disclaimer note** — packages provided as-is, not affiliated with Debian

   Add a new row to the "Available packages" table:
   ```html
   <tr>
     <td><a href="/debian/pool/main/<x>/<name>/"><strong><name></strong></a></td>
     <td><a href="/debian/pool/main/<x>/<name>/<name>_VERSION_amd64.deb">VERSION</a></td>
     <td>Description</td>
     <td><a href="https://github.com/netlinux-ai/<repo>">netlinux-ai/<repo></a></td>
   </tr>
   ```
   (The version and filename will be updated automatically by `update-repo.sh`
   on the first successful build.)

7. **Push to main** — the GitHub Action will build, release, and trigger the
   webhook. The package will appear in the repo within seconds.

8. **Verify:**
   ```bash
   sudo apt update
   apt policy <package-name>
   sudo apt install <package-name>
   ```

---

## Client setup

Pick the suite that matches your host distribution. The signing key and
install flow are identical — only the `deb` line's codename differs.

### Common: import the signing key

```bash
curl -fsSL https://packages.netlinux.co.uk/debian/repo-key.gpg \
  | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/netlinux.gpg
```

### Debian bookworm / NetLinux Desktop / NetLinux Server

```bash
echo "deb https://packages.netlinux.co.uk/debian stable main" \
  | sudo tee /etc/apt/sources.list.d/netlinux.list

sudo apt update
sudo apt install <package-name>
```

### Ubuntu 26.04 LTS (Resolute Raccoon)

```bash
echo "deb https://packages.netlinux.co.uk/debian resolute main" \
  | sudo tee /etc/apt/sources.list.d/netlinux.list

sudo apt update
sudo apt install <package-name>
```

### Other distributions (Debian 13, Ubuntu 24.04, etc.)

Start with the `stable` suite — most packages link only against Debian
bookworm's libraries and will install on Debian 13 / recent Ubuntu releases
without issue. If you hit a dependency version mismatch, file an issue; the
nightly pipeline can be extended with an additional target.

---

## Useful commands

### On the server

```bash
# List all packages in the repo (swap 'stable' for 'resolute' as needed)
reprepro -b /Sites/netlinux/packages/debian list stable
reprepro -b /Sites/netlinux/packages/debian list resolute

# Manually add a .deb to a specific suite
reprepro -b /Sites/netlinux/packages/debian includedeb stable /path/to/file.deb
reprepro -b /Sites/netlinux/packages/debian includedeb resolute /path/to/file.deb

# Remove a package from a specific suite
reprepro -b /Sites/netlinux/packages/debian remove stable <package-name>
reprepro -b /Sites/netlinux/packages/debian remove resolute <package-name>

# Copy a package version across suites (e.g. promote stable → resolute as stand-in)
reprepro -b /Sites/netlinux/packages/debian copy resolute stable <package-name>

# Regenerate Release/InRelease for both suites
reprepro -b /Sites/netlinux/packages/debian export

# Check webhook service
systemctl status repo-webhook
journalctl -u repo-webhook -f

# View webhook log
tail -f /var/log/repo-webhook.log

# Manually trigger an update (from anywhere) — publishes to 'stable' only
curl -sf -X POST \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Signature: sha256=$(echo -n '{"repo":"netlinux-ai/<repo>","tag":"latest"}' \
    | openssl dgst -sha256 -hmac '<secret>' -binary | xxd -p -c 256)" \
  -d '{"repo":"netlinux-ai/<repo>","tag":"latest"}' \
  https://packages.netlinux.co.uk/webhook/update-repo
```

### On dev2 (nightly pipeline)

```bash
# Run the nightly dual-build immediately
/home/graham/nightly/nightly-packages.sh

# Just refresh the build-env containers
/home/graham/nightly/build-images.sh

# Smoke-test only (without rebuilding)
/home/graham/nightly/test-packages.sh \
    --build-num N --repos-conf /home/graham/nightly/repos.conf \
    --output /tmp/test.json --log-dir /tmp/logs
```

### On GitHub

```bash
# Check latest release
gh release list --repo netlinux-ai/<repo>

# View workflow runs
gh run list --repo netlinux-ai/<repo>

# Set webhook secret on a new repo
gh secret set WEBHOOK_SECRET --repo netlinux-ai/<repo>
```

---

## Known issues and gotchas

- **zstd vs xz:** Ubuntu 24.04's checkinstall produces zstd-compressed `.deb`
  files. The server's reprepro requires xz. Both the GitHub Actions workflow
  and `update-repo.sh` include a repack step, but if you're manually adding a
  `.deb`, you must repack it first.

- **checkinstall `-1` suffix:** checkinstall appends `-1` as a Debian release
  number to the version, so `1.0-2netlinux1` becomes `1.0-2netlinux1-1` in the
  filename. The `update-repo.sh` script handles this by looking up the actual
  filename in the pool directory with `ls` rather than constructing it.

- **First push after creating a repo:** The `WEBHOOK_SECRET` must be set before
  the first push, otherwise the webhook notification step will fail (the build
  and release still succeed). If this happens, either push again or manually
  trigger the webhook.

- **Pool directory naming:** The pool directory path uses the **package name**
  (e.g. `s/simplescreenrecorder`), not the GitHub repo name (which may differ,
  e.g. `ssr`). The `POOL_DIR` in the case statement must match exactly.

- **Dual-build version collision:** bookworm and resolute builds of the same
  upstream must carry different Debian version suffixes (`~bookworm1` vs
  `~resolute1`) so both `.deb`s coexist in the shared pool. The nightly
  pipeline enforces this; a manual `dpkg-deb` invocation must do the same.

- **GHA-only packages won't reach `resolute`:** webhook-triggered builds only
  publish to `stable`. Until the nightly runs, `apt update` on a resolute
  client won't see a just-pushed upstream change. Run the nightly manually
  or `reprepro copy resolute stable <pkg>` to bridge.

---

## Architecture diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                        GitHub (netlinux-ai)                     │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │kio-rsync │  │tightvnc  │  │pavucontrol│  │qpwgraph  │  ...  │
│  └────┬─────┘  └────┬─────┘  └────┬──────┘  └────┬─────┘       │
│       │              │             │               │             │
│       └──────────────┴─────────────┴───────────────┘             │
│                          │                                       │
│              push to main triggers                               │
│              .github/workflows/release-deb.yml                   │
│                          │                                       │
│                          v                                       │
│              GitHub Actions (ubuntu-24.04)                        │
│              ┌──────────────────────────┐                        │
│              │ 1. apt install build-deps│                        │
│              │ 2. cmake/make/meson build│                        │
│              │ 3. checkinstall → .deb   │                        │
│              │ 4. repack zstd → xz      │                        │
│              │ 5. gh-release with .deb  │                        │
│              │ 6. HMAC-signed webhook   │────────────────────┐   │
│              └──────────────────────────┘                    │   │
└──────────────────────────────────────────────────────────────┼───┘
                                                               │
                    HTTPS POST /webhook/update-repo            │
                    X-Webhook-Signature: sha256=<hmac>         │
                    {"repo":"netlinux-ai/X","tag":"vY"}        │
                                                               │
┌──────────────────────────────────────────────────────────────┼───┐
│                packages.netlinux.co.uk                       │   │
│                                                              v   │
│  ┌─────────┐    ┌─────────────┐    ┌──────────────────────┐     │
│  │ Apache2 │───>│ listener.py │───>│   update-repo.sh     │     │
│  │ :443    │    │ :9090       │    │                      │     │
│  └─────────┘    └─────────────┘    │ 1. curl GitHub API   │     │
│       │                            │ 2. download .deb     │     │
│       │                            │ 3. repack if needed  │     │
│       v                            │ 4. reprepro remove   │     │
│  ┌──────────┐                      │ 5. reprepro include  │     │
│  │  /Sites/ │                      │ 6. update HTML       │     │
│  │  netlinux│                      └──────────────────────┘     │
│  │ /packages│                                                    │
│  │ /debian/ │  ← reprepro repo (GPG-signed, xz-compressed)     │
│  │          │     • dists/stable/   ← from GHA + dev2 nightly   │
│  │          │     • dists/resolute/ ← from dev2 nightly only   │
│  └──────────┘                                                    │
│       ^                                                          │
│       │ reprepro includedeb <suite>                              │
└───────┼──────────────────────────────────────────────────────────┘
        │
        │
┌───────┼──────────────────────────────────────────────────────────┐
│       │             dev2 nightly pipeline                        │
│       │             (03:00 UTC, cron)                            │
│       │                                                          │
│  ┌────┴─────────┐   For each repo with new commits:              │
│  │ nightly-     │                                                │
│  │ packages.sh  │───> docker run netlinux-build:bookworm         │
│  │              │       brain-code-agent → .deb ~bookworm1 → stable│
│  │              │───> docker run netlinux-build:resolute         │
│  └──────────────┘       brain-code-agent → .deb ~resolute1 → resolute│
└──────────────────────────────────────────────────────────────────┘

                         │
                         │  apt update / apt install
                         │  (suite depends on client distro)
                         v
                ┌─────────────────┐
                │  Client machine │
                │  /etc/apt/      │
                │  sources.list.d/│
                │  netlinux.list  │  — stable for bookworm clients
                └─────────────────┘  — resolute for Ubuntu 26.04 clients
```