Runbook
Day-to-day operation of a netstation server. Run the ansible-playbook
commands from your configuration repository. Add --ask-vault-pass if you
haven't set up a vault password file.
Updating the clients
-
Build a staging image. This also happens automatically every week.
bashansible-playbook nickmous.netstation.image_build -
Check what changed compared with production.
image_buildshows this at the end; after a scheduled build, run:bashansible-playbook nickmous.netstation.image_diffIt lists updated, added and removed packages, and the changes to the files netstation puts in the image (settings, Firefox policies, scripts). It also copies both versions to
build/image-diff/in your configuration repository, to open in a diff viewer you install yourself:bashgit diff --no-index build/image-diff/production build/image-diff/staging meld build/image-diff/production build/image-diff/stagingOn the server,
sudo netstation-image diffshows the same, colored and paged likegit diff. -
Test it: reboot the test client. It boots staging by default; its boot menu also lets you pick production.
-
Promote it. Clients switch to it on their next reboot.
bashansible-playbook nickmous.netstation.image_promote -
Something wrong after all? Roll back to the previous production image:
bashansible-playbook nickmous.netstation.image_rollback # or pick one from the status list: ansible-playbook nickmous.netstation.image_rollback -e netstation_rollback_build=20261015T0300ZRolling back archives the current production image, so running it twice brings you back to where you started.
Changing settings
| What | Where | Then run |
|---|---|---|
| Homepage, wallpaper, boot splash, desktop shortcuts, RDP options and pass-through, idle logout, packages, Firefox | group_vars (netstation_homepage, ...) |
image_build, test, image_promote |
| Files in the image | files/rootfs/... |
image_build, test, image_promote |
| Known clients, test client, per-client settings | netstation_clients |
server |
| DHCP, firewall, WireGuard peers, other server settings | group_vars |
server |
| Remote Mac on or off | netstation_remote_mac_enabled |
server, then image_build, test, image_promote |
Always preview server changes first:
ansible-playbook nickmous.netstation.server --check --diffStatus
ansible-playbook nickmous.netstation.statusOnly the clients that are on, with IP and MAC address:
ansible-playbook nickmous.netstation.clientsOr on the server: sudo netstation-image status. It shows the production and
staging builds, the rollback targets, the scheduled build, which clients are
booted right now, and whether dnsmasq, NFS and nginx are running.
Cockpit is at https://<server>:9090 (log in with your server account): logs,
services, updates and a terminal.
Alerts
With netstation_pushover_app_token and netstation_pushover_user_key set,
the server sends Pushover alerts:
| Alert | When |
|---|---|
| unit failed | The scheduled build, dnsmasq, NFS or nginx failed, with its last log lines |
| Problem / Resolved | Every 10 minutes the server checks: a service stopped, a disk over 85% full, the automatic security updates failed, or the server needs a reboot for a new kernel. It alerts once when a problem appears and once when it's gone. |
| dnsmasq restarted / still hangs / works again | The dnsmasq watchdog (every 2 minutes) found dnsmasq hanging and restarted it |
| Staging build ready to test | A staging build finished, with what changed compared with production |
| Clients still on | At the netstation_power_report times, when any client is still on |
| client: Remote Desktop crashed | xfreerdp3 crashed on a client (with a sound), with its last errors |
| client: Remote Desktop error / can't connect / account blocked / password expired | xfreerdp3 stopped with an error (without a sound). Wrong passwords and normal disconnects aren't reported. |
Alerts about a client name it by label and hostname from
netstation_clients, or by its hostname or IP address. A client sends at most
one alert every 10 minutes (netstation_alerts_client_quiet_minutes); the
others are only logged: sudo journalctl -t netstation-client-report.
Security updates install by themselves, but the server never reboots by itself: clients stop working while it restarts. Reboot it outside opening hours when the reboot alert comes:
ssh <server> sudo systemctl rebootTo see the alerts the server sent, also when Pushover couldn't be reached:
sudo journalctl -t netstation-alertRun a check by hand with sudo netstation-health.
Waking clients
Clients listed in netstation_clients can be switched on from the server:
ansible-playbook nickmous.netstation.wake # all of them
ansible-playbook nickmous.netstation.wake -e '{"netstation_wake": ["pc01"]}' # by hostname or MACEach client's BIOS/UEFI must allow Wake-on-LAN (often under power
management). The image keeps the network card armed when the client shuts
down. If clients don't wake, set netstation_wake_broadcast to the
broadcast address of their network (e.g. 192.168.1.255).
Power schedule
With a power schedule the server wakes the clients and the clients switch themselves off when nobody uses them:
netstation_power_wake: ["08:45"] # Wake-on-LAN for all known clients
netstation_power_off:
- at: "10:00" # not used since switched on
- at: "18:30"
idle_minutes: 15 # no input for 15 minutes
netstation_power_report: ["22:00"] # Pushover: clients still onTimes are systemd calendar times, so "Mon..Fri 08:45" works too. The wake
reaches the clients in netstation_clients and every client the server has
seen before (/var/lib/netstation/known-clients).
A client in netstation_clients can have its own wake times. They replace
netstation_power_wake for that client:
netstation_power_wake: ["Mon..Fri 08:45"]
netstation_clients:
- mac: "d8:9e:f3:37:53:a1"
label: Reception
wake: ["Mon..Fri 07:30", "Sat 08:45"] # earlier, and also on Saturday
- mac: "d8:9e:f3:37:53:a2"
label: Spare
wake: [] # never woken by the scheduleClients with the same wake list share a timer
(netstation-power-wake-N.timer); systemctl list-timers 'netstation-power-*'
shows when each fires next. Switching off is part of
the image, so after changing netstation_power_off run image_build, test,
and image_promote; the other two only need server.
Try it by hand on the server:
sudo netstation-power wake # or: wake-1 (a group), learn, reportVeyon computer list
Veyon Master needs a list of the clients. Generate it from the clients that are booted right now, then import it with the two commands the playbook prints:
ansible-playbook nickmous.netstation.veyon_computersClient IP addresses can change, so run it again when Veyon shows clients as offline that are switched on. DHCP reservations for the clients avoid that.
Remote Mac
With netstation_remote_mac_enabled, people control their own Mac from any
client: the client's keyboard, mouse and screens work the Mac, as with
Remote Desktop. It uses RustDesk: the server runs a
RustDesk ID and relay server (rustdesk-hbbs, rustdesk-hbbr), reachable
only from netstation_remote_mac_networks, and the image has the RustDesk
client and a desktop icon (netstation_remote_mac_icon_name).
At the client: click the icon, type the Mac's RustDesk ID, then the password RustDesk on the Mac shows. Each Mac screen opens full screen on its own client screen. The XFCE panels are gone during the session, and Alt and the Windows key swap places, so the key next to the space bar is Cmd. Close the RustDesk windows (the toolbar at the top of the screen) to get the desktop back.
Turning it on. Set netstation_remote_mac_enabled: true, run the
server playbook, then build, test and promote an image. The server
playbook prints the ID server and key for the Macs. It makes a key on its
first run; save it in the vault right away, or a reinstalled server gets a
new one and every Mac has to be set up again:
ssh netadmin@<server> sudo cat /var/lib/rustdesk-server/id_ed25519# vault.yml
vault_rustdesk_private_key: "..."
# main.yml
netstation_rustdesk_private_key: "{{ vault_rustdesk_private_key }}"Setting up a Mac, once:
- Install RustDesk from https://rustdesk.com and let it start at login.
- In RustDesk, Settings > Network > ID/Relay server: set the ID server to
the server's address and the key printed by the
serverplaybook. Leave the relay server empty. - In macOS System Settings > Privacy & Security, allow RustDesk under Screen Recording and Accessibility, then quit and reopen RustDesk. Without Accessibility the client sees the screen but can't control it.
Updating RustDesk. Change netstation_rustdesk_client_version and
netstation_rustdesk_client_checksum (image), or
netstation_rustdesk_server_version and netstation_rustdesk_server_checksums
(server). The checksums are on the GitHub release pages.
Adding a client
PXE boot must be enabled in the BIOS/UEFI and Secure Boot disabled. A new
client then works without any configuration. Add it to netstation_clients
only to give it a fixed name, label it, make it a test client, or
give it its own settings:
netstation_clients:
- mac: "d8:9e:f3:37:53:a1"
hostname: reception01
label: Anna de Vries # who or what it's for: shown in alerts and by the clients playbookThen run the server playbook.
Troubleshooting
A client doesn't network-boot
Watch the server while the client boots:
sudo journalctl -u dnsmasq -f- Nothing about the client's MAC address: its boot request doesn't reach the server. Check that it really network-boots (boot menu, often F12) and that it's on the same network segment.
- The request arrives but the client gets no answer: set
netstation_dnsmasq_log_dhcp: true, runserver, and look at the detailed log. In proxy mode, also check that the other DHCP server doesn't set options 60, 66 or 67. - dnsmasq hangs (high CPU, clients stop booting): some dnsmasq versions
hang in proxy mode. A watchdog checks every 2 minutes and restarts it (see
sudo journalctl -u netstation-dnsmasq-watchdog). Check by hand withsudo netstation-dnsmasq-checkafter a client has booted; it measures CPU use, unread packets and whether TFTP still answers. If it keeps happening, pin a working version withnetstation_dnsmasq_version.
A client boots but stops with an error
Add a debug shell for one client, run server, and reboot that client:
netstation_clients:
- mac: "d8:9e:f3:37:53:a1"
settings:
DEBUG_SHELL: 1A client hangs while it starts
The boot splash hides the messages. Press Esc on the client to see them.
To boot one client without the splash, give it its own kernel parameters,
run server, and reboot that client:
netstation_clients:
- mac: "d8:9e:f3:37:53:a1"
settings:
KERNEL_PARAMETERS: nosplashThe client shows a login screen or loops back to it
On the client, press Ctrl+Alt+F2, log in with your server account, and run:
ls -la /home/kiosk # must exist and be owned by kiosk
findmnt /home/kiosk # must print nothing (home in RAM)
grep -rn autologin /etc/lightdm/
sudo tail -40 /var/log/lightdm/lightdm.logRemote Desktop fails on a client
The dialog on the client says roughly what went wrong, and crashes and errors come in as alerts. For xfreerdp3's full output, log in to the client (below) while it's still on, and run:
sudo journalctl -t netstation-rdpThe client keeps its journal in memory, so it's gone after a reboot.
Remote Mac doesn't work
- "ID does not exist" or it can't connect: the Mac isn't using this
server (Settings > Network on the Mac), isn't on the network, or RustDesk
isn't running on it. Check the server with
sudo systemctl status rustdesk-hbbs rustdesk-hbbrandsudo journalctl -u rustdesk-hbbs. - "Key mismatch": the key on the Mac isn't the server's. Show the right
one with
sudo cat /var/lib/rustdesk-server/id_ed25519.pub. - The screen shows, but mouse and keyboard do nothing: RustDesk on the Mac lacks Accessibility (see Setting up a Mac). Reopen RustDesk after allowing it.
- Lag: usually the Mac's Wi-Fi. A network cable fixes it; Retina resolutions on the Mac also cost speed.
- Alt and Cmd aren't swapped, or still swapped after the session: log in
to the client and check
systemctl status keyd. A logout always stops it.
On the client, journalctl -t netstation-mac shows each session and
RustDesk's output.
Logging in to a client
ssh <your-server-account>@<client-ip>Clients get new SSH host keys on every boot, so expect a host key warning.
Something can't reach the server
The server's firewall only lets in SSH, network boot, NFS, and Cockpit and WireGuard when they're on. It logs what it drops:
sudo journalctl -k --grep netstation-firewallAllow more with netstation_firewall_extra_rules (nftables rules, e.g.
tcp dport 8080 accept) and run server.
The scheduled staging build failed
sudo journalctl -u netstation-staging-build.service -n 100Production is never touched by a failed build.