build01: building 9hh0amvxk323lg28jmyhwgxqvbz4m7ag-container-test-run-dm-dns.drv
Machine state will be reset. To keep it, pass --keep-machine-state
start all VLans
(finished: start all VLans, in 0.00 seconds)

Test will time out and terminate in 3600.0 seconds
run the VM test script
additionally exposed symbols:
    client, server,
    vlan1,
    start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh
start all VMs
client: systemd-nspawn running (pid 52)
server: systemd-nspawn running (pid 53)
client: Waiting for journal at /build/vm-state-client/var/log/journal...
server: Waiting for journal at /build/vm-state-server/var/log/journal...
(finished: start all VMs, in 0.00 seconds)
server: waiting for unit unbound.service
nixos-nspawn(client): TAP vde-tap1 not found; container will be isolated from VDE
nixos-nspawn(client): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.
nixos-nspawn(server): TAP vde-tap1 not found; container will be isolated from VDE
nixos-nspawn(server): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths.
Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.
Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file.
░ Spawning container client on /build/vm-state-client.
░ Spawning container server on /build/vm-state-server.
client # No journal files were found.
client # No journal boot entry found for the specified boot (+0).
server # No journal files were found.
server # No journal boot entry found for the specified boot (+0).
client # [2101820.713699] client systemd-journald[90]: Journal started
client # [2101820.713755] client systemd-journald[90]: Runtime Journal (/run/log/journal/0f3b85bd73054470bfd0dcaac021c63a) is 8M, max 2.5G, 2.4G free.
server # [2101820.724249] server systemd-journald[99]: Journal started
client # [2101820.723575] client systemd[1]: Finished Apply Kernel Variables.
server # [2101820.724305] server systemd-journald[99]: Runtime Journal (/run/log/journal/e53c54c23e9a495797acd26f3d349f0b) is 8M, max 2.5G, 2.4G free.
client # [2101820.740137] client systemd[1]: Finished Create Static Device Nodes in /dev gracefully.
server # [2101820.725136] server systemd[1]: Finished Apply Kernel Variables.
client # [2101820.754967] client systemd[1]: Starting Flush Journal to Persistent Storage...
server # [2101820.736429] server systemd[1]: Finished Create Static Device Nodes in /dev gracefully.
client # [2101820.755788] client systemd[1]: Starting Network Name Resolution...
server # [2101820.749548] server systemd[1]: Starting Flush Journal to Persistent Storage...
client # [2101820.756461] client systemd[1]: Starting Create Static Device Nodes in /dev...
server # [2101820.750317] server systemd[1]: Starting Network Name Resolution...
client # [2101820.765504] client systemd-journald[90]: Time spent on flushing to /var/log/journal/0f3b85bd73054470bfd0dcaac021c63a is 1.444ms for 7 entries.
server # [2101820.751138] server systemd[1]: Starting Create Static Device Nodes in /dev...
client # [2101820.765504] client systemd-journald[90]: System Journal (/var/log/journal/0f3b85bd73054470bfd0dcaac021c63a) is 8M, max 4G, 3.9G free.
server # [2101820.760353] server systemd-journald[99]: Time spent on flushing to /var/log/journal/e53c54c23e9a495797acd26f3d349f0b is 1.157ms for 7 entries.
client # [2101820.786609] client systemd[1]: Finished Flush Journal to Persistent Storage.
server # [2101820.760353] server systemd-journald[99]: System Journal (/var/log/journal/e53c54c23e9a495797acd26f3d349f0b) is 8M, max 4G, 3.9G free.
client # [2101820.787117] client systemd[1]: Finished Create Static Device Nodes in /dev.
server # [2101820.768990] server systemd[1]: Finished Flush Journal to Persistent Storage.
client # [2101820.788057] client systemd[1]: Reached target Preparation for Local File Systems.
server # [2101820.769832] server systemd[1]: Finished Create Static Device Nodes in /dev.
client # [2101820.788187] client systemd[1]: Reached target Local File Systems.
server # [2101820.770803] server systemd[1]: Reached target Preparation for Local File Systems.
client # [2101820.789030] client systemd[1]: Listening on Boot Loader Control Service Socket.
server # [2101820.770939] server systemd[1]: Reached target Local File Systems.
client # [2101820.789082] client systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container
server # [2101820.771769] server systemd[1]: Listening on Boot Loader Control Service Socket.
client # [2101820.789856] client systemd[1]: Starting Save Transient machine-id to Disk...
server # [2101820.771819] server systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container
client # [2101820.790521] client systemd[1]: Starting Create System Files and Directories...
server # [2101820.772950] server systemd[1]: Starting Save Transient machine-id to Disk...
client # [2101820.790553] client systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys
server # [2101820.773716] server systemd[1]: Starting Create System Files and Directories...
client # [2101820.807265] client systemd-tmpfiles[149]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
server # [2101820.773749] server systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys
client # [2101820.807493] client systemd-tmpfiles[149]: fchmod() of /var/log/journal failed: Operation not permitted
server # [2101820.789121] server systemd-tmpfiles[148]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted
client # [2101820.807660] client systemd-tmpfiles[149]: fchmod() of /var/log/journal/0f3b85bd73054470bfd0dcaac021c63a failed: Operation not permitted
server # [2101820.789300] server systemd-tmpfiles[148]: fchmod() of /var/log/journal failed: Operation not permitted
client # [2101820.807901] client systemd-tmpfiles[149]: fchmod() of /run/log/journal failed: Operation not permitted
server # [2101820.789423] server systemd-tmpfiles[148]: fchmod() of /var/log/journal/e53c54c23e9a495797acd26f3d349f0b failed: Operation not permitted
client # [2101820.810037] client systemd[1]: Finished Create System Files and Directories.
server # [2101820.789609] server systemd-tmpfiles[148]: fchmod() of /run/log/journal failed: Operation not permitted
client # [2101820.811263] client systemd[1]: Starting Rebuild Journal Catalog...
server # [2101820.790907] server systemd[1]: Finished Create System Files and Directories.
client # [2101820.812434] client systemd[1]: Starting Record System Boot/Shutdown in UTMP...
server # [2101820.792170] server systemd[1]: Starting Rebuild Journal Catalog...
client # [2101820.816853] client systemd[1]: Finished Save Transient machine-id to Disk.
server # [2101820.792970] server systemd[1]: Starting Record System Boot/Shutdown in UTMP...
client # [2101820.825888] client systemd[1]: Finished Record System Boot/Shutdown in UTMP.
server # [2101820.806877] server systemd[1]: Finished Record System Boot/Shutdown in UTMP.
client # [2101820.835264] client systemd[1]: Finished Rebuild Journal Catalog.
server # [2101820.815331] server systemd[1]: Finished Rebuild Journal Catalog.
client # [2101820.836279] client systemd[1]: Starting Update is Completed...
server # [2101820.816576] server systemd[1]: Starting Update is Completed...
client # [2101820.845382] client systemd[1]: Finished Update is Completed.
server # [2101820.816843] server systemd[1]: Finished Save Transient machine-id to Disk.
client # [2101820.891131] client systemd[1]: Finished Firewall.
server # [2101820.827988] server systemd[1]: Finished Update is Completed.
client # [2101820.891354] client systemd[1]: Reached target Preparation for Network.
server # [2101820.891437] server systemd[1]: Finished Firewall.
client # [2101820.891665] client systemd[1]: Listening on Network Management Resolve Hook Socket.
server # [2101820.891568] server systemd[1]: Reached target Preparation for Network.
client # [2101820.893004] client systemd[1]: Starting Network Management...
server # [2101820.891786] server systemd[1]: Listening on Network Management Resolve Hook Socket.
client # [2101821.584843] client systemd-networkd[208]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
server # [2101820.892874] server systemd[1]: Starting Network Management...
client # [2101821.584940] client systemd-networkd[208]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted
server # [2101821.591296] server systemd-networkd[217]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
client # [2101821.593886] client systemd-networkd[208]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section.
server # [2101821.591385] server systemd-networkd[217]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted
client # [2101821.594051] client systemd-networkd[208]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section.
server # [2101821.600370] server systemd-networkd[217]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section.
client # [2101821.594199] client systemd-networkd[208]: lo: Link UP
server # [2101821.600538] server systemd-networkd[217]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section.
client # [2101821.594203] client systemd-networkd[208]: lo: Gained carrier
server # [2101821.600676] server systemd-networkd[217]: lo: Link UP
server # [2101821.600680] server systemd-networkd[217]: lo: Gained carrier
client # [2101821.594377] client systemd-networkd[208]: eth1: Configuring with /etc/systemd/network/40-eth1.network.
client # [2101821.594842] client systemd-networkd[208]: eth1: Link UP
client # [2101821.594939] client systemd[1]: Started Network Management.
client # [2101821.596015] client systemd-networkd[208]: eth1: Gained carrier
server # [2101821.600870] server systemd-networkd[217]: eth1: Configuring with /etc/systemd/network/40-eth1.network.
client # [2101821.596369] client systemd[1]: Starting Enable Persistent Storage in systemd-networkd...
server # [2101821.601240] server systemd[1]: Started Network Management.
client # [2101821.645542] client systemd[1]: Finished Enable Persistent Storage in systemd-networkd.
server # [2101821.636307] server systemd-networkd[217]: eth1: Link UP
client # [2101821.722566] client systemd[1]: etc-machine\x2did.mount: Deactivated successfully.
server # [2101821.636611] server systemd-networkd[217]: eth1: Gained carrier
server # [2101821.636697] server systemd[1]: Starting Enable Persistent Storage in systemd-networkd...
server # [2101821.678582] server systemd[1]: Finished Enable Persistent Storage in systemd-networkd.
server # [2101821.731458] server systemd[1]: etc-machine\x2did.mount: Deactivated successfully.
server # [2101821.939173] server systemd-resolved[129]: Positive Trust Anchors:
server # [2101821.939185] server systemd-resolved[129]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
server # [2101821.939188] server systemd-resolved[129]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16
server # [2101821.939223] server systemd-resolved[129]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test
server # [2101821.960839] server systemd-resolved[129]: Using system hostname 'server'.
server # [2101821.962204] server systemd[1]: Started Network Name Resolution.
server # [2101821.962335] server systemd[1]: Reached target Network.
server # [2101821.962448] server systemd[1]: Reached target System Initialization.
server # [2101821.962609] server systemd[1]: Started Watch for zone file changes.
server # [2101821.962663] server systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container
server # [2101821.962715] server systemd[1]: Started Daily Cleanup of Temporary Directories.
server # [2101821.962756] server systemd[1]: Reached target Path Units.
server # [2101821.962825] server systemd[1]: Reached target Timer Units.
server # [2101821.963033] server systemd[1]: Listening on D-Bus System Message Bus Socket.
server # [2101821.963232] server systemd[1]: Listening on Nix Daemon Socket.
server # [2101821.963445] server systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.
server # [2101821.963496] server systemd[1]: Reached target Socket Units.
server # [2101821.963571] server systemd[1]: Reached target Basic System.
server # [2101821.966493] server systemd[1]: Starting data mesher daemon...
server # [2101821.968121] server systemd[1]: Starting Import lastlog data into lastlog2 database...
server # [2101821.969356] server systemd[1]: Starting Name Service Cache Daemon (nsncd)...
server # [2101821.971660] server systemd[1]: Starting D-Bus System Message Bus...
server # [2101822.024232] server systemd[1]: Finished Import lastlog data into lastlog2 database.
server # [2101822.199964] server nsncd[224]: Sep 28 15:17:22.185 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
server # [2101822.199895] server systemd[1]: Started Name Service Cache Daemon (nsncd).
server # [2101822.200038] server systemd[1]: Reached target User and Group Name Lookups.
client # [2101821.939229] client systemd-resolved[126]: Positive Trust Anchors:
client # [2101821.939239] client systemd-resolved[126]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
client # [2101821.939244] client systemd-resolved[126]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16
client # [2101821.939278] client systemd-resolved[126]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test
client # [2101821.960805] client systemd-resolved[126]: Using system hostname 'client'.
client # [2101821.962164] client systemd[1]: Started Network Name Resolution.
client # [2101821.962292] client systemd[1]: Reached target Network.
client # [2101821.962407] client systemd[1]: Reached target System Initialization.
client # [2101821.962557] client systemd[1]: Started Watch for zone file changes.
client # [2101821.962608] client systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container
client # [2101821.962662] client systemd[1]: Started Daily Cleanup of Temporary Directories.
client # [2101821.962708] client systemd[1]: Reached target Path Units.
client # [2101821.962775] client systemd[1]: Reached target Timer Units.
client # [2101821.962989] client systemd[1]: Listening on D-Bus System Message Bus Socket.
client # [2101821.963197] client systemd[1]: Listening on Nix Daemon Socket.
client # [2101821.963414] client systemd[1]: Listening on Virtual Machine and Container Registration Service Socket.
client # [2101821.963461] client systemd[1]: Reached target Socket Units.
client # [2101821.963542] client systemd[1]: Reached target Basic System.
client # [2101821.966394] client systemd[1]: Starting data mesher daemon...
client # [2101821.967821] client systemd[1]: Starting Import lastlog data into lastlog2 database...
client # [2101821.969340] client systemd[1]: Starting Name Service Cache Daemon (nsncd)...
client # [2101821.971488] client systemd[1]: Starting D-Bus System Message Bus...
client # [2101822.024704] client systemd[1]: Finished Import lastlog data into lastlog2 database.
client # [2101822.235467] client systemd[1]: Started Name Service Cache Daemon (nsncd).
client # [2101822.235926] client nsncd[215]: Sep 28 15:17:22.221 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
client # [2101822.235595] client systemd[1]: Reached target User and Group Name Lookups.
server # [2101822.202157] server systemd[1]: Starting User Login Management...
server # [2101822.203578] server systemd[1]: Starting Permit User Sessions...
server # [2101822.240401] server systemd[1]: Finished Permit User Sessions.
server # [2101822.242120] server systemd[1]: Started Console Getty.
server # [2101822.242198] server systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0
server # [2101822.242249] server systemd[1]: Reached target Login Prompts.
server # [2101822.403528] server dbus-broker-launch[225]: Looking up NSS user entry for 'systemd-timesync'...
server # [2101822.405016] server dbus-broker-launch[225]: NSS returned no entry for 'systemd-timesync'
server # [2101822.405016] server dbus-broker-launch[225]: Invalid user-name in /nix/store/g0n529d3yw76mxc12jqzqfrvyqxswrfv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"
server # [2101822.405483] server systemd[1]: Started D-Bus System Message Bus.
server # [2101822.413143] server dbus-broker-launch[225]: Ready
client # [2101822.237575] client systemd[1]: Starting User Login Management...
client # [2101822.238915] client systemd[1]: Starting Permit User Sessions...
client # [2101822.250909] client systemd[1]: Finished Permit User Sessions.
client # [2101822.252512] client systemd[1]: Started Console Getty.
client # [2101822.252594] client systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0
client # [2101822.252633] client systemd[1]: Reached target Login Prompts.
client # [2101822.371082] client dbus-broker-launch[216]: Looking up NSS user entry for 'systemd-timesync'...
client # [2101822.372535] client dbus-broker-launch[216]: NSS returned no entry for 'systemd-timesync'
client # [2101822.372535] client dbus-broker-launch[216]: Invalid user-name in /nix/store/pdpmvp4r97cgcvimwlbhfb5jkb5aja3i-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"
client # [2101822.373314] client systemd[1]: Started D-Bus System Message Bus.
client # [2101822.382440] client dbus-broker-launch[216]: Ready
client # [2101822.818001] client data-mesher[213]: time=2026-09-28T15:17:22.803Z level=DEBUG msg="http config" listen_addresses="[[::1]:7331 127.0.0.1:7331]" port=7331 interfaces=[lo]
client # [2101822.819067] client data-mesher[213]: time=2026-09-28T15:17:22.804Z level=DEBUG msg="cluster config" port=7946 interfaces=[] bootstrap_peers="[{12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE: [/dns/client.test/tcp/7946]} {12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs: [/dns/server.test/tcp/7946]}]" push_pull_interval=5s peer_id=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
client # [2101822.819067] client data-mesher[213]: time=2026-09-28T15:17:22.804Z level=INFO msg="config loaded" config_file=/etc/data-mesher/dm.toml
client # [2101822.820424] client data-mesher[213]: time=2026-09-28T15:17:22.806Z level=INFO msg="checking file integrity"
client # [2101822.820536] client data-mesher[213]: time=2026-09-28T15:17:22.806Z level=INFO msg="file integrity check complete"
client # [2101822.824198] client data-mesher[213]: time=2026-09-28T15:17:22.809Z level=INFO msg="libp2p host created" peer_id=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE addresses="[/ip4/127.0.0.1/tcp/7946 /ip4/192.168.1.1/tcp/7946 /ip6/::1/tcp/7946 /ip6/2001:db8:1::1/tcp/7946]"
client # [2101822.824273] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="registered HTTP route" method=PUT path=/files/:name
client # [2101822.824273] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="registered HTTP route" method=DELETE path=/files/:name
client # [2101822.824273] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="registered HTTP route" method=GET path=/files
client # [2101822.824273] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="starting server"
client # [2101822.824454] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="waiting for DHT to populate" delay=10s
client # [2101822.824454] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="HTTP server listening" address=[::1]:7331
client # [2101822.824454] client data-mesher[213]: time=2026-09-28T15:17:22.810Z level=INFO msg="HTTP server listening" address=127.0.0.1:7331
client # [2101822.829883] client data-mesher[213]: time=2026-09-28T15:17:22.815Z level=INFO msg="peer connected" peer_id=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs remote_addr=/ip4/192.168.1.2/tcp/7946
client # [2101822.859583] client data-mesher[213]: time=2026-09-28T15:17:22.845Z level=INFO msg="peer connected" peer_id=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs remote_addr=/ip4/192.168.1.2/tcp/7946
client # [2101823.018440] client systemd-logind[233]: New seat seat0.
client # [2101823.018635] client systemd[1]: Started User Login Management.
client # [2101823.021183] client systemd[1]: Starting linger-users.service...
client # [2101823.054580] client systemd[1]: linger-users.service: Deactivated successfully.
client # [2101823.054722] client systemd[1]: Finished linger-users.service.
server # [2101822.806824] server data-mesher[222]: time=2026-09-28T15:17:22.792Z level=DEBUG msg="http config" listen_addresses="[[::1]:7331 127.0.0.1:7331]" port=7331 interfaces=[lo]
server # [2101822.807895] server data-mesher[222]: time=2026-09-28T15:17:22.793Z level=DEBUG msg="cluster config" port=7946 interfaces=[] bootstrap_peers="[{12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE: [/dns/client.test/tcp/7946]} {12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs: [/dns/server.test/tcp/7946]}]" push_pull_interval=5s peer_id=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
server # [2101822.807895] server data-mesher[222]: time=2026-09-28T15:17:22.793Z level=INFO msg="config loaded" config_file=/etc/data-mesher/dm.toml
server # [2101822.819062] server data-mesher[222]: time=2026-09-28T15:17:22.804Z level=INFO msg="checking file integrity"
server # [2101822.819184] server data-mesher[222]: time=2026-09-28T15:17:22.804Z level=INFO msg="file integrity check complete"
server # [2101822.823004] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="libp2p host created" peer_id=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs addresses="[/ip4/127.0.0.1/tcp/7946 /ip4/192.168.1.2/tcp/7946 /ip6/::1/tcp/7946 /ip6/2001:db8:1::2/tcp/7946]"
server # [2101822.823066] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="registered HTTP route" method=GET path=/files
server # [2101822.823066] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="registered HTTP route" method=PUT path=/files/:name
server # [2101822.823066] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="registered HTTP route" method=DELETE path=/files/:name
server # [2101822.823066] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="starting server"
server # [2101822.823176] server data-mesher[222]: time=2026-09-28T15:17:22.808Z level=INFO msg="waiting for DHT to populate" delay=10s
server # [2101822.823248] server data-mesher[222]: time=2026-09-28T15:17:22.809Z level=INFO msg="HTTP server listening" address=[::1]:7331
server # [2101822.823288] server data-mesher[222]: time=2026-09-28T15:17:22.809Z level=INFO msg="HTTP server listening" address=127.0.0.1:7331
server # [2101822.829464] server data-mesher[222]: time=2026-09-28T15:17:22.815Z level=INFO msg="peer connected" peer_id=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE remote_addr=/ip4/192.168.1.1/tcp/7946
server # [2101822.860102] server data-mesher[222]: time=2026-09-28T15:17:22.845Z level=INFO msg="peer connected" peer_id=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE remote_addr=/ip4/192.168.1.1/tcp/58508
server # [2101823.018423] server systemd-logind[242]: New seat seat0.
server # [2101823.018671] server systemd[1]: Started User Login Management.
server # [2101823.020942] server systemd[1]: Starting linger-users.service...
server # [2101823.054027] server systemd[1]: linger-users.service: Deactivated successfully.
server # [2101823.054232] server systemd[1]: Finished linger-users.service.
client # [2101823.168181] client systemd-networkd[208]: eth1: Gained IPv6LL
server # [2101823.268131] server systemd-networkd[217]: eth1: Gained IPv6LL
server: still waiting for container 'server' to reach ready state...
client # [2101832.824421] client data-mesher[213]: time=2026-09-28T15:17:32.810Z level=INFO msg="performing state exchange with peers on join" count=1
client # [2101832.825085] client data-mesher[213]: time=2026-09-28T15:17:32.810Z level=DEBUG msg="initiating state exchange" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs timeout=5s
client # [2101832.825085] client data-mesher[213]: time=2026-09-28T15:17:32.810Z level=INFO msg="received state sync from peer" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101832.825085] client data-mesher[213]: time=2026-09-28T15:17:32.810Z level=INFO msg="merging remote state" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101832.825688] client data-mesher[213]: time=2026-09-28T15:17:32.811Z level=INFO msg="merging remote state" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101832.826158] client data-mesher[213]: time=2026-09-28T15:17:32.811Z level=INFO msg="state exchange complete" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs timeout=5s
client # [2101832.826158] client data-mesher[213]: time=2026-09-28T15:17:32.811Z level=INFO msg="server started"
client # [2101832.826316] client data-mesher[213]: time=2026-09-28T15:17:32.812Z level=INFO msg="starting expired-file sweeper" interval=1m0s
client # [2101832.826201] client systemd[1]: Started data mesher daemon.
client # [2101832.828742] client systemd[1]: Starting Unbound recursive Domain Name Server...
server # [2101832.823646] server data-mesher[222]: time=2026-09-28T15:17:32.809Z level=INFO msg="performing state exchange with peers on join" count=1
server # [2101832.824429] server data-mesher[222]: time=2026-09-28T15:17:32.809Z level=DEBUG msg="initiating state exchange" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE timeout=5s
server # [2101832.824833] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="merging remote state" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101832.824833] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="state exchange complete" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE timeout=5s
server # [2101832.824955] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="server started"
server # [2101832.825070] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="starting expired-file sweeper" interval=1m0s
server # [2101832.825224] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="received state sync from peer" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101832.825224] server data-mesher[222]: time=2026-09-28T15:17:32.810Z level=INFO msg="merging remote state" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101832.825237] server systemd[1]: Started data mesher daemon.
server # [2101832.827583] server systemd[1]: Starting Unbound recursive Domain Name Server...
client # [2101833.787846] client unbound-pre-start[275]: Root anchor updated!
client # [2101833.800836] client unbound-pre-start[279]: setup in directory /var/lib/unbound
server # [2101833.785854] server unbound-pre-start[285]: Root anchor updated!
server # [2101833.798091] server unbound-pre-start[289]: setup in directory /var/lib/unbound
client # [2101835.138517] client unbound-pre-start[288]: Certificate request self-signature ok
client # [2101835.138517] client unbound-pre-start[288]: subject=CN=unbound-control
client # [2101835.159098] client unbound-pre-start[279]: removing artifacts
client # [2101835.161613] client unbound-pre-start[279]: Setup success. Certificates created. Enable in unbound.conf file to use
server # [2101835.463383] server unbound-pre-start[298]: Certificate request self-signature ok
server # [2101835.463383] server unbound-pre-start[298]: subject=CN=unbound-control
server # [2101835.483018] server unbound-pre-start[289]: removing artifacts
server # [2101835.485283] server unbound-pre-start[289]: Setup success. Certificates created. Enable in unbound.conf file to use
client # [2101836.093447] client unbound[292]: [292:0] notice: init module 0: validator
client # [2101836.093564] client unbound[292]: [292:0] notice: init module 1: iterator
client # [2101836.099321] client unbound[292]: [292:0] info: start of service (unbound 1.26.0).
client # [2101836.099463] client systemd[1]: Started Unbound recursive Domain Name Server.
client # [2101836.099969] client systemd[1]: Reached target Multi-User System.
client # [2101836.100267] client systemd[1]: Reached target Host and Network Name Lookups.
client # [2101836.102117] client systemd[1]: Starting Reload unbound zone configuration...
client # [2101836.154351] client unbound[292]: [292:0] info: service stopped (unbound 1.26.0).
client # [2101836.154667] client unbound-control[296]: ok
client # [2101836.154689] client unbound[292]: [292:0] info: server stats for thread 0: 1 queries, 0 answers from cache, 1 recursions, 0 prefetch, 0 rejected by ip ratelimiting
client # [2101836.154694] client unbound[292]: [292:0] info: server stats for thread 0: requestlist max 1 avg 1 exceeded 0 jostled 0
client # [2101836.156312] client systemd[1]: unbound-reload-zones.service: Deactivated successfully.
client # [2101836.156491] client unbound[292]: [292:0] notice: Restart of unbound 1.26.0.
client # [2101836.156652] client systemd[1]: Finished Reload unbound zone configuration.
client # [2101836.157133] client systemd[1]: Startup finished in 15.844s.
client # [2101836.157401] client unbound[292]: [292:0] notice: init module 0: validator
client # [2101836.157466] client unbound[292]: [292:0] notice: init module 1: iterator
client # [2101836.162033] client unbound[292]: [292:0] info: start of service (unbound 1.26.0).
server # [2101836.359893] server unbound[302]: [302:0] notice: init module 0: validator
server # [2101836.360013] server unbound[302]: [302:0] notice: init module 1: iterator
server # [2101836.365520] server unbound[302]: [302:0] info: start of service (unbound 1.26.0).
server # [2101836.365664] server systemd[1]: Started Unbound recursive Domain Name Server.
server # [2101836.366147] server systemd[1]: Reached target Multi-User System.
server # [2101836.366414] server systemd[1]: Reached target Host and Network Name Lookups.
server # [2101836.381451] server systemd[1]: Starting Reload unbound zone configuration...
server # [2101836.394702] server unbound[302]: [302:0] info: service stopped (unbound 1.26.0).
server # [2101836.394972] server unbound[302]: [302:0] info: server stats for thread 0: 1 queries, 0 answers from cache, 1 recursions, 0 prefetch, 0 rejected by ip ratelimiting
server # [2101836.394977] server unbound[302]: [302:0] info: server stats for thread 0: requestlist max 1 avg 1 exceeded 0 jostled 0
server # [2101836.395135] server unbound-control[306]: ok
server # [2101836.396392] server unbound[302]: [302:0] notice: Restart of unbound 1.26.0.
server # [2101836.396646] server systemd[1]: unbound-reload-zones.service: Deactivated successfully.
server # [2101836.396948] server systemd[1]: Finished Reload unbound zone configuration.
server # [2101836.397211] server unbound[302]: [302:0] notice: init module 0: validator
server # [2101836.397267] server unbound[302]: [302:0] notice: init module 1: iterator
server # [2101836.397397] server systemd[1]: Startup finished in 16.067s.
server # [2101836.401630] server unbound[302]: [302:0] info: start of service (unbound 1.26.0).
server: (finished: waiting for unit unbound.service, in 17.17 seconds)
client: waiting for unit unbound.service
client: (finished: waiting for unit unbound.service, in 0.02 seconds)
server: waiting for unit data-mesher.service
server: (finished: waiting for unit data-mesher.service, in 0.02 seconds)
server: waiting for success: dig +short @127.0.0.1 -p 5353 server.test A | grep 10.0.0.1
server: (finished: waiting for success: dig +short @127.0.0.1 -p 5353 server.test A | grep 10.0.0.1, in 0.03 seconds)
server: must succeed: data-mesher file update --network-id /nix/store/xa27jjjrwdjp58yprvlqv8nxdcqcmydh-shared-data-mesher-network_network.pub /nix/store/4g9kvmaqs34gisvxgf61p42p665lx8dx-shared-dm-dns_zone.conf --url http://localhost:7331 --key /run/secrets/shared/dm-dns-signing-key/signing.key --name dns/cnames
server: (finished: must succeed: data-mesher file update --network-id /nix/store/xa27jjjrwdjp58yprvlqv8nxdcqcmydh-shared-data-mesher-network_network.pub /nix/store/4g9kvmaqs34gisvxgf61p42p665lx8dx-shared-dm-dns_zone.conf --url http://localhost:7331 --key /run/secrets/shared/dm-dns-signing-key/signing.key --name dns/cnames, in 0.03 seconds)
??? Warning (UserWarning): wait_until_succeeds(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.
    File "/nix/store/5k53g9a603cl82vz5vigz2fdam974b1d-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39
server: waiting for success: dig +short @127.0.0.1 -p 5353 myapp.test CNAME | grep server.test
??? Warning (UserWarning): execute(): passing a bare int/float as a duration is deprecated. Use datetime.timedelta instead.
    File "/nix/store/5k53g9a603cl82vz5vigz2fdam974b1d-nixos-test-driver-1.1/lib/python3.14/site-packages/test_driver/duration.py", line 39
server # [2101837.003706] server systemd[1]: Starting Reload unbound zone configuration...
server # [2101837.004233] server data-mesher[222]: time=2026-09-28T15:17:36.990Z level=INFO msg=http_request uri=/files/dns/cnames status=204
server # [2101837.052955] server unbound[302]: [302:0] info: service stopped (unbound 1.26.0).
server # [2101837.053287] server unbound-control[341]: ok
server # [2101837.053373] server unbound[302]: [302:0] info: server stats for thread 0: 5 queries, 2 answers from cache, 3 recursions, 0 prefetch, 0 rejected by ip ratelimiting
server # [2101837.053378] server unbound[302]: [302:0] info: server stats for thread 0: requestlist max 2 avg 1.33333 exceeded 0 jostled 0
server # [2101837.054502] server systemd[1]: unbound-reload-zones.service: Deactivated successfully.
server # [2101837.054716] server systemd[1]: Finished Reload unbound zone configuration.
server # [2101837.054728] server unbound[302]: [302:0] notice: Restart of unbound 1.26.0.
server # [2101837.055581] server unbound[302]: [302:0] notice: init module 0: validator
server # [2101837.055636] server unbound[302]: [302:0] notice: init module 1: iterator
server # [2101837.059775] server unbound[302]: [302:0] info: start of service (unbound 1.26.0).
server: (finished: waiting for success: dig +short @127.0.0.1 -p 5353 myapp.test CNAME | grep server.test, in 1.05 seconds)
(finished: run the VM test script, in 18.33 seconds)
test script finished in 18.40s
cleanup
kill NspawnMachine (pid 52)
client # [2101837.826613] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=DEBUG msg="attempting push/pull" peer_count=1
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=DEBUG msg="initiating state exchange" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs timeout=5s
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=INFO msg="received state sync from peer" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=INFO msg="merging remote state" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=DEBUG msg="new file detected" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs name=dns/cnames name=dns/cnames
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.812Z level=INFO msg="scheduling file download" name=dns/cnames
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.813Z level=INFO msg="downloading file" name=dns/cnames signed_at="2026-09-28 15:17:36.983 +0000 UTC" signed_by="h2DrT3x3MhJxa6W8GPTEYUiZxFNmpntN04na/RT6qvw=" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.813Z level=INFO msg="merging remote state" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.813Z level=DEBUG msg="new file detected" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs name=dns/cnames name=dns/cnames
client # [2101837.827280] client data-mesher[213]: time=2026-09-28T15:17:37.813Z level=INFO msg="state exchange complete" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs timeout=5s
client # [2101837.827709] client data-mesher[213]: time=2026-09-28T15:17:37.813Z level=DEBUG msg="push/pull successful" interval=5s
client # [2101837.833573] client data-mesher[213]: time=2026-09-28T15:17:37.819Z level=INFO msg="download complete" name=dns/cnames signed_at="2026-09-28 15:17:36.983 +0000 UTC" signed_by="h2DrT3x3MhJxa6W8GPTEYUiZxFNmpntN04na/RT6qvw=" peer=12D3KooWDsHyY5iTVTnuRBfkCBupwbokSS86L6MNChcXiJWXFKVs written=true elapsed=6.397931ms
client # [2101837.833716] client systemd[1]: Starting Reload unbound zone configuration...
client # [2101837.877720] client unbound[292]: [292:0] info: service stopped (unbound 1.26.0).
client # [2101837.878121] client unbound[292]: [292:0] info: server stats for thread 0: 5 queries, 0 answers from cache, 5 recursions, 0 prefetch, 0 rejected by ip ratelimiting
client # [2101837.878127] client unbound[292]: [292:0] info: server stats for thread 0: requestlist max 2 avg 1.6 exceeded 0 jostled 0
client # [2101837.879471] client unbound[292]: [292:0] notice: Restart of unbound 1.26.0.
client # [2101837.879755] client unbound-control[300]: ok
client # [2101837.880350] client unbound[292]: [292:0] notice: init module 0: validator
client # [2101837.880407] client unbound[292]: [292:0] notice: init module 1: iterator
client # [2101837.881019] client systemd[1]: unbound-reload-zones.service: Deactivated successfully.
client # [2101837.881268] client systemd[1]: Finished Reload unbound zone configuration.
client # [2101837.884552] client unbound[292]: [292:0] info: start of service (unbound 1.26.0).
kill NspawnMachine (pid 53)
server # [2101837.826073] server data-mesher[222]: time=2026-09-28T15:17:37.811Z level=DEBUG msg="attempting push/pull" peer_count=1
server # [2101837.826246] server data-mesher[222]: time=2026-09-28T15:17:37.811Z level=DEBUG msg="initiating state exchange" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE timeout=5s
server # [2101837.826990] server data-mesher[222]: time=2026-09-28T15:17:37.812Z level=INFO msg="received state sync from peer" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101837.826990] server data-mesher[222]: time=2026-09-28T15:17:37.812Z level=INFO msg="merging remote state" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101837.827197] server data-mesher[222]: time=2026-09-28T15:17:37.813Z level=INFO msg="merging remote state" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE
server # [2101837.827197] server data-mesher[222]: time=2026-09-28T15:17:37.813Z level=INFO msg="state exchange complete" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE timeout=5s
server # [2101837.827305] server data-mesher[222]: time=2026-09-28T15:17:37.813Z level=DEBUG msg="push/pull successful" interval=5s
server # [2101837.827697] server data-mesher[222]: time=2026-09-28T15:17:37.813Z level=INFO msg="received file request" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE network="Jsv02cI5J+gAg97cqfI9+mLtArUpsAt+ws2DJHjn4xw=" name=dns/cnames
server # [2101837.830076] server data-mesher[222]: time=2026-09-28T15:17:37.815Z level=INFO msg="file transfer complete" peer=12D3KooWQryehCnsAFKqR4PcfoTesq69tVGZjJzr2YhwLbvG7zAE network="Jsv02cI5J+gAg97cqfI9+mLtArUpsAt+ws2DJHjn4xw=" name=dns/cnames
Container client terminated by signal KILL.
(finished: cleanup, in 0.33 seconds)
Container server terminated by signal KILL.