these 32 derivations will be built: /nix/store/1zlf28lha5j8k312g86nf0plx23bykc3-testScript.drv /nix/store/0ka1a77viiapwc04if91k55wfdfa7xb5-user-units.drv /nix/store/0m7a1qdcpl0vl66373n3m326lyaaqncf-system-units.drv /nix/store/xi1jag0nqv42mvqnz26c7nk2zgnipdlz-etc.drv /nix/store/8yi4iqk64jxzx5bz77vgxh49wyb0ds33-activate.drv /nix/store/hvg61yrgf9w9qx4mrb2sp1xscm19xds5-nixos-system-machine2-test.drv /nix/store/0drx5iv2yaanz2r83lh5xv9bpsiddwnr-closure-info.drv /nix/store/789yci37ja9cn9iwnhibgz3nlgy68wq9-datasource.yaml.drv /nix/store/ibbr7sx6l6a2y9cr2csqk8a4hps8hlqq-dashboard.yaml.drv /nix/store/bzi16rghpin40b83h610hjap0915141h-grafana-provisioning.drv /nix/store/1b46dj3pp0kn7yqq6zd424hg7r5m3zrl-config.ini.drv /nix/store/1g86vvvrdml7m98wrrz64yfcl2sifdq4-unit-postgresql.service.drv /nix/store/306s56gydmkb21f6h2i90q76allgs7vv-unit-alloy.service.drv /nix/store/x633khb90nxdh62n1pli6xri77rs9cvc-validate-loki-conf.drv /nix/store/4565nzqhkf4b4vanwlfjffx524gb7l5d-unit-loki.service.drv /nix/store/h60kns5lclr8vwnk0d44cjwha4473dy4-system-path.drv /nix/store/qhx8vmk3f92ndjcmjvffbxq0i4v2s3pc-dbus-1.drv /nix/store/4rkndx5mz3ll11x15lkdqaiawmm3sgan-X-Restart-Triggers-dbus-broker.drv /nix/store/9niyr8ww7cv4lhbv1v38263nash92g3g-unit-dbus-broker.service.drv /nix/store/f02six11ms1ga9wxbmd47nwhn1a8jjrq-user-units.drv /nix/store/p013b69pmqcfs6sxrhq6in51rwm5l7a0-config.yaml.drv /nix/store/7bbkymgrxcalvcidaljx1pj5zx0y4b42-unit-mimir.service.drv /nix/store/9zgpr5ksq6j4chzysh47fcb1kwlkjmcx-unit-dbus-broker.service.drv /nix/store/8vjyqz5yggk3rxci0an1y5bj513rpdv9-unit-script-grafana-start.drv /nix/store/w4ngp7019aln0g29rhansmsq906vsldx-unit-grafana.service.drv /nix/store/mryc78a8766gs5nr89cbf0w7v2p0sb7s-system-units.drv /nix/store/5q0r6i3aag57gplqrjqkkf8n0gm9xyqz-etc.drv /nix/store/k8lm5awzgvaplm5jp0hhqc4fpxcy9hsz-activate.drv /nix/store/p33xs76585hprvnwsfbpv2libddps8mp-nixos-system-machine1-test.drv /nix/store/85h8ggg7q507pgvr3nappfr1dshy1rvq-closure-info.drv /nix/store/s7x125zdwplh2p8yz936zgwq2xh2ls8w-nixos-test-driver-monitoring.drv /nix/store/nd9fb4gmkl3khyyc67n86szi11q4sza8-container-test-run-monitoring.drv these 6 paths will be fetched (115.5 MiB download, 405.9 MiB unpacked): /nix/store/h37m8hkcnylfc9082gyw8fsyhfgc3vg8-contactPoints.yaml /nix/store/gf1dg5qjpb23206lxc0mnr9ljy5rkab8-grafana-loki-3.7.7 /nix/store/gnmsbch2m9l565dj4k6zkbi0i1c9bj5p-muteTimings.yaml /nix/store/k8px4b0ymn0y43i4h83qc4jk6ca5hfgx-policies.yaml /nix/store/01hdzbv7v8m03j7pqhw8hk51p97jcbnf-rules.yaml /nix/store/3g1hqjm5476x8af3s1wl5s3zkns2xnr4-templates.yaml building '/nix/store/0m7a1qdcpl0vl66373n3m326lyaaqncf-system-units.drv' building '/nix/store/0ka1a77viiapwc04if91k55wfdfa7xb5-user-units.drv' building '/nix/store/306s56gydmkb21f6h2i90q76allgs7vv-unit-alloy.service.drv' building '/nix/store/1g86vvvrdml7m98wrrz64yfcl2sifdq4-unit-postgresql.service.drv' building '/nix/store/p013b69pmqcfs6sxrhq6in51rwm5l7a0-config.yaml.drv' building '/nix/store/1zlf28lha5j8k312g86nf0plx23bykc3-testScript.drv' config.yaml> structuredAttrs is enabled building '/nix/store/ibbr7sx6l6a2y9cr2csqk8a4hps8hlqq-dashboard.yaml.drv' building '/nix/store/789yci37ja9cn9iwnhibgz3nlgy68wq9-datasource.yaml.drv' dashboard.yaml> structuredAttrs is enabled building '/nix/store/7bbkymgrxcalvcidaljx1pj5zx0y4b42-unit-mimir.service.drv' datasource.yaml> structuredAttrs is enabled building '/nix/store/bzi16rghpin40b83h610hjap0915141h-grafana-provisioning.drv' on 'ssh-ng://builder@build-x86-01.clan.lol' building '/nix/store/xi1jag0nqv42mvqnz26c7nk2zgnipdlz-etc.drv' building '/nix/store/8yi4iqk64jxzx5bz77vgxh49wyb0ds33-activate.drv' building '/nix/store/hvg61yrgf9w9qx4mrb2sp1xscm19xds5-nixos-system-machine2-test.drv' nixos-system-machine2-test> structuredAttrs is enabled building '/nix/store/0drx5iv2yaanz2r83lh5xv9bpsiddwnr-closure-info.drv' closure-info> structuredAttrs is enabled building '/nix/store/x633khb90nxdh62n1pli6xri77rs9cvc-validate-loki-conf.drv' on 'ssh-ng://builder@build-x86-01.clan.lol' building '/nix/store/h60kns5lclr8vwnk0d44cjwha4473dy4-system-path.drv' system-path> structuredAttrs is enabled system-path> created 1965 symlinks in user environment building '/nix/store/qhx8vmk3f92ndjcmjvffbxq0i4v2s3pc-dbus-1.drv' building '/nix/store/4rkndx5mz3ll11x15lkdqaiawmm3sgan-X-Restart-Triggers-dbus-broker.drv' building '/nix/store/9niyr8ww7cv4lhbv1v38263nash92g3g-unit-dbus-broker.service.drv' building '/nix/store/9zgpr5ksq6j4chzysh47fcb1kwlkjmcx-unit-dbus-broker.service.drv' building '/nix/store/f02six11ms1ga9wxbmd47nwhn1a8jjrq-user-units.drv' building '/nix/store/bzi16rghpin40b83h610hjap0915141h-grafana-provisioning.drv' building '/nix/store/1b46dj3pp0kn7yqq6zd424hg7r5m3zrl-config.ini.drv' building '/nix/store/8vjyqz5yggk3rxci0an1y5bj513rpdv9-unit-script-grafana-start.drv' building '/nix/store/w4ngp7019aln0g29rhansmsq906vsldx-unit-grafana.service.drv' building '/nix/store/x633khb90nxdh62n1pli6xri77rs9cvc-validate-loki-conf.drv' validate-loki-conf> level=info ts=2026-09-10T00:09:47.148196862Z caller=main.go:109 msg="config is valid" building '/nix/store/4565nzqhkf4b4vanwlfjffx524gb7l5d-unit-loki.service.drv' building '/nix/store/mryc78a8766gs5nr89cbf0w7v2p0sb7s-system-units.drv' building '/nix/store/5q0r6i3aag57gplqrjqkkf8n0gm9xyqz-etc.drv' building '/nix/store/k8lm5awzgvaplm5jp0hhqc4fpxcy9hsz-activate.drv' building '/nix/store/p33xs76585hprvnwsfbpv2libddps8mp-nixos-system-machine1-test.drv' nixos-system-machine1-test> structuredAttrs is enabled building '/nix/store/85h8ggg7q507pgvr3nappfr1dshy1rvq-closure-info.drv' closure-info> structuredAttrs is enabled building '/nix/store/s7x125zdwplh2p8yz936zgwq2xh2ls8w-nixos-test-driver-monitoring.drv' nixos-test-driver-monitoring> Running type check (enable/disable: config.skipTypeCheck) nixos-test-driver-monitoring> See https://nixos.org/manual/nixos/stable/#test-opt-skipTypeCheck nixos-test-driver-monitoring> Success: no issues found in 1 source file building '/nix/store/nd9fb4gmkl3khyyc67n86szi11q4sza8-container-test-run-monitoring.drv' container-test-run-monitoring> additionally exposed symbols: container-test-run-monitoring> machine1, machine2, container-test-run-monitoring> start_all, machines, driver, Machine, wait_for_signal container-test-run-monitoring> Starting machine1 container-test-run-monitoring> Starting machine2 container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> <<< NixOS Stage 2 >>> container-test-run-monitoring> container-test-run-monitoring> booting system configuration /nix/store/mr98mdycga9gpx9c9lqaiz3hc8vzxpy6-nixos-system-machine1-test container-test-run-monitoring> running activation script... container-test-run-monitoring> setting up /etc... container-test-run-monitoring> 2: host0@if3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 container-test-run-monitoring> link/ether 5a:94:bb:65:1f:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 container-test-run-monitoring> setting up age secrets... container-test-run-monitoring> starting systemd... container-test-run-monitoring> systemd 260.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) container-test-run-monitoring> Detected virtualization systemd-nspawn. container-test-run-monitoring> Detected architecture x86-64. container-test-run-monitoring> Detected first boot. container-test-run-monitoring> Initializing machine ID from container UUID. container-test-run-monitoring> Applying preset policy. container-test-run-monitoring> Populated /etc with preset unit settings. container-test-run-monitoring> Queued start job for default target Multi-User System. container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> <<< Welcome to NixOS test (x86_64) - console >>> container-test-run-monitoring> + systemd-run /bin/sh -c '/nix/store/61685rbaxmpigwi3z23i33sv3jyfl70q-coreutils-9.11/bin/sleep 999999999 && echo e3e4641b-2ec6-497f-97ef-1581df34a228' container-test-run-monitoring> Running as unit: run-p589-i14751627.service; invocation ID: 1438f0d9093d4261940988b597a6d9a8 container-test-run-monitoring> To attach to container machine1 run on the same machine that runs the test: container-test-run-monitoring> sudo nsenter --user --target $(\pgrep -f '^/bin/sh.*e3e4641b-2ec6-497f-97ef-1581df34a228') --mount --uts --ipc --net --pid --cgroup /bin/sh -c bash container-test-run-monitoring> container-test-run-monitoring> To inject external network and continue test, run: container-test-run-monitoring> sudo /nix/store/h3l4z7p6wny3phbckwwhy1i2g52pdnj4-python3-3.13.15/bin/python3.13 /nix/store/f960jm4nawb9zr6cb9d0vfpb7hifp96b-test-driver-0.0.1/lib/python3.13/site-packages/test_driver/inject_network.py e3e4641b-2ec6-497f-97ef-1581df34a228 container-test-run-monitoring> container-test-run-monitoring> <<< NixOS Stage 2 >>> container-test-run-monitoring> container-test-run-monitoring> booting system configuration /nix/store/i32jh1yhkp2zq35dcrr2x4yvq05ijyab-nixos-system-machine2-test container-test-run-monitoring> running activation script... container-test-run-monitoring> setting up /etc... container-test-run-monitoring> 2: host0@if4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 container-test-run-monitoring> link/ether 46:8d:8e:53:39:af brd ff:ff:ff:ff:ff:ff link-netnsid 0 container-test-run-monitoring> setting up age secrets... container-test-run-monitoring> starting systemd... container-test-run-monitoring> systemd 260.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) container-test-run-monitoring> Detected virtualization systemd-nspawn. container-test-run-monitoring> Detected architecture x86-64. container-test-run-monitoring> Detected first boot. container-test-run-monitoring> Initializing machine ID from container UUID. container-test-run-monitoring> Applying preset policy. container-test-run-monitoring> Populated /etc with preset unit settings. container-test-run-monitoring> Queued start job for default target Multi-User System. container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> container-test-run-monitoring> <<< Welcome to NixOS test (x86_64) - console >>> container-test-run-monitoring> + systemd-run /bin/sh -c '/nix/store/61685rbaxmpigwi3z23i33sv3jyfl70q-coreutils-9.11/bin/sleep 999999999 && echo 8c79af38-aeb4-44ae-9d9e-d2bc4f831212' container-test-run-monitoring> Running as unit: run-p378-i14751675.service; invocation ID: 57db7983556b4c3680e84c19b40817f8 container-test-run-monitoring> To attach to container machine2 run on the same machine that runs the test: container-test-run-monitoring> sudo nsenter --user --target $(\pgrep -f '^/bin/sh.*8c79af38-aeb4-44ae-9d9e-d2bc4f831212') --mount --uts --ipc --net --pid --cgroup /bin/sh -c bash container-test-run-monitoring> container-test-run-monitoring> To inject external network and continue test, run: container-test-run-monitoring> sudo /nix/store/h3l4z7p6wny3phbckwwhy1i2g52pdnj4-python3-3.13.15/bin/python3.13 /nix/store/f960jm4nawb9zr6cb9d0vfpb7hifp96b-test-driver-0.0.1/lib/python3.13/site-packages/test_driver/inject_network.py 8c79af38-aeb4-44ae-9d9e-d2bc4f831212 container-test-run-monitoring> + systemctl --no-pager show multi-user.target container-test-run-monitoring> + systemctl list-jobs --full container-test-run-monitoring> + systemctl --no-pager show multi-user.target container-test-run-monitoring> + systemctl --no-pager show multi-user.target container-test-run-monitoring> + systemctl --no-pager show alloy container-test-run-monitoring> + systemctl --no-pager show alloy container-test-run-monitoring> + test -f /etc/alloy/config.alloy container-test-run-monitoring> + test -f /etc/alloy/config.alloy container-test-run-monitoring> + test -f /etc/alloy/local-extension.alloy container-test-run-monitoring> + systemctl show alloy --property=ExecStart container-test-run-monitoring> + grep -F /etc/alloy container-test-run-monitoring> + systemctl show alloy --property=ExecStart container-test-run-monitoring> + grep -F /etc/alloy container-test-run-monitoring> + systemctl --no-pager show loki container-test-run-monitoring> + systemctl --no-pager show mimir container-test-run-monitoring> + systemctl --no-pager show grafana container-test-run-monitoring> ++ grep -c '^loki.source.journal ' /etc/alloy/config.alloy container-test-run-monitoring> + test 1 = 1 container-test-run-monitoring> + cat /etc/alloy/config.alloy container-test-run-monitoring> container-test-run-monitoring> upload niks3: failed time=2026-09-10T00:34:58.594Z level=WARN msg="Request returned retryable status, retrying" attempt=1 max_attempts=6 backoff=100ms status=502 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T00:36:03.546Z level=WARN msg="Request returned retryable status, retrying" attempt=2 max_attempts=6 backoff=187.155328ms status=502 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T00:41:38.531Z level=WARN msg="Request returned retryable status, retrying" attempt=1 max_attempts=6 backoff=100ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T00:46:38.632Z level=WARN msg="Request returned retryable status, retrying" attempt=2 max_attempts=6 backoff=182.285456ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T00:52:05.427Z level=WARN msg="Request returned retryable status, retrying" attempt=1 max_attempts=6 backoff=100ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T00:57:05.530Z level=WARN msg="Request returned retryable status, retrying" attempt=2 max_attempts=6 backoff=216.110839ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T01:02:05.748Z level=WARN msg="Request returned retryable status, retrying" attempt=3 max_attempts=6 backoff=374.38175ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T01:07:06.123Z level=WARN msg="Request returned retryable status, retrying" attempt=4 max_attempts=6 backoff=789.765369ms status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T01:12:06.915Z level=WARN msg="Request returned retryable status, retrying" attempt=5 max_attempts=6 backoff=1.610297597s status=504 url=https://niks3.clan.lol/api/pending_closures time=2026-09-10T01:17:08.527Z level=ERROR msg="Fatal error" error="pushing paths: creating pending closures: creating pending closure: server returned 504: "