these 57 derivations will be built: /nix/store/0dsq5c1zkd7idyvdrgmli9cnlvwf1anq-postgres-db-restore-command-grafana.drv /nix/store/0xzabzz0sm8bkkbvwv5njmgljwkhlgan-state-commands.drv /nix/store/ziv4qwp8cy60dhpl2cll67lv7jkqdkiz-config.yaml.drv /nix/store/2ad3c7npgvampjrg56fvirv6kchck7c5-unit-mimir.service.drv /nix/store/dm6v205q8jwdf70pkzsdv7i9xp9jv8w1-nginx.conf.drv /nix/store/8fa01983lqydf9awxjqzzvgwgnmdz335-unit-script-nginx-pre-start.drv /nix/store/42pr0qspjjra069psgm8rkvi1q6pg5sx-unit-nginx.service.drv /nix/store/rvc33vgxhm2x5p3k66h9x4iccpg8mal6-system-path.drv /nix/store/syhw7gs16fdpnws12k1wfjn35l70scwk-dbus-1.drv /nix/store/70si73cgij4m9h5khb55iw1k71a4nz8h-X-Restart-Triggers-dbus-broker.drv /nix/store/b05c3cf7vlw4qjmjw8my5r9rw5lxck18-unit-dbus-broker.service.drv /nix/store/x93fqdffh89r9zscc8k86rawnrja487r-X-Reload-Triggers-alloy.drv /nix/store/h2az05n00msixhcv4llqspxdjwljqq36-unit-alloy.service.drv /nix/store/zx0q5mszq0y25k598rv919zvp55b9h5a-firewall-start.drv /nix/store/31v0kib6i2cdk5jv3lzzsv0c4gn0549d-firewall-reload.drv /nix/store/hzdcyd37wl1zx8aqbbkxq9ld7pcnyzy1-unit-firewall.service.drv /nix/store/gnlbrdsp8dsmqh9qn6b9whciy6b5m03j-loki-config.json.drv /nix/store/nq1zm846s1nddds8jvxcqv45jp6118g0-validate-loki-conf.drv /nix/store/l6194hln1402nd706dc5mhmglaqb5r8s-unit-loki.service.drv /nix/store/qar232490kx90mgbvakh95qwm2vbbkph-datasource.yaml.drv /nix/store/z63x2jam3a8v9aks2hsqhgwkldx6j3bi-dashboard.yaml.drv /nix/store/66gy39m6xh6lk42lwv648shqzfj9p1hk-grafana-provisioning.drv /nix/store/dmarxx4v3bag0ppglzsryi8s2gc3jlsc-config.ini.drv /nix/store/makq0g7xr15gfvdlwgqjpz50b8d715r6-unit-grafana.service.drv /nix/store/yxph0kqymq94bq1vprdn9291ki9rys7v-unit-script-postgresql-post-start.drv /nix/store/ng00zf1gfk9wp3dgi4jvfliqcps4vmzd-unit-postgresql.service.drv /nix/store/x0ggr6firn8xfgpb4lb8h0jcnr08y5ny-unit-systemd-tmpfiles-resetup.service.drv /nix/store/1zbfgw1zj97vi4h1ywwdl6ppssmknlqk-system-units.drv /nix/store/k7dla93g1dwafjpxcbr22vx618vhwcfb-string-hosts.drv /nix/store/nf4wbsq66xrj9x51vgpyn17rpx3jq04x-extra-hosts.drv /nix/store/4cfjb63qsyy0il9zxqv8yzzhr7jpggcx-hosts.drv /nix/store/fwf3kycjw6z5ypjk1a267ahvw09028w8-vars-check-monitoring.drv /nix/store/w17x26jc1bqzbigdmm1iz6b7niyfjwcz-unit-dbus-broker.service.drv /nix/store/i4rnrwqsxr24i762ajrfxpijncnwpli1-user-units.drv /nix/store/0zfwwj02f2mcd3p6wxf730qgqk8yz8cw-etc.drv /nix/store/pmz19j5vh9887j0i7fkw80qkkg4mby6x-etc-alloy-local-extension.alloy.drv /nix/store/1iv69qcaxiwh2dxk5m6jbn9xr2l1gymg-X-Reload-Triggers-alloy.drv /nix/store/gz0jqicqzaw5idm5jyx86h8c23bhh61h-string-hosts.drv /nix/store/23azfpwip7kbvlfvish3qwr15m03h85r-hosts.drv /nix/store/xvgfia759zwa8p7yza3pkn6k5p0dzb4b-unit-alloy.service.drv /nix/store/aml404ag271q7m7fzkw58x6pk3hx6jxw-system-units.drv /nix/store/hhaf88bw9diczrfjr3hw33zil6qgng9x-etc.drv /nix/store/hjxg4hx6szc256ww1l2lsq2bdpjvplzl-decrypt-age-secrets.drv /nix/store/ldgv6rd8978w758rz18xpc6dkzajj3p8-activate.drv /nix/store/z7a1zvm648rldmsm8wyfq50rnr5k8xx4-dry-activate.drv /nix/store/vmjh7fy70ssxxmjrvcyr4w5y79xfaw99-nixos-system-machine2-test.drv /nix/store/7w0dzgipgfls2pg45g9d9qdn2agd0fl8-run-machine2-nspawn.drv /nix/store/mzy6z2djmjw6hqsb8r0nkpd0nwvjyj43-test-script.drv /nix/store/fqr45x8x5ihyy2yczybl1fg2rppzcrv5-users-groups.json.drv /nix/store/jd4limj2j755fpykgja82pbpp04dl6yy-decrypt-age-secrets.drv /nix/store/8yihzfpchkwwqlcsmx3nhg8vlkb5c11v-activate.drv /nix/store/i9qqh3ysvsm3zfsnijb8d4054miwrnmp-dry-activate.drv /nix/store/970mmii8g02yranxkzf6r1pk91rf3qyw-nixos-system-machine1-test.drv /nix/store/syxld3z9c724xn0nmgy9dvsgjwqnwvyk-run-machine1-nspawn.drv /nix/store/d093g1narjfdn225gnqxbvv1yq9shfk0-driverConfiguration.json.drv /nix/store/3494zhknp5jzhrh3hz0lhbl8kzaj13p5-nixos-test-driver-monitoring.drv /nix/store/42phjg93y4h2h9hzmi19ldmlkx3hfwjz-container-test-run-monitoring.drv these 9 paths will be fetched (572.3 MiB download, 2.2 GiB unpacked): /nix/store/w0i04v97mqp3snh6gyzzzm77zkqbymcg-contactPoints.yaml /nix/store/0ibj0xmpy88h3a02snv6r529l63cczcn-grafana-13.1.3 /nix/store/2x4bibxhxf0ds7204wh20x66vxb5jikf-grafana-alloy-1.17.1 /nix/store/2as9022q8ki42c4jb7mgwk0c0nxl8v53-grafana-loki-3.7.6 /nix/store/lknk141xgrxbxlg99xllcz9myn4zscxi-mimir-3.1.4 /nix/store/7s8qhn91aj93v6jjniqxxb1qp6kl3ld8-muteTimings.yaml /nix/store/7vwgadwxmwhizzhrsshncia39fng7yjz-policies.yaml /nix/store/kb2n6sh9jbmz11czm7142imhj2001cjq-rules.yaml /nix/store/mfx46y3k28rkvwzr7vx551fkqfs369af-templates.yaml building '/nix/store/mzy6z2djmjw6hqsb8r0nkpd0nwvjyj43-test-script.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/pmz19j5vh9887j0i7fkw80qkkg4mby6x-etc-alloy-local-extension.alloy.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/dm6v205q8jwdf70pkzsdv7i9xp9jv8w1-nginx.conf.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/0dsq5c1zkd7idyvdrgmli9cnlvwf1anq-postgres-db-restore-command-grafana.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/nf4wbsq66xrj9x51vgpyn17rpx3jq04x-extra-hosts.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/gz0jqicqzaw5idm5jyx86h8c23bhh61h-string-hosts.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/k7dla93g1dwafjpxcbr22vx618vhwcfb-string-hosts.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/x93fqdffh89r9zscc8k86rawnrja487r-X-Reload-Triggers-alloy.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/zx0q5mszq0y25k598rv919zvp55b9h5a-firewall-start.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/yxph0kqymq94bq1vprdn9291ki9rys7v-unit-script-postgresql-post-start.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/gz0jqicqzaw5idm5jyx86h8c23bhh61h-string-hosts.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/k7dla93g1dwafjpxcbr22vx618vhwcfb-string-hosts.drv' building '/nix/store/nf4wbsq66xrj9x51vgpyn17rpx3jq04x-extra-hosts.drv' building '/nix/store/pmz19j5vh9887j0i7fkw80qkkg4mby6x-etc-alloy-local-extension.alloy.drv' building '/nix/store/x93fqdffh89r9zscc8k86rawnrja487r-X-Reload-Triggers-alloy.drv' building '/nix/store/mzy6z2djmjw6hqsb8r0nkpd0nwvjyj43-test-script.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/0dsq5c1zkd7idyvdrgmli9cnlvwf1anq-postgres-db-restore-command-grafana.drv' building '/nix/store/ziv4qwp8cy60dhpl2cll67lv7jkqdkiz-config.yaml.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/23azfpwip7kbvlfvish3qwr15m03h85r-hosts.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/4cfjb63qsyy0il9zxqv8yzzhr7jpggcx-hosts.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/1iv69qcaxiwh2dxk5m6jbn9xr2l1gymg-X-Reload-Triggers-alloy.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/dm6v205q8jwdf70pkzsdv7i9xp9jv8w1-nginx.conf.drv' nginx.conf> structuredAttrs is enabled nginx.conf> nginx.conf> ==================== Results =================== nginx.conf> No issues found. nginx.conf> nginx.conf> ==================== Summary =================== nginx.conf> Total issues: nginx.conf> Unspecified: 0 nginx.conf> Low: 0 nginx.conf> Medium: 0 nginx.conf> High: 0 nginx.conf> building '/nix/store/yxph0kqymq94bq1vprdn9291ki9rys7v-unit-script-postgresql-post-start.drv' building '/nix/store/h2az05n00msixhcv4llqspxdjwljqq36-unit-alloy.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/zx0q5mszq0y25k598rv919zvp55b9h5a-firewall-start.drv' building '/nix/store/8fa01983lqydf9awxjqzzvgwgnmdz335-unit-script-nginx-pre-start.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/31v0kib6i2cdk5jv3lzzsv0c4gn0549d-firewall-reload.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/z63x2jam3a8v9aks2hsqhgwkldx6j3bi-dashboard.yaml.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/qar232490kx90mgbvakh95qwm2vbbkph-datasource.yaml.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/4cfjb63qsyy0il9zxqv8yzzhr7jpggcx-hosts.drv' building '/nix/store/ziv4qwp8cy60dhpl2cll67lv7jkqdkiz-config.yaml.drv' config.yaml> structuredAttrs is enabled config.yaml> building '/nix/store/23azfpwip7kbvlfvish3qwr15m03h85r-hosts.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/2ad3c7npgvampjrg56fvirv6kchck7c5-unit-mimir.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/1iv69qcaxiwh2dxk5m6jbn9xr2l1gymg-X-Reload-Triggers-alloy.drv' building '/nix/store/hjxg4hx6szc256ww1l2lsq2bdpjvplzl-decrypt-age-secrets.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/xvgfia759zwa8p7yza3pkn6k5p0dzb4b-unit-alloy.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/31v0kib6i2cdk5jv3lzzsv0c4gn0549d-firewall-reload.drv' building '/nix/store/8fa01983lqydf9awxjqzzvgwgnmdz335-unit-script-nginx-pre-start.drv' building '/nix/store/42pr0qspjjra069psgm8rkvi1q6pg5sx-unit-nginx.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/z63x2jam3a8v9aks2hsqhgwkldx6j3bi-dashboard.yaml.drv' dashboard.yaml> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy dashboard.yaml> building '/nix/store/qar232490kx90mgbvakh95qwm2vbbkph-datasource.yaml.drv' datasource.yaml> structuredAttrs is enabled datasource.yaml> building '/nix/store/h2az05n00msixhcv4llqspxdjwljqq36-unit-alloy.service.drv' unit-alloy.service> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/66gy39m6xh6lk42lwv648shqzfj9p1hk-grafana-provisioning.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/jd4limj2j755fpykgja82pbpp04dl6yy-decrypt-age-secrets.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/2ad3c7npgvampjrg56fvirv6kchck7c5-unit-mimir.service.drv' unit-mimir.service> structuredAttrs is enabled building '/nix/store/42pr0qspjjra069psgm8rkvi1q6pg5sx-unit-nginx.service.drv' unit-nginx.service> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/xvgfia759zwa8p7yza3pkn6k5p0dzb4b-unit-alloy.service.drv' unit-alloy.service> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/aml404ag271q7m7fzkw58x6pk3hx6jxw-system-units.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/hjxg4hx6szc256ww1l2lsq2bdpjvplzl-decrypt-age-secrets.drv' building '/nix/store/z7a1zvm648rldmsm8wyfq50rnr5k8xx4-dry-activate.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/66gy39m6xh6lk42lwv648shqzfj9p1hk-grafana-provisioning.drv' building '/nix/store/dmarxx4v3bag0ppglzsryi8s2gc3jlsc-config.ini.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/jd4limj2j755fpykgja82pbpp04dl6yy-decrypt-age-secrets.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/z7a1zvm648rldmsm8wyfq50rnr5k8xx4-dry-activate.drv' building '/nix/store/gnlbrdsp8dsmqh9qn6b9whciy6b5m03j-loki-config.json.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/aml404ag271q7m7fzkw58x6pk3hx6jxw-system-units.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/dmarxx4v3bag0ppglzsryi8s2gc3jlsc-config.ini.drv' building '/nix/store/makq0g7xr15gfvdlwgqjpz50b8d715r6-unit-grafana.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/gnlbrdsp8dsmqh9qn6b9whciy6b5m03j-loki-config.json.drv' building '/nix/store/nq1zm846s1nddds8jvxcqv45jp6118g0-validate-loki-conf.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/0xzabzz0sm8bkkbvwv5njmgljwkhlgan-state-commands.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/hzdcyd37wl1zx8aqbbkxq9ld7pcnyzy1-unit-firewall.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/ng00zf1gfk9wp3dgi4jvfliqcps4vmzd-unit-postgresql.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/fqr45x8x5ihyy2yczybl1fg2rppzcrv5-users-groups.json.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/makq0g7xr15gfvdlwgqjpz50b8d715r6-unit-grafana.service.drv' unit-grafana.service> structuredAttrs is enabled building '/nix/store/nq1zm846s1nddds8jvxcqv45jp6118g0-validate-loki-conf.drv' validate-loki-conf> level=info ts=2026-08-18T15:04:16.977450369Z caller=main.go:109 msg="config is valid" building '/nix/store/l6194hln1402nd706dc5mhmglaqb5r8s-unit-loki.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/fwf3kycjw6z5ypjk1a267ahvw09028w8-vars-check-monitoring.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/fqr45x8x5ihyy2yczybl1fg2rppzcrv5-users-groups.json.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/i9qqh3ysvsm3zfsnijb8d4054miwrnmp-dry-activate.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/l6194hln1402nd706dc5mhmglaqb5r8s-unit-loki.service.drv' unit-loki.service> structuredAttrs is enabled building '/nix/store/ng00zf1gfk9wp3dgi4jvfliqcps4vmzd-unit-postgresql.service.drv' unit-postgresql.service> structuredAttrs is enabled building '/nix/store/0xzabzz0sm8bkkbvwv5njmgljwkhlgan-state-commands.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/hzdcyd37wl1zx8aqbbkxq9ld7pcnyzy1-unit-firewall.service.drv' unit-firewall.service> structuredAttrs is enabled building '/nix/store/rvc33vgxhm2x5p3k66h9x4iccpg8mal6-system-path.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/fwf3kycjw6z5ypjk1a267ahvw09028w8-vars-check-monitoring.drv' vars-check-monitoring> Running vars check using Nix-based executor... vars-check-monitoring> ✓ All vars checks completed successfully building '/nix/store/hhaf88bw9diczrfjr3hw33zil6qgng9x-etc.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/i9qqh3ysvsm3zfsnijb8d4054miwrnmp-dry-activate.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/rvc33vgxhm2x5p3k66h9x4iccpg8mal6-system-path.drv' system-path> structuredAttrs is enabled system-path> created 1868 symlinks in user environment building '/nix/store/hhaf88bw9diczrfjr3hw33zil6qgng9x-etc.drv' building '/nix/store/ldgv6rd8978w758rz18xpc6dkzajj3p8-activate.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/syhw7gs16fdpnws12k1wfjn35l70scwk-dbus-1.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/ldgv6rd8978w758rz18xpc6dkzajj3p8-activate.drv' building '/nix/store/syhw7gs16fdpnws12k1wfjn35l70scwk-dbus-1.drv' building '/nix/store/70si73cgij4m9h5khb55iw1k71a4nz8h-X-Restart-Triggers-dbus-broker.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/vmjh7fy70ssxxmjrvcyr4w5y79xfaw99-nixos-system-machine2-test.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/vmjh7fy70ssxxmjrvcyr4w5y79xfaw99-nixos-system-machine2-test.drv' nixos-system-machine2-test> structuredAttrs is enabled building '/nix/store/70si73cgij4m9h5khb55iw1k71a4nz8h-X-Restart-Triggers-dbus-broker.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/b05c3cf7vlw4qjmjw8my5r9rw5lxck18-unit-dbus-broker.service.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/w17x26jc1bqzbigdmm1iz6b7niyfjwcz-unit-dbus-broker.service.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/b05c3cf7vlw4qjmjw8my5r9rw5lxck18-unit-dbus-broker.service.drv' unit-dbus-broker.service> structuredAttrs is enabled warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/1zbfgw1zj97vi4h1ywwdl6ppssmknlqk-system-units.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/7w0dzgipgfls2pg45g9d9qdn2agd0fl8-run-machine2-nspawn.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/1zbfgw1zj97vi4h1ywwdl6ppssmknlqk-system-units.drv' building '/nix/store/w17x26jc1bqzbigdmm1iz6b7niyfjwcz-unit-dbus-broker.service.drv' unit-dbus-broker.service> structuredAttrs is enabled building '/nix/store/i4rnrwqsxr24i762ajrfxpijncnwpli1-user-units.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/7w0dzgipgfls2pg45g9d9qdn2agd0fl8-run-machine2-nspawn.drv' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/i4rnrwqsxr24i762ajrfxpijncnwpli1-user-units.drv' building '/nix/store/0zfwwj02f2mcd3p6wxf730qgqk8yz8cw-etc.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/0zfwwj02f2mcd3p6wxf730qgqk8yz8cw-etc.drv' building '/nix/store/8yihzfpchkwwqlcsmx3nhg8vlkb5c11v-activate.drv' on 'ssh-ng://builder@build01.clan.lol' warning: SQLite database '/nix/var/nix/db/db.sqlite' is busy building '/nix/store/8yihzfpchkwwqlcsmx3nhg8vlkb5c11v-activate.drv' building '/nix/store/970mmii8g02yranxkzf6r1pk91rf3qyw-nixos-system-machine1-test.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/970mmii8g02yranxkzf6r1pk91rf3qyw-nixos-system-machine1-test.drv' nixos-system-machine1-test> structuredAttrs is enabled building '/nix/store/syxld3z9c724xn0nmgy9dvsgjwqnwvyk-run-machine1-nspawn.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/syxld3z9c724xn0nmgy9dvsgjwqnwvyk-run-machine1-nspawn.drv' building '/nix/store/d093g1narjfdn225gnqxbvv1yq9shfk0-driverConfiguration.json.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/d093g1narjfdn225gnqxbvv1yq9shfk0-driverConfiguration.json.drv' driverConfiguration.json> structuredAttrs is enabled building '/nix/store/3494zhknp5jzhrh3hz0lhbl8kzaj13p5-nixos-test-driver-monitoring.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/3494zhknp5jzhrh3hz0lhbl8kzaj13p5-nixos-test-driver-monitoring.drv' nixos-test-driver-monitoring> Running type check (enable/disable: config.skipTypeCheck) nixos-test-driver-monitoring> See https://nixos.org/manual/nixos/stable/#test-opt-skipTypeCheck nixos-test-driver-monitoring> All checks passed! nixos-test-driver-monitoring> Linting test script (enable/disable: config.skipLint) nixos-test-driver-monitoring> See https://nixos.org/manual/nixos/stable/#test-opt-skipLint nixos-test-driver-monitoring> All checks passed! building '/nix/store/42phjg93y4h2h9hzmi19ldmlkx3hfwjz-container-test-run-monitoring.drv' on 'ssh-ng://builder@build01.clan.lol' building '/nix/store/42phjg93y4h2h9hzmi19ldmlkx3hfwjz-container-test-run-monitoring.drv' container-test-run-monitoring> Machine state will be reset. To keep it, pass --keep-machine-state container-test-run-monitoring> start all VLans container-test-run-monitoring> (finished: start all VLans, in 0.00 seconds) container-test-run-monitoring> container-test-run-monitoring> Test will time out and terminate in 3600.0 seconds container-test-run-monitoring> run the VM test script container-test-run-monitoring> additionally exposed symbols: container-test-run-monitoring> machine1, machine2, container-test-run-monitoring> vlan1, container-test-run-monitoring> start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh container-test-run-monitoring> start all VMs container-test-run-monitoring> machine2: systemd-nspawn running (pid 52) container-test-run-monitoring> machine2: Waiting for journal at /build/vm-state-machine2/var/log/journal... container-test-run-monitoring> machine1: systemd-nspawn running (pid 55) container-test-run-monitoring> machine1: Waiting for journal at /build/vm-state-machine1/var/log/journal... container-test-run-monitoring> (finished: start all VMs, in 0.00 seconds) container-test-run-monitoring> machine1: waiting for unit multi-user.target container-test-run-monitoring> nixos-nspawn(machine2): TAP vde-tap1 not found; container will be isolated from VDE container-test-run-monitoring> nixos-nspawn(machine2): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. container-test-run-monitoring> nixos-nspawn(machine1): TAP vde-tap1 not found; container will be isolated from VDE container-test-run-monitoring> nixos-nspawn(machine1): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. container-test-run-monitoring> Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. container-test-run-monitoring> Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. container-test-run-monitoring> ░ Spawning container machine2 on /build/vm-state-machine2. container-test-run-monitoring> ░ Spawning container machine1 on /build/vm-state-machine1. container-test-run-monitoring> machine2 # [6104607.333485] machine2 systemd-journald[105]: Journal started container-test-run-monitoring> machine2 # [6104607.333550] machine2 systemd-journald[105]: Runtime Journal (/run/log/journal/c5ec279aec6a46b5b8e5e1c14b1b2b7e) is 8M, max 2.5G, 2.4G free. container-test-run-monitoring> machine2 # [6104607.342582] machine2 systemd[1]: Finished Create Static Device Nodes in /dev gracefully. container-test-run-monitoring> machine2 # [6104607.352841] machine2 systemd[1]: Starting Flush Journal to Persistent Storage... container-test-run-monitoring> machine2 # [6104607.353952] machine2 systemd[1]: Starting Network Name Resolution... container-test-run-monitoring> machine2 # [6104607.354910] machine2 systemd[1]: Starting Create Static Device Nodes in /dev... container-test-run-monitoring> machine2 # [6104607.363680] machine2 systemd-journald[105]: Time spent on flushing to /var/log/journal/c5ec279aec6a46b5b8e5e1c14b1b2b7e is 1.940ms for 6 entries. container-test-run-monitoring> machine2 # [6104607.363680] machine2 systemd-journald[105]: System Journal (/var/log/journal/c5ec279aec6a46b5b8e5e1c14b1b2b7e) is 8M, max 4G, 3.9G free. container-test-run-monitoring> machine2 # [6104607.376302] machine2 systemd[1]: Finished Create Static Device Nodes in /dev. container-test-run-monitoring> machine2 # [6104607.377562] machine2 systemd[1]: Reached target Preparation for Local File Systems. container-test-run-monitoring> machine2 # [6104607.378581] machine2 systemd[1]: Reached target Local File Systems. container-test-run-monitoring> machine2 # [6104607.379662] machine2 systemd[1]: Listening on Boot Loader Control Service Socket. container-test-run-monitoring> machine2 # [6104607.379719] machine2 systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine2 # [6104607.381119] machine2 systemd[1]: Starting Save Transient machine-id to Disk... container-test-run-monitoring> machine2 # [6104607.381172] machine2 systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys container-test-run-monitoring> machine2 # [6104607.382042] machine2 systemd[1]: Finished Flush Journal to Persistent Storage. container-test-run-monitoring> machine2 # [6104607.385221] machine2 systemd[1]: Starting Create System Files and Directories... container-test-run-monitoring> machine2 # [6104607.402876] machine2 systemd-tmpfiles[149]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted container-test-run-monitoring> machine2 # [6104607.403114] machine2 systemd-tmpfiles[149]: fchmod() of /var/log/journal failed: Operation not permitted container-test-run-monitoring> machine2 # [6104607.403270] machine2 systemd-tmpfiles[149]: fchmod() of /var/log/journal/c5ec279aec6a46b5b8e5e1c14b1b2b7e failed: Operation not permitted container-test-run-monitoring> machine2 # [6104607.403503] machine2 systemd-tmpfiles[149]: fchmod() of /run/log/journal failed: Operation not permitted container-test-run-monitoring> machine2 # [6104607.408797] machine2 systemd[1]: Finished Create System Files and Directories. container-test-run-monitoring> machine2 # [6104607.410810] machine2 systemd[1]: Starting Rebuild Journal Catalog... container-test-run-monitoring> machine2 # [6104607.411783] machine2 systemd[1]: Starting Record System Boot/Shutdown in UTMP... container-test-run-monitoring> machine2 # [6104607.427812] machine2 systemd[1]: Finished Record System Boot/Shutdown in UTMP. container-test-run-monitoring> machine2 # [6104607.432341] machine2 systemd[1]: Finished Rebuild Journal Catalog. container-test-run-monitoring> machine2 # [6104607.433635] machine2 systemd[1]: Starting Update is Completed... container-test-run-monitoring> machine2 # [6104607.445698] machine2 systemd[1]: Finished Update is Completed. container-test-run-monitoring> machine2 # [6104607.499401] machine2 systemd[1]: Finished Firewall. container-test-run-monitoring> machine2 # [6104607.500148] machine2 systemd[1]: Reached target Preparation for Network. container-test-run-monitoring> machine2 # [6104607.500499] machine2 systemd[1]: Listening on Network Management Resolve Hook Socket. container-test-run-monitoring> machine2 # [6104607.501864] machine2 systemd[1]: Starting Network Management... container-test-run-monitoring> machine1 # [6104607.364548] machine1 systemd-journald[123]: Journal started container-test-run-monitoring> machine1 # [6104607.364620] machine1 systemd-journald[123]: Runtime Journal (/run/log/journal/4abc9739039d4087ba18f77657bea266) is 8M, max 2.5G, 2.4G free. container-test-run-monitoring> machine1 # [6104607.380913] machine1 systemd[1]: Finished Create Static Device Nodes in /dev gracefully. container-test-run-monitoring> machine1 # [6104607.392612] machine1 systemd[1]: Starting Flush Journal to Persistent Storage... container-test-run-monitoring> machine1 # [6104607.393728] machine1 systemd[1]: Starting Network Name Resolution... container-test-run-monitoring> machine1 # [6104607.394675] machine1 systemd[1]: Starting Create Static Device Nodes in /dev... container-test-run-monitoring> machine1 # [6104607.404378] machine1 systemd-journald[123]: Time spent on flushing to /var/log/journal/4abc9739039d4087ba18f77657bea266 is 1.668ms for 6 entries. container-test-run-monitoring> machine1 # [6104607.404378] machine1 systemd-journald[123]: System Journal (/var/log/journal/4abc9739039d4087ba18f77657bea266) is 8M, max 4G, 3.9G free. container-test-run-monitoring> machine1 # [6104607.414284] machine1 systemd[1]: Finished Create Static Device Nodes in /dev. container-test-run-monitoring> machine1 # [6104607.414588] machine1 systemd[1]: Reached target Preparation for Local File Systems. container-test-run-monitoring> machine1 # [6104607.414682] machine1 systemd[1]: Reached target Local File Systems. container-test-run-monitoring> machine1 # [6104607.415466] machine1 systemd[1]: Listening on Boot Loader Control Service Socket. container-test-run-monitoring> machine1 # [6104607.415512] machine1 systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine1 # [6104607.416475] machine1 systemd[1]: Starting Save Transient machine-id to Disk... container-test-run-monitoring> machine1 # [6104607.416511] machine1 systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys container-test-run-monitoring> machine1 # [6104607.442903] machine1 systemd[1]: Finished Flush Journal to Persistent Storage. container-test-run-monitoring> machine1 # [6104607.444760] machine1 systemd[1]: Starting Create System Files and Directories... container-test-run-monitoring> machine1 # [6104607.461046] machine1 systemd-tmpfiles[184]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [6104607.461273] machine1 systemd-tmpfiles[184]: fchmod() of /var/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [6104607.461425] machine1 systemd-tmpfiles[184]: fchmod() of /var/log/journal/4abc9739039d4087ba18f77657bea266 failed: Operation not permitted container-test-run-monitoring> machine1 # [6104607.461709] machine1 systemd-tmpfiles[184]: fchmod() of /run/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [6104607.463316] machine1 systemd[1]: Finished Create System Files and Directories. container-test-run-monitoring> machine1 # [6104607.464626] machine1 systemd[1]: Starting Rebuild Journal Catalog... container-test-run-monitoring> machine1 # [6104607.465408] machine1 systemd[1]: Starting Record System Boot/Shutdown in UTMP... container-test-run-monitoring> machine1 # [6104607.477503] machine1 systemd[1]: Finished Record System Boot/Shutdown in UTMP. container-test-run-monitoring> machine1 # [6104607.484978] machine1 systemd[1]: Finished Rebuild Journal Catalog. container-test-run-monitoring> machine1 # [6104607.487073] machine1 systemd[1]: Starting Update is Completed... container-test-run-monitoring> machine1 # [6104607.499074] machine1 systemd[1]: Finished Update is Completed. container-test-run-monitoring> machine1 # [6104607.524709] machine1 systemd[1]: Finished Firewall. container-test-run-monitoring> machine1 # [6104607.524878] machine1 systemd[1]: Reached target Preparation for Network. container-test-run-monitoring> machine1 # [6104607.525131] machine1 systemd[1]: Listening on Network Management Resolve Hook Socket. container-test-run-monitoring> machine1 # [6104607.526329] machine1 systemd[1]: Starting Network Management... container-test-run-monitoring> machine2 # [6104607.791645] machine2 systemd[1]: Finished Save Transient machine-id to Disk. container-test-run-monitoring> machine2 # [6104608.031861] machine2 systemd-networkd[218]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine2 # [6104608.031969] machine2 systemd-networkd[218]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine2 # [6104608.039148] machine2 systemd-networkd[218]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine2 # [6104608.039311] machine2 systemd-networkd[218]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine2 # [6104608.039491] machine2 systemd-networkd[218]: lo: Link UP container-test-run-monitoring> machine2 # [6104608.039496] machine2 systemd-networkd[218]: lo: Gained carrier container-test-run-monitoring> machine2 # [6104608.039689] machine2 systemd-networkd[218]: eth1: Configuring with /etc/systemd/network/40-eth1.network. container-test-run-monitoring> machine2 # [6104608.040195] machine2 systemd-networkd[218]: eth1: Link UP container-test-run-monitoring> machine2 # [6104608.040352] machine2 systemd[1]: Started Network Management. container-test-run-monitoring> machine2 # [6104608.041517] machine2 systemd-networkd[218]: eth1: Gained carrier container-test-run-monitoring> machine2 # [6104608.041669] machine2 systemd[1]: Starting Enable Persistent Storage in systemd-networkd... container-test-run-monitoring> machine1 # [6104607.790346] machine1 systemd[1]: Finished Save Transient machine-id to Disk. container-test-run-monitoring> machine2 # [6104608.095703] machine2 systemd[1]: Finished Enable Persistent Storage in systemd-networkd. container-test-run-monitoring> machine2 # [6104608.236278] machine2 systemd-resolved[130]: Positive Trust Anchors: container-test-run-monitoring> machine2 # [6104608.236293] machine2 systemd-resolved[130]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d container-test-run-monitoring> machine2 # [6104608.236296] machine2 systemd-resolved[130]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16 container-test-run-monitoring> machine2 # [6104608.236334] machine2 systemd-resolved[130]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test container-test-run-monitoring> machine2 # [6104608.259116] machine2 systemd-resolved[130]: Using system hostname 'machine2'. container-test-run-monitoring> machine1 # [6104608.044672] machine1 systemd-networkd[238]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine2 # [6104608.260629] machine2 systemd[1]: Started Network Name Resolution. container-test-run-monitoring> machine2 # [6104608.260720] machine2 systemd[1]: Reached target Network. container-test-run-monitoring> machine2 # [6104608.260785] machine2 systemd[1]: Reached target System Initialization. container-test-run-monitoring> machine2 # [6104608.260839] machine2 systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine2 # [6104608.260866] machine2 systemd[1]: Started Daily Cleanup of Temporary Directories. container-test-run-monitoring> machine2 # [6104608.260894] machine2 systemd[1]: Reached target Timer Units. container-test-run-monitoring> machine2 # [6104608.261028] machine2 systemd[1]: Listening on D-Bus System Message Bus Socket. container-test-run-monitoring> machine2 # [6104608.261138] machine2 systemd[1]: Listening on Nix Daemon Socket. container-test-run-monitoring> machine2 # [6104608.261238] machine2 systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. container-test-run-monitoring> machine2 # [6104608.261259] machine2 systemd[1]: Reached target Socket Units. container-test-run-monitoring> machine2 # [6104608.261300] machine2 systemd[1]: Reached target Basic System. container-test-run-monitoring> machine2 # [6104608.262596] machine2 systemd[1]: Started alloy.service. container-test-run-monitoring> machine2 # [6104608.263430] machine2 systemd[1]: Starting Import lastlog data into lastlog2 database... container-test-run-monitoring> machine2 # [6104608.264467] machine2 systemd[1]: Starting Name Service Cache Daemon (nsncd)... container-test-run-monitoring> machine2 # [6104608.265684] machine2 systemd[1]: Starting D-Bus System Message Bus... container-test-run-monitoring> machine1 # [6104608.044767] machine1 systemd-networkd[238]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [6104608.051750] machine1 systemd-networkd[238]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine1 # [6104608.051918] machine1 systemd-networkd[238]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine1 # [6104608.052135] machine1 systemd-networkd[238]: lo: Link UP container-test-run-monitoring> machine1 # [6104608.052138] machine1 systemd-networkd[238]: lo: Gained carrier container-test-run-monitoring> machine1 # [6104608.052347] machine1 systemd-networkd[238]: eth1: Configuring with /etc/systemd/network/40-eth1.network. container-test-run-monitoring> machine1 # [6104608.052754] machine1 systemd[1]: Started Network Management. container-test-run-monitoring> machine1 # [6104608.088328] machine1 systemd-networkd[238]: eth1: Link UP container-test-run-monitoring> machine1 # [6104608.088646] machine1 systemd-networkd[238]: eth1: Gained carrier container-test-run-monitoring> machine1 # [6104608.088650] machine1 systemd[1]: Starting Enable Persistent Storage in systemd-networkd... container-test-run-monitoring> machine1 # [6104608.150824] machine1 systemd[1]: Finished Enable Persistent Storage in systemd-networkd. container-test-run-monitoring> machine1 # [6104608.257041] machine1 systemd-resolved[149]: Positive Trust Anchors: container-test-run-monitoring> machine1 # [6104608.257053] machine1 systemd-resolved[149]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d container-test-run-monitoring> machine1 # [6104608.257057] machine1 systemd-resolved[149]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16 container-test-run-monitoring> machine1 # [6104608.257091] machine1 systemd-resolved[149]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test container-test-run-monitoring> machine1 # [6104608.280240] machine1 systemd-resolved[149]: Using system hostname 'machine1'. container-test-run-monitoring> machine1 # [6104608.282005] machine1 systemd[1]: Started Network Name Resolution. container-test-run-monitoring> machine1 # [6104608.282099] machine1 systemd[1]: Reached target Network. container-test-run-monitoring> machine1 # [6104608.282164] machine1 systemd[1]: Reached target Network is Online. container-test-run-monitoring> machine1 # [6104608.282208] machine1 systemd[1]: Reached target System Initialization. container-test-run-monitoring> machine1 # [6104608.282260] machine1 systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine1 # [6104608.282298] machine1 systemd[1]: Started Daily Cleanup of Temporary Directories. container-test-run-monitoring> machine1 # [6104608.282316] machine1 systemd[1]: Reached target Timer Units. container-test-run-monitoring> machine1 # [6104608.282440] machine1 systemd[1]: Listening on D-Bus System Message Bus Socket. container-test-run-monitoring> machine1 # [6104608.282569] machine1 systemd[1]: Listening on Nix Daemon Socket. container-test-run-monitoring> machine1 # [6104608.282671] machine1 systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. container-test-run-monitoring> machine1 # [6104608.282695] machine1 systemd[1]: Reached target Socket Units. container-test-run-monitoring> machine1 # [6104608.282729] machine1 systemd[1]: Reached target Basic System. container-test-run-monitoring> machine1 # [6104608.304458] machine1 systemd[1]: Started alloy.service. container-test-run-monitoring> machine1 # [6104608.305447] machine1 systemd[1]: Starting Import lastlog data into lastlog2 database... container-test-run-monitoring> machine1 # [6104608.306361] machine1 systemd[1]: Started Loki Service Daemon. container-test-run-monitoring> machine1 # [6104608.307304] machine1 systemd[1]: Started mimir Service Daemon. container-test-run-monitoring> machine1 # [6104608.308442] machine1 systemd[1]: Starting Nginx Web Server... container-test-run-monitoring> machine1 # [6104608.309454] machine1 systemd[1]: Starting Name Service Cache Daemon (nsncd)... container-test-run-monitoring> machine1 # [6104608.310408] machine1 systemd[1]: Starting PostgreSQL Server... container-test-run-monitoring> machine1 # [6104608.312047] machine1 systemd[1]: Starting D-Bus System Message Bus... container-test-run-monitoring> machine1 # [6104608.323488] machine1 systemd[1]: Finished Import lastlog data into lastlog2 database. container-test-run-monitoring> machine1 # [6104608.362725] machine1 systemd[1]: etc-machine\x2did.mount: Deactivated successfully. container-test-run-monitoring> machine1 # [6104608.516748] machine1 nsncd[249]: Aug 18 15:07:14.569 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" container-test-run-monitoring> machine1 # [6104608.517084] machine1 systemd[1]: Started Name Service Cache Daemon (nsncd). container-test-run-monitoring> machine1 # [6104608.517157] machine1 systemd[1]: Reached target Host and Network Name Lookups. container-test-run-monitoring> machine1 # [6104608.517259] machine1 systemd[1]: Reached target User and Group Name Lookups. container-test-run-monitoring> machine1 # [6104608.548406] machine1 systemd[1]: Starting User Login Management... container-test-run-monitoring> machine1 # [6104608.549572] machine1 systemd[1]: Starting Permit User Sessions... container-test-run-monitoring> machine2 # [6104608.343213] machine2 systemd[1]: etc-machine\x2did.mount: Deactivated successfully. container-test-run-monitoring> machine2 # [6104608.344212] machine2 systemd[1]: Finished Import lastlog data into lastlog2 database. container-test-run-monitoring> machine2 # [6104608.505062] machine2 nsncd[226]: Aug 18 15:07:14.558 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" container-test-run-monitoring> machine2 # [6104608.505138] machine2 systemd[1]: Started Name Service Cache Daemon (nsncd). container-test-run-monitoring> machine2 # [6104608.505219] machine2 systemd[1]: Reached target Host and Network Name Lookups. container-test-run-monitoring> machine2 # [6104608.505286] machine2 systemd[1]: Reached target User and Group Name Lookups. container-test-run-monitoring> machine2 # [6104608.507039] machine2 systemd[1]: Starting User Login Management... container-test-run-monitoring> machine2 # [6104608.507814] machine2 systemd[1]: Starting Permit User Sessions... container-test-run-monitoring> machine2 # [6104608.555756] machine2 systemd[1]: Finished Permit User Sessions. container-test-run-monitoring> machine2 # [6104608.557387] machine2 systemd[1]: Started Console Getty. container-test-run-monitoring> machine2 # [6104608.557440] machine2 systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 container-test-run-monitoring> machine2 # [6104608.557460] machine2 systemd[1]: Reached target Login Prompts. container-test-run-monitoring> machine1 # [6104608.559367] machine1 systemd[1]: Finished Permit User Sessions. container-test-run-monitoring> machine1 # [6104608.560507] machine1 systemd[1]: Started Console Getty. container-test-run-monitoring> machine1 # [6104608.560552] machine1 systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 container-test-run-monitoring> machine1 # [6104608.560567] machine1 systemd[1]: Reached target Login Prompts. container-test-run-monitoring> machine1 # [6104608.625049] machine1 dbus-broker-launch[252]: Looking up NSS user entry for 'systemd-timesync'... container-test-run-monitoring> machine1 # [6104608.625730] machine1 dbus-broker-launch[252]: NSS returned no entry for 'systemd-timesync' container-test-run-monitoring> machine1 # [6104608.625730] machine1 dbus-broker-launch[252]: Invalid user-name in /nix/store/dzf3q853lbzyx7f2rlpfqc8rvkrl2a1k-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" container-test-run-monitoring> machine1 # [6104608.626216] machine1 systemd[1]: Started D-Bus System Message Bus. container-test-run-monitoring> machine1 # [6104608.633634] machine1 dbus-broker-launch[252]: Ready container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.707565724Z caller=main.go:232 level=info msg="Starting application" version="(version=3.1.4, branch=unknown, revision=unknown)" container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.712209508Z caller=server.go:401 level=info msg="server listening on addresses" http=[::]:3001 grpc=[::]:9095 container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.72958771Z caller=memberlist_client.go:541 level=info component=memberlist msg="Using memberlist cluster label and node name" cluster_label= node=machine1-7e3e3098 container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.733625126Z caller=inmemory.go:137 level=info msg="created in-memory index cache" maxItemSizeBytes=134217728 maxSizeBytes=1073741824 maxItems=maxInt container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.737656782Z caller=modules.go:936 level=info msg="The query-frontend has not been configured with the query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic frontend configuration. If queries are unresponsive, consider explicitly configuring the query-scheduler address for querier-frontend." address=127.0.0.1:9095 container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.739419727Z caller=ingester.go:359 level=warn msg="-blocks-storage.backend=filesystem is for development and testing only; you should switch to an external object store for production use or use a shared filesystem" container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.740648664Z caller=ingester.go:513 level=info msg="TSDB idle compaction timeout set" timeout=1h6m34.443515597s container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.741861561Z caller=modules.go:1054 level=info msg="The ruler is not being started because you need to configure the ruler storage." container-test-run-monitoring> machine1 # [6104608.752298] machine1 mimir[247]: ts=2026-08-18T15:07:14.742063164Z caller=modules.go:1068 level=info msg="The ruler storage has not been configured. Not starting the ruler." container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.880686173Z caller=modules.go:716 level=info msg="The querier worker has not been configured with the query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-scheduler address for querier worker." address=127.0.0.1:9095 container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.880742054Z caller=worker.go:148 level=info msg="Starting querier worker connected to query-scheduler" scheduler=127.0.0.1:9095 container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.881624506Z caller=module_service.go:82 level=info msg=starting module=active-groups-cleanup-service container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.881647426Z caller=module_service.go:82 level=info msg=starting module=activity-tracker container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.881683747Z caller=module_service.go:82 level=info msg=starting module=sanity-check container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.881707787Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access" container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882115833Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked" container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882131073Z caller=sanity_check.go:39 level=info msg="Checking object storage config" container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882258555Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked" container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882290475Z caller=module_service.go:82 level=info msg=starting module=server container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882332276Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882353716Z caller=module_service.go:82 level=info msg=starting module=store-queryable container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882365276Z caller=module_service.go:82 level=info msg=starting module=query-scheduler container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882368637Z caller=module_service.go:82 level=info msg=starting module=compactor container-test-run-monitoring> machine1 # [6104608.854384] machine1 mimir[247]: ts=2026-08-18T15:07:14.882425557Z caller=module_service.go:82 level=info msg=starting module=querier-ring container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882426037Z caller=module_service.go:82 level=info msg=starting module=ingester-ring container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882474838Z caller=module_service.go:82 level=info msg=starting module=querier-lifecycler container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882481438Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882487118Z caller=module_service.go:82 level=info msg=starting module=store-gateway container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882481358Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882501158Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882516079Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=querier container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882545559Z caller=module_service.go:82 level=info msg=starting module=query-frontend container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882546639Z caller=module_service.go:82 level=info msg=starting module=distributor-service container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882550399Z caller=module_service.go:82 level=info msg=starting module=ingester-service container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.8826048Z caller=frontend_scheduler_worker.go:146 level=info msg="adding connection to query-scheduler" addr=127.0.0.1:9095 container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882661641Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882659681Z caller=ingester_tsdb.go:399 level=info msg="opening existing TSDBs" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882743362Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=store-gateway container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882767682Z caller=basic_lifecycler_delegates.go:63 level=info msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882791642Z caller=lifecycler.go:694 level=info msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [6104608.854721] machine1 mimir[247]: ts=2026-08-18T15:07:14.882813603Z caller=lifecycler.go:721 level=info msg="instance not found in ring, adding with no tokens" ring=ingester container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.882871844Z caller=lifecycler.go:562 level=info msg="auto-joining cluster after timeout" ring=ingester container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883007205Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883042566Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=distributor container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883105767Z caller=ring.go:390 level=info component=compactor msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883108007Z caller=distributor.go:882 level=info msg="waiting until distributor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883130287Z caller=module_service.go:82 level=info msg=starting module=querier container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883150887Z caller=basic_lifecycler.go:324 level=info component=compactor msg="instance not found in the ring" instance=machine1 ring=compactor container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883172168Z caller=worker.go:248 level=info msg="adding connection" addr=127.0.0.1:9095 in-use=true container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883223488Z caller=gateway.go:246 level=info msg="waiting until store-gateway is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883243769Z caller=gateway.go:250 level=info msg="store-gateway is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883265049Z caller=bucket_stores.go:215 level=info msg="synchronizing TSDB blocks for all users" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.88331913Z caller=compactor.go:583 level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104608.854988] machine1 mimir[247]: ts=2026-08-18T15:07:14.883480172Z caller=bucket_stores.go:228 level=info msg="successfully synchronized TSDB blocks for all users" container-test-run-monitoring> machine2 # [6104608.614495] machine2 dbus-broker-launch[227]: Looking up NSS user entry for 'systemd-timesync'... container-test-run-monitoring> machine2 # [6104608.615375] machine2 dbus-broker-launch[227]: NSS returned no entry for 'systemd-timesync' container-test-run-monitoring> machine2 # [6104608.615375] machine2 dbus-broker-launch[227]: Invalid user-name in /nix/store/f22qhc8ziywvxq9q588a7r0mcbjxv1bk-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" container-test-run-monitoring> machine2 # [6104608.615743] machine2 systemd[1]: Started D-Bus System Message Bus. container-test-run-monitoring> machine2 # [6104608.623120] machine2 dbus-broker-launch[227]: Ready container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.826753942Z caller=main.go:146 msg="Starting Loki" version="(version=3.7.6, branch=unknown, revision=unknown)" container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.826817063Z caller=main.go:147 msg="Loading configuration file" filename=/nix/store/4lzyglvw832kxrrj865xvk6cngiz9g3l-validate-loki-conf container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.827005026Z caller=modules.go:1597 msg="Ruler storage is not configured; ruler will not be started." config_key=ruler.storage container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.831453208Z caller=server.go:397 msg="server listening on addresses" http=[::]:3002 grpc=[::]:9096 container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.839620002Z caller=table_manager.go:136 index-store=tsdb-2025-11-01 msg="uploading tables" container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.839787604Z caller=table_manager.go:300 index-store=tsdb-2025-11-01 msg="query readiness setup completed" duration=3.08µs distinct_users_len=0 distinct_users= container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.839835565Z caller=shipper.go:165 index-store=tsdb-2025-11-01 msg="starting index shipper in RW mode" container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.840420213Z caller=head_manager.go:313 index-store=tsdb-2025-11-01 component=tsdb-head-manager msg="loaded wals by period" groups=0 container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.840491414Z caller=manager.go:86 index-store=tsdb-2025-11-01 component=tsdb-manager msg="loaded leftover local indices" err=null successful=true buckets=0 indices=0 failures=0 container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.840533414Z caller=head_manager.go:313 index-store=tsdb-2025-11-01 component=tsdb-head-manager msg="loaded wals by period" groups=0 container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.846234294Z caller=worker.go:134 component=querier msg="Starting querier worker using query-scheduler and scheduler ring for addresses" container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=warn ts=2026-08-18T15:07:14.850769797Z caller=modules.go:1643 msg="RulerStorage is nil. Not starting the ruler." container-test-run-monitoring> machine1 # [6104608.873393] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872270736Z caller=module_service.go:82 msg=starting module=cache-generation-loader container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872390138Z caller=module_service.go:82 msg=starting module=server container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.8725599Z caller=module_service.go:82 msg=starting module=memberlist-kv container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.87257758Z caller=module_service.go:82 msg=starting module=query-frontend-tripperware container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872614341Z caller=module_service.go:82 msg=starting module=query-scheduler-ring container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872713662Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872777543Z caller=module_service.go:82 msg=starting module=store container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872806103Z caller=module_service.go:82 msg=starting module=ring container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872832584Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872909585Z caller=basic_lifecycler.go:324 msg="instance not found in the ring" instance=machine1 ring=scheduler container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.872919785Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873009106Z caller=module_service.go:82 msg=starting module=compactor container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873055027Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873110748Z caller=module_service.go:82 msg=starting module=distributor container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=error ts=2026-08-18T15:07:14.873149868Z caller=ratestore.go:110 msg="error getting ingester clients" err="empty ring" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873225869Z caller=module_service.go:82 msg=starting module=ingester-querier container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.87325359Z caller=module_service.go:82 msg=starting module=rule-evaluator container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873317711Z caller=ring.go:390 component=distributor msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [6104608.873878] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873353071Z caller=ringmanager.go:186 msg="waiting until scheduler is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873362671Z caller=ringmanager.go:190 msg="scheduler is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873361911Z caller=basic_lifecycler.go:324 component=distributor msg="instance not found in the ring" instance=machine1 ring=distributor container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873403232Z caller=ringmanager.go:199 msg="waiting until scheduler is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873178469Z caller=module_service.go:82 msg=starting module=ingester container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873478473Z caller=basic_lifecycler.go:324 msg="instance not found in the ring" instance=machine1 ring=compactor container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873507713Z caller=ingester.go:571 component=ingester msg="recovering from checkpoint" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873514273Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873529154Z caller=checkpoint.go:549 msg="tmp checkpoint cleanup starting" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873607795Z caller=checkpoint.go:554 msg="tmp checkpoint cleanup done" duration=61.801µs success=true container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873630155Z caller=recovery.go:45 component=ingester msg="no checkpoint found, treating as no-op" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873682516Z caller=compactor.go:420 msg="waiting until compactor is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873693196Z caller=compactor.go:424 msg="compactor is JOINING in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.873723836Z caller=compactor.go:434 msg="waiting until compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874212243Z caller=ingester.go:587 component=ingester msg="recovered WAL checkpoint recovery finished" elapsed=690.05µs errors=false container-test-run-monitoring> machine1 # [6104608.874150] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874281004Z caller=ingester.go:593 component=ingester msg="recovering from WAL" container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.87474617Z caller=ingester.go:609 component=ingester msg="WAL segment recovery finished" elapsed=1.239337ms errors=false container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874762731Z caller=ingester.go:557 component=ingester msg="closing recoverer" container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874771651Z caller=ingester.go:565 component=ingester msg="WAL recovery finished" time=1.265458ms container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874802371Z caller=wal.go:173 msg=started component=wal container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874882452Z caller=lifecycler.go:694 component=ingester msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874875132Z caller=ingester.go:778 component=ingester msg="sleeping for initial delay before starting periodic flushing" delay=3.649990118s container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.874939693Z caller=lifecycler.go:721 component=ingester msg="instance not found in ring, adding with no tokens" ring=ingester container-test-run-monitoring> machine1 # [6104608.874367] machine1 loki[246]: level=info ts=2026-08-18T15:07:14.875059255Z caller=lifecycler.go:562 component=ingester msg="auto-joining cluster after timeout" ring=ingester container-test-run-monitoring> machine1 # [6104608.984735] machine1 nginx-pre-start[370]: nginx: the configuration file /nix/store/zms43abcv2kwg70lmrgy8yknka0zkhg5-nginx.conf syntax is ok container-test-run-monitoring> machine1 # [6104608.985394] machine1 nginx-pre-start[370]: nginx: configuration file /nix/store/zms43abcv2kwg70lmrgy8yknka0zkhg5-nginx.conf test is successful container-test-run-monitoring> machine1 # [6104609.005132] machine1 postgresql-pre-start[372]: The files belonging to this database system will be owned by user "postgres". container-test-run-monitoring> machine1 # [6104609.005132] machine1 postgresql-pre-start[372]: This user must also own the server process. container-test-run-monitoring> machine1 # [6104609.005556] machine1 postgresql-pre-start[372]: The database cluster will be initialized with locale "en_US.UTF-8". container-test-run-monitoring> machine1 # [6104609.005556] machine1 postgresql-pre-start[372]: The default database encoding has accordingly been set to "UTF8". container-test-run-monitoring> machine1 # [6104609.005556] machine1 postgresql-pre-start[372]: The default text search configuration will be set to "english". container-test-run-monitoring> machine1 # [6104609.005556] machine1 postgresql-pre-start[372]: Data page checksums are enabled. container-test-run-monitoring> machine1 # [6104609.005556] machine1 postgresql-pre-start[372]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok container-test-run-monitoring> machine1 # [6104609.005928] machine1 postgresql-pre-start[372]: creating subdirectories ... ok container-test-run-monitoring> machine1 # [6104609.006042] machine1 postgresql-pre-start[372]: selecting dynamic shared memory implementation ... posix container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.056606502Z caller=compactor.go:438 msg="compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.061764893Z caller=ringmanager.go:203 msg="scheduler is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.061847175Z caller=module_service.go:82 msg=starting module=query-scheduler container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.061947736Z caller=module_service.go:82 msg=starting module=query-frontend container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.061984056Z caller=module_service.go:82 msg=starting module=querier container-test-run-monitoring> machine1 # [6104609.009095] machine1 loki[246]: level=info ts=2026-08-18T15:07:15.06220598Z caller=loki.go:615 msg="Loki started" startup_time=311.364333ms container-test-run-monitoring> machine1 # [6104609.029669] machine1 postgresql-pre-start[372]: selecting default "max_connections" ... 100 container-test-run-monitoring> machine1 # [6104609.053279] machine1 mimir[247]: ts=2026-08-18T15:07:15.066278876Z caller=compactor.go:588 level=info component=compactor msg="compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104609.053279] machine1 mimir[247]: ts=2026-08-18T15:07:15.066734243Z caller=compactor.go:706 level=info component=compactor msg="discovering users from bucket" container-test-run-monitoring> machine1 # [6104609.053279] machine1 mimir[247]: ts=2026-08-18T15:07:15.066798243Z caller=compactor.go:716 level=info component=compactor msg="discovered users from bucket" users=0 container-test-run-monitoring> machine1 # [6104609.053279] machine1 mimir[247]: ts=2026-08-18T15:07:15.066790643Z caller=blocks_cleaner.go:241 level=info component=cleaner task=clean_up_users_during_startup msg="started blocks cleanup and maintenance" container-test-run-monitoring> machine1 # [6104609.053279] machine1 mimir[247]: ts=2026-08-18T15:07:15.066937405Z caller=blocks_cleaner.go:247 level=info component=cleaner task=clean_up_users_during_startup msg="successfully completed blocks cleanup and maintenance" container-test-run-monitoring> machine1 # [6104609.057510] machine1 postgresql-pre-start[372]: selecting default "shared_buffers" ... 128MB container-test-run-monitoring> machine1 # [6104609.065192] machine1 systemd-logind[274]: New seat seat0. container-test-run-monitoring> machine1 # [6104609.065456] machine1 systemd[1]: Started User Login Management. container-test-run-monitoring> machine1 # [6104609.067632] machine1 systemd[1]: Starting linger-users.service... container-test-run-monitoring> machine1 # [6104609.080081] machine1 systemd[1]: linger-users.service: Deactivated successfully. container-test-run-monitoring> machine1 # [6104609.080175] machine1 systemd[1]: Finished linger-users.service. container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.184336199Z level=info msg="Alloy is starting" container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.184460161Z level=info msg="boringcrypto enabled" enabled=false container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.187117638Z level=info msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.187340801Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.189473791Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.190370243Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=loki.write.loki duration=882.172µs container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.198107351Z level=info msg="replaying WAL, this may take a while" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal dir=data-alloy/prometheus.remote_write.mimir/wal container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.198317354Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal segment=0 maxSegment=0 container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.199105405Z level=info msg="Starting WAL watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=35e5e8 url=http://machine1.clan/mimir/api/v1/push queue=35e5e8 container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.199110805Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=35e5e8 url=http://machine1.clan/mimir/api/v1/push container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.199123165Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=prometheus.remote_write.mimir duration=8.744842ms container-test-run-monitoring> machine1 # [6104609.148086] machine1 alloy[244]: ts=2026-08-18T15:07:15.199231727Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=35e5e8 url=http://machine1.clan/mimir/api/v1/push queue=35e5e8 container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199514451Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=prometheus.relabel.create_nixos_services_metric duration=380.646µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199522731Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=otel duration=3.44µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199583772Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=remotecfg duration=58.041µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199607452Z level=info msg="applying non-TLS config to HTTP server" service=http container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199608772Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=http duration=21.8µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199613812Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=cluster duration=2.24µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199622812Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=livedebugging duration=6.68µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199628732Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=ui duration=2.88µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199799575Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=logging duration=167.763µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.199814055Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=labelstore duration=6.8µs container-test-run-monitoring> machine1 # [6104609.148610] machine1 alloy[244]: ts=2026-08-18T15:07:15.200279141Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=loki.relabel.journal duration=457.806µs container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.200507584Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=loki.source.journal.all duration=208.923µs container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.200690067Z level=info msg="Parsed flag --collector.filesystem.mount-points-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.200707867Z level=info msg="Parsed flag --collector.filesystem.fs-types-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.200964951Z level=info msg="Parsed flag --collector.systemd.unit-include" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+ container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.200997631Z level=info msg="Parsed flag --collector.systemd.unit-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+\.(automount|device|mount|scope|slice) container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.201026312Z level=info msg="Enabled node_exporter collectors" component_path=/ component_id=prometheus.exporter.unix.local_system container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.201036752Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=cpu container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.201040232Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=filesystem container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.201043752Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=meminfo container-test-run-monitoring> machine1 # [6104609.148790] machine1 alloy[244]: ts=2026-08-18T15:07:15.201046792Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=systemd container-test-run-monitoring> machine1 # [6104609.148946] machine1 alloy[244]: ts=2026-08-18T15:07:15.201065072Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=prometheus.exporter.unix.local_system duration=538.647µs container-test-run-monitoring> machine2 # [6104609.065624] machine2 systemd-logind[243]: New seat seat0. container-test-run-monitoring> machine2 # [6104609.065862] machine2 systemd[1]: Started User Login Management. container-test-run-monitoring> machine2 # [6104609.067057] machine2 systemd[1]: Starting linger-users.service... container-test-run-monitoring> machine2 # [6104609.079737] machine2 systemd[1]: linger-users.service: Deactivated successfully. container-test-run-monitoring> machine2 # [6104609.080047] machine2 systemd[1]: Finished linger-users.service. container-test-run-monitoring> machine2 # [6104609.080602] machine2 systemd[1]: Reached target Multi-User System. container-test-run-monitoring> machine2 # [6104609.080856] machine2 systemd[1]: Startup finished in 2.087s. container-test-run-monitoring> machine2 # [6104609.140934] machine2 alloy[224]: ts=2026-08-18T15:07:15.184332279Z level=info msg="Alloy is starting" container-test-run-monitoring> machine2 # [6104609.140934] machine2 alloy[224]: ts=2026-08-18T15:07:15.184461721Z level=info msg="boringcrypto enabled" enabled=false container-test-run-monitoring> machine2 # [6104609.140934] machine2 alloy[224]: ts=2026-08-18T15:07:15.18724152Z level=info msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit container-test-run-monitoring> machine2 # [6104609.140934] machine2 alloy[224]: ts=2026-08-18T15:07:15.187339961Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster container-test-run-monitoring> machine2 # [6104609.140934] machine2 alloy[224]: ts=2026-08-18T15:07:15.189482271Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.191996706Z level=info msg="replaying WAL, this may take a while" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal dir=data-alloy/prometheus.remote_write.mimir/wal container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.192533433Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal segment=0 maxSegment=0 container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.193639609Z level=info msg="Starting WAL watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=055dac url=http://machine1.clan/mimir/api/v1/push queue=055dac container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.193644649Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=055dac url=http://machine1.clan/mimir/api/v1/push container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.193661729Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=prometheus.remote_write.mimir duration=4.172258ms container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.194022854Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=prometheus.relabel.create_nixos_services_metric duration=351.205µs container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.194163536Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=logging duration=135.282µs container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.194241577Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=remotecfg duration=69.961µs container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.194329498Z level=info msg="applying non-TLS config to HTTP server" service=http container-test-run-monitoring> machine2 # [6104609.141418] machine2 alloy[224]: ts=2026-08-18T15:07:15.194339179Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=http duration=86.882µs container-test-run-monitoring> machine2 # [6104609.141612] machine2 alloy[224]: ts=2026-08-18T15:07:15.194348379Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=cluster duration=2.92µs container-test-run-monitoring> machine2 # [6104609.141612] machine2 alloy[224]: ts=2026-08-18T15:07:15.194747824Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=loki.write.loki duration=391.645µs container-test-run-monitoring> machine2 # [6104609.141912] machine2 alloy[224]: ts=2026-08-18T15:07:15.195042788Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=loki.relabel.journal duration=270.923µs container-test-run-monitoring> machine2 # [6104609.142006] machine2 alloy[224]: ts=2026-08-18T15:07:15.194625543Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=055dac url=http://machine1.clan/mimir/api/v1/push queue=055dac container-test-run-monitoring> machine2 # [6104609.142110] machine2 alloy[224]: ts=2026-08-18T15:07:15.195268271Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=loki.source.journal.all duration=205.762µs container-test-run-monitoring> machine2 # [6104609.142145] machine2 alloy[224]: ts=2026-08-18T15:07:15.195303992Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=tracing duration=15.88µs container-test-run-monitoring> machine2 # [6104609.142164] machine2 alloy[224]: ts=2026-08-18T15:07:15.195327912Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=livedebugging duration=15.16µs container-test-run-monitoring> machine2 # [6104609.142164] machine2 alloy[224]: ts=2026-08-18T15:07:15.195341472Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=ui duration=5µs container-test-run-monitoring> machine2 # [6104609.142548] machine2 alloy[224]: ts=2026-08-18T15:07:15.195701037Z level=info msg="Parsed flag --collector.systemd.unit-include" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+ container-test-run-monitoring> machine2 # [6104609.142593] machine2 alloy[224]: ts=2026-08-18T15:07:15.195731158Z level=info msg="Parsed flag --collector.systemd.unit-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+\.(automount|device|mount|scope|slice) container-test-run-monitoring> machine2 # [6104609.142867] machine2 alloy[224]: ts=2026-08-18T15:07:15.196002882Z level=info msg="Parsed flag --collector.filesystem.mount-points-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) container-test-run-monitoring> machine2 # [6104609.142867] machine2 alloy[224]: ts=2026-08-18T15:07:15.196023802Z level=info msg="Parsed flag --collector.filesystem.fs-types-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ container-test-run-monitoring> machine2 # [6104609.142979] machine2 alloy[224]: ts=2026-08-18T15:07:15.196136044Z level=info msg="Enabled node_exporter collectors" component_path=/ component_id=prometheus.exporter.unix.local_system container-test-run-monitoring> machine2 # [6104609.142979] machine2 alloy[224]: ts=2026-08-18T15:07:15.196153764Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=cpu container-test-run-monitoring> machine2 # [6104609.142979] machine2 alloy[224]: ts=2026-08-18T15:07:15.196158324Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=filesystem container-test-run-monitoring> machine2 # [6104609.142979] machine2 alloy[224]: ts=2026-08-18T15:07:15.196161884Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=meminfo container-test-run-monitoring> machine2 # [6104609.143072] machine2 alloy[224]: ts=2026-08-18T15:07:15.196165364Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=systemd container-test-run-monitoring> machine2 # [6104609.143072] machine2 alloy[224]: ts=2026-08-18T15:07:15.196184564Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=prometheus.exporter.unix.local_system duration=835.851µs container-test-run-monitoring> machine2 # [6104609.144183] machine2 alloy[224]: ts=2026-08-18T15:07:15.19732782Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=prometheus.scrape.scrape_metrics duration=1.124736ms container-test-run-monitoring> machine2 # [6104609.144234] machine2 alloy[224]: ts=2026-08-18T15:07:15.197377101Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=otel duration=13.6µs container-test-run-monitoring> machine2 # [6104609.144234] machine2 alloy[224]: ts=2026-08-18T15:07:15.197396741Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca node_id=labelstore duration=10.6µs container-test-run-monitoring> machine2 # [6104609.144234] machine2 alloy[224]: ts=2026-08-18T15:07:15.197403941Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=a26cc88b14ae334f3d7b799aec4d11ca duration=8.114393ms container-test-run-monitoring> machine2 # [6104609.144469] machine2 alloy[224]: ts=2026-08-18T15:07:15.197577544Z level=info msg="scheduling loaded components and services" controller_id="" container-test-run-monitoring> machine2 # [6104609.144518] machine2 alloy[224]: ts=2026-08-18T15:07:15.197661745Z level=info msg="{^_^} Alloy is running" container-test-run-monitoring> machine2 # [6104609.144944] machine2 alloy[224]: ts=2026-08-18T15:07:15.198073951Z level=error msg="failed to register collector with remote server" service=remotecfg id=75eca959-0345-4dcd-89e8-e4507fa8e700 name="" err="noop client" container-test-run-monitoring> machine2 # [6104609.145428] machine2 alloy[224]: ts=2026-08-18T15:07:15.198540717Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s container-test-run-monitoring> machine2 # [6104609.145799] machine2 alloy[224]: ts=2026-08-18T15:07:15.198962403Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=machine2 container-test-run-monitoring> machine2 # [6104609.145938] machine2 alloy[224]: ts=2026-08-18T15:07:15.199027444Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345 container-test-run-monitoring> machine2 # [6104609.416086] machine2 systemd-networkd[218]: eth1: Gained IPv6LL container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.202237768Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=prometheus.scrape.scrape_metrics duration=1.155576ms container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.202308969Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 node_id=tracing duration=16.68µs container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.20232425Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=21e4ccba8ea67f532628c1a261f86c07 duration=13.027222ms container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.202597373Z level=info msg="{^_^} Alloy is running" container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.202722455Z level=info msg="scheduling loaded components and services" controller_id="" container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.202862577Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.203026379Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=machine1 container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.203114061Z level=error msg="failed to register collector with remote server" service=remotecfg id=1f6dc217-f15c-4215-90d7-9997478611a7 name="" err="noop client" container-test-run-monitoring> machine1 # [6104609.152044] machine1 alloy[244]: ts=2026-08-18T15:07:15.203418025Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345 container-test-run-monitoring> machine1 # [6104609.307476] machine1 postgresql-pre-start[372]: selecting default time zone ... UTC container-test-run-monitoring> machine1 # [6104609.308251] machine1 postgresql-pre-start[372]: creating configuration files ... ok container-test-run-monitoring> machine1 # [6104609.452741] machine1 postgresql-pre-start[372]: running bootstrap script ... ok container-test-run-monitoring> machine1 # [6104609.625190] machine1 systemd[1]: Started Nginx Web Server. container-test-run-monitoring> machine1 # [6104609.853644] machine1 mimir[247]: ts=2026-08-18T15:07:15.884114459Z caller=gateway.go:286 level=info msg="waiting until store-gateway is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104609.860629] machine1 systemd-networkd[238]: eth1: Gained IPv6LL container-test-run-monitoring> machine1 # [6104609.867533] machine1 postgresql-pre-start[372]: performing post-bootstrap initialization ... ok container-test-run-monitoring> machine1 # [6104609.952570] machine1 postgresql-pre-start[372]: syncing data to disk ... ok container-test-run-monitoring> machine1 # [6104609.952570] machine1 postgresql-pre-start[372]: initdb: warning: enabling "trust" authentication for local connections container-test-run-monitoring> machine1 # [6104609.952570] machine1 postgresql-pre-start[372]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. container-test-run-monitoring> machine1 # [6104609.952570] machine1 postgresql-pre-start[372]: Success. You can now start the database server using: container-test-run-monitoring> machine1 # [6104609.952570] machine1 postgresql-pre-start[372]: pg_ctl -D /var/lib/postgresql/18 -l logfile start container-test-run-monitoring> machine1 # [6104610.053222] machine1 mimir[247]: ts=2026-08-18T15:07:16.081124961Z caller=gateway.go:290 level=info msg="store-gateway is ACTIVE in the ring" container-test-run-monitoring> machine1 # [6104610.053222] machine1 mimir[247]: ts=2026-08-18T15:07:16.081210442Z caller=mimir.go:1103 level=info msg="Application started" container-test-run-monitoring> machine1 # [6104610.651095] machine1 postgres[395]: [395] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit container-test-run-monitoring> machine1 # [6104610.653926] machine1 postgres[395]: [395] LOG: listening on IPv6 address "::1", port 5432 container-test-run-monitoring> machine1 # [6104610.653926] machine1 postgres[395]: [395] LOG: listening on IPv4 address "127.0.0.1", port 5432 container-test-run-monitoring> machine1 # [6104610.656630] machine1 postgres[395]: [395] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" container-test-run-monitoring> machine1 # [6104610.661738] machine1 postgres[408]: [408] LOG: database system was shut down at 2026-08-18 15:07:15 GMT container-test-run-monitoring> machine1 # [6104610.666577] machine1 postgres[395]: [395] LOG: database system is ready to accept connections container-test-run-monitoring> machine1 # [6104611.411718] machine1 postgresql-post-start[421]: CREATE ROLE container-test-run-monitoring> machine1 # [6104611.484814] machine1 postgresql-post-start[426]: CREATE DATABASE container-test-run-monitoring> machine1 # [6104611.488785] machine1 systemd[1]: Started PostgreSQL Server. container-test-run-monitoring> machine1 # [6104611.490077] machine1 systemd[1]: Starting PostgreSQL Setup Scripts... container-test-run-monitoring> machine1 # [6104611.551213] machine1 systemd[1]: Finished PostgreSQL Setup Scripts. container-test-run-monitoring> machine1 # [6104611.551650] machine1 systemd[1]: Reached target PostgreSQL. container-test-run-monitoring> machine1 # [6104611.553090] machine1 systemd[1]: Starting Grafana Service Daemon... container-test-run-monitoring> machine1 # [6104612.073491] machine1 loki[246]: level=info ts=2026-08-18T15:07:18.06203073Z caller=scheduler.go:658 msg="this scheduler is in the ReplicationSet, will now accept requests." container-test-run-monitoring> machine1 # [6104612.073491] machine1 loki[246]: level=info ts=2026-08-18T15:07:18.063208147Z caller=worker.go:235 component=querier msg="adding connection" addr=127.0.0.1:9096 container-test-run-monitoring> machine1: (finished: waiting for unit multi-user.target, in 7.68 seconds) container-test-run-monitoring> machine2: waiting for unit multi-user.target container-test-run-monitoring> machine2: (finished: waiting for unit multi-user.target, in 0.02 seconds) container-test-run-monitoring> machine1: waiting for unit alloy container-test-run-monitoring> machine1: (finished: waiting for unit alloy, in 0.01 seconds) container-test-run-monitoring> machine2: waiting for unit alloy container-test-run-monitoring> machine2: (finished: waiting for unit alloy, in 0.01 seconds) container-test-run-monitoring> machine1: must succeed: test -f /etc/alloy/config.alloy container-test-run-monitoring> machine1: (finished: must succeed: test -f /etc/alloy/config.alloy, in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: test -f /etc/alloy/config.alloy container-test-run-monitoring> machine2: (finished: must succeed: test -f /etc/alloy/config.alloy, in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: test -f /etc/alloy/local-extension.alloy container-test-run-monitoring> machine2: (finished: must succeed: test -f /etc/alloy/local-extension.alloy, in 0.01 seconds) container-test-run-monitoring> machine1: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy' container-test-run-monitoring> machine1: (finished: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy', in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy' container-test-run-monitoring> machine2: (finished: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy', in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit loki container-test-run-monitoring> machine1: (finished: waiting for unit loki, in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit mimir container-test-run-monitoring> machine1: (finished: waiting for unit mimir, in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit grafana container-test-run-monitoring> machine1: (finished: waiting for unit grafana, in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: test "$(grep -c '^loki.source.journal ' /etc/alloy/config.alloy)" = 1 container-test-run-monitoring> machine1 # [6104614.073844] machine1 loki[246]: level=info ts=2026-08-18T15:07:20.057483102Z caller=compactor.go:497 msg="this instance has been chosen to run the compactor, starting compactor" container-test-run-monitoring> machine1 # [6104614.073844] machine1 loki[246]: level=info ts=2026-08-18T15:07:20.057553263Z caller=tables_manager.go:70 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor" container-test-run-monitoring> machine1 # [6104614.101651] machine1 systemd[1]: Started Grafana Service Daemon. container-test-run-monitoring> machine1 # [6104614.102020] machine1 systemd[1]: Reached target Multi-User System. container-test-run-monitoring> machine1 # [6104614.102263] machine1 systemd[1]: Startup finished in 7.079s. container-test-run-monitoring> machine2: (finished: must succeed: test "$(grep -c '^loki.source.journal ' /etc/alloy/config.alloy)" = 1, in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: cat /etc/alloy/config.alloy container-test-run-monitoring> machine2: (finished: must succeed: cat /etc/alloy/config.alloy, in 0.01 seconds) container-test-run-monitoring> (finished: run the VM test script, in 7.84 seconds) container-test-run-monitoring> test script finished in 7.90s container-test-run-monitoring> cleanup container-test-run-monitoring> kill NspawnMachine (pid 55) container-test-run-monitoring> kill NspawnMachine (pid 52) container-test-run-monitoring> machine1 # [6104614.331489] machine1 grafana[440]: logger=settings t=2026-08-18T15:07:20.384567334Z level=info msg="Starting Grafana" version=13.1.3 commit=NA branch=main compiled=2026-08-18T15:07:20Z container-test-run-monitoring> Container machine1 terminated by signal KILL. container-test-run-monitoring> Container machine2 terminated by signal KILL. container-test-run-monitoring> (finished: cleanup, in 0.48 seconds) post-build step Upload to niks3: ok time=2026-08-18T15:07:21.430Z level=INFO msg="Uploading 0 paths to niks3.clan.lol (1 already cached)" time=2026-08-18T15:07:22.085Z level=WARN msg="Request returned retryable status, retrying" attempt=1 max_attempts=6 backoff=100ms status=500 url="https://s3.eu-central-003.backblazeb2.com/clan-cache-geninf/di5q9wx2xhnldiybn4wlxnbbx8hb4ahc.ls?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=003eaae730241090000000009%2F20260818%2Feu-central-003%2Fs3%2Faws4_request&X-Amz-Date=20260818T150721Z&X-Amz-Expires=18000&X-Amz-SignedHeaders=host&X-Amz-Signature=05bbacfc3dddeaf291d795576229a94549e96e839b89a5eb9897864b0d7a4072" time=2026-08-18T15:07:24.305Z level=INFO msg="Uploading 1 narinfos" time=2026-08-18T15:07:24.787Z level=INFO msg="Upload complete. (3.409s)"