don't know how to build these paths: /nix/store/d7jgpnfd10y81m3vzbs1ppz9hzpiafvr-container-test-run-monitoring.drv warning: ignoring the client-specified setting 'store', because it is a restricted setting and you are not a trusted user building '/nix/store/r4ga6aaxk69aiciis4yaa9ldpdprighr-state-commands.drv' on nix-store.geninf.io:50051 state-commands> gi-build-x86-02: building r4ga6aaxk69aiciis4yaa9ldpdprighr-state-commands.drv building '/nix/store/224xghi75a1b7k8gswdcmb1d0npw98w3-system-path.drv' on nix-store.geninf.io:50051 system-path> gi-build-x86-02: building 224xghi75a1b7k8gswdcmb1d0npw98w3-system-path.drv system-path> structuredAttrs is enabled system-path> created 1873 symlinks in user environment building '/nix/store/2vhrxmiwfm5xza599nyn6xzxz4zdr0fg-dbus-1.drv' on nix-store.geninf.io:50051 dbus-1> gi-build-x86-02: building 2vhrxmiwfm5xza599nyn6xzxz4zdr0fg-dbus-1.drv building '/nix/store/fgk704469876byjzfifqvnfgswr9zhdn-X-Restart-Triggers-dbus-broker.drv' on nix-store.geninf.io:50051 X-Restart-Triggers-dbus-broker> gi-build-x86-02: building fgk704469876byjzfifqvnfgswr9zhdn-X-Restart-Triggers-dbus-broker.drv building '/nix/store/akpcy90hqk64h5y7k3a6zriv38466a76-unit-dbus-broker.service.drv' on nix-store.geninf.io:50051 unit-dbus-broker.service> gi-build-x86-02: building akpcy90hqk64h5y7k3a6zriv38466a76-unit-dbus-broker.service.drv building '/nix/store/x08yv6idwq342dajq7qsbqpix16wchva-unit-dbus-broker.service.drv' on nix-store.geninf.io:50051 unit-dbus-broker.service> build-x86-01: building x08yv6idwq342dajq7qsbqpix16wchva-unit-dbus-broker.service.drv unit-dbus-broker.service> structuredAttrs is enabled unit-dbus-broker.service> structuredAttrs is enabled building '/nix/store/a72lsha0v6yd01m8d8pbdr7iyawk1mhz-user-units.drv' on nix-store.geninf.io:50051 user-units> gi-build-x86-02: building a72lsha0v6yd01m8d8pbdr7iyawk1mhz-user-units.drv building '/nix/store/mj8f37kzflvxj3z8l89y6zlyvzf79brd-system-units.drv' on nix-store.geninf.io:50051 system-units> build-x86-01: building mj8f37kzflvxj3z8l89y6zlyvzf79brd-system-units.drv building '/nix/store/bfnifq7mbc66q0g2kz5b06cbm3p6g8r7-etc.drv' on nix-store.geninf.io:50051 etc> gi-build-x86-02: building bfnifq7mbc66q0g2kz5b06cbm3p6g8r7-etc.drv building '/nix/store/hi80fn51p0c44r0f5qz3dl6p5w5q39ss-activate.drv' on nix-store.geninf.io:50051 activate> build-x86-01: building hi80fn51p0c44r0f5qz3dl6p5w5q39ss-activate.drv building '/nix/store/vphg07c54c2n9bkazv2a75didn00gllw-nixos-system-machine1-test.drv' on nix-store.geninf.io:50051 nixos-system-machine1-test> build-x86-01: building vphg07c54c2n9bkazv2a75didn00gllw-nixos-system-machine1-test.drv nixos-system-machine1-test> structuredAttrs is enabled building '/nix/store/16zqkfw1fvgkrv8s6mlg2m19cv719ac5-run-machine1-nspawn.drv' on nix-store.geninf.io:50051 run-machine1-nspawn> build-x86-01: building 16zqkfw1fvgkrv8s6mlg2m19cv719ac5-run-machine1-nspawn.drv building '/nix/store/yf1snc0m9kgwms7bncbjzrr34l6r2l74-driverConfiguration.json.drv' on nix-store.geninf.io:50051 driverConfiguration.json> build-x86-01: building yf1snc0m9kgwms7bncbjzrr34l6r2l74-driverConfiguration.json.drv driverConfiguration.json> structuredAttrs is enabled building '/nix/store/9ccxram683749xq5a5ds19h502c4w1m5-nixos-test-driver-monitoring.drv' on nix-store.geninf.io:50051 nixos-test-driver-monitoring> build-x86-01: building 9ccxram683749xq5a5ds19h502c4w1m5-nixos-test-driver-monitoring.drv nixos-test-driver-monitoring> Running type check (enable/disable: config.skipTypeCheck) nixos-test-driver-monitoring> See https://nixos.org/manual/nixos/stable/#test-opt-skipTypeCheck nixos-test-driver-monitoring> All checks passed! nixos-test-driver-monitoring> Linting test script (enable/disable: config.skipLint) nixos-test-driver-monitoring> See https://nixos.org/manual/nixos/stable/#test-opt-skipLint nixos-test-driver-monitoring> All checks passed! building '/nix/store/d7jgpnfd10y81m3vzbs1ppz9hzpiafvr-container-test-run-monitoring.drv' on nix-store.geninf.io:50051 container-test-run-monitoring> build-x86-01: building d7jgpnfd10y81m3vzbs1ppz9hzpiafvr-container-test-run-monitoring.drv container-test-run-monitoring> Machine state will be reset. To keep it, pass --keep-machine-state container-test-run-monitoring> start all VLans container-test-run-monitoring> (finished: start all VLans, in 0.00 seconds) container-test-run-monitoring> Initializing machine ID from random generator. container-test-run-monitoring> Test will time out and terminate in 3600.0 seconds container-test-run-monitoring> run the VM test script container-test-run-monitoring> additionally exposed symbols: container-test-run-monitoring> machine1, machine2, container-test-run-monitoring> vlan1, container-test-run-monitoring> start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh container-test-run-monitoring> start all VMs container-test-run-monitoring> machine2: systemd-nspawn running (pid 52) container-test-run-monitoring> machine1: systemd-nspawn running (pid 53) container-test-run-monitoring> machine1: Waiting for journal at /build/vm-state-machine1/var/log/journal... container-test-run-monitoring> machine2: Waiting for journal at /build/vm-state-machine2/var/log/journal... container-test-run-monitoring> (finished: start all VMs, in 0.00 seconds) container-test-run-monitoring> machine1: waiting for unit multi-user.target container-test-run-monitoring> nixos-nspawn(machine1): TAP vde-tap1 not found; container will be isolated from VDE container-test-run-monitoring> nixos-nspawn(machine1): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. container-test-run-monitoring> nixos-nspawn(machine2): TAP vde-tap1 not found; container will be isolated from VDE container-test-run-monitoring> nixos-nspawn(machine2): A common reason for this is that /dev/net is not available in the Nix sandbox. Try adding /dev/net to extra-sandbox-paths. container-test-run-monitoring> Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. container-test-run-monitoring> Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. container-test-run-monitoring> ░ Spawning container machine1 on /build/vm-state-machine1. container-test-run-monitoring> ░ Spawning container machine2 on /build/vm-state-machine2. container-test-run-monitoring> machine1 # [880780.203734] machine1 systemd-journald[123]: Journal started container-test-run-monitoring> machine1 # [880780.203759] machine1 systemd-journald[123]: Runtime Journal (/run/log/journal/687710f1bdf64438b76da043f813ae96) is 8M, max 3.7G, 3.7G free. container-test-run-monitoring> machine1 # [880780.206405] machine1 systemd[1]: Finished Create Static Device Nodes in /dev gracefully. container-test-run-monitoring> machine1 # [880780.210868] machine1 systemd[1]: Starting Flush Journal to Persistent Storage... container-test-run-monitoring> machine1 # [880780.211209] machine1 systemd[1]: Starting Network Name Resolution... container-test-run-monitoring> machine1 # [880780.211515] machine1 systemd[1]: Starting Create Static Device Nodes in /dev... container-test-run-monitoring> machine1 # [880780.215664] machine1 systemd-journald[123]: Time spent on flushing to /var/log/journal/687710f1bdf64438b76da043f813ae96 is 1.185ms for 6 entries. container-test-run-monitoring> machine1 # [880780.215664] machine1 systemd-journald[123]: System Journal (/var/log/journal/687710f1bdf64438b76da043f813ae96) is 8M, max 4G, 3.9G free. container-test-run-monitoring> machine1 # [880780.219208] machine1 systemd[1]: Finished Create Static Device Nodes in /dev. container-test-run-monitoring> machine1 # [880780.219309] machine1 systemd[1]: Reached target Preparation for Local File Systems. container-test-run-monitoring> machine1 # [880780.219350] machine1 systemd[1]: Reached target Local File Systems. container-test-run-monitoring> machine1 # [880780.219720] machine1 systemd[1]: Listening on Boot Loader Control Service Socket. container-test-run-monitoring> machine1 # [880780.219743] machine1 systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine1 # [880780.220097] machine1 systemd[1]: Starting Save Transient machine-id to Disk... container-test-run-monitoring> machine1 # [880780.220112] machine1 systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys container-test-run-monitoring> machine1 # [880780.220318] machine1 systemd[1]: Finished Flush Journal to Persistent Storage. container-test-run-monitoring> machine1 # [880780.220926] machine1 systemd[1]: Starting Create System Files and Directories... container-test-run-monitoring> machine1 # [880780.230402] machine1 systemd[1]: Finished Save Transient machine-id to Disk. container-test-run-monitoring> machine1 # [880780.230796] machine1 systemd-tmpfiles[162]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.230940] machine1 systemd-tmpfiles[162]: fchmod() of /var/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.231046] machine1 systemd-tmpfiles[162]: fchmod() of /var/log/journal/687710f1bdf64438b76da043f813ae96 failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.231191] machine1 systemd-tmpfiles[162]: fchmod() of /run/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.231993] machine1 systemd[1]: Finished Create System Files and Directories. container-test-run-monitoring> machine1 # [880780.232479] machine1 systemd[1]: Starting Rebuild Journal Catalog... container-test-run-monitoring> machine1 # [880780.232806] machine1 systemd[1]: Starting Record System Boot/Shutdown in UTMP... container-test-run-monitoring> machine1 # [880780.240617] machine1 systemd[1]: Finished Record System Boot/Shutdown in UTMP. container-test-run-monitoring> machine1 # [880780.246743] machine1 systemd[1]: Finished Rebuild Journal Catalog. container-test-run-monitoring> machine1 # [880780.247150] machine1 systemd[1]: Starting Update is Completed... container-test-run-monitoring> machine1 # [880780.252235] machine1 systemd[1]: Finished Update is Completed. container-test-run-monitoring> machine1 # [880780.312095] machine1 systemd[1]: Finished Firewall. container-test-run-monitoring> machine1 # [880780.312225] machine1 systemd[1]: Reached target Preparation for Network. container-test-run-monitoring> machine1 # [880780.312430] machine1 systemd[1]: Listening on Network Management Resolve Hook Socket. container-test-run-monitoring> machine1 # [880780.312985] machine1 systemd[1]: Starting Network Management... container-test-run-monitoring> machine1 # [880780.560143] machine1 systemd-networkd[239]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.560234] machine1 systemd-networkd[239]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.565264] machine1 systemd-networkd[239]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine1 # [880780.565426] machine1 systemd-networkd[239]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine1 # [880780.565495] machine1 systemd-networkd[239]: lo: Link UP container-test-run-monitoring> machine1 # [880780.565499] machine1 systemd-networkd[239]: lo: Gained carrier container-test-run-monitoring> machine2 # [880780.199917] machine2 systemd-journald[105]: Journal started container-test-run-monitoring> machine2 # [880780.199947] machine2 systemd-journald[105]: Runtime Journal (/run/log/journal/4202005adca443a6b2bd8d37fe10680c) is 8M, max 3.7G, 3.7G free. container-test-run-monitoring> machine2 # [880780.200281] machine2 systemd[1]: Finished Create Static Device Nodes in /dev gracefully. container-test-run-monitoring> machine2 # [880780.204651] machine2 systemd[1]: Starting Flush Journal to Persistent Storage... container-test-run-monitoring> machine2 # [880780.204990] machine2 systemd[1]: Starting Network Name Resolution... container-test-run-monitoring> machine2 # [880780.205333] machine2 systemd[1]: Starting Create Static Device Nodes in /dev... container-test-run-monitoring> machine2 # [880780.209525] machine2 systemd-journald[105]: Time spent on flushing to /var/log/journal/4202005adca443a6b2bd8d37fe10680c is 1.002ms for 6 entries. container-test-run-monitoring> machine2 # [880780.209525] machine2 systemd-journald[105]: System Journal (/var/log/journal/4202005adca443a6b2bd8d37fe10680c) is 8M, max 4G, 3.9G free. container-test-run-monitoring> machine2 # [880780.214323] machine2 systemd[1]: Finished Flush Journal to Persistent Storage. container-test-run-monitoring> machine2 # [880780.214524] machine2 systemd[1]: Finished Create Static Device Nodes in /dev. container-test-run-monitoring> machine2 # [880780.215220] machine2 systemd[1]: Reached target Preparation for Local File Systems. container-test-run-monitoring> machine2 # [880780.215282] machine2 systemd[1]: Reached target Local File Systems. container-test-run-monitoring> machine2 # [880780.215832] machine2 systemd[1]: Listening on Boot Loader Control Service Socket. container-test-run-monitoring> machine1 # [880780.565632] machine1 systemd-networkd[239]: eth1: Configuring with /etc/systemd/network/40-eth1.network. container-test-run-monitoring> machine2 # [880780.215856] machine2 systemd[1]: Update Boot Loader Random Seed skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine1 # [880780.565868] machine1 systemd[1]: Started Network Management. container-test-run-monitoring> machine2 # [880780.216219] machine2 systemd[1]: Starting Save Transient machine-id to Disk... container-test-run-monitoring> machine2 # [880780.216560] machine2 systemd[1]: Starting Create System Files and Directories... container-test-run-monitoring> machine1 # [880780.580115] machine1 systemd-networkd[239]: eth1: Link UP container-test-run-monitoring> machine2 # [880780.216579] machine2 systemd[1]: Rule-based Manager for Device Events and Files skipped, unmet condition check ConditionPathIsReadWrite=/sys container-test-run-monitoring> machine1 # [880780.580150] machine1 systemd[1]: Starting Enable Persistent Storage in systemd-networkd... container-test-run-monitoring> machine2 # [880780.228292] machine2 systemd-tmpfiles[146]: Cannot set file attributes for '/var/empty', value=0x00000010, mask=0x00000010, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.580267] machine1 systemd-networkd[239]: eth1: Gained carrier container-test-run-monitoring> machine2 # [880780.228442] machine2 systemd-tmpfiles[146]: fchmod() of /var/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.600076] machine1 systemd[1]: Finished Enable Persistent Storage in systemd-networkd. container-test-run-monitoring> machine2 # [880780.228952] machine2 systemd-tmpfiles[146]: fchmod() of /var/log/journal/4202005adca443a6b2bd8d37fe10680c failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.673413] machine1 systemd-resolved[147]: Positive Trust Anchors: container-test-run-monitoring> machine2 # [880780.229323] machine2 systemd-tmpfiles[146]: fchmod() of /run/log/journal failed: Operation not permitted container-test-run-monitoring> machine1 # [880780.673419] machine1 systemd-resolved[147]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d container-test-run-monitoring> machine2 # [880780.229923] machine2 systemd[1]: Finished Save Transient machine-id to Disk. container-test-run-monitoring> machine1 # [880780.673423] machine1 systemd-resolved[147]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16 container-test-run-monitoring> machine2 # [880780.230120] machine2 systemd[1]: Finished Create System Files and Directories. container-test-run-monitoring> machine1 # [880780.673438] machine1 systemd-resolved[147]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test container-test-run-monitoring> machine2 # [880780.230801] machine2 systemd[1]: Starting Rebuild Journal Catalog... container-test-run-monitoring> machine1 # [880780.683510] machine1 systemd-resolved[147]: Using system hostname 'machine1'. container-test-run-monitoring> machine2 # [880780.231113] machine2 systemd[1]: Starting Record System Boot/Shutdown in UTMP... container-test-run-monitoring> machine1 # [880780.684280] machine1 systemd[1]: Started Network Name Resolution. container-test-run-monitoring> machine2 # [880780.236770] machine2 systemd[1]: Finished Record System Boot/Shutdown in UTMP. container-test-run-monitoring> machine1 # [880780.684316] machine1 systemd[1]: Reached target Network. container-test-run-monitoring> machine1 # [880780.684340] machine1 systemd[1]: Reached target Network is Online. container-test-run-monitoring> machine2 # [880780.242831] machine2 systemd[1]: Finished Rebuild Journal Catalog. container-test-run-monitoring> machine2 # [880780.243295] machine2 systemd[1]: Starting Update is Completed... container-test-run-monitoring> machine1 # [880780.684364] machine1 systemd[1]: Reached target System Initialization. container-test-run-monitoring> machine2 # [880780.248932] machine2 systemd[1]: Finished Update is Completed. container-test-run-monitoring> machine1 # [880780.684395] machine1 systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine2 # [880780.280924] machine2 systemd[1]: Finished Firewall. container-test-run-monitoring> machine1 # [880780.684408] machine1 systemd[1]: Started Daily Cleanup of Temporary Directories. container-test-run-monitoring> machine2 # [880780.280987] machine2 systemd[1]: Reached target Preparation for Network. container-test-run-monitoring> machine1 # [880780.684417] machine1 systemd[1]: Reached target Timer Units. container-test-run-monitoring> machine2 # [880780.281130] machine2 systemd[1]: Listening on Network Management Resolve Hook Socket. container-test-run-monitoring> machine1 # [880780.684479] machine1 systemd[1]: Listening on D-Bus System Message Bus Socket. container-test-run-monitoring> machine2 # [880780.281696] machine2 systemd[1]: Starting Network Management... container-test-run-monitoring> machine1 # [880780.684537] machine1 systemd[1]: Listening on Nix Daemon Socket. container-test-run-monitoring> machine2 # [880780.551292] machine2 systemd-networkd[219]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.684604] machine1 systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. container-test-run-monitoring> machine2 # [880780.551359] machine2 systemd-networkd[219]: Failed to increase receive buffer size for nftables netlink socket, ignoring: Operation not permitted container-test-run-monitoring> machine1 # [880780.684616] machine1 systemd[1]: Reached target Socket Units. container-test-run-monitoring> machine2 # [880780.556832] machine2 systemd-networkd[219]: /etc/systemd/network/99-ethernet-default-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine1 # [880780.684635] machine1 systemd[1]: Reached target Basic System. container-test-run-monitoring> machine2 # [880780.556977] machine2 systemd-networkd[219]: /etc/systemd/network/99-wireless-client-dhcp.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section. container-test-run-monitoring> machine2 # [880780.557039] machine2 systemd-networkd[219]: lo: Link UP container-test-run-monitoring> machine2 # [880780.557042] machine2 systemd-networkd[219]: lo: Gained carrier container-test-run-monitoring> machine2 # [880780.557176] machine2 systemd-networkd[219]: eth1: Configuring with /etc/systemd/network/40-eth1.network. container-test-run-monitoring> machine2 # [880780.557399] machine2 systemd[1]: Started Network Management. container-test-run-monitoring> machine2 # [880780.557450] machine2 systemd-networkd[219]: eth1: Link UP container-test-run-monitoring> machine2 # [880780.557583] machine2 systemd-networkd[219]: eth1: Gained carrier container-test-run-monitoring> machine2 # [880780.557891] machine2 systemd[1]: Starting Enable Persistent Storage in systemd-networkd... container-test-run-monitoring> machine2 # [880780.586982] machine2 systemd[1]: Finished Enable Persistent Storage in systemd-networkd. container-test-run-monitoring> machine2 # [880780.666005] machine2 systemd-resolved[127]: Positive Trust Anchors: container-test-run-monitoring> machine2 # [880780.666013] machine2 systemd-resolved[127]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d container-test-run-monitoring> machine2 # [880780.666016] machine2 systemd-resolved[127]: . IN DS 38696 8 2 683d2d0acb8c9b712a1948b27f741219298d0a450d612c483af444a4c0fb2b16 container-test-run-monitoring> machine2 # [880780.666031] machine2 systemd-resolved[127]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test container-test-run-monitoring> machine2 # [880780.678302] machine2 systemd-resolved[127]: Using system hostname 'machine2'. container-test-run-monitoring> machine2 # [880780.679149] machine2 systemd[1]: Started Network Name Resolution. container-test-run-monitoring> machine2 # [880780.679189] machine2 systemd[1]: Reached target Network. container-test-run-monitoring> machine2 # [880780.679221] machine2 systemd[1]: Reached target System Initialization. container-test-run-monitoring> machine2 # [880780.679249] machine2 systemd[1]: Discard unused filesystem blocks once a week skipped, unmet condition check ConditionVirtualization=!container container-test-run-monitoring> machine2 # [880780.679266] machine2 systemd[1]: Started Daily Cleanup of Temporary Directories. container-test-run-monitoring> machine2 # [880780.679274] machine2 systemd[1]: Reached target Timer Units. container-test-run-monitoring> machine2 # [880780.679344] machine2 systemd[1]: Listening on D-Bus System Message Bus Socket. container-test-run-monitoring> machine2 # [880780.679399] machine2 systemd[1]: Listening on Nix Daemon Socket. container-test-run-monitoring> machine2 # [880780.679463] machine2 systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. container-test-run-monitoring> machine2 # [880780.679472] machine2 systemd[1]: Reached target Socket Units. container-test-run-monitoring> machine2 # [880780.679491] machine2 systemd[1]: Reached target Basic System. container-test-run-monitoring> machine2 # [880780.679972] machine2 systemd[1]: Started alloy.service. container-test-run-monitoring> machine2 # [880780.680353] machine2 systemd[1]: Starting Import lastlog data into lastlog2 database... container-test-run-monitoring> machine2 # [880780.680742] machine2 systemd[1]: Starting Name Service Cache Daemon (nsncd)... container-test-run-monitoring> machine2 # [880780.681345] machine2 systemd[1]: Starting D-Bus System Message Bus... container-test-run-monitoring> machine2 # [880780.710821] machine2 systemd[1]: Finished Import lastlog data into lastlog2 database. container-test-run-monitoring> machine2 # [880780.780477] machine2 nsncd[226]: Nov 13 10:13:19.222 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" container-test-run-monitoring> machine2 # [880780.780521] machine2 systemd[1]: Started Name Service Cache Daemon (nsncd). container-test-run-monitoring> machine2 # [880780.780553] machine2 systemd[1]: Reached target Host and Network Name Lookups. container-test-run-monitoring> machine2 # [880780.780583] machine2 systemd[1]: Reached target User and Group Name Lookups. container-test-run-monitoring> machine2 # [880780.781217] machine2 systemd[1]: Starting User Login Management... container-test-run-monitoring> machine2 # [880780.781602] machine2 systemd[1]: Starting Permit User Sessions... container-test-run-monitoring> machine1 # [880780.703161] machine1 systemd[1]: Started alloy.service. container-test-run-monitoring> machine1 # [880780.703473] machine1 systemd[1]: Starting Import lastlog data into lastlog2 database... container-test-run-monitoring> machine1 # [880780.703842] machine1 systemd[1]: Started Loki Service Daemon. container-test-run-monitoring> machine1 # [880780.704163] machine1 systemd[1]: Started mimir Service Daemon. container-test-run-monitoring> machine1 # [880780.704543] machine1 systemd[1]: Starting Nginx Web Server... container-test-run-monitoring> machine1 # [880780.704875] machine1 systemd[1]: Starting Name Service Cache Daemon (nsncd)... container-test-run-monitoring> machine1 # [880780.705311] machine1 systemd[1]: Starting PostgreSQL Server... container-test-run-monitoring> machine1 # [880780.705942] machine1 systemd[1]: Starting D-Bus System Message Bus... container-test-run-monitoring> machine1 # [880780.712876] machine1 systemd[1]: Finished Import lastlog data into lastlog2 database. container-test-run-monitoring> machine1 # [880780.789099] machine1 nsncd[249]: Nov 13 10:13:19.231 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" container-test-run-monitoring> machine1 # [880780.789164] machine1 systemd[1]: Started Name Service Cache Daemon (nsncd). container-test-run-monitoring> machine1 # [880780.789206] machine1 systemd[1]: Reached target Host and Network Name Lookups. container-test-run-monitoring> machine1 # [880780.789239] machine1 systemd[1]: Reached target User and Group Name Lookups. container-test-run-monitoring> machine1 # [880780.806173] machine1 systemd[1]: Starting User Login Management... container-test-run-monitoring> machine1 # [880780.806737] machine1 systemd[1]: Starting Permit User Sessions... container-test-run-monitoring> machine1 # [880780.811609] machine1 systemd[1]: Finished Permit User Sessions. container-test-run-monitoring> machine1 # [880780.812069] machine1 systemd[1]: Started Console Getty. container-test-run-monitoring> machine1 # [880780.812089] machine1 systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 container-test-run-monitoring> machine1 # [880780.812098] machine1 systemd[1]: Reached target Login Prompts. container-test-run-monitoring> machine1 # [880780.849832] machine1 dbus-broker-launch[252]: Looking up NSS user entry for 'systemd-timesync'... container-test-run-monitoring> machine1 # [880780.850205] machine1 dbus-broker-launch[252]: NSS returned no entry for 'systemd-timesync' container-test-run-monitoring> machine1 # [880780.850205] machine1 dbus-broker-launch[252]: Invalid user-name in /nix/store/jw08xhndd6c39x51k7yhca3vi65zs3m5-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" container-test-run-monitoring> machine1 # [880780.850424] machine1 systemd[1]: Started D-Bus System Message Bus. container-test-run-monitoring> machine1 # [880780.853777] machine1 dbus-broker-launch[252]: Ready container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.396160973Z caller=main.go:232 level=info msg="Starting application" version="(version=3.2.0, branch=unknown, revision=unknown)" container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.397007548Z caller=server.go:401 level=info msg="server listening on addresses" http=[::]:3001 grpc=[::]:9095 container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.409104869Z caller=memberlist_client.go:562 level=info component=memberlist msg="Using memberlist cluster label and node name" cluster_label= node=machine1-f6757be5 container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.410410048Z caller=modules.go:1217 level=info msg="The ruler is not being started because you need to configure the ruler storage." container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.411488029Z caller=inmemory.go:137 level=info msg="created in-memory index cache" maxItemSizeBytes=134217728 maxSizeBytes=1073741824 maxItems=maxInt container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.413488338Z caller=ingester.go:414 level=warn msg="-blocks-storage.backend=filesystem is for development and testing only; you should switch to an external object store for production use or use a shared filesystem" container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.414478533Z caller=ingester.go:584 level=info msg="TSDB idle compaction timeout set" timeout=1h11m51.159787901s container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.415171108Z caller=modules.go:1240 level=info msg="The ruler storage has not been configured. Not starting the ruler." container-test-run-monitoring> machine1 # [880781.053807] machine1 mimir[247]: ts=2026-11-13T10:13:19.415632448Z caller=modules.go:1038 level=info msg="The query-frontend has not been configured with the query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic frontend configuration. If queries are unresponsive, consider explicitly configuring the query-scheduler address for querier-frontend." address=127.0.0.1:9095 container-test-run-monitoring> machine2 # [880780.809459] machine2 systemd[1]: Finished Permit User Sessions. container-test-run-monitoring> machine2 # [880780.809909] machine2 systemd[1]: Started Console Getty. container-test-run-monitoring> machine2 # [880780.809930] machine2 systemd[1]: Getty on tty1 skipped, unmet condition check ConditionPathExists=/dev/tty0 container-test-run-monitoring> machine2 # [880780.809942] machine2 systemd[1]: Reached target Login Prompts. container-test-run-monitoring> machine2 # [880780.836520] machine2 dbus-broker-launch[227]: Looking up NSS user entry for 'systemd-timesync'... container-test-run-monitoring> machine2 # [880780.837086] machine2 dbus-broker-launch[227]: NSS returned no entry for 'systemd-timesync' container-test-run-monitoring> machine2 # [880780.837086] machine2 dbus-broker-launch[227]: Invalid user-name in /nix/store/fg9ykcz7w5yknifs6hkggpr98zz440lc-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" container-test-run-monitoring> machine2 # [880780.837486] machine2 systemd[1]: Started D-Bus System Message Bus. container-test-run-monitoring> machine2 # [880780.841203] machine2 dbus-broker-launch[227]: Ready container-test-run-monitoring> machine1 # [880781.072207] machine1 nginx-pre-start[346]: nginx: the configuration file /nix/store/dnma0np070wfb8g2y94bygqgv7kv4znm-nginx.conf syntax is ok container-test-run-monitoring> machine1 # [880781.072466] machine1 nginx-pre-start[346]: nginx: configuration file /nix/store/dnma0np070wfb8g2y94bygqgv7kv4znm-nginx.conf test is successful container-test-run-monitoring> machine1 # [880781.079519] machine1 postgresql-pre-start[348]: The files belonging to this database system will be owned by user "postgres". container-test-run-monitoring> machine1 # [880781.079519] machine1 postgresql-pre-start[348]: This user must also own the server process. container-test-run-monitoring> machine1 # [880781.079742] machine1 postgresql-pre-start[348]: The database cluster will be initialized with locale "en_US.UTF-8". container-test-run-monitoring> machine1 # [880781.079742] machine1 postgresql-pre-start[348]: The default database encoding has accordingly been set to "UTF8". container-test-run-monitoring> machine1 # [880781.079742] machine1 postgresql-pre-start[348]: The default text search configuration will be set to "english". container-test-run-monitoring> machine1 # [880781.079742] machine1 postgresql-pre-start[348]: Data page checksums are enabled. container-test-run-monitoring> machine1 # [880781.079742] machine1 postgresql-pre-start[348]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok container-test-run-monitoring> machine1 # [880781.080012] machine1 postgresql-pre-start[348]: creating subdirectories ... ok container-test-run-monitoring> machine1 # [880781.080070] machine1 postgresql-pre-start[348]: selecting dynamic shared memory implementation ... posix container-test-run-monitoring> machine1 # [880781.091900] machine1 postgresql-pre-start[348]: selecting default "max_connections" ... 100 container-test-run-monitoring> machine1 # [880781.107446] machine1 postgresql-pre-start[348]: selecting default "shared_buffers" ... 128MB container-test-run-monitoring> machine1 # [880781.120060] machine1 systemd-logind[274]: New seat seat0. container-test-run-monitoring> machine1 # [880781.120162] machine1 systemd[1]: Started User Login Management. container-test-run-monitoring> machine1 # [880781.120947] machine1 systemd[1]: Starting linger-users.service... container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.48971578Z caller=main.go:146 msg="Starting Loki" version="(version=3.7.7, branch=unknown, revision=unknown)" container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.489767478Z caller=main.go:147 msg="Loading configuration file" filename=/nix/store/h3rq2czfsg8d12mf7j7347149hymsixs-validate-loki-conf container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.490136012Z caller=modules.go:1597 msg="Ruler storage is not configured; ruler will not be started." config_key=ruler.storage container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.493096509Z caller=server.go:397 msg="server listening on addresses" http=[::]:3002 grpc=[::]:9096 container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.49665326Z caller=table_manager.go:136 index-store=tsdb-2025-11-01 msg="uploading tables" container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.496751826Z caller=table_manager.go:300 index-store=tsdb-2025-11-01 msg="query readiness setup completed" duration=2.104µs distinct_users_len=0 distinct_users= container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.496996457Z caller=shipper.go:165 index-store=tsdb-2025-11-01 msg="starting index shipper in RW mode" container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.497281293Z caller=head_manager.go:313 index-store=tsdb-2025-11-01 component=tsdb-head-manager msg="loaded wals by period" groups=0 container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.497313674Z caller=manager.go:86 index-store=tsdb-2025-11-01 component=tsdb-manager msg="loaded leftover local indices" err=null successful=true buckets=0 indices=0 failures=0 container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.497329244Z caller=head_manager.go:313 index-store=tsdb-2025-11-01 component=tsdb-head-manager msg="loaded wals by period" groups=0 container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.502532737Z caller=worker.go:134 component=querier msg="Starting querier worker using query-scheduler and scheduler ring for addresses" container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=warn ts=2026-11-13T10:13:19.503714424Z caller=modules.go:1643 msg="RulerStorage is nil. Not starting the ruler." container-test-run-monitoring> machine1 # [880781.146863] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513138729Z caller=module_service.go:82 msg=starting module=cache-generation-loader container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513145682Z caller=module_service.go:82 msg=starting module=server container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513184826Z caller=module_service.go:82 msg=starting module=memberlist-kv container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513189725Z caller=module_service.go:82 msg=starting module=query-frontend-tripperware container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513212488Z caller=module_service.go:82 msg=starting module=store container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513223579Z caller=module_service.go:82 msg=starting module=query-scheduler-ring container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513244989Z caller=module_service.go:82 msg=starting module=ring container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513321002Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513340299Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513350468Z caller=module_service.go:82 msg=starting module=compactor container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513372489Z caller=module_service.go:82 msg=starting module=ingester-querier container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513374072Z caller=module_service.go:82 msg=starting module=distributor container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513384933Z caller=module_service.go:82 msg=starting module=rule-evaluator container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513386486Z caller=module_service.go:82 msg=starting module=ingester container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51339933Z caller=ring.go:390 component=distributor msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=error ts=2026-11-13T10:13:19.513406223Z caller=ratestore.go:110 msg="error getting ingester clients" err="empty ring" container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513424728Z caller=basic_lifecycler.go:324 msg="instance not found in the ring" instance=machine1 ring=scheduler container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513430729Z caller=ingester.go:571 component=ingester msg="recovering from checkpoint" container-test-run-monitoring> machine1 # [880781.147178] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513432252Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [880781.146965] machine1 systemd[1]: linger-users.service: Deactivated successfully. container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513446709Z caller=basic_lifecycler.go:324 component=distributor msg="instance not found in the ring" instance=machine1 ring=distributor container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513504999Z caller=ringmanager.go:186 msg="waiting until scheduler is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51357975Z caller=basic_lifecycler.go:324 msg="instance not found in the ring" instance=machine1 ring=compactor container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513583156Z caller=ring.go:390 msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51358501Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513626959Z caller=compactor.go:420 msg="waiting until compactor is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513438594Z caller=checkpoint.go:549 msg="tmp checkpoint cleanup starting" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513676452Z caller=checkpoint.go:554 msg="tmp checkpoint cleanup done" duration=40.446µs success=true container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513703834Z caller=recovery.go:45 component=ingester msg="no checkpoint found, treating as no-op" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.513943505Z caller=ingester.go:587 component=ingester msg="recovered WAL checkpoint recovery finished" elapsed=506.785µs errors=false container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514002747Z caller=ingester.go:593 component=ingester msg="recovering from WAL" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514242098Z caller=ingester.go:609 component=ingester msg="WAL segment recovery finished" elapsed=822.37µs errors=false container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514258999Z caller=ingester.go:557 component=ingester msg="closing recoverer" container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514263628Z caller=ingester.go:565 component=ingester msg="WAL recovery finished" time=843.63µs container-test-run-monitoring> machine1 # [880781.147363] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514359238Z caller=wal.go:173 msg=started component=wal container-test-run-monitoring> machine1 # [880781.147106] machine1 systemd[1]: Finished linger-users.service. container-test-run-monitoring> machine1 # [880781.147516] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.514428449Z caller=lifecycler.go:694 component=ingester msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [880781.147516] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51445576Z caller=lifecycler.go:721 component=ingester msg="instance not found in ring, adding with no tokens" ring=ingester container-test-run-monitoring> machine1 # [880781.147516] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51449269Z caller=lifecycler.go:562 component=ingester msg="auto-joining cluster after timeout" ring=ingester container-test-run-monitoring> machine1 # [880781.147516] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.51442385Z caller=ingester.go:778 component=ingester msg="sleeping for initial delay before starting periodic flushing" delay=7.483111595s container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506037169Z caller=modules.go:755 level=info msg="The querier worker has not been configured with the query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-scheduler address for querier worker." address=127.0.0.1:9095 container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506070342Z caller=worker.go:148 level=info msg="Starting querier worker connected to query-scheduler" scheduler=127.0.0.1:9095 container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506658049Z caller=module_service.go:82 level=info msg=starting module=active-groups-cleanup-service container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506691322Z caller=module_service.go:82 level=info msg=starting module=sanity-check container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506711009Z caller=module_service.go:82 level=info msg=starting module=activity-tracker container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506739192Z caller=sanity_check.go:34 level=info msg="Checking directories read/write access" container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506929911Z caller=sanity_check.go:39 level=info msg="Directories read/write access successfully checked" container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.506935853Z caller=sanity_check.go:41 level=info msg="Checking object storage config" container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.507027375Z caller=sanity_check.go:46 level=info msg="Object storage config successfully checked" container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.507041542Z caller=module_service.go:82 level=info msg=starting module=server container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.50706724Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.50708336Z caller=module_service.go:82 level=info msg=starting module=ingester-ring container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.507087508Z caller=module_service.go:82 level=info msg=starting module=querier-lifecycler container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.507096415Z caller=module_service.go:82 level=info msg=starting module=query-scheduler container-test-run-monitoring> machine1 # [880781.153181] machine1 mimir[247]: ts=2026-11-13T10:13:19.507103318Z caller=module_service.go:82 level=info msg=starting module=compactor container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507116663Z caller=module_service.go:82 level=info msg=starting module=store-queryable container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507114369Z caller=module_service.go:82 level=info msg=starting module=store-gateway container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507113938Z caller=module_service.go:82 level=info msg=starting module=querier-ring container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507127514Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507140859Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507154504Z caller=module_service.go:82 level=info msg=starting module=distributor-service container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507147241Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=querier container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507163542Z caller=module_service.go:82 level=info msg=starting module=ingester-service container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507183098Z caller=module_service.go:82 level=info msg=starting module=query-frontend container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507202495Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507220789Z caller=frontend_scheduler_worker.go:146 level=info msg="adding connection to query-scheduler" addr=127.0.0.1:9095 container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507223855Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=distributor container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507226791Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507323533Z caller=ingester_tsdb.go:415 level=info msg="opening existing TSDBs" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507395649Z caller=ring.go:390 level=info msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507395388Z caller=basic_lifecycler.go:324 level=info msg="instance not found in the ring" instance=machine1 ring=store-gateway container-test-run-monitoring> machine1 # [880781.153418] machine1 mimir[247]: ts=2026-11-13T10:13:19.507405127Z caller=basic_lifecycler_delegates.go:63 level=info msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507403995Z caller=lifecycler.go:694 level=info msg="not loading tokens from file, tokens file path is empty" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507431416Z caller=lifecycler.go:721 level=info msg="instance not found in ring, adding with no tokens" ring=ingester container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507525503Z caller=lifecycler.go:562 level=info msg="auto-joining cluster after timeout" ring=ingester container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507582551Z caller=gateway.go:290 level=info msg="waiting until store-gateway is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507596227Z caller=gateway.go:294 level=info msg="store-gateway is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507498192Z caller=distributor.go:952 level=info msg="waiting until distributor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.50770424Z caller=bucket_stores.go:231 level=info msg="synchronizing TSDB blocks for all users" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507748403Z caller=ring.go:390 level=info component=compactor msg="ring doesn't exist in KV store yet" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507776466Z caller=basic_lifecycler.go:324 level=info component=compactor msg="instance not found in the ring" instance=machine1 ring=compactor container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507956185Z caller=bucket_stores.go:244 level=info msg="successfully synchronized TSDB blocks for all users" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.507925657Z caller=compactor.go:616 level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.508104955Z caller=compactor.go:621 level=info component=compactor msg="compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.50817124Z caller=compactor.go:743 level=info component=compactor msg="discovering users from bucket" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.508174326Z caller=blocks_cleaner.go:244 level=info component=cleaner task=clean_up_users_during_startup msg="started blocks cleanup and maintenance" container-test-run-monitoring> machine1 # [880781.153591] machine1 mimir[247]: ts=2026-11-13T10:13:19.508184024Z caller=compactor.go:753 level=info component=compactor msg="discovered users from bucket" users=0 container-test-run-monitoring> machine1 # [880781.153742] machine1 mimir[247]: ts=2026-11-13T10:13:19.50835197Z caller=blocks_cleaner.go:250 level=info component=cleaner task=clean_up_users_during_startup msg="successfully completed blocks cleanup and maintenance" container-test-run-monitoring> machine1 # [880781.198483] machine1 systemd[1]: etc-machine\x2did.mount: Deactivated successfully. container-test-run-monitoring> machine1 # [880781.233234] machine1 alloy[244]: ts=2026-11-13T10:13:19.667235877Z level=info msg="Alloy is starting" container-test-run-monitoring> machine1 # [880781.233234] machine1 alloy[244]: ts=2026-11-13T10:13:19.667378145Z level=info msg="boringcrypto enabled" enabled=false container-test-run-monitoring> machine1 # [880781.233234] machine1 alloy[244]: ts=2026-11-13T10:13:19.66964725Z level=info msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit container-test-run-monitoring> machine1 # [880781.233234] machine1 alloy[244]: ts=2026-11-13T10:13:19.669781864Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.672230888Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.672267296Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=remotecfg duration=33.092µs container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.672273829Z level=info msg="applying non-TLS config to HTTP server" service=http container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.67227445Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=http duration=5.67µs container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.672276253Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=cluster duration=531ns container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.673451547Z level=info msg="replaying WAL, this may take a while" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal dir=data-alloy/prometheus.remote_write.mimir/wal container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.673541968Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal segment=0 maxSegment=0 container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.673876147Z level=info msg="Starting WAL watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=89eafe url=http://machine1.clan/mimir/api/v1/push queue=89eafe container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.673879534Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=89eafe url=http://machine1.clan/mimir/api/v1/push container-test-run-monitoring> machine1 # [880781.233464] machine1 alloy[244]: ts=2026-11-13T10:13:19.673884814Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=prometheus.remote_write.mimir duration=1.607479ms container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.673961198Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=89eafe url=http://machine1.clan/mimir/api/v1/push queue=89eafe container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674226157Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=prometheus.relabel.create_nixos_services_metric duration=338.478µs container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674438306Z level=info msg="Parsed flag --collector.filesystem.mount-points-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674439619Z level=info msg="Parsed flag --collector.filesystem.fs-types-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674530821Z level=info msg="Parsed flag --collector.systemd.unit-include" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+ container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674534738Z level=info msg="Parsed flag --collector.systemd.unit-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+\.(automount|device|mount|scope|slice) container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674582568Z level=info msg="Enabled node_exporter collectors" component_path=/ component_id=prometheus.exporter.unix.local_system container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.67458336Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=cpu container-test-run-monitoring> machine1 # [880781.233601] machine1 alloy[244]: ts=2026-11-13T10:13:19.674584372Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=filesystem container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.674584652Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=meminfo container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.674584963Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=systemd container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.674590122Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=prometheus.exporter.unix.local_system duration=361.492µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675552295Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=prometheus.scrape.scrape_metrics duration=959.698µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675653325Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=logging duration=98.356µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675663455Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=livedebugging duration=6.092µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675668264Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=ui duration=1.463µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675674315Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=tracing duration=3.256µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675678964Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=otel duration=1.783µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675684494Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=labelstore duration=2.605µs container-test-run-monitoring> machine1 # [880781.233734] machine1 alloy[244]: ts=2026-11-13T10:13:19.675822194Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=loki.write.loki duration=134.173µs container-test-run-monitoring> machine1 # [880781.233870] machine1 alloy[244]: ts=2026-11-13T10:13:19.675944294Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=loki.relabel.journal duration=112.622µs container-test-run-monitoring> machine1 # [880781.233870] machine1 alloy[244]: ts=2026-11-13T10:13:19.676037119Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db node_id=loki.source.journal.all duration=84.8µs container-test-run-monitoring> machine1 # [880781.233870] machine1 alloy[244]: ts=2026-11-13T10:13:19.676043461Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=f3f49c63dfd3c34ac404265d73b6e0db duration=4.419376ms container-test-run-monitoring> machine1 # [880781.233870] machine1 alloy[244]: ts=2026-11-13T10:13:19.676124674Z level=info msg="scheduling loaded components and services" controller_id="" container-test-run-monitoring> machine1 # [880781.233870] machine1 alloy[244]: ts=2026-11-13T10:13:19.676136386Z level=info msg="{^_^} Alloy is running" container-test-run-monitoring> machine1 # [880781.234683] machine1 alloy[244]: ts=2026-11-13T10:13:19.67699836Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s container-test-run-monitoring> machine1 # [880781.234849] machine1 alloy[244]: ts=2026-11-13T10:13:19.677169763Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=machine1 container-test-run-monitoring> machine1 # [880781.234996] machine1 alloy[244]: ts=2026-11-13T10:13:19.677242309Z level=error msg="failed to register collector with remote server" service=remotecfg id=15531ef7-277f-4093-84d0-4abf4436cc2b name="" err="noop client" container-test-run-monitoring> machine1 # [880781.235598] machine1 alloy[244]: ts=2026-11-13T10:13:19.677787076Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345 container-test-run-monitoring> machine1 # [880781.246318] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.632475148Z caller=ringmanager.go:190 msg="scheduler is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.246318] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.632556031Z caller=ringmanager.go:199 msg="waiting until scheduler is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.246318] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.657591617Z caller=compactor.go:424 msg="compactor is JOINING in the ring" container-test-run-monitoring> machine1 # [880781.246318] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.657666759Z caller=compactor.go:434 msg="waiting until compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.296086] machine1 postgresql-pre-start[348]: selecting default time zone ... UTC container-test-run-monitoring> machine1 # [880781.296442] machine1 postgresql-pre-start[348]: creating configuration files ... ok container-test-run-monitoring> machine1 # [880781.346749] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.774945371Z caller=ringmanager.go:203 msg="scheduler is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.346749] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.774984335Z caller=module_service.go:82 msg=starting module=query-scheduler container-test-run-monitoring> machine1 # [880781.346749] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.775023108Z caller=module_service.go:82 msg=starting module=querier container-test-run-monitoring> machine1 # [880781.346749] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.775071148Z caller=module_service.go:82 msg=starting module=query-frontend container-test-run-monitoring> machine2 # [880781.127186] machine2 systemd-logind[243]: New seat seat0. container-test-run-monitoring> machine2 # [880781.127322] machine2 systemd[1]: Started User Login Management. container-test-run-monitoring> machine2 # [880781.142260] machine2 systemd[1]: Starting linger-users.service... container-test-run-monitoring> machine2 # [880781.148847] machine2 systemd[1]: linger-users.service: Deactivated successfully. container-test-run-monitoring> machine2 # [880781.148907] machine2 systemd[1]: Finished linger-users.service. container-test-run-monitoring> machine2 # [880781.149152] machine2 systemd[1]: Reached target Multi-User System. container-test-run-monitoring> machine2 # [880781.149266] machine2 systemd[1]: Startup finished in 1.181s. container-test-run-monitoring> machine2 # [880781.193151] machine2 systemd[1]: etc-machine\x2did.mount: Deactivated successfully. container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.667233823Z level=info msg="Alloy is starting" container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.667376102Z level=info msg="boringcrypto enabled" enabled=false container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.669645777Z level=info msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.669782886Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.672230908Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.672240095Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=labelstore duration=5.651µs container-test-run-monitoring> machine2 # [880781.232402] machine2 alloy[224]: ts=2026-11-13T10:13:19.672245415Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=livedebugging duration=3.387µs container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.67310814Z level=info msg="replaying WAL, this may take a while" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal dir=data-alloy/prometheus.remote_write.mimir/wal container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.673392005Z level=info msg="WAL segment loaded" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=wal segment=0 maxSegment=0 container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674012424Z level=info msg="Starting WAL watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=2d899e url=http://machine1.clan/mimir/api/v1/push queue=2d899e container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674013606Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=2d899e url=http://machine1.clan/mimir/api/v1/push container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674017423Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=prometheus.remote_write.mimir duration=1.770465ms container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.67410563Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.mimir subcomponent=rw remote_name=2d899e url=http://machine1.clan/mimir/api/v1/push queue=2d899e container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674180962Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=prometheus.relabel.create_nixos_services_metric duration=160.894µs container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674527013Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=loki.write.loki duration=343.247µs container-test-run-monitoring> machine2 # [880781.232795] machine2 alloy[224]: ts=2026-11-13T10:13:19.674622543Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=loki.relabel.journal duration=93.186µs container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.67470007Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=loki.source.journal.all duration=75.723µs container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.674706131Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=tracing duration=4.438µs container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.674790019Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=logging duration=82.165µs container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.674866343Z level=info msg="Parsed flag --collector.filesystem.mount-points-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.67487535Z level=info msg="Parsed flag --collector.filesystem.fs-types-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.674950491Z level=info msg="Parsed flag --collector.systemd.unit-include" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+ container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.674961242Z level=info msg="Parsed flag --collector.systemd.unit-exclude" component_path=/ component_id=prometheus.exporter.unix.local_system collector=systemd flag=.+\.(automount|device|mount|scope|slice) container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.675017938Z level=info msg="Enabled node_exporter collectors" component_path=/ component_id=prometheus.exporter.unix.local_system container-test-run-monitoring> machine2 # [880781.232932] machine2 alloy[224]: ts=2026-11-13T10:13:19.675023318Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=cpu container-test-run-monitoring> machine2 # [880781.233064] machine2 alloy[224]: ts=2026-11-13T10:13:19.675026034Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=filesystem container-test-run-monitoring> machine2 # [880781.233064] machine2 alloy[224]: ts=2026-11-13T10:13:19.675028208Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=meminfo container-test-run-monitoring> machine2 # [880781.233064] machine2 alloy[224]: ts=2026-11-13T10:13:19.675030452Z level=info msg=collector component_path=/ component_id=prometheus.exporter.unix.local_system name=systemd container-test-run-monitoring> machine2 # [880781.233064] machine2 alloy[224]: ts=2026-11-13T10:13:19.675036774Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=prometheus.exporter.unix.local_system duration=243.178µs container-test-run-monitoring> machine2 # [880781.233361] machine2 alloy[224]: ts=2026-11-13T10:13:19.675684755Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=prometheus.scrape.scrape_metrics duration=642.15µs container-test-run-monitoring> machine2 # [880781.233406] machine2 alloy[224]: ts=2026-11-13T10:13:19.675744517Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=remotecfg duration=37.46µs container-test-run-monitoring> machine2 # [880781.233422] machine2 alloy[224]: ts=2026-11-13T10:13:19.675757361Z level=info msg="applying non-TLS config to HTTP server" service=http container-test-run-monitoring> machine2 # [880781.233422] machine2 alloy[224]: ts=2026-11-13T10:13:19.675765447Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=http duration=15.539µs container-test-run-monitoring> machine2 # [880781.233453] machine2 alloy[224]: ts=2026-11-13T10:13:19.675771107Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=ui duration=2.194µs container-test-run-monitoring> machine2 # [880781.233453] machine2 alloy[224]: ts=2026-11-13T10:13:19.675777349Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=cluster duration=2.114µs container-test-run-monitoring> machine2 # [880781.233453] machine2 alloy[224]: ts=2026-11-13T10:13:19.675782128Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 node_id=otel duration=1.884µs container-test-run-monitoring> machine2 # [880781.233453] machine2 alloy[224]: ts=2026-11-13T10:13:19.675785134Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=0b517f9796734862d41872bdd0f27168 duration=4.022228ms container-test-run-monitoring> machine2 # [880781.233530] machine2 alloy[224]: ts=2026-11-13T10:13:19.67585206Z level=info msg="scheduling loaded components and services" controller_id="" container-test-run-monitoring> machine2 # [880781.233570] machine2 alloy[224]: ts=2026-11-13T10:13:19.675900862Z level=info msg="{^_^} Alloy is running" container-test-run-monitoring> machine2 # [880781.233727] machine2 alloy[224]: ts=2026-11-13T10:13:19.676047258Z level=error msg="failed to register collector with remote server" service=remotecfg id=1be5df2c-270d-492a-9e88-cbc519deac48 name="" err="noop client" container-test-run-monitoring> machine2 # [880781.233777] machine2 alloy[224]: ts=2026-11-13T10:13:19.67595824Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s container-test-run-monitoring> machine2 # [880781.233824] machine2 alloy[224]: ts=2026-11-13T10:13:19.676161232Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=machine2 container-test-run-monitoring> machine2 # [880781.234040] machine2 alloy[224]: ts=2026-11-13T10:13:19.67636712Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345 container-test-run-monitoring> machine1 # [880781.368532] machine1 postgresql-pre-start[348]: running bootstrap script ... ok container-test-run-monitoring> machine1 # [880781.394045] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.836316845Z caller=compactor.go:438 msg="compactor is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880781.394045] machine1 loki[246]: level=info ts=2026-11-13T10:13:19.836332855Z caller=loki.go:615 msg="Loki started" startup_time=389.041944ms container-test-run-monitoring> machine1 # [880781.402532] machine1 systemd[1]: Started Nginx Web Server. container-test-run-monitoring> machine1 # [880781.552887] machine1 postgresql-pre-start[348]: performing post-bootstrap initialization ... ok container-test-run-monitoring> machine1 # [880781.575662] machine1 postgresql-pre-start[348]: syncing data to disk ... ok container-test-run-monitoring> machine1 # [880781.575662] machine1 postgresql-pre-start[348]: initdb: warning: enabling "trust" authentication for local connections container-test-run-monitoring> machine1 # [880781.575662] machine1 postgresql-pre-start[348]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. container-test-run-monitoring> machine1 # [880781.575662] machine1 postgresql-pre-start[348]: Success. You can now start the database server using: container-test-run-monitoring> machine1 # [880781.575662] machine1 postgresql-pre-start[348]: pg_ctl -D /var/lib/postgresql/18 -l logfile start container-test-run-monitoring> machine2 # [880781.888098] machine2 systemd-networkd[219]: eth1: Gained IPv6LL container-test-run-monitoring> machine1 # [880781.919438] machine1 postgres[380]: [380] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit container-test-run-monitoring> machine1 # [880781.920357] machine1 postgres[380]: [380] LOG: listening on IPv6 address "::1", port 5432 container-test-run-monitoring> machine1 # [880781.920394] machine1 postgres[380]: [380] LOG: listening on IPv4 address "127.0.0.1", port 5432 container-test-run-monitoring> machine1 # [880781.920539] machine1 postgres[380]: [380] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" container-test-run-monitoring> machine1 # [880781.922603] machine1 postgres[389]: [389] LOG: database system was shut down at 2026-11-13 10:13:19 GMT container-test-run-monitoring> machine1 # [880781.924211] machine1 postgres[380]: [380] LOG: database system is ready to accept connections container-test-run-monitoring> machine1 # [880782.144033] machine1 systemd-networkd[239]: eth1: Gained IPv6LL container-test-run-monitoring> machine1 # [880782.153442] machine1 mimir[247]: ts=2026-11-13T10:13:20.508669524Z caller=gateway.go:330 level=info msg="waiting until store-gateway is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880782.253868] machine1 mimir[247]: ts=2026-11-13T10:13:20.685061466Z caller=gateway.go:334 level=info msg="store-gateway is ACTIVE in the ring" container-test-run-monitoring> machine1 # [880782.264262] machine1 postgresql-post-start[402]: CREATE ROLE container-test-run-monitoring> machine1 # [880782.300366] machine1 postgresql-post-start[407]: CREATE DATABASE container-test-run-monitoring> machine1 # [880782.302123] machine1 systemd[1]: Started PostgreSQL Server. container-test-run-monitoring> machine1 # [880782.302764] machine1 systemd[1]: Starting PostgreSQL Setup Scripts... container-test-run-monitoring> machine1 # [880782.332131] machine1 systemd[1]: Finished PostgreSQL Setup Scripts. container-test-run-monitoring> machine1 # [880782.332321] machine1 systemd[1]: Reached target PostgreSQL. container-test-run-monitoring> machine1 # [880782.332847] machine1 systemd[1]: Starting Grafana Service Daemon... container-test-run-monitoring> machine1: (finished: waiting for unit multi-user.target, in 3.64 seconds) container-test-run-monitoring> machine2: waiting for unit multi-user.target container-test-run-monitoring> machine2: (finished: waiting for unit multi-user.target, in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit alloy container-test-run-monitoring> machine1: (finished: waiting for unit alloy, in 0.01 seconds) container-test-run-monitoring> machine2: waiting for unit alloy container-test-run-monitoring> machine2: (finished: waiting for unit alloy, in 0.01 seconds) container-test-run-monitoring> machine1: must succeed: test -f /etc/alloy/config.alloy container-test-run-monitoring> machine1: (finished: must succeed: test -f /etc/alloy/config.alloy, in 0.00 seconds) container-test-run-monitoring> machine2: must succeed: test -f /etc/alloy/config.alloy container-test-run-monitoring> machine2: (finished: must succeed: test -f /etc/alloy/config.alloy, in 0.00 seconds) container-test-run-monitoring> machine2: must succeed: test -f /etc/alloy/local-extension.alloy container-test-run-monitoring> machine2: (finished: must succeed: test -f /etc/alloy/local-extension.alloy, in 0.00 seconds) container-test-run-monitoring> machine1: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy' container-test-run-monitoring> machine1: (finished: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy', in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy' container-test-run-monitoring> machine2: (finished: must succeed: systemctl show alloy --property=ExecStart | grep -F '/etc/alloy', in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit loki container-test-run-monitoring> machine1: (finished: waiting for unit loki, in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit mimir container-test-run-monitoring> machine1: (finished: waiting for unit mimir, in 0.01 seconds) container-test-run-monitoring> machine1: waiting for unit grafana container-test-run-monitoring> machine1: (finished: waiting for unit grafana, in 0.01 seconds) container-test-run-monitoring> machine2: must succeed: test "$(grep -c '^loki.source.journal ' /etc/alloy/config.alloy)" = 1 container-test-run-monitoring> machine2: (finished: must succeed: test "$(grep -c '^loki.source.journal ' /etc/alloy/config.alloy)" = 1, in 0.00 seconds) container-test-run-monitoring> machine2: must succeed: cat /etc/alloy/config.alloy container-test-run-monitoring> machine2: (finished: must succeed: cat /etc/alloy/config.alloy, in 0.00 seconds) container-test-run-monitoring> (finished: run the VM test script, in 3.71 seconds) container-test-run-monitoring> test script finished in 3.73s container-test-run-monitoring> cleanup container-test-run-monitoring> kill NspawnMachine (pid 53) container-test-run-monitoring> kill NspawnMachine (pid 52) container-test-run-monitoring> machine1 # [880783.229938] machine1 systemd[1]: Started Grafana Service Daemon. container-test-run-monitoring> (finished: cleanup, in 0.09 seconds) container-test-run-monitoring> Container machine1 terminated by signal KILL. container-test-run-monitoring> Container machine2 terminated by signal KILL.