mirror/orchestrator.log
[11:24:05] ╔══════════════════════════════════════════════════════════════╗ [11:24:05] ║ F16 NVMe/ZFS Full Benchmark Suite ║ [11:24:05] ╠══════════════════════════════════════════════════════════════╣ [11:24:05] ║ Pool layout : mirror [11:24:05] ║ Target : fs:/tank-mirror/bench [11:24:05] ║ Target type : file [11:24:05] ║ Runs/test : 3 [11:24:05] ║ Cooldown : 30s [11:24:05] ║ fio size : 64G [11:24:05] ║ fio runtime : 180s + 10s ramp [11:24:05] ║ Results : /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [11:24:05] ╚══════════════════════════════════════════════════════════════╝ [11:24:05] Capturing Phase 2 pool documentation for pool: tank-mirror [11:24:05] Pool documentation saved → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/pool-config/ [11:24:05] Generating Sequential Throughput jobs … [11:24:05] Generating Random IOPS jobs … [11:24:05] Generating Latency jobs … [11:24:05] Generating Mixed 70/30 jobs … [11:24:05] [11:24:05] Generated 53 fio jobs × 3 runs = 159 total runs [11:24:05] Estimated time: ~10h 22m [11:24:05] [11:24:05] [11:24:05] ╔══════════════════════════════════════════════════════════╗ [11:24:05] ║ Job 1/53: seq-read-qd1-nj1 [11:24:05] ║ Category: seq [11:24:05] ╚══════════════════════════════════════════════════════════╝ [11:24:05] ╔══════════════════════════════════════════════════════════╗ [11:24:05] ║ Benchmark: seq-read-qd1-nj1 [11:24:05] ║ Category: seq [11:24:05] ║ Runs: 3 × cooldown 30s [11:24:05] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [11:24:05] ╚══════════════════════════════════════════════════════════╝ [11:24:05] ──────────────────────────────────────────────────── [11:24:05] Run 1/3: seq-read-qd1-nj1_run1 [11:24:05] ──────────────────────────────────────────────────── [11:24:05] Cleaning fio test files from /tank-mirror/bench … [11:24:05] Test files removed [11:24:05] Dropping Linux page cache + ZFS ARC … [11:24:06] Flushing ZFS ARC (was 1) … [11:24:11] ARC flushed — current size: 6149448 bytes [11:24:11] NVMe volatile write caches flushed [11:24:13] pmcd healthy (core ✓ pmdazfs ✓) [11:24:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj1_run1.ndjson [11:24:14] Collector running (PID 2428412), settling 5s … [11:24:19] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj1.fio [11:27:47] fio completed in 208s [11:27:47] Post-fio settle 5s … [11:27:52] Stopping collector [11:27:53] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj1_run1.manifest.json [11:27:53] Cooldown: 30s [11:28:23] ──────────────────────────────────────────────────── [11:28:23] Run 2/3: seq-read-qd1-nj1_run2 [11:28:23] ──────────────────────────────────────────────────── [11:28:23] Cleaning fio test files from /tank-mirror/bench … [11:28:23] Test files removed [11:28:23] Dropping Linux page cache + ZFS ARC … [11:28:23] Flushing ZFS ARC (was 1) … [11:28:28] ARC flushed — current size: 4160776 bytes [11:28:28] NVMe volatile write caches flushed [11:28:30] pmcd healthy (core ✓ pmdazfs ✓) [11:28:30] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj1_run2.ndjson [11:28:31] Collector running (PID 2432454), settling 5s … [11:28:36] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj1.fio [11:32:04] fio completed in 208s [11:32:04] Post-fio settle 5s … [11:32:09] Stopping collector [11:32:10] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj1_run2.manifest.json [11:32:10] Cooldown: 30s [11:32:40] ──────────────────────────────────────────────────── [11:32:40] Run 3/3: seq-read-qd1-nj1_run3 [11:32:40] ──────────────────────────────────────────────────── [11:32:40] Cleaning fio test files from /tank-mirror/bench … [11:32:40] Test files removed [11:32:40] Dropping Linux page cache + ZFS ARC … [11:32:40] Flushing ZFS ARC (was 1) … [11:32:45] ARC flushed — current size: 4371544 bytes [11:32:45] NVMe volatile write caches flushed [11:32:47] pmcd healthy (core ✓ pmdazfs ✓) [11:32:47] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj1_run3.ndjson [11:32:48] Collector running (PID 2435142), settling 5s … [11:32:53] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj1.fio [11:36:21] fio completed in 208s [11:36:21] Post-fio settle 5s … [11:36:26] Stopping collector [11:36:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj1_run3.manifest.json [11:36:27] ═══════════════════════════════════════════════════════════ [11:36:27] All 3 runs complete for: seq-read-qd1-nj1 [11:36:27] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [11:36:27] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [11:36:27] ═══════════════════════════════════════════════════════════ [11:36:27] Progress: 1/53 jobs | Elapsed: 0h12m | ETA: ~10h43m [11:36:27] [11:36:27] ╔══════════════════════════════════════════════════════════╗ [11:36:27] ║ Job 2/53: seq-read-qd1-nj4 [11:36:27] ║ Category: seq [11:36:27] ╚══════════════════════════════════════════════════════════╝ [11:36:27] ╔══════════════════════════════════════════════════════════╗ [11:36:27] ║ Benchmark: seq-read-qd1-nj4 [11:36:27] ║ Category: seq [11:36:27] ║ Runs: 3 × cooldown 30s [11:36:27] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [11:36:27] ╚══════════════════════════════════════════════════════════╝ [11:36:27] ──────────────────────────────────────────────────── [11:36:27] Run 1/3: seq-read-qd1-nj4_run1 [11:36:27] ──────────────────────────────────────────────────── [11:36:27] Cleaning fio test files from /tank-mirror/bench … [11:36:27] Test files removed [11:36:27] Dropping Linux page cache + ZFS ARC … [11:36:27] Flushing ZFS ARC (was 1) … [11:36:32] ARC flushed — current size: 4651496 bytes [11:36:32] NVMe volatile write caches flushed [11:36:34] pmcd healthy (core ✓ pmdazfs ✓) [11:36:34] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj4_run1.ndjson [11:36:35] Collector running (PID 2437756), settling 5s … [11:36:40] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj4.fio [11:41:03] fio completed in 263s [11:41:03] Post-fio settle 5s … [11:41:08] Stopping collector [11:41:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj4_run1.manifest.json [11:41:09] Cooldown: 30s [11:41:39] ──────────────────────────────────────────────────── [11:41:39] Run 2/3: seq-read-qd1-nj4_run2 [11:41:39] ──────────────────────────────────────────────────── [11:41:39] Cleaning fio test files from /tank-mirror/bench … [11:41:39] Test files removed [11:41:39] Dropping Linux page cache + ZFS ARC … [11:41:39] Flushing ZFS ARC (was 1) … [11:41:44] ARC flushed — current size: 15894076240 bytes [11:41:44] NVMe volatile write caches flushed [11:41:46] pmcd healthy (core ✓ pmdazfs ✓) [11:41:46] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj4_run2.ndjson [11:41:47] Collector running (PID 2441227), settling 5s … [11:41:52] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj4.fio [11:46:14] fio completed in 262s [11:46:14] Post-fio settle 5s … [11:46:19] Stopping collector [11:46:21] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj4_run2.manifest.json [11:46:21] Cooldown: 30s [11:46:51] ──────────────────────────────────────────────────── [11:46:51] Run 3/3: seq-read-qd1-nj4_run3 [11:46:51] ──────────────────────────────────────────────────── [11:46:51] Cleaning fio test files from /tank-mirror/bench … [11:46:51] Test files removed [11:46:51] Dropping Linux page cache + ZFS ARC … [11:46:51] Flushing ZFS ARC (was 1) … [11:46:56] ARC flushed — current size: 6043200 bytes [11:46:56] NVMe volatile write caches flushed [11:46:58] pmcd healthy (core ✓ pmdazfs ✓) [11:46:58] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj4_run3.ndjson [11:46:59] Collector running (PID 2444707), settling 5s … [11:47:04] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj4.fio [11:51:26] fio completed in 262s [11:51:26] Post-fio settle 5s … [11:51:31] Stopping collector [11:51:32] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj4_run3.manifest.json [11:51:32] ═══════════════════════════════════════════════════════════ [11:51:32] All 3 runs complete for: seq-read-qd1-nj4 [11:51:32] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [11:51:32] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [11:51:32] ═══════════════════════════════════════════════════════════ [11:51:32] Progress: 2/53 jobs | Elapsed: 0h27m | ETA: ~11h39m [11:51:32] [11:51:32] ╔══════════════════════════════════════════════════════════╗ [11:51:32] ║ Job 3/53: seq-read-qd1-nj8 [11:51:32] ║ Category: seq [11:51:32] ╚══════════════════════════════════════════════════════════╝ [11:51:32] ╔══════════════════════════════════════════════════════════╗ [11:51:32] ║ Benchmark: seq-read-qd1-nj8 [11:51:32] ║ Category: seq [11:51:32] ║ Runs: 3 × cooldown 30s [11:51:32] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [11:51:32] ╚══════════════════════════════════════════════════════════╝ [11:51:32] ──────────────────────────────────────────────────── [11:51:32] Run 1/3: seq-read-qd1-nj8_run1 [11:51:32] ──────────────────────────────────────────────────── [11:51:32] Cleaning fio test files from /tank-mirror/bench … [11:51:32] Test files removed [11:51:32] Dropping Linux page cache + ZFS ARC … [11:51:32] Flushing ZFS ARC (was 1) … [11:51:37] ARC flushed — current size: 879010808 bytes [11:51:37] NVMe volatile write caches flushed [11:51:39] pmcd healthy (core ✓ pmdazfs ✓) [11:51:39] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj8_run1.ndjson [11:51:40] Collector running (PID 2448059), settling 5s … [11:51:45] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj8.fio [11:57:21] fio completed in 336s [11:57:21] Post-fio settle 5s … [11:57:26] Stopping collector [11:57:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj8_run1.manifest.json [11:57:27] Cooldown: 30s [11:57:57] ──────────────────────────────────────────────────── [11:57:57] Run 2/3: seq-read-qd1-nj8_run2 [11:57:57] ──────────────────────────────────────────────────── [11:57:57] Cleaning fio test files from /tank-mirror/bench … [11:57:57] Test files removed [11:57:57] Dropping Linux page cache + ZFS ARC … [11:57:57] Flushing ZFS ARC (was 1) … [11:58:02] ARC flushed — current size: 12247537248 bytes [11:58:02] NVMe volatile write caches flushed [11:58:04] pmcd healthy (core ✓ pmdazfs ✓) [11:58:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj8_run2.ndjson [11:58:05] Collector running (PID 2452710), settling 5s … [11:58:10] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj8.fio [12:03:45] fio completed in 335s [12:03:45] Post-fio settle 5s … [12:03:50] Stopping collector [12:03:52] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj8_run2.manifest.json [12:03:52] Cooldown: 30s [12:04:22] ──────────────────────────────────────────────────── [12:04:22] Run 3/3: seq-read-qd1-nj8_run3 [12:04:22] ──────────────────────────────────────────────────── [12:04:22] Cleaning fio test files from /tank-mirror/bench … [12:04:23] Test files removed [12:04:23] Dropping Linux page cache + ZFS ARC … [12:04:23] Flushing ZFS ARC (was 1) … [12:04:28] ARC flushed — current size: 31540396680 bytes [12:04:28] NVMe volatile write caches flushed [12:04:29] pmcd healthy (core ✓ pmdazfs ✓) [12:04:29] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj8_run3.ndjson [12:04:30] Collector running (PID 2456664), settling 5s … [12:04:35] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj8.fio [12:10:11] fio completed in 336s [12:10:11] Post-fio settle 5s … [12:10:16] Stopping collector [12:10:18] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj8_run3.manifest.json [12:10:18] ═══════════════════════════════════════════════════════════ [12:10:18] All 3 runs complete for: seq-read-qd1-nj8 [12:10:18] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [12:10:18] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [12:10:18] ═══════════════════════════════════════════════════════════ [12:10:18] Progress: 3/53 jobs | Elapsed: 0h46m | ETA: ~12h50m [12:10:18] [12:10:18] ╔══════════════════════════════════════════════════════════╗ [12:10:18] ║ Job 4/53: seq-read-qd1-nj16 [12:10:18] ║ Category: seq [12:10:18] ╚══════════════════════════════════════════════════════════╝ [12:10:18] ╔══════════════════════════════════════════════════════════╗ [12:10:18] ║ Benchmark: seq-read-qd1-nj16 [12:10:18] ║ Category: seq [12:10:18] ║ Runs: 3 × cooldown 30s [12:10:18] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [12:10:18] ╚══════════════════════════════════════════════════════════╝ [12:10:18] ──────────────────────────────────────────────────── [12:10:18] Run 1/3: seq-read-qd1-nj16_run1 [12:10:18] ──────────────────────────────────────────────────── [12:10:18] Cleaning fio test files from /tank-mirror/bench … [12:10:18] Test files removed [12:10:18] Dropping Linux page cache + ZFS ARC … [12:10:19] Flushing ZFS ARC (was 1) … [12:10:24] ARC flushed — current size: 9863363784 bytes [12:10:24] NVMe volatile write caches flushed [12:10:25] pmcd healthy (core ✓ pmdazfs ✓) [12:10:25] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj16_run1.ndjson [12:10:26] Collector running (PID 2460567), settling 5s … [12:10:31] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj16.fio [12:18:30] fio completed in 479s [12:18:30] Post-fio settle 5s … [12:18:35] Stopping collector [12:18:36] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj16_run1.manifest.json [12:18:36] Cooldown: 30s [12:19:06] ──────────────────────────────────────────────────── [12:19:06] Run 2/3: seq-read-qd1-nj16_run2 [12:19:06] ──────────────────────────────────────────────────── [12:19:06] Cleaning fio test files from /tank-mirror/bench … [12:19:06] Test files removed [12:19:06] Dropping Linux page cache + ZFS ARC … [12:19:06] Flushing ZFS ARC (was 1) … [12:19:12] ARC flushed — current size: 49834840880 bytes [12:19:12] NVMe volatile write caches flushed [12:19:13] pmcd healthy (core ✓ pmdazfs ✓) [12:19:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj16_run2.ndjson [12:19:14] Collector running (PID 2465991), settling 5s … [12:19:19] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj16.fio [12:27:16] fio completed in 477s [12:27:16] Post-fio settle 5s … [12:27:21] Stopping collector [12:27:22] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj16_run2.manifest.json [12:27:22] Cooldown: 30s [12:27:52] ──────────────────────────────────────────────────── [12:27:52] Run 3/3: seq-read-qd1-nj16_run3 [12:27:52] ──────────────────────────────────────────────────── [12:27:52] Cleaning fio test files from /tank-mirror/bench … [12:27:52] Test files removed [12:27:52] Dropping Linux page cache + ZFS ARC … [12:27:53] Flushing ZFS ARC (was 1) … [12:27:58] ARC flushed — current size: 41595844320 bytes [12:27:58] NVMe volatile write caches flushed [12:27:59] pmcd healthy (core ✓ pmdazfs ✓) [12:27:59] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd1-nj16_run3.ndjson [12:28:00] Collector running (PID 2472010), settling 5s … [12:28:05] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd1-nj16.fio [12:36:03] fio completed in 478s [12:36:03] Post-fio settle 5s … [12:36:08] Stopping collector [12:36:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd1-nj16_run3.manifest.json [12:36:09] ═══════════════════════════════════════════════════════════ [12:36:09] All 3 runs complete for: seq-read-qd1-nj16 [12:36:09] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [12:36:09] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [12:36:09] ═══════════════════════════════════════════════════════════ [12:36:09] Progress: 4/53 jobs | Elapsed: 1h12m | ETA: ~14h42m [12:36:09] [12:36:09] ╔══════════════════════════════════════════════════════════╗ [12:36:09] ║ Job 5/53: seq-read-qd32-nj1 [12:36:09] ║ Category: seq [12:36:09] ╚══════════════════════════════════════════════════════════╝ [12:36:09] ╔══════════════════════════════════════════════════════════╗ [12:36:09] ║ Benchmark: seq-read-qd32-nj1 [12:36:09] ║ Category: seq [12:36:09] ║ Runs: 3 × cooldown 30s [12:36:09] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [12:36:09] ╚══════════════════════════════════════════════════════════╝ [12:36:09] ──────────────────────────────────────────────────── [12:36:09] Run 1/3: seq-read-qd32-nj1_run1 [12:36:09] ──────────────────────────────────────────────────── [12:36:09] Cleaning fio test files from /tank-mirror/bench … [12:36:09] Test files removed [12:36:09] Dropping Linux page cache + ZFS ARC … [12:36:10] Flushing ZFS ARC (was 1) … [12:36:15] ARC flushed — current size: 49103104200 bytes [12:36:15] NVMe volatile write caches flushed [12:36:16] pmcd healthy (core ✓ pmdazfs ✓) [12:36:16] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj1_run1.ndjson [12:36:17] Collector running (PID 2477332), settling 5s … [12:36:22] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj1.fio [12:39:50] fio completed in 208s [12:39:50] Post-fio settle 5s … [12:39:55] Stopping collector [12:39:56] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj1_run1.manifest.json [12:39:56] Cooldown: 30s [12:40:26] ──────────────────────────────────────────────────── [12:40:26] Run 2/3: seq-read-qd32-nj1_run2 [12:40:26] ──────────────────────────────────────────────────── [12:40:26] Cleaning fio test files from /tank-mirror/bench … [12:40:27] Test files removed [12:40:27] Dropping Linux page cache + ZFS ARC … [12:40:27] Flushing ZFS ARC (was 1) … [12:40:32] ARC flushed — current size: 17035568 bytes [12:40:32] NVMe volatile write caches flushed [12:40:33] pmcd healthy (core ✓ pmdazfs ✓) [12:40:33] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj1_run2.ndjson [12:40:34] Collector running (PID 2480015), settling 5s … [12:40:39] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj1.fio [12:44:07] fio completed in 208s [12:44:07] Post-fio settle 5s … [12:44:12] Stopping collector [12:44:13] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj1_run2.manifest.json [12:44:13] Cooldown: 30s [12:44:43] ──────────────────────────────────────────────────── [12:44:43] Run 3/3: seq-read-qd32-nj1_run3 [12:44:43] ──────────────────────────────────────────────────── [12:44:43] Cleaning fio test files from /tank-mirror/bench … [12:44:43] Test files removed [12:44:43] Dropping Linux page cache + ZFS ARC … [12:44:44] Flushing ZFS ARC (was 1) … [12:44:49] ARC flushed — current size: 346046896 bytes [12:44:49] NVMe volatile write caches flushed [12:44:50] pmcd healthy (core ✓ pmdazfs ✓) [12:44:50] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj1_run3.ndjson [12:44:51] Collector running (PID 2482702), settling 5s … [12:44:56] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj1.fio [12:48:24] fio completed in 208s [12:48:24] Post-fio settle 5s … [12:48:29] Stopping collector [12:48:31] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj1_run3.manifest.json [12:48:31] ═══════════════════════════════════════════════════════════ [12:48:31] All 3 runs complete for: seq-read-qd32-nj1 [12:48:31] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [12:48:31] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [12:48:31] ═══════════════════════════════════════════════════════════ [12:48:31] Progress: 5/53 jobs | Elapsed: 1h24m | ETA: ~13h30m [12:48:31] [12:48:31] ╔══════════════════════════════════════════════════════════╗ [12:48:31] ║ Job 6/53: seq-read-qd32-nj4 [12:48:31] ║ Category: seq [12:48:31] ╚══════════════════════════════════════════════════════════╝ [12:48:31] ╔══════════════════════════════════════════════════════════╗ [12:48:31] ║ Benchmark: seq-read-qd32-nj4 [12:48:31] ║ Category: seq [12:48:31] ║ Runs: 3 × cooldown 30s [12:48:31] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [12:48:31] ╚══════════════════════════════════════════════════════════╝ [12:48:31] ──────────────────────────────────────────────────── [12:48:31] Run 1/3: seq-read-qd32-nj4_run1 [12:48:31] ──────────────────────────────────────────────────── [12:48:31] Cleaning fio test files from /tank-mirror/bench … [12:48:31] Test files removed [12:48:31] Dropping Linux page cache + ZFS ARC … [12:48:32] Flushing ZFS ARC (was 1) … [12:48:37] ARC flushed — current size: 17880872 bytes [12:48:37] NVMe volatile write caches flushed [12:48:38] pmcd healthy (core ✓ pmdazfs ✓) [12:48:38] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj4_run1.ndjson [12:48:39] Collector running (PID 2485322), settling 5s … [12:48:44] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj4.fio [12:53:06] fio completed in 262s [12:53:06] Post-fio settle 5s … [12:53:11] Stopping collector [12:53:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj4_run1.manifest.json [12:53:12] Cooldown: 30s [12:53:42] ──────────────────────────────────────────────────── [12:53:42] Run 2/3: seq-read-qd32-nj4_run2 [12:53:42] ──────────────────────────────────────────────────── [12:53:42] Cleaning fio test files from /tank-mirror/bench … [12:53:42] Test files removed [12:53:42] Dropping Linux page cache + ZFS ARC … [12:53:43] Flushing ZFS ARC (was 1) … [12:53:48] ARC flushed — current size: 39618526184 bytes [12:53:48] NVMe volatile write caches flushed [12:53:49] pmcd healthy (core ✓ pmdazfs ✓) [12:53:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj4_run2.ndjson [12:53:50] Collector running (PID 2488718), settling 5s … [12:53:55] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj4.fio [12:58:17] fio completed in 262s [12:58:17] Post-fio settle 5s … [12:58:22] Stopping collector [12:58:24] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj4_run2.manifest.json [12:58:24] Cooldown: 30s [12:58:54] ──────────────────────────────────────────────────── [12:58:54] Run 3/3: seq-read-qd32-nj4_run3 [12:58:54] ──────────────────────────────────────────────────── [12:58:54] Cleaning fio test files from /tank-mirror/bench … [12:58:54] Test files removed [12:58:54] Dropping Linux page cache + ZFS ARC … [12:58:55] Flushing ZFS ARC (was 1) … [12:59:00] ARC flushed — current size: 17684440 bytes [12:59:00] NVMe volatile write caches flushed [12:59:01] pmcd healthy (core ✓ pmdazfs ✓) [12:59:01] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj4_run3.ndjson [12:59:02] Collector running (PID 2492838), settling 5s … [12:59:07] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj4.fio [13:03:29] fio completed in 262s [13:03:29] Post-fio settle 5s … [13:03:34] Stopping collector [13:03:36] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj4_run3.manifest.json [13:03:36] ═══════════════════════════════════════════════════════════ [13:03:36] All 3 runs complete for: seq-read-qd32-nj4 [13:03:36] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [13:03:36] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [13:03:36] ═══════════════════════════════════════════════════════════ [13:03:36] Progress: 6/53 jobs | Elapsed: 1h39m | ETA: ~12h59m [13:03:36] [13:03:36] ╔══════════════════════════════════════════════════════════╗ [13:03:36] ║ Job 7/53: seq-read-qd32-nj8 [13:03:36] ║ Category: seq [13:03:36] ╚══════════════════════════════════════════════════════════╝ [13:03:36] ╔══════════════════════════════════════════════════════════╗ [13:03:36] ║ Benchmark: seq-read-qd32-nj8 [13:03:36] ║ Category: seq [13:03:36] ║ Runs: 3 × cooldown 30s [13:03:36] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [13:03:36] ╚══════════════════════════════════════════════════════════╝ [13:03:36] ──────────────────────────────────────────────────── [13:03:36] Run 1/3: seq-read-qd32-nj8_run1 [13:03:36] ──────────────────────────────────────────────────── [13:03:36] Cleaning fio test files from /tank-mirror/bench … [13:03:36] Test files removed [13:03:36] Dropping Linux page cache + ZFS ARC … [13:03:37] Flushing ZFS ARC (was 1) … [13:03:42] ARC flushed — current size: 958434936 bytes [13:03:42] NVMe volatile write caches flushed [13:03:43] pmcd healthy (core ✓ pmdazfs ✓) [13:03:43] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj8_run1.ndjson [13:03:44] Collector running (PID 2496204), settling 5s … [13:03:49] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj8.fio [13:09:24] fio completed in 335s [13:09:24] Post-fio settle 5s … [13:09:29] Stopping collector [13:09:30] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj8_run1.manifest.json [13:09:30] Cooldown: 30s [13:10:00] ──────────────────────────────────────────────────── [13:10:00] Run 2/3: seq-read-qd32-nj8_run2 [13:10:00] ──────────────────────────────────────────────────── [13:10:00] Cleaning fio test files from /tank-mirror/bench … [13:10:00] Test files removed [13:10:00] Dropping Linux page cache + ZFS ARC … [13:10:01] Flushing ZFS ARC (was 1) … [13:10:06] ARC flushed — current size: 37691172512 bytes [13:10:06] NVMe volatile write caches flushed [13:10:07] pmcd healthy (core ✓ pmdazfs ✓) [13:10:07] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj8_run2.ndjson [13:10:08] Collector running (PID 2500175), settling 5s … [13:10:13] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj8.fio [13:15:48] fio completed in 335s [13:15:48] Post-fio settle 5s … [13:15:53] Stopping collector [13:15:55] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj8_run2.manifest.json [13:15:55] Cooldown: 30s [13:16:25] ──────────────────────────────────────────────────── [13:16:25] Run 3/3: seq-read-qd32-nj8_run3 [13:16:25] ──────────────────────────────────────────────────── [13:16:25] Cleaning fio test files from /tank-mirror/bench … [13:16:26] Test files removed [13:16:26] Dropping Linux page cache + ZFS ARC … [13:16:26] Flushing ZFS ARC (was 1) … [13:16:31] ARC flushed — current size: 19768691656 bytes [13:16:31] NVMe volatile write caches flushed [13:16:32] pmcd healthy (core ✓ pmdazfs ✓) [13:16:32] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj8_run3.ndjson [13:16:33] Collector running (PID 2504098), settling 5s … [13:16:38] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj8.fio [13:22:13] fio completed in 335s [13:22:13] Post-fio settle 5s … [13:22:18] Stopping collector [13:22:20] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj8_run3.manifest.json [13:22:20] ═══════════════════════════════════════════════════════════ [13:22:20] All 3 runs complete for: seq-read-qd32-nj8 [13:22:20] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [13:22:20] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [13:22:20] ═══════════════════════════════════════════════════════════ [13:22:20] Progress: 7/53 jobs | Elapsed: 1h58m | ETA: ~12h56m [13:22:20] [13:22:20] ╔══════════════════════════════════════════════════════════╗ [13:22:20] ║ Job 8/53: seq-read-qd32-nj16 [13:22:20] ║ Category: seq [13:22:20] ╚══════════════════════════════════════════════════════════╝ [13:22:20] ╔══════════════════════════════════════════════════════════╗ [13:22:20] ║ Benchmark: seq-read-qd32-nj16 [13:22:20] ║ Category: seq [13:22:20] ║ Runs: 3 × cooldown 30s [13:22:20] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [13:22:20] ╚══════════════════════════════════════════════════════════╝ [13:22:20] ──────────────────────────────────────────────────── [13:22:20] Run 1/3: seq-read-qd32-nj16_run1 [13:22:20] ──────────────────────────────────────────────────── [13:22:20] Cleaning fio test files from /tank-mirror/bench … [13:22:21] Test files removed [13:22:21] Dropping Linux page cache + ZFS ARC … [13:22:21] Flushing ZFS ARC (was 1) … [13:22:26] ARC flushed — current size: 34151335424 bytes [13:22:26] NVMe volatile write caches flushed [13:22:27] pmcd healthy (core ✓ pmdazfs ✓) [13:22:27] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj16_run1.ndjson [13:22:28] Collector running (PID 2507960), settling 5s … [13:22:33] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj16.fio [13:30:30] fio completed in 477s [13:30:30] Post-fio settle 5s … [13:30:35] Stopping collector [13:30:37] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj16_run1.manifest.json [13:30:37] Cooldown: 30s [13:31:07] ──────────────────────────────────────────────────── [13:31:07] Run 2/3: seq-read-qd32-nj16_run2 [13:31:07] ──────────────────────────────────────────────────── [13:31:07] Cleaning fio test files from /tank-mirror/bench … [13:31:08] Test files removed [13:31:08] Dropping Linux page cache + ZFS ARC … [13:31:08] Flushing ZFS ARC (was 1) … [13:31:13] ARC flushed — current size: 40161579296 bytes [13:31:13] NVMe volatile write caches flushed [13:31:14] pmcd healthy (core ✓ pmdazfs ✓) [13:31:14] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj16_run2.ndjson [13:31:15] Collector running (PID 2513946), settling 5s … [13:31:20] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj16.fio [13:39:17] fio completed in 477s [13:39:17] Post-fio settle 5s … [13:39:22] Stopping collector [13:39:23] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj16_run2.manifest.json [13:39:23] Cooldown: 30s [13:39:53] ──────────────────────────────────────────────────── [13:39:53] Run 3/3: seq-read-qd32-nj16_run3 [13:39:53] ──────────────────────────────────────────────────── [13:39:53] Cleaning fio test files from /tank-mirror/bench … [13:39:54] Test files removed [13:39:54] Dropping Linux page cache + ZFS ARC … [13:39:54] Flushing ZFS ARC (was 1) … [13:39:59] ARC flushed — current size: 36735729528 bytes [13:39:59] NVMe volatile write caches flushed [13:40:00] pmcd healthy (core ✓ pmdazfs ✓) [13:40:00] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-read-qd32-nj16_run3.ndjson [13:40:01] Collector running (PID 2519290), settling 5s … [13:40:06] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-read-qd32-nj16.fio [13:48:03] fio completed in 477s [13:48:03] Post-fio settle 5s … [13:48:08] Stopping collector [13:48:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-read-qd32-nj16_run3.manifest.json [13:48:09] ═══════════════════════════════════════════════════════════ [13:48:09] All 3 runs complete for: seq-read-qd32-nj16 [13:48:09] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [13:48:09] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [13:48:09] ═══════════════════════════════════════════════════════════ [13:48:09] Progress: 8/53 jobs | Elapsed: 2h24m | ETA: ~13h30m [13:48:09] [13:48:09] ╔══════════════════════════════════════════════════════════╗ [13:48:09] ║ Job 9/53: seq-write-qd1-nj1 [13:48:09] ║ Category: seq [13:48:09] ╚══════════════════════════════════════════════════════════╝ [13:48:09] ╔══════════════════════════════════════════════════════════╗ [13:48:09] ║ Benchmark: seq-write-qd1-nj1 [13:48:09] ║ Category: seq [13:48:09] ║ Runs: 3 × cooldown 30s [13:48:09] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [13:48:09] ╚══════════════════════════════════════════════════════════╝ [13:48:09] ──────────────────────────────────────────────────── [13:48:09] Run 1/3: seq-write-qd1-nj1_run1 [13:48:09] ──────────────────────────────────────────────────── [13:48:09] Cleaning fio test files from /tank-mirror/bench … [13:48:09] Test files removed [13:48:09] Dropping Linux page cache + ZFS ARC … [13:48:09] Flushing ZFS ARC (was 1) … [13:48:15] ARC flushed — current size: 39473825144 bytes [13:48:15] NVMe volatile write caches flushed [13:48:16] pmcd healthy (core ✓ pmdazfs ✓) [13:48:16] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj1_run1.ndjson [13:48:17] Collector running (PID 2524530), settling 5s … [13:48:22] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj1.fio [13:51:32] fio completed in 190s [13:51:32] Post-fio settle 5s … [13:51:37] Stopping collector [13:51:38] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj1_run1.manifest.json [13:51:38] Cooldown: 30s [13:52:08] ──────────────────────────────────────────────────── [13:52:08] Run 2/3: seq-write-qd1-nj1_run2 [13:52:08] ──────────────────────────────────────────────────── [13:52:08] Cleaning fio test files from /tank-mirror/bench … [13:52:09] Test files removed [13:52:09] Dropping Linux page cache + ZFS ARC … [13:52:09] Flushing ZFS ARC (was 1) … [13:52:14] ARC flushed — current size: 33590153648 bytes [13:52:14] NVMe volatile write caches flushed [13:52:15] pmcd healthy (core ✓ pmdazfs ✓) [13:52:15] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj1_run2.ndjson [13:52:16] Collector running (PID 2526957), settling 5s … [13:52:21] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj1.fio [13:55:32] fio completed in 191s [13:55:32] Post-fio settle 5s … [13:55:37] Stopping collector [13:55:38] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj1_run2.manifest.json [13:55:38] Cooldown: 30s [13:56:08] ──────────────────────────────────────────────────── [13:56:08] Run 3/3: seq-write-qd1-nj1_run3 [13:56:08] ──────────────────────────────────────────────────── [13:56:08] Cleaning fio test files from /tank-mirror/bench … [13:56:08] Test files removed [13:56:08] Dropping Linux page cache + ZFS ARC … [13:56:08] Flushing ZFS ARC (was 1) … [13:56:13] ARC flushed — current size: 50498464496 bytes [13:56:13] NVMe volatile write caches flushed [13:56:14] pmcd healthy (core ✓ pmdazfs ✓) [13:56:14] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj1_run3.ndjson [13:56:15] Collector running (PID 2530033), settling 5s … [13:56:20] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj1.fio [13:59:31] fio completed in 191s [13:59:31] Post-fio settle 5s … [13:59:36] Stopping collector [13:59:37] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj1_run3.manifest.json [13:59:37] ═══════════════════════════════════════════════════════════ [13:59:37] All 3 runs complete for: seq-write-qd1-nj1 [13:59:37] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [13:59:37] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [13:59:37] ═══════════════════════════════════════════════════════════ [13:59:37] Progress: 9/53 jobs | Elapsed: 2h35m | ETA: ~12h39m [13:59:37] [13:59:37] ╔══════════════════════════════════════════════════════════╗ [13:59:37] ║ Job 10/53: seq-write-qd1-nj4 [13:59:37] ║ Category: seq [13:59:37] ╚══════════════════════════════════════════════════════════╝ [13:59:37] ╔══════════════════════════════════════════════════════════╗ [13:59:37] ║ Benchmark: seq-write-qd1-nj4 [13:59:37] ║ Category: seq [13:59:37] ║ Runs: 3 × cooldown 30s [13:59:37] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [13:59:37] ╚══════════════════════════════════════════════════════════╝ [13:59:37] ──────────────────────────────────────────────────── [13:59:37] Run 1/3: seq-write-qd1-nj4_run1 [13:59:37] ──────────────────────────────────────────────────── [13:59:37] Cleaning fio test files from /tank-mirror/bench … [13:59:37] Test files removed [13:59:37] Dropping Linux page cache + ZFS ARC … [13:59:37] Flushing ZFS ARC (was 1) … [13:59:42] ARC flushed — current size: 12005824208 bytes [13:59:42] NVMe volatile write caches flushed [13:59:44] pmcd healthy (core ✓ pmdazfs ✓) [13:59:44] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj4_run1.ndjson [13:59:45] Collector running (PID 2532407), settling 5s … [13:59:50] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj4.fio [14:03:00] fio completed in 190s [14:03:00] Post-fio settle 5s … [14:03:05] Stopping collector [14:03:06] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj4_run1.manifest.json [14:03:06] Cooldown: 30s [14:03:36] ──────────────────────────────────────────────────── [14:03:36] Run 2/3: seq-write-qd1-nj4_run2 [14:03:36] ──────────────────────────────────────────────────── [14:03:36] Cleaning fio test files from /tank-mirror/bench … [14:03:37] Test files removed [14:03:37] Dropping Linux page cache + ZFS ARC … [14:03:37] Flushing ZFS ARC (was 1) … [14:03:42] ARC flushed — current size: 22056890248 bytes [14:03:42] NVMe volatile write caches flushed [14:03:43] pmcd healthy (core ✓ pmdazfs ✓) [14:03:43] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj4_run2.ndjson [14:03:44] Collector running (PID 2534814), settling 5s … [14:03:49] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj4.fio [14:07:00] fio completed in 191s [14:07:00] Post-fio settle 5s … [14:07:05] Stopping collector [14:07:07] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj4_run2.manifest.json [14:07:07] Cooldown: 30s [14:07:37] ──────────────────────────────────────────────────── [14:07:37] Run 3/3: seq-write-qd1-nj4_run3 [14:07:37] ──────────────────────────────────────────────────── [14:07:37] Cleaning fio test files from /tank-mirror/bench … [14:07:37] Test files removed [14:07:37] Dropping Linux page cache + ZFS ARC … [14:07:37] Flushing ZFS ARC (was 1) … [14:07:43] ARC flushed — current size: 19478741104 bytes [14:07:43] NVMe volatile write caches flushed [14:07:44] pmcd healthy (core ✓ pmdazfs ✓) [14:07:44] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj4_run3.ndjson [14:07:45] Collector running (PID 2537270), settling 5s … [14:07:50] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj4.fio [14:11:00] fio completed in 190s [14:11:00] Post-fio settle 5s … [14:11:05] Stopping collector [14:11:06] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj4_run3.manifest.json [14:11:06] ═══════════════════════════════════════════════════════════ [14:11:06] All 3 runs complete for: seq-write-qd1-nj4 [14:11:06] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [14:11:06] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [14:11:06] ═══════════════════════════════════════════════════════════ [14:11:06] Progress: 10/53 jobs | Elapsed: 2h47m | ETA: ~11h58m [14:11:06] [14:11:06] ╔══════════════════════════════════════════════════════════╗ [14:11:06] ║ Job 11/53: seq-write-qd1-nj8 [14:11:06] ║ Category: seq [14:11:06] ╚══════════════════════════════════════════════════════════╝ [14:11:06] ╔══════════════════════════════════════════════════════════╗ [14:11:06] ║ Benchmark: seq-write-qd1-nj8 [14:11:06] ║ Category: seq [14:11:06] ║ Runs: 3 × cooldown 30s [14:11:06] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:11:06] ╚══════════════════════════════════════════════════════════╝ [14:11:06] ──────────────────────────────────────────────────── [14:11:06] Run 1/3: seq-write-qd1-nj8_run1 [14:11:06] ──────────────────────────────────────────────────── [14:11:06] Cleaning fio test files from /tank-mirror/bench … [14:11:07] Test files removed [14:11:07] Dropping Linux page cache + ZFS ARC … [14:11:07] Flushing ZFS ARC (was 1) … [14:11:12] ARC flushed — current size: 85664565624 bytes [14:11:12] NVMe volatile write caches flushed [14:11:13] pmcd healthy (core ✓ pmdazfs ✓) [14:11:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj8_run1.ndjson [14:11:14] Collector running (PID 2539583), settling 5s … [14:11:19] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj8.fio [14:14:30] fio completed in 191s [14:14:30] Post-fio settle 5s … [14:14:35] Stopping collector [14:14:36] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj8_run1.manifest.json [14:14:36] Cooldown: 30s [14:15:06] ──────────────────────────────────────────────────── [14:15:06] Run 2/3: seq-write-qd1-nj8_run2 [14:15:06] ──────────────────────────────────────────────────── [14:15:06] Cleaning fio test files from /tank-mirror/bench … [14:15:06] Test files removed [14:15:06] Dropping Linux page cache + ZFS ARC … [14:15:07] Flushing ZFS ARC (was 1) … [14:15:12] ARC flushed — current size: 129438480216 bytes [14:15:12] NVMe volatile write caches flushed [14:15:13] pmcd healthy (core ✓ pmdazfs ✓) [14:15:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj8_run2.ndjson [14:15:14] Collector running (PID 2542047), settling 5s … [14:15:19] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj8.fio [14:18:29] fio completed in 190s [14:18:30] Post-fio settle 5s … [14:18:35] Stopping collector [14:18:37] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj8_run2.manifest.json [14:18:37] Cooldown: 30s [14:19:07] ──────────────────────────────────────────────────── [14:19:07] Run 3/3: seq-write-qd1-nj8_run3 [14:19:07] ──────────────────────────────────────────────────── [14:19:07] Cleaning fio test files from /tank-mirror/bench … [14:19:07] Test files removed [14:19:07] Dropping Linux page cache + ZFS ARC … [14:19:07] Flushing ZFS ARC (was 1) … [14:19:12] ARC flushed — current size: 43231077216 bytes [14:19:12] NVMe volatile write caches flushed [14:19:14] pmcd healthy (core ✓ pmdazfs ✓) [14:19:14] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj8_run3.ndjson [14:19:15] Collector running (PID 2544494), settling 5s … [14:19:20] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj8.fio [14:22:30] fio completed in 190s [14:22:30] Post-fio settle 5s … [14:22:35] Stopping collector [14:22:37] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj8_run3.manifest.json [14:22:37] ═══════════════════════════════════════════════════════════ [14:22:37] All 3 runs complete for: seq-write-qd1-nj8 [14:22:37] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [14:22:37] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [14:22:37] ═══════════════════════════════════════════════════════════ [14:22:37] Progress: 11/53 jobs | Elapsed: 2h58m | ETA: ~11h21m [14:22:37] [14:22:37] ╔══════════════════════════════════════════════════════════╗ [14:22:37] ║ Job 12/53: seq-write-qd1-nj16 [14:22:37] ║ Category: seq [14:22:37] ╚══════════════════════════════════════════════════════════╝ [14:22:37] ╔══════════════════════════════════════════════════════════╗ [14:22:37] ║ Benchmark: seq-write-qd1-nj16 [14:22:37] ║ Category: seq [14:22:37] ║ Runs: 3 × cooldown 30s [14:22:37] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:22:37] ╚══════════════════════════════════════════════════════════╝ [14:22:37] ──────────────────────────────────────────────────── [14:22:37] Run 1/3: seq-write-qd1-nj16_run1 [14:22:37] ──────────────────────────────────────────────────── [14:22:37] Cleaning fio test files from /tank-mirror/bench … [14:22:38] Test files removed [14:22:38] Dropping Linux page cache + ZFS ARC … [14:22:38] Flushing ZFS ARC (was 1) … [14:22:43] ARC flushed — current size: 33163054392 bytes [14:22:43] NVMe volatile write caches flushed [14:22:44] pmcd healthy (core ✓ pmdazfs ✓) [14:22:44] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj16_run1.ndjson [14:22:45] Collector running (PID 2546896), settling 5s … [14:22:50] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj16.fio [14:26:01] fio completed in 191s [14:26:01] Post-fio settle 5s … [14:26:06] Stopping collector [14:26:08] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj16_run1.manifest.json [14:26:08] Cooldown: 30s [14:26:38] ──────────────────────────────────────────────────── [14:26:38] Run 2/3: seq-write-qd1-nj16_run2 [14:26:38] ──────────────────────────────────────────────────── [14:26:38] Cleaning fio test files from /tank-mirror/bench … [14:26:39] Test files removed [14:26:39] Dropping Linux page cache + ZFS ARC … [14:26:39] Flushing ZFS ARC (was 1) … [14:26:44] ARC flushed — current size: 129790775024 bytes [14:26:44] NVMe volatile write caches flushed [14:26:45] pmcd healthy (core ✓ pmdazfs ✓) [14:26:45] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj16_run2.ndjson [14:26:46] Collector running (PID 2550010), settling 5s … [14:26:51] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj16.fio [14:30:02] fio completed in 191s [14:30:02] Post-fio settle 5s … [14:30:07] Stopping collector [14:30:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj16_run2.manifest.json [14:30:09] Cooldown: 30s [14:30:39] ──────────────────────────────────────────────────── [14:30:39] Run 3/3: seq-write-qd1-nj16_run3 [14:30:39] ──────────────────────────────────────────────────── [14:30:39] Cleaning fio test files from /tank-mirror/bench … [14:30:39] Test files removed [14:30:39] Dropping Linux page cache + ZFS ARC … [14:30:39] Flushing ZFS ARC (was 1) … [14:30:44] ARC flushed — current size: 129855246488 bytes [14:30:44] NVMe volatile write caches flushed [14:30:46] pmcd healthy (core ✓ pmdazfs ✓) [14:30:46] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd1-nj16_run3.ndjson [14:30:47] Collector running (PID 2552464), settling 5s … [14:30:52] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd1-nj16.fio [14:34:02] fio completed in 190s [14:34:02] Post-fio settle 5s … [14:34:07] Stopping collector [14:34:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd1-nj16_run3.manifest.json [14:34:09] ═══════════════════════════════════════════════════════════ [14:34:09] All 3 runs complete for: seq-write-qd1-nj16 [14:34:09] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [14:34:09] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [14:34:09] ═══════════════════════════════════════════════════════════ [14:34:09] Progress: 12/53 jobs | Elapsed: 3h10m | ETA: ~10h49m [14:34:09] [14:34:09] ╔══════════════════════════════════════════════════════════╗ [14:34:09] ║ Job 13/53: seq-write-qd32-nj1 [14:34:09] ║ Category: seq [14:34:09] ╚══════════════════════════════════════════════════════════╝ [14:34:09] ╔══════════════════════════════════════════════════════════╗ [14:34:09] ║ Benchmark: seq-write-qd32-nj1 [14:34:09] ║ Category: seq [14:34:09] ║ Runs: 3 × cooldown 30s [14:34:09] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:34:09] ╚══════════════════════════════════════════════════════════╝ [14:34:09] ──────────────────────────────────────────────────── [14:34:09] Run 1/3: seq-write-qd32-nj1_run1 [14:34:09] ──────────────────────────────────────────────────── [14:34:09] Cleaning fio test files from /tank-mirror/bench … [14:34:10] Test files removed [14:34:10] Dropping Linux page cache + ZFS ARC … [14:34:10] Flushing ZFS ARC (was 1) … [14:34:15] ARC flushed — current size: 129853619992 bytes [14:34:15] NVMe volatile write caches flushed [14:34:17] pmcd healthy (core ✓ pmdazfs ✓) [14:34:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj1_run1.ndjson [14:34:18] Collector running (PID 2554841), settling 5s … [14:34:23] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj1.fio [14:37:33] fio completed in 190s [14:37:33] Post-fio settle 5s … [14:37:38] Stopping collector [14:37:40] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj1_run1.manifest.json [14:37:40] Cooldown: 30s [14:38:10] ──────────────────────────────────────────────────── [14:38:10] Run 2/3: seq-write-qd32-nj1_run2 [14:38:10] ──────────────────────────────────────────────────── [14:38:10] Cleaning fio test files from /tank-mirror/bench … [14:38:11] Test files removed [14:38:11] Dropping Linux page cache + ZFS ARC … [14:38:11] Flushing ZFS ARC (was 1) … [14:38:16] ARC flushed — current size: 27059896 bytes [14:38:16] NVMe volatile write caches flushed [14:38:17] pmcd healthy (core ✓ pmdazfs ✓) [14:38:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj1_run2.ndjson [14:38:18] Collector running (PID 2557311), settling 5s … [14:38:23] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj1.fio [14:41:34] fio completed in 191s [14:41:34] Post-fio settle 5s … [14:41:39] Stopping collector [14:41:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj1_run2.manifest.json [14:41:41] Cooldown: 30s [14:42:11] ──────────────────────────────────────────────────── [14:42:11] Run 3/3: seq-write-qd32-nj1_run3 [14:42:11] ──────────────────────────────────────────────────── [14:42:11] Cleaning fio test files from /tank-mirror/bench … [14:42:11] Test files removed [14:42:11] Dropping Linux page cache + ZFS ARC … [14:42:11] Flushing ZFS ARC (was 1) … [14:42:16] ARC flushed — current size: 26041472 bytes [14:42:16] NVMe volatile write caches flushed [14:42:17] pmcd healthy (core ✓ pmdazfs ✓) [14:42:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj1_run3.ndjson [14:42:18] Collector running (PID 2559731), settling 5s … [14:42:23] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj1.fio [14:45:34] fio completed in 191s [14:45:34] Post-fio settle 5s … [14:45:39] Stopping collector [14:45:40] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj1_run3.manifest.json [14:45:40] ═══════════════════════════════════════════════════════════ [14:45:40] All 3 runs complete for: seq-write-qd32-nj1 [14:45:40] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [14:45:40] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [14:45:40] ═══════════════════════════════════════════════════════════ [14:45:40] Progress: 13/53 jobs | Elapsed: 3h21m | ETA: ~10h20m [14:45:40] [14:45:40] ╔══════════════════════════════════════════════════════════╗ [14:45:40] ║ Job 14/53: seq-write-qd32-nj4 [14:45:40] ║ Category: seq [14:45:40] ╚══════════════════════════════════════════════════════════╝ [14:45:40] ╔══════════════════════════════════════════════════════════╗ [14:45:40] ║ Benchmark: seq-write-qd32-nj4 [14:45:40] ║ Category: seq [14:45:40] ║ Runs: 3 × cooldown 30s [14:45:40] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:45:40] ╚══════════════════════════════════════════════════════════╝ [14:45:40] ──────────────────────────────────────────────────── [14:45:40] Run 1/3: seq-write-qd32-nj4_run1 [14:45:40] ──────────────────────────────────────────────────── [14:45:40] Cleaning fio test files from /tank-mirror/bench … [14:45:40] Test files removed [14:45:40] Dropping Linux page cache + ZFS ARC … [14:45:40] Flushing ZFS ARC (was 1) … [14:45:45] ARC flushed — current size: 807633720 bytes [14:45:45] NVMe volatile write caches flushed [14:45:47] pmcd healthy (core ✓ pmdazfs ✓) [14:45:47] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj4_run1.ndjson [14:45:48] Collector running (PID 2562075), settling 5s … [14:45:53] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj4.fio [14:49:03] fio completed in 190s [14:49:03] Post-fio settle 5s … [14:49:08] Stopping collector [14:49:10] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj4_run1.manifest.json [14:49:10] Cooldown: 30s [14:49:40] ──────────────────────────────────────────────────── [14:49:40] Run 2/3: seq-write-qd32-nj4_run2 [14:49:40] ──────────────────────────────────────────────────── [14:49:40] Cleaning fio test files from /tank-mirror/bench … [14:49:41] Test files removed [14:49:41] Dropping Linux page cache + ZFS ARC … [14:49:41] Flushing ZFS ARC (was 1) … [14:49:46] ARC flushed — current size: 4879705208 bytes [14:49:46] NVMe volatile write caches flushed [14:49:47] pmcd healthy (core ✓ pmdazfs ✓) [14:49:47] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj4_run2.ndjson [14:49:48] Collector running (PID 2564520), settling 5s … [14:49:53] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj4.fio [14:53:04] fio completed in 191s [14:53:04] Post-fio settle 5s … [14:53:09] Stopping collector [14:53:11] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj4_run2.manifest.json [14:53:11] Cooldown: 30s [14:53:41] ──────────────────────────────────────────────────── [14:53:41] Run 3/3: seq-write-qd32-nj4_run3 [14:53:41] ──────────────────────────────────────────────────── [14:53:41] Cleaning fio test files from /tank-mirror/bench … [14:53:41] Test files removed [14:53:41] Dropping Linux page cache + ZFS ARC … [14:53:42] Flushing ZFS ARC (was 1) … [14:53:47] ARC flushed — current size: 15839119296 bytes [14:53:47] NVMe volatile write caches flushed [14:53:48] pmcd healthy (core ✓ pmdazfs ✓) [14:53:48] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj4_run3.ndjson [14:53:49] Collector running (PID 2566946), settling 5s … [14:53:54] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj4.fio [14:57:04] fio completed in 190s [14:57:04] Post-fio settle 5s … [14:57:09] Stopping collector [14:57:10] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj4_run3.manifest.json [14:57:10] ═══════════════════════════════════════════════════════════ [14:57:10] All 3 runs complete for: seq-write-qd32-nj4 [14:57:10] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [14:57:10] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [14:57:10] ═══════════════════════════════════════════════════════════ [14:57:10] Progress: 14/53 jobs | Elapsed: 3h33m | ETA: ~9h53m [14:57:10] [14:57:10] ╔══════════════════════════════════════════════════════════╗ [14:57:10] ║ Job 15/53: seq-write-qd32-nj8 [14:57:10] ║ Category: seq [14:57:10] ╚══════════════════════════════════════════════════════════╝ [14:57:10] ╔══════════════════════════════════════════════════════════╗ [14:57:10] ║ Benchmark: seq-write-qd32-nj8 [14:57:10] ║ Category: seq [14:57:10] ║ Runs: 3 × cooldown 30s [14:57:10] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:57:10] ╚══════════════════════════════════════════════════════════╝ [14:57:10] ──────────────────────────────────────────────────── [14:57:10] Run 1/3: seq-write-qd32-nj8_run1 [14:57:10] ──────────────────────────────────────────────────── [14:57:10] Cleaning fio test files from /tank-mirror/bench … [14:57:11] Test files removed [14:57:11] Dropping Linux page cache + ZFS ARC … [14:57:11] Flushing ZFS ARC (was 1) … [14:57:16] ARC flushed — current size: 75228474168 bytes [14:57:16] NVMe volatile write caches flushed [14:57:17] pmcd healthy (core ✓ pmdazfs ✓) [14:57:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj8_run1.ndjson [14:57:19] Collector running (PID 2569977), settling 5s … [14:57:24] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj8.fio [15:00:34] fio completed in 190s [15:00:34] Post-fio settle 5s … [15:00:39] Stopping collector [15:00:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj8_run1.manifest.json [15:00:41] Cooldown: 30s [15:01:11] ──────────────────────────────────────────────────── [15:01:11] Run 2/3: seq-write-qd32-nj8_run2 [15:01:11] ──────────────────────────────────────────────────── [15:01:11] Cleaning fio test files from /tank-mirror/bench … [15:01:12] Test files removed [15:01:12] Dropping Linux page cache + ZFS ARC … [15:01:12] Flushing ZFS ARC (was 1) … [15:01:17] ARC flushed — current size: 20205076944 bytes [15:01:17] NVMe volatile write caches flushed [15:01:18] pmcd healthy (core ✓ pmdazfs ✓) [15:01:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj8_run2.ndjson [15:01:19] Collector running (PID 2572411), settling 5s … [15:01:24] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj8.fio [15:04:35] fio completed in 191s [15:04:35] Post-fio settle 5s … [15:04:40] Stopping collector [15:04:42] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj8_run2.manifest.json [15:04:42] Cooldown: 30s [15:05:12] ──────────────────────────────────────────────────── [15:05:12] Run 3/3: seq-write-qd32-nj8_run3 [15:05:12] ──────────────────────────────────────────────────── [15:05:12] Cleaning fio test files from /tank-mirror/bench … [15:05:12] Test files removed [15:05:12] Dropping Linux page cache + ZFS ARC … [15:05:12] Flushing ZFS ARC (was 1) … [15:05:18] ARC flushed — current size: 9275023112 bytes [15:05:18] NVMe volatile write caches flushed [15:05:19] pmcd healthy (core ✓ pmdazfs ✓) [15:05:19] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj8_run3.ndjson [15:05:20] Collector running (PID 2574856), settling 5s … [15:05:25] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj8.fio [15:08:35] fio completed in 190s [15:08:35] Post-fio settle 5s … [15:08:40] Stopping collector [15:08:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj8_run3.manifest.json [15:08:41] ═══════════════════════════════════════════════════════════ [15:08:41] All 3 runs complete for: seq-write-qd32-nj8 [15:08:41] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [15:08:41] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [15:08:41] ═══════════════════════════════════════════════════════════ [15:08:41] Progress: 15/53 jobs | Elapsed: 3h44m | ETA: ~9h28m [15:08:41] [15:08:41] ╔══════════════════════════════════════════════════════════╗ [15:08:41] ║ Job 16/53: seq-write-qd32-nj16 [15:08:41] ║ Category: seq [15:08:41] ╚══════════════════════════════════════════════════════════╝ [15:08:41] ╔══════════════════════════════════════════════════════════╗ [15:08:41] ║ Benchmark: seq-write-qd32-nj16 [15:08:41] ║ Category: seq [15:08:41] ║ Runs: 3 × cooldown 30s [15:08:41] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [15:08:41] ╚══════════════════════════════════════════════════════════╝ [15:08:41] ──────────────────────────────────────────────────── [15:08:41] Run 1/3: seq-write-qd32-nj16_run1 [15:08:41] ──────────────────────────────────────────────────── [15:08:41] Cleaning fio test files from /tank-mirror/bench … [15:08:42] Test files removed [15:08:42] Dropping Linux page cache + ZFS ARC … [15:08:42] Flushing ZFS ARC (was 1) … [15:08:47] ARC flushed — current size: 32981942848 bytes [15:08:47] NVMe volatile write caches flushed [15:08:49] pmcd healthy (core ✓ pmdazfs ✓) [15:08:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj16_run1.ndjson [15:08:50] Collector running (PID 2577207), settling 5s … [15:08:55] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj16.fio [15:12:05] fio completed in 190s [15:12:05] Post-fio settle 5s … [15:12:10] Stopping collector [15:12:11] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj16_run1.manifest.json [15:12:11] Cooldown: 30s [15:12:41] ──────────────────────────────────────────────────── [15:12:41] Run 2/3: seq-write-qd32-nj16_run2 [15:12:41] ──────────────────────────────────────────────────── [15:12:41] Cleaning fio test files from /tank-mirror/bench … [15:12:42] Test files removed [15:12:42] Dropping Linux page cache + ZFS ARC … [15:12:42] Flushing ZFS ARC (was 1) … [15:12:47] ARC flushed — current size: 97773941088 bytes [15:12:47] NVMe volatile write caches flushed [15:12:48] pmcd healthy (core ✓ pmdazfs ✓) [15:12:48] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj16_run2.ndjson [15:12:49] Collector running (PID 2579687), settling 5s … [15:12:54] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj16.fio [15:16:05] fio completed in 191s [15:16:05] Post-fio settle 5s … [15:16:10] Stopping collector [15:16:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj16_run2.manifest.json [15:16:12] Cooldown: 30s [15:16:42] ──────────────────────────────────────────────────── [15:16:42] Run 3/3: seq-write-qd32-nj16_run3 [15:16:42] ──────────────────────────────────────────────────── [15:16:42] Cleaning fio test files from /tank-mirror/bench … [15:16:43] Test files removed [15:16:43] Dropping Linux page cache + ZFS ARC … [15:16:43] Flushing ZFS ARC (was 1) … [15:16:48] ARC flushed — current size: 129765315176 bytes [15:16:48] NVMe volatile write caches flushed [15:16:49] pmcd healthy (core ✓ pmdazfs ✓) [15:16:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq/seq-write-qd32-nj16_run3.ndjson [15:16:50] Collector running (PID 2582113), settling 5s … [15:16:55] Starting fio: /scripts_f16_with_data_deletion/jobs/seq-write-qd32-nj16.fio [15:20:06] fio completed in 191s [15:20:06] Post-fio settle 5s … [15:20:11] Stopping collector [15:20:13] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq/seq-write-qd32-nj16_run3.manifest.json [15:20:13] ═══════════════════════════════════════════════════════════ [15:20:13] All 3 runs complete for: seq-write-qd32-nj16 [15:20:13] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/seq [15:20:13] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/seq [15:20:13] ═══════════════════════════════════════════════════════════ [15:20:13] Progress: 16/53 jobs | Elapsed: 3h56m | ETA: ~9h5m [15:20:13] [15:20:13] ╔══════════════════════════════════════════════════════════╗ [15:20:13] ║ Job 17/53: rand-read-qd1-nj1 [15:20:13] ║ Category: rand [15:20:13] ╚══════════════════════════════════════════════════════════╝ [15:20:13] ╔══════════════════════════════════════════════════════════╗ [15:20:13] ║ Benchmark: rand-read-qd1-nj1 [15:20:13] ║ Category: rand [15:20:13] ║ Runs: 3 × cooldown 30s [15:20:13] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [15:20:13] ╚══════════════════════════════════════════════════════════╝ [15:20:13] ──────────────────────────────────────────────────── [15:20:13] Run 1/3: rand-read-qd1-nj1_run1 [15:20:13] ──────────────────────────────────────────────────── [15:20:13] Cleaning fio test files from /tank-mirror/bench … [15:20:13] Test files removed [15:20:13] Dropping Linux page cache + ZFS ARC … [15:20:13] Flushing ZFS ARC (was 1) … [15:20:19] ARC flushed — current size: 125147928400 bytes [15:20:19] NVMe volatile write caches flushed [15:20:20] pmcd healthy (core ✓ pmdazfs ✓) [15:20:20] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj1_run1.ndjson [15:20:21] Collector running (PID 2584524), settling 5s … [15:20:26] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj1.fio [15:26:03] fio completed in 337s [15:26:03] Post-fio settle 5s … [15:26:08] Stopping collector [15:26:10] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj1_run1.manifest.json [15:26:10] Cooldown: 30s [15:26:40] ──────────────────────────────────────────────────── [15:26:40] Run 2/3: rand-read-qd1-nj1_run2 [15:26:40] ──────────────────────────────────────────────────── [15:26:40] Cleaning fio test files from /tank-mirror/bench … [15:26:40] Test files removed [15:26:40] Dropping Linux page cache + ZFS ARC … [15:26:40] Flushing ZFS ARC (was 1) … [15:26:46] ARC flushed — current size: 25818520 bytes [15:26:46] NVMe volatile write caches flushed [15:26:47] pmcd healthy (core ✓ pmdazfs ✓) [15:26:47] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj1_run2.ndjson [15:26:48] Collector running (PID 2589421), settling 5s … [15:26:53] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj1.fio [15:32:30] fio completed in 337s [15:32:30] Post-fio settle 5s … [15:32:35] Stopping collector [15:32:36] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj1_run2.manifest.json [15:32:36] Cooldown: 30s [15:33:06] ──────────────────────────────────────────────────── [15:33:06] Run 3/3: rand-read-qd1-nj1_run3 [15:33:06] ──────────────────────────────────────────────────── [15:33:06] Cleaning fio test files from /tank-mirror/bench … [15:33:07] Test files removed [15:33:07] Dropping Linux page cache + ZFS ARC … [15:33:07] Flushing ZFS ARC (was 1) … [15:33:12] ARC flushed — current size: 25663368 bytes [15:33:12] NVMe volatile write caches flushed [15:33:13] pmcd healthy (core ✓ pmdazfs ✓) [15:33:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj1_run3.ndjson [15:33:14] Collector running (PID 2593620), settling 5s … [15:33:19] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj1.fio [15:38:55] fio completed in 336s [15:38:55] Post-fio settle 5s … [15:39:00] Stopping collector [15:39:01] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj1_run3.manifest.json [15:39:01] ═══════════════════════════════════════════════════════════ [15:39:01] All 3 runs complete for: rand-read-qd1-nj1 [15:39:01] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [15:39:01] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [15:39:01] ═══════════════════════════════════════════════════════════ [15:39:01] Progress: 17/53 jobs | Elapsed: 4h14m | ETA: ~8h59m [15:39:01] [15:39:01] ╔══════════════════════════════════════════════════════════╗ [15:39:01] ║ Job 18/53: rand-read-qd1-nj8 [15:39:01] ║ Category: rand [15:39:01] ╚══════════════════════════════════════════════════════════╝ [15:39:01] ╔══════════════════════════════════════════════════════════╗ [15:39:01] ║ Benchmark: rand-read-qd1-nj8 [15:39:01] ║ Category: rand [15:39:01] ║ Runs: 3 × cooldown 30s [15:39:01] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [15:39:01] ╚══════════════════════════════════════════════════════════╝ [15:39:01] ──────────────────────────────────────────────────── [15:39:02] Run 1/3: rand-read-qd1-nj8_run1 [15:39:02] ──────────────────────────────────────────────────── [15:39:02] Cleaning fio test files from /tank-mirror/bench … [15:39:02] Test files removed [15:39:02] Dropping Linux page cache + ZFS ARC … [15:39:02] Flushing ZFS ARC (was 1) … [15:39:07] ARC flushed — current size: 25667224 bytes [15:39:07] NVMe volatile write caches flushed [15:39:08] pmcd healthy (core ✓ pmdazfs ✓) [15:39:08] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj8_run1.ndjson [15:39:09] Collector running (PID 2597753), settling 5s … [15:39:14] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj8.fio [16:01:43] fio completed in 1349s [16:01:43] Post-fio settle 5s … [16:01:48] Stopping collector [16:01:49] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj8_run1.manifest.json [16:01:49] Cooldown: 30s [16:02:19] ──────────────────────────────────────────────────── [16:02:19] Run 2/3: rand-read-qd1-nj8_run2 [16:02:19] ──────────────────────────────────────────────────── [16:02:19] Cleaning fio test files from /tank-mirror/bench … [16:02:19] Test files removed [16:02:19] Dropping Linux page cache + ZFS ARC … [16:02:19] Flushing ZFS ARC (was 1) … [16:02:24] ARC flushed — current size: 48668621872 bytes [16:02:24] NVMe volatile write caches flushed [16:02:26] pmcd healthy (core ✓ pmdazfs ✓) [16:02:26] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj8_run2.ndjson [16:02:27] Collector running (PID 2614699), settling 5s … [16:02:32] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj8.fio [16:25:20] fio completed in 1368s [16:25:20] Post-fio settle 5s … [16:25:25] Stopping collector [16:25:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj8_run2.manifest.json [16:25:27] Cooldown: 30s [16:25:57] ──────────────────────────────────────────────────── [16:25:57] Run 3/3: rand-read-qd1-nj8_run3 [16:25:57] ──────────────────────────────────────────────────── [16:25:57] Cleaning fio test files from /tank-mirror/bench … [16:25:57] Test files removed [16:25:57] Dropping Linux page cache + ZFS ARC … [16:25:57] Flushing ZFS ARC (was 1) … [16:26:02] ARC flushed — current size: 39721988312 bytes [16:26:02] NVMe volatile write caches flushed [16:26:04] pmcd healthy (core ✓ pmdazfs ✓) [16:26:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj8_run3.ndjson [16:26:05] Collector running (PID 2631879), settling 5s … [16:26:10] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj8.fio [16:48:10] fio completed in 1320s [16:48:10] Post-fio settle 5s … [16:48:15] Stopping collector [16:48:16] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj8_run3.manifest.json [16:48:16] ═══════════════════════════════════════════════════════════ [16:48:16] All 3 runs complete for: rand-read-qd1-nj8 [16:48:16] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [16:48:16] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [16:48:16] ═══════════════════════════════════════════════════════════ [16:48:16] Progress: 18/53 jobs | Elapsed: 5h24m | ETA: ~10h30m [16:48:16] [16:48:16] ╔══════════════════════════════════════════════════════════╗ [16:48:16] ║ Job 19/53: rand-read-qd1-nj16 [16:48:16] ║ Category: rand [16:48:16] ╚══════════════════════════════════════════════════════════╝ [16:48:16] ╔══════════════════════════════════════════════════════════╗ [16:48:16] ║ Benchmark: rand-read-qd1-nj16 [16:48:16] ║ Category: rand [16:48:16] ║ Runs: 3 × cooldown 30s [16:48:16] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [16:48:16] ╚══════════════════════════════════════════════════════════╝ [16:48:16] ──────────────────────────────────────────────────── [16:48:16] Run 1/3: rand-read-qd1-nj16_run1 [16:48:16] ──────────────────────────────────────────────────── [16:48:16] Cleaning fio test files from /tank-mirror/bench … [16:48:16] Test files removed [16:48:16] Dropping Linux page cache + ZFS ARC … [16:48:17] Flushing ZFS ARC (was 1) … [16:48:22] ARC flushed — current size: 30858085440 bytes [16:48:22] NVMe volatile write caches flushed [16:48:23] pmcd healthy (core ✓ pmdazfs ✓) [16:48:23] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj16_run1.ndjson [16:48:24] Collector running (PID 2647855), settling 5s … [16:48:29] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj16.fio [17:30:44] fio completed in 2535s [17:30:44] Post-fio settle 5s … [17:30:49] Stopping collector [17:30:51] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj16_run1.manifest.json [17:30:51] Cooldown: 30s [17:31:21] ──────────────────────────────────────────────────── [17:31:21] Run 2/3: rand-read-qd1-nj16_run2 [17:31:21] ──────────────────────────────────────────────────── [17:31:21] Cleaning fio test files from /tank-mirror/bench … [17:31:22] Test files removed [17:31:22] Dropping Linux page cache + ZFS ARC … [17:31:22] Flushing ZFS ARC (was 1) … [17:31:27] ARC flushed — current size: 48408640648 bytes [17:31:27] NVMe volatile write caches flushed [17:31:28] pmcd healthy (core ✓ pmdazfs ✓) [17:31:28] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj16_run2.ndjson [17:31:29] Collector running (PID 2679393), settling 5s … [17:31:34] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj16.fio [18:13:43] fio completed in 2529s [18:13:43] Post-fio settle 5s … [18:13:49] Stopping collector [18:13:50] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj16_run2.manifest.json [18:13:50] Cooldown: 30s [18:14:20] ──────────────────────────────────────────────────── [18:14:20] Run 3/3: rand-read-qd1-nj16_run3 [18:14:20] ──────────────────────────────────────────────────── [18:14:20] Cleaning fio test files from /tank-mirror/bench … [18:14:20] Test files removed [18:14:20] Dropping Linux page cache + ZFS ARC … [18:14:20] Flushing ZFS ARC (was 1) … [18:14:25] ARC flushed — current size: 48578797832 bytes [18:14:25] NVMe volatile write caches flushed [18:14:26] pmcd healthy (core ✓ pmdazfs ✓) [18:14:26] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj16_run3.ndjson [18:14:27] Collector running (PID 2710089), settling 5s … [18:14:32] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj16.fio [18:56:50] fio completed in 2538s [18:56:50] Post-fio settle 5s … [18:56:55] Stopping collector [18:56:57] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj16_run3.manifest.json [18:56:57] ═══════════════════════════════════════════════════════════ [18:56:57] All 3 runs complete for: rand-read-qd1-nj16 [18:56:57] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [18:56:57] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [18:56:57] ═══════════════════════════════════════════════════════════ [18:56:57] Progress: 19/53 jobs | Elapsed: 7h32m | ETA: ~13h30m [18:56:57] [18:56:57] ╔══════════════════════════════════════════════════════════╗ [18:56:57] ║ Job 20/53: rand-read-qd1-nj32 [18:56:57] ║ Category: rand [18:56:57] ╚══════════════════════════════════════════════════════════╝ [18:56:57] ╔══════════════════════════════════════════════════════════╗ [18:56:57] ║ Benchmark: rand-read-qd1-nj32 [18:56:57] ║ Category: rand [18:56:57] ║ Runs: 3 × cooldown 30s [18:56:57] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [18:56:57] ╚══════════════════════════════════════════════════════════╝ [18:56:57] ──────────────────────────────────────────────────── [18:56:57] Run 1/3: rand-read-qd1-nj32_run1 [18:56:57] ──────────────────────────────────────────────────── [18:56:57] Cleaning fio test files from /tank-mirror/bench … [18:56:57] Test files removed [18:56:57] Dropping Linux page cache + ZFS ARC … [18:56:58] Flushing ZFS ARC (was 1) … [18:57:03] ARC flushed — current size: 48963619648 bytes [18:57:03] NVMe volatile write caches flushed [18:57:04] pmcd healthy (core ✓ pmdazfs ✓) [18:57:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj32_run1.ndjson [18:57:05] Collector running (PID 2741527), settling 5s … [18:57:10] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj32.fio [20:18:54] fio completed in 4904s [20:18:54] Post-fio settle 5s … [20:18:59] Stopping collector [20:19:00] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj32_run1.manifest.json [20:19:00] Cooldown: 30s [20:19:30] ──────────────────────────────────────────────────── [20:19:30] Run 2/3: rand-read-qd1-nj32_run2 [20:19:30] ──────────────────────────────────────────────────── [20:19:30] Cleaning fio test files from /tank-mirror/bench … [20:19:30] Test files removed [20:19:30] Dropping Linux page cache + ZFS ARC … [20:19:31] Flushing ZFS ARC (was 1) … [20:19:36] ARC flushed — current size: 87403607368 bytes [20:19:36] NVMe volatile write caches flushed [20:19:37] pmcd healthy (core ✓ pmdazfs ✓) [20:19:37] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj32_run2.ndjson [20:19:38] Collector running (PID 2800813), settling 5s … [20:19:43] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj32.fio [21:41:42] fio completed in 4919s [21:41:42] Post-fio settle 5s … [21:41:47] Stopping collector [21:41:48] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj32_run2.manifest.json [21:41:48] Cooldown: 30s [21:42:18] ──────────────────────────────────────────────────── [21:42:18] Run 3/3: rand-read-qd1-nj32_run3 [21:42:18] ──────────────────────────────────────────────────── [21:42:18] Cleaning fio test files from /tank-mirror/bench … [21:42:18] Test files removed [21:42:18] Dropping Linux page cache + ZFS ARC … [21:42:18] Flushing ZFS ARC (was 1) … [21:42:23] ARC flushed — current size: 88266692128 bytes [21:42:23] NVMe volatile write caches flushed [21:42:25] pmcd healthy (core ✓ pmdazfs ✓) [21:42:25] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd1-nj32_run3.ndjson [21:42:26] Collector running (PID 2860550), settling 5s … [21:42:31] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd1-nj32.fio [23:03:27] fio completed in 4856s [23:03:27] Post-fio settle 5s … [23:03:32] Stopping collector [23:03:33] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd1-nj32_run3.manifest.json [23:03:33] ═══════════════════════════════════════════════════════════ [23:03:33] All 3 runs complete for: rand-read-qd1-nj32 [23:03:33] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:03:33] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:03:33] ═══════════════════════════════════════════════════════════ [23:03:33] Progress: 20/53 jobs | Elapsed: 11h39m | ETA: ~19h13m [23:03:33] [23:03:33] ╔══════════════════════════════════════════════════════════╗ [23:03:33] ║ Job 21/53: rand-read-qd4-nj1 [23:03:33] ║ Category: rand [23:03:33] ╚══════════════════════════════════════════════════════════╝ [23:03:34] ╔══════════════════════════════════════════════════════════╗ [23:03:34] ║ Benchmark: rand-read-qd4-nj1 [23:03:34] ║ Category: rand [23:03:34] ║ Runs: 3 × cooldown 30s [23:03:34] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:03:34] ╚══════════════════════════════════════════════════════════╝ [23:03:34] ──────────────────────────────────────────────────── [23:03:34] Run 1/3: rand-read-qd4-nj1_run1 [23:03:34] ──────────────────────────────────────────────────── [23:03:34] Cleaning fio test files from /tank-mirror/bench … [23:03:34] Test files removed [23:03:34] Dropping Linux page cache + ZFS ARC … [23:03:34] Flushing ZFS ARC (was 1) … [23:03:39] ARC flushed — current size: 88133907080 bytes [23:03:39] NVMe volatile write caches flushed [23:03:40] pmcd healthy (core ✓ pmdazfs ✓) [23:03:40] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj1_run1.ndjson [23:03:41] Collector running (PID 2919656), settling 5s … [23:03:46] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj1.fio [23:09:27] fio completed in 341s [23:09:27] Post-fio settle 5s … [23:09:32] Stopping collector [23:09:33] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj1_run1.manifest.json [23:09:33] Cooldown: 30s [23:10:03] ──────────────────────────────────────────────────── [23:10:03] Run 2/3: rand-read-qd4-nj1_run2 [23:10:03] ──────────────────────────────────────────────────── [23:10:03] Cleaning fio test files from /tank-mirror/bench … [23:10:04] Test files removed [23:10:04] Dropping Linux page cache + ZFS ARC … [23:10:04] Flushing ZFS ARC (was 1) … [23:10:09] ARC flushed — current size: 25405968 bytes [23:10:09] NVMe volatile write caches flushed [23:10:10] pmcd healthy (core ✓ pmdazfs ✓) [23:10:10] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj1_run2.ndjson [23:10:11] Collector running (PID 2923947), settling 5s … [23:10:16] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj1.fio [23:15:52] fio completed in 336s [23:15:52] Post-fio settle 5s … [23:15:57] Stopping collector [23:15:59] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj1_run2.manifest.json [23:15:59] Cooldown: 30s [23:16:29] ──────────────────────────────────────────────────── [23:16:29] Run 3/3: rand-read-qd4-nj1_run3 [23:16:29] ──────────────────────────────────────────────────── [23:16:29] Cleaning fio test files from /tank-mirror/bench … [23:16:29] Test files removed [23:16:29] Dropping Linux page cache + ZFS ARC … [23:16:29] Flushing ZFS ARC (was 1) … [23:16:34] ARC flushed — current size: 25528656 bytes [23:16:34] NVMe volatile write caches flushed [23:16:35] pmcd healthy (core ✓ pmdazfs ✓) [23:16:35] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj1_run3.ndjson [23:16:36] Collector running (PID 2928129), settling 5s … [23:16:41] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj1.fio [23:22:19] fio completed in 338s [23:22:19] Post-fio settle 5s … [23:22:24] Stopping collector [23:22:25] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj1_run3.manifest.json [23:22:25] ═══════════════════════════════════════════════════════════ [23:22:25] All 3 runs complete for: rand-read-qd4-nj1 [23:22:25] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:22:25] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:22:25] ═══════════════════════════════════════════════════════════ [23:22:25] Progress: 21/53 jobs | Elapsed: 11h58m | ETA: ~18h14m [23:22:25] [23:22:25] ╔══════════════════════════════════════════════════════════╗ [23:22:25] ║ Job 22/53: rand-read-qd4-nj8 [23:22:25] ║ Category: rand [23:22:25] ╚══════════════════════════════════════════════════════════╝ [23:22:25] ╔══════════════════════════════════════════════════════════╗ [23:22:25] ║ Benchmark: rand-read-qd4-nj8 [23:22:25] ║ Category: rand [23:22:25] ║ Runs: 3 × cooldown 30s [23:22:25] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:22:25] ╚══════════════════════════════════════════════════════════╝ [23:22:25] ──────────────────────────────────────────────────── [23:22:25] Run 1/3: rand-read-qd4-nj8_run1 [23:22:25] ──────────────────────────────────────────────────── [23:22:25] Cleaning fio test files from /tank-mirror/bench … [23:22:25] Test files removed [23:22:25] Dropping Linux page cache + ZFS ARC … [23:22:25] Flushing ZFS ARC (was 1) … [23:22:30] ARC flushed — current size: 25933888 bytes [23:22:30] NVMe volatile write caches flushed [23:22:32] pmcd healthy (core ✓ pmdazfs ✓) [23:22:32] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj8_run1.ndjson [23:22:33] Collector running (PID 2932273), settling 5s … [23:22:38] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj8.fio [23:45:29] fio completed in 1371s [23:45:29] Post-fio settle 5s … [23:45:34] Stopping collector [23:45:36] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj8_run1.manifest.json [23:45:36] Cooldown: 30s [23:46:06] ──────────────────────────────────────────────────── [23:46:06] Run 2/3: rand-read-qd4-nj8_run2 [23:46:06] ──────────────────────────────────────────────────── [23:46:06] Cleaning fio test files from /tank-mirror/bench … [23:46:06] Test files removed [23:46:06] Dropping Linux page cache + ZFS ARC … [23:46:06] Flushing ZFS ARC (was 1) … [23:46:12] ARC flushed — current size: 32802842992 bytes [23:46:12] NVMe volatile write caches flushed [23:46:13] pmcd healthy (core ✓ pmdazfs ✓) [23:46:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj8_run2.ndjson [23:46:14] Collector running (PID 2949393), settling 5s … [23:46:19] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj8.fio [00:09:11] fio completed in 1372s [00:09:11] Post-fio settle 5s … [00:09:16] Stopping collector [00:09:17] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj8_run2.manifest.json [00:09:17] Cooldown: 30s [00:09:47] ──────────────────────────────────────────────────── [00:09:47] Run 3/3: rand-read-qd4-nj8_run3 [00:09:47] ──────────────────────────────────────────────────── [00:09:47] Cleaning fio test files from /tank-mirror/bench … [00:09:47] Test files removed [00:09:47] Dropping Linux page cache + ZFS ARC … [00:09:47] Flushing ZFS ARC (was 1) … [00:09:52] ARC flushed — current size: 59120629984 bytes [00:09:52] NVMe volatile write caches flushed [00:09:54] pmcd healthy (core ✓ pmdazfs ✓) [00:09:54] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj8_run3.ndjson [00:09:55] Collector running (PID 2966593), settling 5s … [00:10:00] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj8.fio [00:32:45] fio completed in 1365s [00:32:45] Post-fio settle 5s … [00:32:50] Stopping collector [00:32:51] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj8_run3.manifest.json [00:32:51] ═══════════════════════════════════════════════════════════ [00:32:51] All 3 runs complete for: rand-read-qd4-nj8 [00:32:51] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:32:51] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:32:51] ═══════════════════════════════════════════════════════════ [00:32:51] Progress: 22/53 jobs | Elapsed: 13h8m | ETA: ~18h31m [00:32:51] [00:32:51] ╔══════════════════════════════════════════════════════════╗ [00:32:51] ║ Job 23/53: rand-read-qd4-nj16 [00:32:51] ║ Category: rand [00:32:51] ╚══════════════════════════════════════════════════════════╝ [00:32:51] ╔══════════════════════════════════════════════════════════╗ [00:32:51] ║ Benchmark: rand-read-qd4-nj16 [00:32:51] ║ Category: rand [00:32:51] ║ Runs: 3 × cooldown 30s [00:32:51] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:32:51] ╚══════════════════════════════════════════════════════════╝ [00:32:51] ──────────────────────────────────────────────────── [00:32:51] Run 1/3: rand-read-qd4-nj16_run1 [00:32:51] ──────────────────────────────────────────────────── [00:32:51] Cleaning fio test files from /tank-mirror/bench … [00:32:51] Test files removed [00:32:51] Dropping Linux page cache + ZFS ARC … [00:32:51] Flushing ZFS ARC (was 1) … [00:32:56] ARC flushed — current size: 56587740600 bytes [00:32:56] NVMe volatile write caches flushed [00:32:58] pmcd healthy (core ✓ pmdazfs ✓) [00:32:58] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj16_run1.ndjson [00:32:59] Collector running (PID 2983926), settling 5s … [00:33:04] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj16.fio [01:15:37] fio completed in 2553s [01:15:37] Post-fio settle 5s … [01:15:42] Stopping collector [01:15:43] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj16_run1.manifest.json [01:15:43] Cooldown: 30s [01:16:13] ──────────────────────────────────────────────────── [01:16:13] Run 2/3: rand-read-qd4-nj16_run2 [01:16:13] ──────────────────────────────────────────────────── [01:16:13] Cleaning fio test files from /tank-mirror/bench … [01:16:13] Test files removed [01:16:13] Dropping Linux page cache + ZFS ARC … [01:16:14] Flushing ZFS ARC (was 1) … [01:16:19] ARC flushed — current size: 47846708408 bytes [01:16:19] NVMe volatile write caches flushed [01:16:20] pmcd healthy (core ✓ pmdazfs ✓) [01:16:20] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj16_run2.ndjson [01:16:21] Collector running (PID 3014839), settling 5s … [01:16:26] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj16.fio [01:58:53] fio completed in 2547s [01:58:53] Post-fio settle 5s … [01:58:58] Stopping collector [01:58:59] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj16_run2.manifest.json [01:58:59] Cooldown: 30s [01:59:29] ──────────────────────────────────────────────────── [01:59:29] Run 3/3: rand-read-qd4-nj16_run3 [01:59:29] ──────────────────────────────────────────────────── [01:59:29] Cleaning fio test files from /tank-mirror/bench … [01:59:29] Test files removed [01:59:29] Dropping Linux page cache + ZFS ARC … [01:59:30] Flushing ZFS ARC (was 1) … [01:59:35] ARC flushed — current size: 48991617120 bytes [01:59:35] NVMe volatile write caches flushed [01:59:36] pmcd healthy (core ✓ pmdazfs ✓) [01:59:36] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj16_run3.ndjson [01:59:37] Collector running (PID 3046316), settling 5s … [01:59:42] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj16.fio [02:42:12] fio completed in 2550s [02:42:12] Post-fio settle 5s … [02:42:17] Stopping collector [02:42:18] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj16_run3.manifest.json [02:42:18] ═══════════════════════════════════════════════════════════ [02:42:18] All 3 runs complete for: rand-read-qd4-nj16 [02:42:18] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [02:42:18] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [02:42:18] ═══════════════════════════════════════════════════════════ [02:42:18] Progress: 23/53 jobs | Elapsed: 15h18m | ETA: ~19h57m [02:42:18] [02:42:18] ╔══════════════════════════════════════════════════════════╗ [02:42:18] ║ Job 24/53: rand-read-qd4-nj32 [02:42:18] ║ Category: rand [02:42:18] ╚══════════════════════════════════════════════════════════╝ [02:42:18] ╔══════════════════════════════════════════════════════════╗ [02:42:18] ║ Benchmark: rand-read-qd4-nj32 [02:42:18] ║ Category: rand [02:42:18] ║ Runs: 3 × cooldown 30s [02:42:18] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [02:42:18] ╚══════════════════════════════════════════════════════════╝ [02:42:18] ──────────────────────────────────────────────────── [02:42:18] Run 1/3: rand-read-qd4-nj32_run1 [02:42:18] ──────────────────────────────────────────────────── [02:42:18] Cleaning fio test files from /tank-mirror/bench … [02:42:19] Test files removed [02:42:19] Dropping Linux page cache + ZFS ARC … [02:42:19] Flushing ZFS ARC (was 1) … [02:42:24] ARC flushed — current size: 48634976928 bytes [02:42:24] NVMe volatile write caches flushed [02:42:25] pmcd healthy (core ✓ pmdazfs ✓) [02:42:25] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj32_run1.ndjson [02:42:26] Collector running (PID 3077196), settling 5s … [02:42:31] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj32.fio [04:04:03] fio completed in 4892s [04:04:03] Post-fio settle 5s … [04:04:08] Stopping collector [04:04:09] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj32_run1.manifest.json [04:04:09] Cooldown: 30s [04:04:39] ──────────────────────────────────────────────────── [04:04:39] Run 2/3: rand-read-qd4-nj32_run2 [04:04:39] ──────────────────────────────────────────────────── [04:04:39] Cleaning fio test files from /tank-mirror/bench … [04:04:39] Test files removed [04:04:39] Dropping Linux page cache + ZFS ARC … [04:04:39] Flushing ZFS ARC (was 1) … [04:04:44] ARC flushed — current size: 90941327560 bytes [04:04:44] NVMe volatile write caches flushed [04:04:46] pmcd healthy (core ✓ pmdazfs ✓) [04:04:46] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj32_run2.ndjson [04:04:47] Collector running (PID 3136680), settling 5s … [04:04:52] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj32.fio [05:26:28] fio completed in 4896s [05:26:28] Post-fio settle 5s … [05:26:33] Stopping collector [05:26:34] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj32_run2.manifest.json [05:26:34] Cooldown: 30s [05:27:04] ──────────────────────────────────────────────────── [05:27:04] Run 3/3: rand-read-qd4-nj32_run3 [05:27:04] ──────────────────────────────────────────────────── [05:27:04] Cleaning fio test files from /tank-mirror/bench … [05:27:05] Test files removed [05:27:05] Dropping Linux page cache + ZFS ARC … [05:27:05] Flushing ZFS ARC (was 1) … [05:27:10] ARC flushed — current size: 87253665272 bytes [05:27:10] NVMe volatile write caches flushed [05:27:11] pmcd healthy (core ✓ pmdazfs ✓) [05:27:11] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd4-nj32_run3.ndjson [05:27:12] Collector running (PID 3196330), settling 5s … [05:27:17] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd4-nj32.fio [06:49:07] fio completed in 4910s [06:49:07] Post-fio settle 5s … [06:49:12] Stopping collector [06:49:13] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd4-nj32_run3.manifest.json [06:49:13] ═══════════════════════════════════════════════════════════ [06:49:13] All 3 runs complete for: rand-read-qd4-nj32 [06:49:13] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [06:49:13] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [06:49:13] ═══════════════════════════════════════════════════════════ [06:49:13] Progress: 24/53 jobs | Elapsed: 19h25m | ETA: ~23h27m [06:49:13] [06:49:13] ╔══════════════════════════════════════════════════════════╗ [06:49:13] ║ Job 25/53: rand-read-qd16-nj1 [06:49:13] ║ Category: rand [06:49:13] ╚══════════════════════════════════════════════════════════╝ [06:49:13] ╔══════════════════════════════════════════════════════════╗ [06:49:13] ║ Benchmark: rand-read-qd16-nj1 [06:49:13] ║ Category: rand [06:49:13] ║ Runs: 3 × cooldown 30s [06:49:13] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [06:49:13] ╚══════════════════════════════════════════════════════════╝ [06:49:13] ──────────────────────────────────────────────────── [06:49:13] Run 1/3: rand-read-qd16-nj1_run1 [06:49:13] ──────────────────────────────────────────────────── [06:49:13] Cleaning fio test files from /tank-mirror/bench … [06:49:14] Test files removed [06:49:14] Dropping Linux page cache + ZFS ARC … [06:49:14] Flushing ZFS ARC (was 1) … [06:49:19] ARC flushed — current size: 87168797848 bytes [06:49:19] NVMe volatile write caches flushed [06:49:20] pmcd healthy (core ✓ pmdazfs ✓) [06:49:20] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj1_run1.ndjson [06:49:21] Collector running (PID 3255401), settling 5s … [06:49:26] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj1.fio [06:55:07] fio completed in 341s [06:55:07] Post-fio settle 5s … [06:55:12] Stopping collector [06:55:13] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj1_run1.manifest.json [06:55:13] Cooldown: 30s [06:55:43] ──────────────────────────────────────────────────── [06:55:43] Run 2/3: rand-read-qd16-nj1_run2 [06:55:43] ──────────────────────────────────────────────────── [06:55:43] Cleaning fio test files from /tank-mirror/bench … [06:55:44] Test files removed [06:55:44] Dropping Linux page cache + ZFS ARC … [06:55:44] Flushing ZFS ARC (was 1) … [06:55:49] ARC flushed — current size: 25871096 bytes [06:55:49] NVMe volatile write caches flushed [06:55:50] pmcd healthy (core ✓ pmdazfs ✓) [06:55:50] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj1_run2.ndjson [06:55:51] Collector running (PID 3260311), settling 5s … [06:55:56] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj1.fio [07:01:34] fio completed in 338s [07:01:34] Post-fio settle 5s … [07:01:39] Stopping collector [07:01:40] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj1_run2.manifest.json [07:01:40] Cooldown: 30s [07:02:10] ──────────────────────────────────────────────────── [07:02:10] Run 3/3: rand-read-qd16-nj1_run3 [07:02:10] ──────────────────────────────────────────────────── [07:02:10] Cleaning fio test files from /tank-mirror/bench … [07:02:10] Test files removed [07:02:10] Dropping Linux page cache + ZFS ARC … [07:02:10] Flushing ZFS ARC (was 1) … [07:02:16] ARC flushed — current size: 631386976 bytes [07:02:16] NVMe volatile write caches flushed [07:02:17] pmcd healthy (core ✓ pmdazfs ✓) [07:02:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj1_run3.ndjson [07:02:18] Collector running (PID 3264519), settling 5s … [07:02:23] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj1.fio [07:08:00] fio completed in 337s [07:08:00] Post-fio settle 5s … [07:08:05] Stopping collector [07:08:06] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj1_run3.manifest.json [07:08:06] ═══════════════════════════════════════════════════════════ [07:08:06] All 3 runs complete for: rand-read-qd16-nj1 [07:08:06] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [07:08:06] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [07:08:06] ═══════════════════════════════════════════════════════════ [07:08:06] Progress: 25/53 jobs | Elapsed: 19h44m | ETA: ~22h5m [07:08:06] [07:08:06] ╔══════════════════════════════════════════════════════════╗ [07:08:06] ║ Job 26/53: rand-read-qd16-nj8 [07:08:06] ║ Category: rand [07:08:06] ╚══════════════════════════════════════════════════════════╝ [07:08:07] ╔══════════════════════════════════════════════════════════╗ [07:08:07] ║ Benchmark: rand-read-qd16-nj8 [07:08:07] ║ Category: rand [07:08:07] ║ Runs: 3 × cooldown 30s [07:08:07] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [07:08:07] ╚══════════════════════════════════════════════════════════╝ [07:08:07] ──────────────────────────────────────────────────── [07:08:07] Run 1/3: rand-read-qd16-nj8_run1 [07:08:07] ──────────────────────────────────────────────────── [07:08:07] Cleaning fio test files from /tank-mirror/bench … [07:08:07] Test files removed [07:08:07] Dropping Linux page cache + ZFS ARC … [07:08:07] Flushing ZFS ARC (was 1) … [07:08:12] ARC flushed — current size: 26426944 bytes [07:08:12] NVMe volatile write caches flushed [07:08:13] pmcd healthy (core ✓ pmdazfs ✓) [07:08:13] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj8_run1.ndjson [07:08:14] Collector running (PID 3268656), settling 5s … [07:08:19] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj8.fio [07:31:10] fio completed in 1371s [07:31:10] Post-fio settle 5s … [07:31:15] Stopping collector [07:31:16] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj8_run1.manifest.json [07:31:16] Cooldown: 30s [07:31:46] ──────────────────────────────────────────────────── [07:31:46] Run 2/3: rand-read-qd16-nj8_run2 [07:31:46] ──────────────────────────────────────────────────── [07:31:46] Cleaning fio test files from /tank-mirror/bench … [07:31:46] Test files removed [07:31:46] Dropping Linux page cache + ZFS ARC … [07:31:46] Flushing ZFS ARC (was 1) … [07:31:51] ARC flushed — current size: 54226529424 bytes [07:31:51] NVMe volatile write caches flushed [07:31:52] pmcd healthy (core ✓ pmdazfs ✓) [07:31:52] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj8_run2.ndjson [07:31:53] Collector running (PID 3285820), settling 5s … [07:31:58] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj8.fio [07:54:44] fio completed in 1366s [07:54:44] Post-fio settle 5s … [07:54:49] Stopping collector [07:54:50] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj8_run2.manifest.json [07:54:50] Cooldown: 30s [07:55:20] ──────────────────────────────────────────────────── [07:55:20] Run 3/3: rand-read-qd16-nj8_run3 [07:55:20] ──────────────────────────────────────────────────── [07:55:20] Cleaning fio test files from /tank-mirror/bench … [07:55:20] Test files removed [07:55:20] Dropping Linux page cache + ZFS ARC … [07:55:21] Flushing ZFS ARC (was 1) … [07:55:26] ARC flushed — current size: 68333345104 bytes [07:55:26] NVMe volatile write caches flushed [07:55:27] pmcd healthy (core ✓ pmdazfs ✓) [07:55:27] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj8_run3.ndjson [07:55:28] Collector running (PID 3302862), settling 5s … [07:55:33] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj8.fio [08:18:02] fio completed in 1349s [08:18:02] Post-fio settle 5s … [08:18:07] Stopping collector [08:18:08] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj8_run3.manifest.json [08:18:08] ═══════════════════════════════════════════════════════════ [08:18:08] All 3 runs complete for: rand-read-qd16-nj8 [08:18:08] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [08:18:08] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [08:18:08] ═══════════════════════════════════════════════════════════ [08:18:08] Progress: 26/53 jobs | Elapsed: 20h54m | ETA: ~21h41m [08:18:08] [08:18:08] ╔══════════════════════════════════════════════════════════╗ [08:18:08] ║ Job 27/53: rand-read-qd16-nj16 [08:18:08] ║ Category: rand [08:18:08] ╚══════════════════════════════════════════════════════════╝ [08:18:08] ╔══════════════════════════════════════════════════════════╗ [08:18:08] ║ Benchmark: rand-read-qd16-nj16 [08:18:08] ║ Category: rand [08:18:08] ║ Runs: 3 × cooldown 30s [08:18:08] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [08:18:08] ╚══════════════════════════════════════════════════════════╝ [08:18:08] ──────────────────────────────────────────────────── [08:18:08] Run 1/3: rand-read-qd16-nj16_run1 [08:18:08] ──────────────────────────────────────────────────── [08:18:08] Cleaning fio test files from /tank-mirror/bench … [08:18:09] Test files removed [08:18:09] Dropping Linux page cache + ZFS ARC … [08:18:09] Flushing ZFS ARC (was 1) … [08:18:14] ARC flushed — current size: 1059041472 bytes [08:18:14] NVMe volatile write caches flushed [08:18:15] pmcd healthy (core ✓ pmdazfs ✓) [08:18:15] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj16_run1.ndjson [08:18:16] Collector running (PID 3319000), settling 5s … [08:18:21] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj16.fio [09:00:14] fio completed in 2513s [09:00:14] Post-fio settle 5s … [09:00:19] Stopping collector [09:00:21] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj16_run1.manifest.json [09:00:21] Cooldown: 30s [09:00:51] ──────────────────────────────────────────────────── [09:00:51] Run 2/3: rand-read-qd16-nj16_run2 [09:00:51] ──────────────────────────────────────────────────── [09:00:51] Cleaning fio test files from /tank-mirror/bench … [09:00:51] Test files removed [09:00:51] Dropping Linux page cache + ZFS ARC … [09:00:51] Flushing ZFS ARC (was 1) … [09:00:56] ARC flushed — current size: 48010584336 bytes [09:00:56] NVMe volatile write caches flushed [09:00:57] pmcd healthy (core ✓ pmdazfs ✓) [09:00:57] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj16_run2.ndjson [09:00:58] Collector running (PID 3350156), settling 5s … [09:01:03] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj16.fio [09:43:28] fio completed in 2545s [09:43:28] Post-fio settle 5s … [09:43:33] Stopping collector [09:43:34] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj16_run2.manifest.json [09:43:34] Cooldown: 30s [09:44:04] ──────────────────────────────────────────────────── [09:44:04] Run 3/3: rand-read-qd16-nj16_run3 [09:44:04] ──────────────────────────────────────────────────── [09:44:04] Cleaning fio test files from /tank-mirror/bench … [09:44:04] Test files removed [09:44:04] Dropping Linux page cache + ZFS ARC … [09:44:04] Flushing ZFS ARC (was 1) … [09:44:09] ARC flushed — current size: 48955968208 bytes [09:44:09] NVMe volatile write caches flushed [09:44:10] pmcd healthy (core ✓ pmdazfs ✓) [09:44:10] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj16_run3.ndjson [09:44:11] Collector running (PID 3380975), settling 5s … [09:44:16] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj16.fio [10:26:21] fio completed in 2525s [10:26:21] Post-fio settle 5s … [10:26:26] Stopping collector [10:26:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj16_run3.manifest.json [10:26:27] ═══════════════════════════════════════════════════════════ [10:26:27] All 3 runs complete for: rand-read-qd16-nj16 [10:26:27] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [10:26:27] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [10:26:27] ═══════════════════════════════════════════════════════════ [10:26:27] Progress: 27/53 jobs | Elapsed: 23h2m | ETA: ~22h10m [10:26:27] [10:26:27] ╔══════════════════════════════════════════════════════════╗ [10:26:27] ║ Job 28/53: rand-read-qd16-nj32 [10:26:27] ║ Category: rand [10:26:27] ╚══════════════════════════════════════════════════════════╝ [10:26:27] ╔══════════════════════════════════════════════════════════╗ [10:26:27] ║ Benchmark: rand-read-qd16-nj32 [10:26:27] ║ Category: rand [10:26:27] ║ Runs: 3 × cooldown 30s [10:26:27] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [10:26:27] ╚══════════════════════════════════════════════════════════╝ [10:26:27] ──────────────────────────────────────────────────── [10:26:27] Run 1/3: rand-read-qd16-nj32_run1 [10:26:27] ──────────────────────────────────────────────────── [10:26:27] Cleaning fio test files from /tank-mirror/bench … [10:26:27] Test files removed [10:26:27] Dropping Linux page cache + ZFS ARC … [10:26:27] Flushing ZFS ARC (was 1) … [10:26:32] ARC flushed — current size: 48500339336 bytes [10:26:32] NVMe volatile write caches flushed [10:26:34] pmcd healthy (core ✓ pmdazfs ✓) [10:26:34] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj32_run1.ndjson [10:26:35] Collector running (PID 3412351), settling 5s … [10:26:40] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj32.fio [11:47:59] fio completed in 4879s [11:47:59] Post-fio settle 5s … [11:48:04] Stopping collector [11:48:05] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj32_run1.manifest.json [11:48:05] Cooldown: 30s [11:48:35] ──────────────────────────────────────────────────── [11:48:35] Run 2/3: rand-read-qd16-nj32_run2 [11:48:35] ──────────────────────────────────────────────────── [11:48:35] Cleaning fio test files from /tank-mirror/bench … [11:48:35] Test files removed [11:48:35] Dropping Linux page cache + ZFS ARC … [11:48:36] Flushing ZFS ARC (was 1) … [11:48:41] ARC flushed — current size: 85318430904 bytes [11:48:41] NVMe volatile write caches flushed [11:48:42] pmcd healthy (core ✓ pmdazfs ✓) [11:48:42] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj32_run2.ndjson [11:48:43] Collector running (PID 3471742), settling 5s … [11:48:48] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj32.fio [13:10:16] fio completed in 4888s [13:10:16] Post-fio settle 5s … [13:10:21] Stopping collector [13:10:23] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj32_run2.manifest.json [13:10:23] Cooldown: 30s [13:10:53] ──────────────────────────────────────────────────── [13:10:53] Run 3/3: rand-read-qd16-nj32_run3 [13:10:53] ──────────────────────────────────────────────────── [13:10:53] Cleaning fio test files from /tank-mirror/bench … [13:10:54] Test files removed [13:10:54] Dropping Linux page cache + ZFS ARC … [13:10:54] Flushing ZFS ARC (was 1) … [13:10:59] ARC flushed — current size: 85755607648 bytes [13:10:59] NVMe volatile write caches flushed [13:11:00] pmcd healthy (core ✓ pmdazfs ✓) [13:11:00] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd16-nj32_run3.ndjson [13:11:01] Collector running (PID 3531393), settling 5s … [13:11:06] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd16-nj32.fio [14:33:03] fio completed in 4917s [14:33:03] Post-fio settle 5s … [14:33:08] Stopping collector [14:33:10] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd16-nj32_run3.manifest.json [14:33:10] ═══════════════════════════════════════════════════════════ [14:33:10] All 3 runs complete for: rand-read-qd16-nj32 [14:33:10] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [14:33:10] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [14:33:10] ═══════════════════════════════════════════════════════════ [14:33:10] Progress: 28/53 jobs | Elapsed: 27h9m | ETA: ~24h14m [14:33:10] [14:33:10] ╔══════════════════════════════════════════════════════════╗ [14:33:10] ║ Job 29/53: rand-read-qd32-nj1 [14:33:10] ║ Category: rand [14:33:10] ╚══════════════════════════════════════════════════════════╝ [14:33:10] ╔══════════════════════════════════════════════════════════╗ [14:33:10] ║ Benchmark: rand-read-qd32-nj1 [14:33:10] ║ Category: rand [14:33:10] ║ Runs: 3 × cooldown 30s [14:33:10] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:33:10] ╚══════════════════════════════════════════════════════════╝ [14:33:10] ──────────────────────────────────────────────────── [14:33:10] Run 1/3: rand-read-qd32-nj1_run1 [14:33:10] ──────────────────────────────────────────────────── [14:33:10] Cleaning fio test files from /tank-mirror/bench … [14:33:11] Test files removed [14:33:11] Dropping Linux page cache + ZFS ARC … [14:33:11] Flushing ZFS ARC (was 1) … [14:33:16] ARC flushed — current size: 87671369440 bytes [14:33:16] NVMe volatile write caches flushed [14:33:17] pmcd healthy (core ✓ pmdazfs ✓) [14:33:17] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj1_run1.ndjson [14:33:18] Collector running (PID 3591100), settling 5s … [14:33:23] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj1.fio [14:39:01] fio completed in 338s [14:39:01] Post-fio settle 5s … [14:39:06] Stopping collector [14:39:07] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj1_run1.manifest.json [14:39:07] Cooldown: 30s [14:39:37] ──────────────────────────────────────────────────── [14:39:37] Run 2/3: rand-read-qd32-nj1_run2 [14:39:37] ──────────────────────────────────────────────────── [14:39:37] Cleaning fio test files from /tank-mirror/bench … [14:39:37] Test files removed [14:39:37] Dropping Linux page cache + ZFS ARC … [14:39:37] Flushing ZFS ARC (was 1) … [14:39:43] ARC flushed — current size: 26416032 bytes [14:39:43] NVMe volatile write caches flushed [14:39:44] pmcd healthy (core ✓ pmdazfs ✓) [14:39:44] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj1_run2.ndjson [14:39:45] Collector running (PID 3595286), settling 5s … [14:39:50] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj1.fio [14:45:21] fio completed in 331s [14:45:21] Post-fio settle 5s … [14:45:26] Stopping collector [14:45:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj1_run2.manifest.json [14:45:27] Cooldown: 30s [14:45:57] ──────────────────────────────────────────────────── [14:45:57] Run 3/3: rand-read-qd32-nj1_run3 [14:45:57] ──────────────────────────────────────────────────── [14:45:57] Cleaning fio test files from /tank-mirror/bench … [14:45:58] Test files removed [14:45:58] Dropping Linux page cache + ZFS ARC … [14:45:58] Flushing ZFS ARC (was 1) … [14:46:03] ARC flushed — current size: 26712976 bytes [14:46:03] NVMe volatile write caches flushed [14:46:04] pmcd healthy (core ✓ pmdazfs ✓) [14:46:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj1_run3.ndjson [14:46:05] Collector running (PID 3599433), settling 5s … [14:46:10] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj1.fio [14:51:48] fio completed in 338s [14:51:48] Post-fio settle 5s … [14:51:53] Stopping collector [14:51:54] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj1_run3.manifest.json [14:51:54] ═══════════════════════════════════════════════════════════ [14:51:54] All 3 runs complete for: rand-read-qd32-nj1 [14:51:54] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [14:51:54] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [14:51:54] ═══════════════════════════════════════════════════════════ [14:51:54] Progress: 29/53 jobs | Elapsed: 27h27m | ETA: ~22h43m [14:51:54] [14:51:54] ╔══════════════════════════════════════════════════════════╗ [14:51:54] ║ Job 30/53: rand-read-qd32-nj8 [14:51:54] ║ Category: rand [14:51:54] ╚══════════════════════════════════════════════════════════╝ [14:51:54] ╔══════════════════════════════════════════════════════════╗ [14:51:54] ║ Benchmark: rand-read-qd32-nj8 [14:51:54] ║ Category: rand [14:51:54] ║ Runs: 3 × cooldown 30s [14:51:54] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [14:51:54] ╚══════════════════════════════════════════════════════════╝ [14:51:54] ──────────────────────────────────────────────────── [14:51:54] Run 1/3: rand-read-qd32-nj8_run1 [14:51:54] ──────────────────────────────────────────────────── [14:51:54] Cleaning fio test files from /tank-mirror/bench … [14:51:55] Test files removed [14:51:55] Dropping Linux page cache + ZFS ARC … [14:51:55] Flushing ZFS ARC (was 1) … [14:52:00] ARC flushed — current size: 27004720 bytes [14:52:00] NVMe volatile write caches flushed [14:52:01] pmcd healthy (core ✓ pmdazfs ✓) [14:52:01] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj8_run1.ndjson [14:52:02] Collector running (PID 3603584), settling 5s … [14:52:07] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj8.fio [15:14:57] fio completed in 1370s [15:14:57] Post-fio settle 5s … [15:15:02] Stopping collector [15:15:03] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj8_run1.manifest.json [15:15:03] Cooldown: 30s [15:15:33] ──────────────────────────────────────────────────── [15:15:33] Run 2/3: rand-read-qd32-nj8_run2 [15:15:33] ──────────────────────────────────────────────────── [15:15:33] Cleaning fio test files from /tank-mirror/bench … [15:15:34] Test files removed [15:15:34] Dropping Linux page cache + ZFS ARC … [15:15:34] Flushing ZFS ARC (was 1) … [15:15:39] ARC flushed — current size: 60682848016 bytes [15:15:39] NVMe volatile write caches flushed [15:15:40] pmcd healthy (core ✓ pmdazfs ✓) [15:15:40] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj8_run2.ndjson [15:15:41] Collector running (PID 3620715), settling 5s … [15:15:46] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj8.fio [15:38:35] fio completed in 1369s [15:38:35] Post-fio settle 5s … [15:38:40] Stopping collector [15:38:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj8_run2.manifest.json [15:38:41] Cooldown: 30s [15:39:11] ──────────────────────────────────────────────────── [15:39:11] Run 3/3: rand-read-qd32-nj8_run3 [15:39:11] ──────────────────────────────────────────────────── [15:39:11] Cleaning fio test files from /tank-mirror/bench … [15:39:11] Test files removed [15:39:11] Dropping Linux page cache + ZFS ARC … [15:39:11] Flushing ZFS ARC (was 1) … [15:39:16] ARC flushed — current size: 46978042096 bytes [15:39:16] NVMe volatile write caches flushed [15:39:18] pmcd healthy (core ✓ pmdazfs ✓) [15:39:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj8_run3.ndjson [15:39:19] Collector running (PID 3637886), settling 5s … [15:39:24] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj8.fio [16:01:31] fio completed in 1327s [16:01:31] Post-fio settle 5s … [16:01:36] Stopping collector [16:01:37] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj8_run3.manifest.json [16:01:37] ═══════════════════════════════════════════════════════════ [16:01:37] All 3 runs complete for: rand-read-qd32-nj8 [16:01:37] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [16:01:37] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [16:01:37] ═══════════════════════════════════════════════════════════ [16:01:37] Progress: 30/53 jobs | Elapsed: 28h37m | ETA: ~21h56m [16:01:37] [16:01:37] ╔══════════════════════════════════════════════════════════╗ [16:01:37] ║ Job 31/53: rand-read-qd32-nj16 [16:01:37] ║ Category: rand [16:01:37] ╚══════════════════════════════════════════════════════════╝ [16:01:37] ╔══════════════════════════════════════════════════════════╗ [16:01:37] ║ Benchmark: rand-read-qd32-nj16 [16:01:37] ║ Category: rand [16:01:37] ║ Runs: 3 × cooldown 30s [16:01:37] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [16:01:37] ╚══════════════════════════════════════════════════════════╝ [16:01:37] ──────────────────────────────────────────────────── [16:01:37] Run 1/3: rand-read-qd32-nj16_run1 [16:01:37] ──────────────────────────────────────────────────── [16:01:37] Cleaning fio test files from /tank-mirror/bench … [16:01:38] Test files removed [16:01:38] Dropping Linux page cache + ZFS ARC … [16:01:38] Flushing ZFS ARC (was 1) … [16:01:43] ARC flushed — current size: 12181414776 bytes [16:01:43] NVMe volatile write caches flushed [16:01:44] pmcd healthy (core ✓ pmdazfs ✓) [16:01:44] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj16_run1.ndjson [16:01:45] Collector running (PID 3654492), settling 5s … [16:01:50] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj16.fio [16:44:26] fio completed in 2556s [16:44:26] Post-fio settle 5s … [16:44:31] Stopping collector [16:44:33] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj16_run1.manifest.json [16:44:33] Cooldown: 30s [16:45:03] ──────────────────────────────────────────────────── [16:45:03] Run 2/3: rand-read-qd32-nj16_run2 [16:45:03] ──────────────────────────────────────────────────── [16:45:03] Cleaning fio test files from /tank-mirror/bench … [16:45:03] Test files removed [16:45:03] Dropping Linux page cache + ZFS ARC … [16:45:03] Flushing ZFS ARC (was 1) … [16:45:08] ARC flushed — current size: 45085820488 bytes [16:45:08] NVMe volatile write caches flushed [16:45:10] pmcd healthy (core ✓ pmdazfs ✓) [16:45:10] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj16_run2.ndjson [16:45:11] Collector running (PID 3685333), settling 5s … [16:45:16] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj16.fio [17:27:51] fio completed in 2555s [17:27:51] Post-fio settle 5s … [17:27:56] Stopping collector [17:27:57] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj16_run2.manifest.json [17:27:57] Cooldown: 30s [17:28:27] ──────────────────────────────────────────────────── [17:28:27] Run 3/3: rand-read-qd32-nj16_run3 [17:28:27] ──────────────────────────────────────────────────── [17:28:27] Cleaning fio test files from /tank-mirror/bench … [17:28:27] Test files removed [17:28:27] Dropping Linux page cache + ZFS ARC … [17:28:28] Flushing ZFS ARC (was 1) … [17:28:33] ARC flushed — current size: 48633809304 bytes [17:28:33] NVMe volatile write caches flushed [17:28:34] pmcd healthy (core ✓ pmdazfs ✓) [17:28:34] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj16_run3.ndjson [17:28:35] Collector running (PID 3716911), settling 5s … [17:28:40] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj16.fio [18:10:56] fio completed in 2536s [18:10:56] Post-fio settle 5s … [18:11:01] Stopping collector [18:11:02] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj16_run3.manifest.json [18:11:02] ═══════════════════════════════════════════════════════════ [18:11:02] All 3 runs complete for: rand-read-qd32-nj16 [18:11:02] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [18:11:02] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [18:11:02] ═══════════════════════════════════════════════════════════ [18:11:02] Progress: 31/53 jobs | Elapsed: 30h46m | ETA: ~21h50m [18:11:02] [18:11:02] ╔══════════════════════════════════════════════════════════╗ [18:11:02] ║ Job 32/53: rand-read-qd32-nj32 [18:11:02] ║ Category: rand [18:11:02] ╚══════════════════════════════════════════════════════════╝ [18:11:02] ╔══════════════════════════════════════════════════════════╗ [18:11:02] ║ Benchmark: rand-read-qd32-nj32 [18:11:02] ║ Category: rand [18:11:02] ║ Runs: 3 × cooldown 30s [18:11:02] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [18:11:02] ╚══════════════════════════════════════════════════════════╝ [18:11:02] ──────────────────────────────────────────────────── [18:11:02] Run 1/3: rand-read-qd32-nj32_run1 [18:11:02] ──────────────────────────────────────────────────── [18:11:02] Cleaning fio test files from /tank-mirror/bench … [18:11:02] Test files removed [18:11:02] Dropping Linux page cache + ZFS ARC … [18:11:02] Flushing ZFS ARC (was 1) … [18:11:07] ARC flushed — current size: 51703313232 bytes [18:11:07] NVMe volatile write caches flushed [18:11:09] pmcd healthy (core ✓ pmdazfs ✓) [18:11:09] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj32_run1.ndjson [18:11:10] Collector running (PID 3747683), settling 5s … [18:11:15] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj32.fio [19:32:20] fio completed in 4865s [19:32:20] Post-fio settle 5s … [19:32:25] Stopping collector [19:32:26] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj32_run1.manifest.json [19:32:26] Cooldown: 30s [19:32:56] ──────────────────────────────────────────────────── [19:32:56] Run 2/3: rand-read-qd32-nj32_run2 [19:32:56] ──────────────────────────────────────────────────── [19:32:56] Cleaning fio test files from /tank-mirror/bench … [19:32:56] Test files removed [19:32:56] Dropping Linux page cache + ZFS ARC … [19:32:56] Flushing ZFS ARC (was 1) … [19:33:01] ARC flushed — current size: 87529411392 bytes [19:33:01] NVMe volatile write caches flushed [19:33:03] pmcd healthy (core ✓ pmdazfs ✓) [19:33:03] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj32_run2.ndjson [19:33:04] Collector running (PID 3806779), settling 5s … [19:33:09] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj32.fio [20:54:00] fio completed in 4851s [20:54:00] Post-fio settle 5s … [20:54:05] Stopping collector [20:54:06] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj32_run2.manifest.json [20:54:06] Cooldown: 30s [20:54:36] ──────────────────────────────────────────────────── [20:54:36] Run 3/3: rand-read-qd32-nj32_run3 [20:54:36] ──────────────────────────────────────────────────── [20:54:36] Cleaning fio test files from /tank-mirror/bench … [20:54:37] Test files removed [20:54:37] Dropping Linux page cache + ZFS ARC … [20:54:37] Flushing ZFS ARC (was 1) … [20:54:42] ARC flushed — current size: 85920853928 bytes [20:54:42] NVMe volatile write caches flushed [20:54:43] pmcd healthy (core ✓ pmdazfs ✓) [20:54:43] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-read-qd32-nj32_run3.ndjson [20:54:44] Collector running (PID 3865191), settling 5s … [20:54:49] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-read-qd32-nj32.fio [22:15:53] fio completed in 4864s [22:15:53] Post-fio settle 5s … [22:15:58] Stopping collector [22:15:59] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-read-qd32-nj32_run3.manifest.json [22:15:59] ═══════════════════════════════════════════════════════════ [22:15:59] All 3 runs complete for: rand-read-qd32-nj32 [22:15:59] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [22:15:59] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [22:15:59] ═══════════════════════════════════════════════════════════ [22:15:59] Progress: 32/53 jobs | Elapsed: 34h51m | ETA: ~22h52m [22:15:59] [22:15:59] ╔══════════════════════════════════════════════════════════╗ [22:15:59] ║ Job 33/53: rand-write-qd1-nj1 [22:15:59] ║ Category: rand [22:15:59] ╚══════════════════════════════════════════════════════════╝ [22:15:59] ╔══════════════════════════════════════════════════════════╗ [22:15:59] ║ Benchmark: rand-write-qd1-nj1 [22:15:59] ║ Category: rand [22:15:59] ║ Runs: 3 × cooldown 30s [22:15:59] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [22:15:59] ╚══════════════════════════════════════════════════════════╝ [22:15:59] ──────────────────────────────────────────────────── [22:15:59] Run 1/3: rand-write-qd1-nj1_run1 [22:15:59] ──────────────────────────────────────────────────── [22:15:59] Cleaning fio test files from /tank-mirror/bench … [22:15:59] Test files removed [22:15:59] Dropping Linux page cache + ZFS ARC … [22:16:00] Flushing ZFS ARC (was 1) … [22:16:05] ARC flushed — current size: 87867829720 bytes [22:16:05] NVMe volatile write caches flushed [22:16:06] pmcd healthy (core ✓ pmdazfs ✓) [22:16:06] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj1_run1.ndjson [22:16:07] Collector running (PID 3924207), settling 5s … [22:16:12] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj1.fio [22:19:22] fio completed in 190s [22:19:22] Post-fio settle 5s … [22:19:27] Stopping collector [22:19:28] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj1_run1.manifest.json [22:19:28] Cooldown: 30s [22:19:58] ──────────────────────────────────────────────────── [22:19:58] Run 2/3: rand-write-qd1-nj1_run2 [22:19:58] ──────────────────────────────────────────────────── [22:19:58] Cleaning fio test files from /tank-mirror/bench … [22:19:59] Test files removed [22:19:59] Dropping Linux page cache + ZFS ARC … [22:19:59] Flushing ZFS ARC (was 1) … [22:20:04] ARC flushed — current size: 41787460496 bytes [22:20:04] NVMe volatile write caches flushed [22:20:05] pmcd healthy (core ✓ pmdazfs ✓) [22:20:05] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj1_run2.ndjson [22:20:06] Collector running (PID 3926668), settling 5s … [22:20:11] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj1.fio [22:23:22] fio completed in 191s [22:23:22] Post-fio settle 5s … [22:23:27] Stopping collector [22:23:28] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj1_run2.manifest.json [22:23:28] Cooldown: 30s [22:23:58] ──────────────────────────────────────────────────── [22:23:58] Run 3/3: rand-write-qd1-nj1_run3 [22:23:58] ──────────────────────────────────────────────────── [22:23:58] Cleaning fio test files from /tank-mirror/bench … [22:23:58] Test files removed [22:23:58] Dropping Linux page cache + ZFS ARC … [22:23:58] Flushing ZFS ARC (was 1) … [22:24:03] ARC flushed — current size: 53803070480 bytes [22:24:03] NVMe volatile write caches flushed [22:24:04] pmcd healthy (core ✓ pmdazfs ✓) [22:24:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj1_run3.ndjson [22:24:06] Collector running (PID 3929109), settling 5s … [22:24:11] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj1.fio [22:27:21] fio completed in 190s [22:27:21] Post-fio settle 5s … [22:27:26] Stopping collector [22:27:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj1_run3.manifest.json [22:27:27] ═══════════════════════════════════════════════════════════ [22:27:27] All 3 runs complete for: rand-write-qd1-nj1 [22:27:27] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [22:27:27] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [22:27:27] ═══════════════════════════════════════════════════════════ [22:27:27] Progress: 33/53 jobs | Elapsed: 35h3m | ETA: ~21h14m [22:27:27] [22:27:27] ╔══════════════════════════════════════════════════════════╗ [22:27:27] ║ Job 34/53: rand-write-qd1-nj8 [22:27:27] ║ Category: rand [22:27:27] ╚══════════════════════════════════════════════════════════╝ [22:27:27] ╔══════════════════════════════════════════════════════════╗ [22:27:27] ║ Benchmark: rand-write-qd1-nj8 [22:27:27] ║ Category: rand [22:27:27] ║ Runs: 3 × cooldown 30s [22:27:27] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [22:27:27] ╚══════════════════════════════════════════════════════════╝ [22:27:27] ──────────────────────────────────────────────────── [22:27:27] Run 1/3: rand-write-qd1-nj8_run1 [22:27:27] ──────────────────────────────────────────────────── [22:27:27] Cleaning fio test files from /tank-mirror/bench … [22:27:27] Test files removed [22:27:27] Dropping Linux page cache + ZFS ARC … [22:27:27] Flushing ZFS ARC (was 1) … [22:27:33] ARC flushed — current size: 22995942400 bytes [22:27:33] NVMe volatile write caches flushed [22:27:34] pmcd healthy (core ✓ pmdazfs ✓) [22:27:34] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj8_run1.ndjson [22:27:35] Collector running (PID 3932143), settling 5s … [22:27:40] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj8.fio [22:30:50] fio completed in 190s [22:30:50] Post-fio settle 5s … [22:30:55] Stopping collector [22:30:57] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj8_run1.manifest.json [22:30:57] Cooldown: 30s [22:31:27] ──────────────────────────────────────────────────── [22:31:27] Run 2/3: rand-write-qd1-nj8_run2 [22:31:27] ──────────────────────────────────────────────────── [22:31:27] Cleaning fio test files from /tank-mirror/bench … [22:31:28] Test files removed [22:31:28] Dropping Linux page cache + ZFS ARC … [22:31:28] Flushing ZFS ARC (was 1) … [22:31:33] ARC flushed — current size: 54531795992 bytes [22:31:33] NVMe volatile write caches flushed [22:31:34] pmcd healthy (core ✓ pmdazfs ✓) [22:31:34] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj8_run2.ndjson [22:31:35] Collector running (PID 3934772), settling 5s … [22:31:40] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj8.fio [22:34:51] fio completed in 191s [22:34:51] Post-fio settle 5s … [22:34:56] Stopping collector [22:34:58] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj8_run2.manifest.json [22:34:58] Cooldown: 30s [22:35:28] ──────────────────────────────────────────────────── [22:35:28] Run 3/3: rand-write-qd1-nj8_run3 [22:35:28] ──────────────────────────────────────────────────── [22:35:28] Cleaning fio test files from /tank-mirror/bench … [22:35:28] Test files removed [22:35:28] Dropping Linux page cache + ZFS ARC … [22:35:28] Flushing ZFS ARC (was 1) … [22:35:33] ARC flushed — current size: 57440015288 bytes [22:35:33] NVMe volatile write caches flushed [22:35:35] pmcd healthy (core ✓ pmdazfs ✓) [22:35:35] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj8_run3.ndjson [22:35:36] Collector running (PID 3937407), settling 5s … [22:35:41] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj8.fio [22:38:51] fio completed in 190s [22:38:51] Post-fio settle 5s … [22:38:56] Stopping collector [22:38:58] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj8_run3.manifest.json [22:38:58] ═══════════════════════════════════════════════════════════ [22:38:58] All 3 runs complete for: rand-write-qd1-nj8 [22:38:58] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [22:38:58] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [22:38:58] ═══════════════════════════════════════════════════════════ [22:38:58] Progress: 34/53 jobs | Elapsed: 35h14m | ETA: ~19h41m [22:38:58] [22:38:58] ╔══════════════════════════════════════════════════════════╗ [22:38:58] ║ Job 35/53: rand-write-qd1-nj16 [22:38:58] ║ Category: rand [22:38:58] ╚══════════════════════════════════════════════════════════╝ [22:38:58] ╔══════════════════════════════════════════════════════════╗ [22:38:58] ║ Benchmark: rand-write-qd1-nj16 [22:38:58] ║ Category: rand [22:38:58] ║ Runs: 3 × cooldown 30s [22:38:58] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [22:38:58] ╚══════════════════════════════════════════════════════════╝ [22:38:58] ──────────────────────────────────────────────────── [22:38:58] Run 1/3: rand-write-qd1-nj16_run1 [22:38:58] ──────────────────────────────────────────────────── [22:38:58] Cleaning fio test files from /tank-mirror/bench … [22:38:58] Test files removed [22:38:58] Dropping Linux page cache + ZFS ARC … [22:38:59] Flushing ZFS ARC (was 1) … [22:39:04] ARC flushed — current size: 20384368120 bytes [22:39:04] NVMe volatile write caches flushed [22:39:05] pmcd healthy (core ✓ pmdazfs ✓) [22:39:05] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj16_run1.ndjson [22:39:06] Collector running (PID 3939970), settling 5s … [22:39:11] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj16.fio [22:42:22] fio completed in 191s [22:42:22] Post-fio settle 5s … [22:42:27] Stopping collector [22:42:28] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj16_run1.manifest.json [22:42:28] Cooldown: 30s [22:42:58] ──────────────────────────────────────────────────── [22:42:58] Run 2/3: rand-write-qd1-nj16_run2 [22:42:58] ──────────────────────────────────────────────────── [22:42:58] Cleaning fio test files from /tank-mirror/bench … [22:42:58] Test files removed [22:42:58] Dropping Linux page cache + ZFS ARC … [22:42:58] Flushing ZFS ARC (was 1) … [22:43:03] ARC flushed — current size: 15585687640 bytes [22:43:03] NVMe volatile write caches flushed [22:43:04] pmcd healthy (core ✓ pmdazfs ✓) [22:43:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj16_run2.ndjson [22:43:05] Collector running (PID 3942538), settling 5s … [22:43:10] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj16.fio [22:46:21] fio completed in 191s [22:46:21] Post-fio settle 5s … [22:46:26] Stopping collector [22:46:27] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj16_run2.manifest.json [22:46:27] Cooldown: 30s [22:46:57] ──────────────────────────────────────────────────── [22:46:57] Run 3/3: rand-write-qd1-nj16_run3 [22:46:57] ──────────────────────────────────────────────────── [22:46:57] Cleaning fio test files from /tank-mirror/bench … [22:46:57] Test files removed [22:46:57] Dropping Linux page cache + ZFS ARC … [22:46:57] Flushing ZFS ARC (was 1) … [22:47:02] ARC flushed — current size: 16521933584 bytes [22:47:02] NVMe volatile write caches flushed [22:47:04] pmcd healthy (core ✓ pmdazfs ✓) [22:47:04] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj16_run3.ndjson [22:47:05] Collector running (PID 3945168), settling 5s … [22:47:10] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj16.fio [22:50:20] fio completed in 190s [22:50:20] Post-fio settle 5s … [22:50:25] Stopping collector [22:50:26] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj16_run3.manifest.json [22:50:26] ═══════════════════════════════════════════════════════════ [22:50:26] All 3 runs complete for: rand-write-qd1-nj16 [22:50:26] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [22:50:26] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [22:50:26] ═══════════════════════════════════════════════════════════ [22:50:26] Progress: 35/53 jobs | Elapsed: 35h26m | ETA: ~18h13m [22:50:26] [22:50:26] ╔══════════════════════════════════════════════════════════╗ [22:50:26] ║ Job 36/53: rand-write-qd1-nj32 [22:50:26] ║ Category: rand [22:50:26] ╚══════════════════════════════════════════════════════════╝ [22:50:26] ╔══════════════════════════════════════════════════════════╗ [22:50:26] ║ Benchmark: rand-write-qd1-nj32 [22:50:26] ║ Category: rand [22:50:26] ║ Runs: 3 × cooldown 30s [22:50:26] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [22:50:26] ╚══════════════════════════════════════════════════════════╝ [22:50:26] ──────────────────────────────────────────────────── [22:50:26] Run 1/3: rand-write-qd1-nj32_run1 [22:50:26] ──────────────────────────────────────────────────── [22:50:26] Cleaning fio test files from /tank-mirror/bench … [22:50:27] Test files removed [22:50:27] Dropping Linux page cache + ZFS ARC … [22:50:27] Flushing ZFS ARC (was 1) … [22:50:32] ARC flushed — current size: 14065350424 bytes [22:50:32] NVMe volatile write caches flushed [22:50:33] pmcd healthy (core ✓ pmdazfs ✓) [22:50:33] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj32_run1.ndjson [22:50:34] Collector running (PID 3947692), settling 5s … [22:50:39] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj32.fio [22:53:50] fio completed in 191s [22:53:50] Post-fio settle 5s … [22:53:55] Stopping collector [22:53:56] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj32_run1.manifest.json [22:53:56] Cooldown: 30s [22:54:26] ──────────────────────────────────────────────────── [22:54:26] Run 2/3: rand-write-qd1-nj32_run2 [22:54:26] ──────────────────────────────────────────────────── [22:54:26] Cleaning fio test files from /tank-mirror/bench … [22:54:26] Test files removed [22:54:26] Dropping Linux page cache + ZFS ARC … [22:54:26] Flushing ZFS ARC (was 1) … [22:54:31] ARC flushed — current size: 36853456 bytes [22:54:31] NVMe volatile write caches flushed [22:54:32] pmcd healthy (core ✓ pmdazfs ✓) [22:54:32] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj32_run2.ndjson [22:54:33] Collector running (PID 3950201), settling 5s … [22:54:38] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj32.fio [22:57:49] fio completed in 191s [22:57:49] Post-fio settle 5s … [22:57:54] Stopping collector [22:57:56] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj32_run2.manifest.json [22:57:56] Cooldown: 30s [22:58:26] ──────────────────────────────────────────────────── [22:58:26] Run 3/3: rand-write-qd1-nj32_run3 [22:58:26] ──────────────────────────────────────────────────── [22:58:26] Cleaning fio test files from /tank-mirror/bench … [22:58:26] Test files removed [22:58:26] Dropping Linux page cache + ZFS ARC … [22:58:26] Flushing ZFS ARC (was 1) … [22:58:31] ARC flushed — current size: 41069072 bytes [22:58:31] NVMe volatile write caches flushed [22:58:33] pmcd healthy (core ✓ pmdazfs ✓) [22:58:33] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd1-nj32_run3.ndjson [22:58:34] Collector running (PID 3953357), settling 5s … [22:58:39] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd1-nj32.fio [23:01:49] fio completed in 190s [23:01:49] Post-fio settle 5s … [23:01:54] Stopping collector [23:01:56] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd1-nj32_run3.manifest.json [23:01:56] ═══════════════════════════════════════════════════════════ [23:01:56] All 3 runs complete for: rand-write-qd1-nj32 [23:01:56] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:01:56] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:01:56] ═══════════════════════════════════════════════════════════ [23:01:56] Progress: 36/53 jobs | Elapsed: 35h37m | ETA: ~16h49m [23:01:56] [23:01:56] ╔══════════════════════════════════════════════════════════╗ [23:01:56] ║ Job 37/53: rand-write-qd4-nj1 [23:01:56] ║ Category: rand [23:01:56] ╚══════════════════════════════════════════════════════════╝ [23:01:56] ╔══════════════════════════════════════════════════════════╗ [23:01:56] ║ Benchmark: rand-write-qd4-nj1 [23:01:56] ║ Category: rand [23:01:56] ║ Runs: 3 × cooldown 30s [23:01:56] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:01:56] ╚══════════════════════════════════════════════════════════╝ [23:01:56] ──────────────────────────────────────────────────── [23:01:56] Run 1/3: rand-write-qd4-nj1_run1 [23:01:56] ──────────────────────────────────────────────────── [23:01:56] Cleaning fio test files from /tank-mirror/bench … [23:01:56] Test files removed [23:01:56] Dropping Linux page cache + ZFS ARC … [23:01:57] Flushing ZFS ARC (was 1) … [23:02:02] ARC flushed — current size: 38794776 bytes [23:02:02] NVMe volatile write caches flushed [23:02:03] pmcd healthy (core ✓ pmdazfs ✓) [23:02:03] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj1_run1.ndjson [23:02:04] Collector running (PID 3955773), settling 5s … [23:02:09] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj1.fio [23:05:19] fio completed in 190s [23:05:19] Post-fio settle 5s … [23:05:24] Stopping collector [23:05:25] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj1_run1.manifest.json [23:05:25] Cooldown: 30s [23:05:55] ──────────────────────────────────────────────────── [23:05:55] Run 2/3: rand-write-qd4-nj1_run2 [23:05:55] ──────────────────────────────────────────────────── [23:05:55] Cleaning fio test files from /tank-mirror/bench … [23:05:56] Test files removed [23:05:56] Dropping Linux page cache + ZFS ARC … [23:05:56] Flushing ZFS ARC (was 1) … [23:06:01] ARC flushed — current size: 42184406040 bytes [23:06:01] NVMe volatile write caches flushed [23:06:02] pmcd healthy (core ✓ pmdazfs ✓) [23:06:02] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj1_run2.ndjson [23:06:03] Collector running (PID 3958222), settling 5s … [23:06:08] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj1.fio [23:09:19] fio completed in 191s [23:09:19] Post-fio settle 5s … [23:09:24] Stopping collector [23:09:25] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj1_run2.manifest.json [23:09:25] Cooldown: 30s [23:09:55] ──────────────────────────────────────────────────── [23:09:55] Run 3/3: rand-write-qd4-nj1_run3 [23:09:55] ──────────────────────────────────────────────────── [23:09:55] Cleaning fio test files from /tank-mirror/bench … [23:09:55] Test files removed [23:09:55] Dropping Linux page cache + ZFS ARC … [23:09:55] Flushing ZFS ARC (was 1) … [23:10:00] ARC flushed — current size: 54011387512 bytes [23:10:00] NVMe volatile write caches flushed [23:10:02] pmcd healthy (core ✓ pmdazfs ✓) [23:10:02] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj1_run3.ndjson [23:10:03] Collector running (PID 3960658), settling 5s … [23:10:08] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj1.fio [23:13:18] fio completed in 190s [23:13:18] Post-fio settle 5s … [23:13:23] Stopping collector [23:13:24] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj1_run3.manifest.json [23:13:24] ═══════════════════════════════════════════════════════════ [23:13:24] All 3 runs complete for: rand-write-qd4-nj1 [23:13:24] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:13:24] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:13:24] ═══════════════════════════════════════════════════════════ [23:13:24] Progress: 37/53 jobs | Elapsed: 35h49m | ETA: ~15h29m [23:13:24] [23:13:24] ╔══════════════════════════════════════════════════════════╗ [23:13:24] ║ Job 38/53: rand-write-qd4-nj8 [23:13:24] ║ Category: rand [23:13:24] ╚══════════════════════════════════════════════════════════╝ [23:13:24] ╔══════════════════════════════════════════════════════════╗ [23:13:24] ║ Benchmark: rand-write-qd4-nj8 [23:13:24] ║ Category: rand [23:13:24] ║ Runs: 3 × cooldown 30s [23:13:24] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:13:24] ╚══════════════════════════════════════════════════════════╝ [23:13:24] ──────────────────────────────────────────────────── [23:13:24] Run 1/3: rand-write-qd4-nj8_run1 [23:13:24] ──────────────────────────────────────────────────── [23:13:24] Cleaning fio test files from /tank-mirror/bench … [23:13:24] Test files removed [23:13:24] Dropping Linux page cache + ZFS ARC … [23:13:25] Flushing ZFS ARC (was 1) … [23:13:30] ARC flushed — current size: 21628363000 bytes [23:13:30] NVMe volatile write caches flushed [23:13:31] pmcd healthy (core ✓ pmdazfs ✓) [23:13:31] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj8_run1.ndjson [23:13:32] Collector running (PID 3963059), settling 5s … [23:13:37] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj8.fio [23:16:47] fio completed in 190s [23:16:47] Post-fio settle 5s … [23:16:52] Stopping collector [23:16:53] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj8_run1.manifest.json [23:16:53] Cooldown: 30s [23:17:23] ──────────────────────────────────────────────────── [23:17:23] Run 2/3: rand-write-qd4-nj8_run2 [23:17:23] ──────────────────────────────────────────────────── [23:17:23] Cleaning fio test files from /tank-mirror/bench … [23:17:24] Test files removed [23:17:24] Dropping Linux page cache + ZFS ARC … [23:17:24] Flushing ZFS ARC (was 1) … [23:17:29] ARC flushed — current size: 98007618896 bytes [23:17:29] NVMe volatile write caches flushed [23:17:30] pmcd healthy (core ✓ pmdazfs ✓) [23:17:30] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj8_run2.ndjson [23:17:31] Collector running (PID 3965695), settling 5s … [23:17:36] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj8.fio [23:20:47] fio completed in 191s [23:20:47] Post-fio settle 5s … [23:20:52] Stopping collector [23:20:53] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj8_run2.manifest.json [23:20:53] Cooldown: 30s [23:21:23] ──────────────────────────────────────────────────── [23:21:23] Run 3/3: rand-write-qd4-nj8_run3 [23:21:23] ──────────────────────────────────────────────────── [23:21:23] Cleaning fio test files from /tank-mirror/bench … [23:21:23] Test files removed [23:21:23] Dropping Linux page cache + ZFS ARC … [23:21:23] Flushing ZFS ARC (was 1) … [23:21:28] ARC flushed — current size: 59460245488 bytes [23:21:28] NVMe volatile write caches flushed [23:21:30] pmcd healthy (core ✓ pmdazfs ✓) [23:21:30] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj8_run3.ndjson [23:21:31] Collector running (PID 3968294), settling 5s … [23:21:36] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj8.fio [23:24:46] fio completed in 190s [23:24:46] Post-fio settle 5s … [23:24:51] Stopping collector [23:24:52] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj8_run3.manifest.json [23:24:52] ═══════════════════════════════════════════════════════════ [23:24:52] All 3 runs complete for: rand-write-qd4-nj8 [23:24:52] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:24:52] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:24:52] ═══════════════════════════════════════════════════════════ [23:24:52] Progress: 38/53 jobs | Elapsed: 36h0m | ETA: ~14h12m [23:24:52] [23:24:52] ╔══════════════════════════════════════════════════════════╗ [23:24:52] ║ Job 39/53: rand-write-qd4-nj16 [23:24:52] ║ Category: rand [23:24:52] ╚══════════════════════════════════════════════════════════╝ [23:24:52] ╔══════════════════════════════════════════════════════════╗ [23:24:52] ║ Benchmark: rand-write-qd4-nj16 [23:24:52] ║ Category: rand [23:24:52] ║ Runs: 3 × cooldown 30s [23:24:52] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:24:52] ╚══════════════════════════════════════════════════════════╝ [23:24:52] ──────────────────────────────────────────────────── [23:24:52] Run 1/3: rand-write-qd4-nj16_run1 [23:24:52] ──────────────────────────────────────────────────── [23:24:52] Cleaning fio test files from /tank-mirror/bench … [23:24:53] Test files removed [23:24:53] Dropping Linux page cache + ZFS ARC … [23:24:53] Flushing ZFS ARC (was 1) … [23:24:58] ARC flushed — current size: 79444522624 bytes [23:24:58] NVMe volatile write caches flushed [23:24:59] pmcd healthy (core ✓ pmdazfs ✓) [23:24:59] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj16_run1.ndjson [23:25:00] Collector running (PID 3970865), settling 5s … [23:25:05] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj16.fio [23:28:16] fio completed in 191s [23:28:16] Post-fio settle 5s … [23:28:21] Stopping collector [23:28:22] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj16_run1.manifest.json [23:28:22] Cooldown: 30s [23:28:52] ──────────────────────────────────────────────────── [23:28:52] Run 2/3: rand-write-qd4-nj16_run2 [23:28:52] ──────────────────────────────────────────────────── [23:28:52] Cleaning fio test files from /tank-mirror/bench … [23:28:52] Test files removed [23:28:52] Dropping Linux page cache + ZFS ARC … [23:28:52] Flushing ZFS ARC (was 1) … [23:28:57] ARC flushed — current size: 14392516640 bytes [23:28:57] NVMe volatile write caches flushed [23:28:59] pmcd healthy (core ✓ pmdazfs ✓) [23:28:59] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj16_run2.ndjson [23:29:00] Collector running (PID 3974107), settling 5s … [23:29:05] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj16.fio [23:32:15] fio completed in 190s [23:32:15] Post-fio settle 5s … [23:32:20] Stopping collector [23:32:21] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj16_run2.manifest.json [23:32:21] Cooldown: 30s [23:32:51] ──────────────────────────────────────────────────── [23:32:51] Run 3/3: rand-write-qd4-nj16_run3 [23:32:51] ──────────────────────────────────────────────────── [23:32:51] Cleaning fio test files from /tank-mirror/bench … [23:32:51] Test files removed [23:32:51] Dropping Linux page cache + ZFS ARC … [23:32:52] Flushing ZFS ARC (was 1) … [23:32:57] ARC flushed — current size: 13745862504 bytes [23:32:57] NVMe volatile write caches flushed [23:32:58] pmcd healthy (core ✓ pmdazfs ✓) [23:32:58] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj16_run3.ndjson [23:32:59] Collector running (PID 3976695), settling 5s … [23:33:04] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj16.fio [23:36:14] fio completed in 190s [23:36:14] Post-fio settle 5s … [23:36:19] Stopping collector [23:36:20] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj16_run3.manifest.json [23:36:20] ═══════════════════════════════════════════════════════════ [23:36:20] All 3 runs complete for: rand-write-qd4-nj16 [23:36:20] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:36:20] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:36:20] ═══════════════════════════════════════════════════════════ [23:36:20] Progress: 39/53 jobs | Elapsed: 36h12m | ETA: ~12h59m [23:36:20] [23:36:20] ╔══════════════════════════════════════════════════════════╗ [23:36:20] ║ Job 40/53: rand-write-qd4-nj32 [23:36:20] ║ Category: rand [23:36:20] ╚══════════════════════════════════════════════════════════╝ [23:36:20] ╔══════════════════════════════════════════════════════════╗ [23:36:20] ║ Benchmark: rand-write-qd4-nj32 [23:36:20] ║ Category: rand [23:36:20] ║ Runs: 3 × cooldown 30s [23:36:20] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:36:20] ╚══════════════════════════════════════════════════════════╝ [23:36:20] ──────────────────────────────────────────────────── [23:36:20] Run 1/3: rand-write-qd4-nj32_run1 [23:36:20] ──────────────────────────────────────────────────── [23:36:20] Cleaning fio test files from /tank-mirror/bench … [23:36:21] Test files removed [23:36:21] Dropping Linux page cache + ZFS ARC … [23:36:21] Flushing ZFS ARC (was 1) … [23:36:26] ARC flushed — current size: 16130670000 bytes [23:36:26] NVMe volatile write caches flushed [23:36:27] pmcd healthy (core ✓ pmdazfs ✓) [23:36:27] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj32_run1.ndjson [23:36:28] Collector running (PID 3979196), settling 5s … [23:36:33] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj32.fio [23:39:44] fio completed in 191s [23:39:44] Post-fio settle 5s … [23:39:49] Stopping collector [23:39:50] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj32_run1.manifest.json [23:39:50] Cooldown: 30s [23:40:20] ──────────────────────────────────────────────────── [23:40:20] Run 2/3: rand-write-qd4-nj32_run2 [23:40:20] ──────────────────────────────────────────────────── [23:40:20] Cleaning fio test files from /tank-mirror/bench … [23:40:20] Test files removed [23:40:20] Dropping Linux page cache + ZFS ARC … [23:40:20] Flushing ZFS ARC (was 1) … [23:40:25] ARC flushed — current size: 384555264 bytes [23:40:25] NVMe volatile write caches flushed [23:40:27] pmcd healthy (core ✓ pmdazfs ✓) [23:40:27] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj32_run2.ndjson [23:40:28] Collector running (PID 3981629), settling 5s … [23:40:33] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj32.fio [23:43:43] fio completed in 190s [23:43:43] Post-fio settle 5s … [23:43:48] Stopping collector [23:43:49] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj32_run2.manifest.json [23:43:49] Cooldown: 30s [23:44:19] ──────────────────────────────────────────────────── [23:44:19] Run 3/3: rand-write-qd4-nj32_run3 [23:44:19] ──────────────────────────────────────────────────── [23:44:19] Cleaning fio test files from /tank-mirror/bench … [23:44:19] Test files removed [23:44:19] Dropping Linux page cache + ZFS ARC … [23:44:20] Flushing ZFS ARC (was 1) … [23:44:25] ARC flushed — current size: 40989752 bytes [23:44:25] NVMe volatile write caches flushed [23:44:26] pmcd healthy (core ✓ pmdazfs ✓) [23:44:26] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd4-nj32_run3.ndjson [23:44:27] Collector running (PID 3984142), settling 5s … [23:44:32] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd4-nj32.fio [23:47:42] fio completed in 190s [23:47:42] Post-fio settle 5s … [23:47:47] Stopping collector [23:47:49] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd4-nj32_run3.manifest.json [23:47:49] ═══════════════════════════════════════════════════════════ [23:47:49] All 3 runs complete for: rand-write-qd4-nj32 [23:47:49] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:47:49] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:47:49] ═══════════════════════════════════════════════════════════ [23:47:49] Progress: 40/53 jobs | Elapsed: 36h23m | ETA: ~11h49m [23:47:49] [23:47:49] ╔══════════════════════════════════════════════════════════╗ [23:47:49] ║ Job 41/53: rand-write-qd16-nj1 [23:47:49] ║ Category: rand [23:47:49] ╚══════════════════════════════════════════════════════════╝ [23:47:49] ╔══════════════════════════════════════════════════════════╗ [23:47:49] ║ Benchmark: rand-write-qd16-nj1 [23:47:49] ║ Category: rand [23:47:49] ║ Runs: 3 × cooldown 30s [23:47:49] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:47:49] ╚══════════════════════════════════════════════════════════╝ [23:47:49] ──────────────────────────────────────────────────── [23:47:49] Run 1/3: rand-write-qd16-nj1_run1 [23:47:49] ──────────────────────────────────────────────────── [23:47:49] Cleaning fio test files from /tank-mirror/bench … [23:47:50] Test files removed [23:47:50] Dropping Linux page cache + ZFS ARC … [23:47:50] Flushing ZFS ARC (was 1) … [23:47:55] ARC flushed — current size: 38703520 bytes [23:47:55] NVMe volatile write caches flushed [23:47:56] pmcd healthy (core ✓ pmdazfs ✓) [23:47:56] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj1_run1.ndjson [23:47:57] Collector running (PID 3986680), settling 5s … [23:48:02] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj1.fio [23:51:13] fio completed in 191s [23:51:13] Post-fio settle 5s … [23:51:18] Stopping collector [23:51:19] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj1_run1.manifest.json [23:51:19] Cooldown: 30s [23:51:49] ──────────────────────────────────────────────────── [23:51:49] Run 2/3: rand-write-qd16-nj1_run2 [23:51:49] ──────────────────────────────────────────────────── [23:51:49] Cleaning fio test files from /tank-mirror/bench … [23:51:49] Test files removed [23:51:49] Dropping Linux page cache + ZFS ARC … [23:51:49] Flushing ZFS ARC (was 1) … [23:51:54] ARC flushed — current size: 37367253480 bytes [23:51:54] NVMe volatile write caches flushed [23:51:56] pmcd healthy (core ✓ pmdazfs ✓) [23:51:56] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj1_run2.ndjson [23:51:57] Collector running (PID 3989076), settling 5s … [23:52:02] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj1.fio [23:55:12] fio completed in 190s [23:55:12] Post-fio settle 5s … [23:55:17] Stopping collector [23:55:18] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj1_run2.manifest.json [23:55:18] Cooldown: 30s [23:55:48] ──────────────────────────────────────────────────── [23:55:48] Run 3/3: rand-write-qd16-nj1_run3 [23:55:48] ──────────────────────────────────────────────────── [23:55:48] Cleaning fio test files from /tank-mirror/bench … [23:55:48] Test files removed [23:55:48] Dropping Linux page cache + ZFS ARC … [23:55:48] Flushing ZFS ARC (was 1) … [23:55:53] ARC flushed — current size: 49531261272 bytes [23:55:53] NVMe volatile write caches flushed [23:55:55] pmcd healthy (core ✓ pmdazfs ✓) [23:55:55] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj1_run3.ndjson [23:55:56] Collector running (PID 3992214), settling 5s … [23:56:01] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj1.fio [23:59:11] fio completed in 190s [23:59:11] Post-fio settle 5s … [23:59:16] Stopping collector [23:59:17] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj1_run3.manifest.json [23:59:17] ═══════════════════════════════════════════════════════════ [23:59:17] All 3 runs complete for: rand-write-qd16-nj1 [23:59:17] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [23:59:17] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [23:59:17] ═══════════════════════════════════════════════════════════ [23:59:17] Progress: 41/53 jobs | Elapsed: 36h35m | ETA: ~10h42m [23:59:17] [23:59:17] ╔══════════════════════════════════════════════════════════╗ [23:59:17] ║ Job 42/53: rand-write-qd16-nj8 [23:59:17] ║ Category: rand [23:59:17] ╚══════════════════════════════════════════════════════════╝ [23:59:17] ╔══════════════════════════════════════════════════════════╗ [23:59:17] ║ Benchmark: rand-write-qd16-nj8 [23:59:17] ║ Category: rand [23:59:17] ║ Runs: 3 × cooldown 30s [23:59:17] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [23:59:17] ╚══════════════════════════════════════════════════════════╝ [23:59:17] ──────────────────────────────────────────────────── [23:59:17] Run 1/3: rand-write-qd16-nj8_run1 [23:59:17] ──────────────────────────────────────────────────── [23:59:17] Cleaning fio test files from /tank-mirror/bench … [23:59:18] Test files removed [23:59:18] Dropping Linux page cache + ZFS ARC … [23:59:18] Flushing ZFS ARC (was 1) … [23:59:23] ARC flushed — current size: 18960373816 bytes [23:59:23] NVMe volatile write caches flushed [23:59:24] pmcd healthy (core ✓ pmdazfs ✓) [23:59:24] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj8_run1.ndjson [23:59:25] Collector running (PID 3994590), settling 5s … [23:59:30] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj8.fio [00:02:41] fio completed in 191s [00:02:41] Post-fio settle 5s … [00:02:46] Stopping collector [00:02:48] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj8_run1.manifest.json [00:02:48] Cooldown: 30s [00:03:18] ──────────────────────────────────────────────────── [00:03:18] Run 2/3: rand-write-qd16-nj8_run2 [00:03:18] ──────────────────────────────────────────────────── [00:03:18] Cleaning fio test files from /tank-mirror/bench … [00:03:18] Test files removed [00:03:18] Dropping Linux page cache + ZFS ARC … [00:03:18] Flushing ZFS ARC (was 1) … [00:03:23] ARC flushed — current size: 70870794800 bytes [00:03:23] NVMe volatile write caches flushed [00:03:24] pmcd healthy (core ✓ pmdazfs ✓) [00:03:24] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj8_run2.ndjson [00:03:26] Collector running (PID 3997262), settling 5s … [00:03:31] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj8.fio [00:06:41] fio completed in 190s [00:06:41] Post-fio settle 5s … [00:06:46] Stopping collector [00:06:47] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj8_run2.manifest.json [00:06:47] Cooldown: 30s [00:07:17] ──────────────────────────────────────────────────── [00:07:17] Run 3/3: rand-write-qd16-nj8_run3 [00:07:17] ──────────────────────────────────────────────────── [00:07:17] Cleaning fio test files from /tank-mirror/bench … [00:07:17] Test files removed [00:07:17] Dropping Linux page cache + ZFS ARC … [00:07:18] Flushing ZFS ARC (was 1) … [00:07:23] ARC flushed — current size: 102560277032 bytes [00:07:23] NVMe volatile write caches flushed [00:07:24] pmcd healthy (core ✓ pmdazfs ✓) [00:07:24] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj8_run3.ndjson [00:07:25] Collector running (PID 3999879), settling 5s … [00:07:30] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj8.fio [00:10:40] fio completed in 190s [00:10:40] Post-fio settle 5s … [00:10:45] Stopping collector [00:10:46] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj8_run3.manifest.json [00:10:46] ═══════════════════════════════════════════════════════════ [00:10:46] All 3 runs complete for: rand-write-qd16-nj8 [00:10:46] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:10:46] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:10:46] ═══════════════════════════════════════════════════════════ [00:10:46] Progress: 42/53 jobs | Elapsed: 36h46m | ETA: ~9h37m [00:10:46] [00:10:46] ╔══════════════════════════════════════════════════════════╗ [00:10:46] ║ Job 43/53: rand-write-qd16-nj16 [00:10:46] ║ Category: rand [00:10:46] ╚══════════════════════════════════════════════════════════╝ [00:10:46] ╔══════════════════════════════════════════════════════════╗ [00:10:46] ║ Benchmark: rand-write-qd16-nj16 [00:10:46] ║ Category: rand [00:10:46] ║ Runs: 3 × cooldown 30s [00:10:46] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:10:46] ╚══════════════════════════════════════════════════════════╝ [00:10:46] ──────────────────────────────────────────────────── [00:10:46] Run 1/3: rand-write-qd16-nj16_run1 [00:10:46] ──────────────────────────────────────────────────── [00:10:46] Cleaning fio test files from /tank-mirror/bench … [00:10:47] Test files removed [00:10:47] Dropping Linux page cache + ZFS ARC … [00:10:47] Flushing ZFS ARC (was 1) … [00:10:52] ARC flushed — current size: 85704446976 bytes [00:10:52] NVMe volatile write caches flushed [00:10:53] pmcd healthy (core ✓ pmdazfs ✓) [00:10:53] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj16_run1.ndjson [00:10:54] Collector running (PID 4002713), settling 5s … [00:10:59] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj16.fio [00:14:10] fio completed in 191s [00:14:10] Post-fio settle 5s … [00:14:15] Stopping collector [00:14:16] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj16_run1.manifest.json [00:14:16] Cooldown: 30s [00:14:46] ──────────────────────────────────────────────────── [00:14:46] Run 2/3: rand-write-qd16-nj16_run2 [00:14:46] ──────────────────────────────────────────────────── [00:14:46] Cleaning fio test files from /tank-mirror/bench … [00:14:46] Test files removed [00:14:46] Dropping Linux page cache + ZFS ARC … [00:14:46] Flushing ZFS ARC (was 1) … [00:14:51] ARC flushed — current size: 15589051776 bytes [00:14:51] NVMe volatile write caches flushed [00:14:53] pmcd healthy (core ✓ pmdazfs ✓) [00:14:53] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj16_run2.ndjson [00:14:54] Collector running (PID 4005281), settling 5s … [00:14:59] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj16.fio [00:18:09] fio completed in 190s [00:18:09] Post-fio settle 5s … [00:18:14] Stopping collector [00:18:15] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj16_run2.manifest.json [00:18:15] Cooldown: 30s [00:18:45] ──────────────────────────────────────────────────── [00:18:45] Run 3/3: rand-write-qd16-nj16_run3 [00:18:45] ──────────────────────────────────────────────────── [00:18:45] Cleaning fio test files from /tank-mirror/bench … [00:18:46] Test files removed [00:18:46] Dropping Linux page cache + ZFS ARC … [00:18:46] Flushing ZFS ARC (was 1) … [00:18:51] ARC flushed — current size: 15107843192 bytes [00:18:51] NVMe volatile write caches flushed [00:18:52] pmcd healthy (core ✓ pmdazfs ✓) [00:18:52] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj16_run3.ndjson [00:18:53] Collector running (PID 4007879), settling 5s … [00:18:58] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj16.fio [00:22:09] fio completed in 191s [00:22:09] Post-fio settle 5s … [00:22:14] Stopping collector [00:22:16] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj16_run3.manifest.json [00:22:16] ═══════════════════════════════════════════════════════════ [00:22:16] All 3 runs complete for: rand-write-qd16-nj16 [00:22:16] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:22:16] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:22:16] ═══════════════════════════════════════════════════════════ [00:22:16] Progress: 43/53 jobs | Elapsed: 36h58m | ETA: ~8h35m [00:22:16] [00:22:16] ╔══════════════════════════════════════════════════════════╗ [00:22:16] ║ Job 44/53: rand-write-qd16-nj32 [00:22:16] ║ Category: rand [00:22:16] ╚══════════════════════════════════════════════════════════╝ [00:22:16] ╔══════════════════════════════════════════════════════════╗ [00:22:16] ║ Benchmark: rand-write-qd16-nj32 [00:22:16] ║ Category: rand [00:22:16] ║ Runs: 3 × cooldown 30s [00:22:16] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:22:16] ╚══════════════════════════════════════════════════════════╝ [00:22:16] ──────────────────────────────────────────────────── [00:22:16] Run 1/3: rand-write-qd16-nj32_run1 [00:22:16] ──────────────────────────────────────────────────── [00:22:16] Cleaning fio test files from /tank-mirror/bench … [00:22:16] Test files removed [00:22:16] Dropping Linux page cache + ZFS ARC … [00:22:16] Flushing ZFS ARC (was 1) … [00:22:21] ARC flushed — current size: 13962661600 bytes [00:22:21] NVMe volatile write caches flushed [00:22:22] pmcd healthy (core ✓ pmdazfs ✓) [00:22:22] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj32_run1.ndjson [00:22:23] Collector running (PID 4010351), settling 5s … [00:22:28] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj32.fio [00:25:39] fio completed in 191s [00:25:39] Post-fio settle 5s … [00:25:44] Stopping collector [00:25:45] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj32_run1.manifest.json [00:25:45] Cooldown: 30s [00:26:15] ──────────────────────────────────────────────────── [00:26:15] Run 2/3: rand-write-qd16-nj32_run2 [00:26:15] ──────────────────────────────────────────────────── [00:26:15] Cleaning fio test files from /tank-mirror/bench … [00:26:15] Test files removed [00:26:15] Dropping Linux page cache + ZFS ARC … [00:26:15] Flushing ZFS ARC (was 1) … [00:26:20] ARC flushed — current size: 3115504504 bytes [00:26:20] NVMe volatile write caches flushed [00:26:22] pmcd healthy (core ✓ pmdazfs ✓) [00:26:22] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj32_run2.ndjson [00:26:23] Collector running (PID 4013595), settling 5s … [00:26:28] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj32.fio [00:29:38] fio completed in 190s [00:29:38] Post-fio settle 5s … [00:29:43] Stopping collector [00:29:44] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj32_run2.manifest.json [00:29:44] Cooldown: 30s [00:30:14] ──────────────────────────────────────────────────── [00:30:14] Run 3/3: rand-write-qd16-nj32_run3 [00:30:14] ──────────────────────────────────────────────────── [00:30:14] Cleaning fio test files from /tank-mirror/bench … [00:30:15] Test files removed [00:30:15] Dropping Linux page cache + ZFS ARC … [00:30:15] Flushing ZFS ARC (was 1) … [00:30:20] ARC flushed — current size: 40283800 bytes [00:30:20] NVMe volatile write caches flushed [00:30:21] pmcd healthy (core ✓ pmdazfs ✓) [00:30:21] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd16-nj32_run3.ndjson [00:30:22] Collector running (PID 4016142), settling 5s … [00:30:27] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd16-nj32.fio [00:33:38] fio completed in 191s [00:33:38] Post-fio settle 5s … [00:33:43] Stopping collector [00:33:44] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd16-nj32_run3.manifest.json [00:33:44] ═══════════════════════════════════════════════════════════ [00:33:44] All 3 runs complete for: rand-write-qd16-nj32 [00:33:44] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:33:44] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:33:44] ═══════════════════════════════════════════════════════════ [00:33:44] Progress: 44/53 jobs | Elapsed: 37h9m | ETA: ~7h36m [00:33:44] [00:33:44] ╔══════════════════════════════════════════════════════════╗ [00:33:44] ║ Job 45/53: rand-write-qd32-nj1 [00:33:44] ║ Category: rand [00:33:44] ╚══════════════════════════════════════════════════════════╝ [00:33:44] ╔══════════════════════════════════════════════════════════╗ [00:33:44] ║ Benchmark: rand-write-qd32-nj1 [00:33:44] ║ Category: rand [00:33:44] ║ Runs: 3 × cooldown 30s [00:33:44] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:33:44] ╚══════════════════════════════════════════════════════════╝ [00:33:44] ──────────────────────────────────────────────────── [00:33:44] Run 1/3: rand-write-qd32-nj1_run1 [00:33:44] ──────────────────────────────────────────────────── [00:33:44] Cleaning fio test files from /tank-mirror/bench … [00:33:44] Test files removed [00:33:44] Dropping Linux page cache + ZFS ARC … [00:33:44] Flushing ZFS ARC (was 1) … [00:33:49] ARC flushed — current size: 36724680 bytes [00:33:49] NVMe volatile write caches flushed [00:33:50] pmcd healthy (core ✓ pmdazfs ✓) [00:33:50] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj1_run1.ndjson [00:33:51] Collector running (PID 4018611), settling 5s … [00:33:56] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj1.fio [00:37:07] fio completed in 191s [00:37:07] Post-fio settle 5s … [00:37:12] Stopping collector [00:37:13] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj1_run1.manifest.json [00:37:13] Cooldown: 30s [00:37:43] ──────────────────────────────────────────────────── [00:37:43] Run 2/3: rand-write-qd32-nj1_run2 [00:37:43] ──────────────────────────────────────────────────── [00:37:43] Cleaning fio test files from /tank-mirror/bench … [00:37:43] Test files removed [00:37:43] Dropping Linux page cache + ZFS ARC … [00:37:43] Flushing ZFS ARC (was 1) … [00:37:49] ARC flushed — current size: 46075749440 bytes [00:37:49] NVMe volatile write caches flushed [00:37:50] pmcd healthy (core ✓ pmdazfs ✓) [00:37:50] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj1_run2.ndjson [00:37:51] Collector running (PID 4021022), settling 5s … [00:37:56] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj1.fio [00:41:06] fio completed in 190s [00:41:06] Post-fio settle 5s … [00:41:11] Stopping collector [00:41:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj1_run2.manifest.json [00:41:12] Cooldown: 30s [00:41:42] ──────────────────────────────────────────────────── [00:41:42] Run 3/3: rand-write-qd32-nj1_run3 [00:41:42] ──────────────────────────────────────────────────── [00:41:42] Cleaning fio test files from /tank-mirror/bench … [00:41:42] Test files removed [00:41:42] Dropping Linux page cache + ZFS ARC … [00:41:43] Flushing ZFS ARC (was 1) … [00:41:48] ARC flushed — current size: 60559640 bytes [00:41:48] NVMe volatile write caches flushed [00:41:49] pmcd healthy (core ✓ pmdazfs ✓) [00:41:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj1_run3.ndjson [00:41:50] Collector running (PID 4023460), settling 5s … [00:41:55] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj1.fio [00:45:06] fio completed in 191s [00:45:06] Post-fio settle 5s … [00:45:11] Stopping collector [00:45:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj1_run3.manifest.json [00:45:12] ═══════════════════════════════════════════════════════════ [00:45:12] All 3 runs complete for: rand-write-qd32-nj1 [00:45:12] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:45:12] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:45:12] ═══════════════════════════════════════════════════════════ [00:45:12] Progress: 45/53 jobs | Elapsed: 37h21m | ETA: ~6h38m [00:45:12] [00:45:12] ╔══════════════════════════════════════════════════════════╗ [00:45:12] ║ Job 46/53: rand-write-qd32-nj8 [00:45:12] ║ Category: rand [00:45:12] ╚══════════════════════════════════════════════════════════╝ [00:45:12] ╔══════════════════════════════════════════════════════════╗ [00:45:12] ║ Benchmark: rand-write-qd32-nj8 [00:45:12] ║ Category: rand [00:45:12] ║ Runs: 3 × cooldown 30s [00:45:12] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:45:12] ╚══════════════════════════════════════════════════════════╝ [00:45:12] ──────────────────────────────────────────────────── [00:45:12] Run 1/3: rand-write-qd32-nj8_run1 [00:45:12] ──────────────────────────────────────────────────── [00:45:12] Cleaning fio test files from /tank-mirror/bench … [00:45:12] Test files removed [00:45:12] Dropping Linux page cache + ZFS ARC … [00:45:12] Flushing ZFS ARC (was 1) … [00:45:17] ARC flushed — current size: 29515037816 bytes [00:45:17] NVMe volatile write caches flushed [00:45:18] pmcd healthy (core ✓ pmdazfs ✓) [00:45:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj8_run1.ndjson [00:45:19] Collector running (PID 4025844), settling 5s … [00:45:24] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj8.fio [00:48:35] fio completed in 191s [00:48:35] Post-fio settle 5s … [00:48:40] Stopping collector [00:48:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj8_run1.manifest.json [00:48:41] Cooldown: 30s [00:49:11] ──────────────────────────────────────────────────── [00:49:11] Run 2/3: rand-write-qd32-nj8_run2 [00:49:11] ──────────────────────────────────────────────────── [00:49:11] Cleaning fio test files from /tank-mirror/bench … [00:49:11] Test files removed [00:49:11] Dropping Linux page cache + ZFS ARC … [00:49:11] Flushing ZFS ARC (was 1) … [00:49:16] ARC flushed — current size: 25202951056 bytes [00:49:16] NVMe volatile write caches flushed [00:49:18] pmcd healthy (core ✓ pmdazfs ✓) [00:49:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj8_run2.ndjson [00:49:19] Collector running (PID 4028474), settling 5s … [00:49:24] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj8.fio [00:52:34] fio completed in 190s [00:52:34] Post-fio settle 5s … [00:52:39] Stopping collector [00:52:41] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj8_run2.manifest.json [00:52:41] Cooldown: 30s [00:53:11] ──────────────────────────────────────────────────── [00:53:11] Run 3/3: rand-write-qd32-nj8_run3 [00:53:11] ──────────────────────────────────────────────────── [00:53:11] Cleaning fio test files from /tank-mirror/bench … [00:53:12] Test files removed [00:53:12] Dropping Linux page cache + ZFS ARC … [00:53:12] Flushing ZFS ARC (was 1) … [00:53:17] ARC flushed — current size: 66264307376 bytes [00:53:17] NVMe volatile write caches flushed [00:53:18] pmcd healthy (core ✓ pmdazfs ✓) [00:53:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj8_run3.ndjson [00:53:19] Collector running (PID 4031107), settling 5s … [00:53:24] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj8.fio [00:56:35] fio completed in 191s [00:56:35] Post-fio settle 5s … [00:56:40] Stopping collector [00:56:42] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj8_run3.manifest.json [00:56:42] ═══════════════════════════════════════════════════════════ [00:56:42] All 3 runs complete for: rand-write-qd32-nj8 [00:56:42] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [00:56:42] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [00:56:42] ═══════════════════════════════════════════════════════════ [00:56:42] Progress: 46/53 jobs | Elapsed: 37h32m | ETA: ~5h42m [00:56:42] [00:56:42] ╔══════════════════════════════════════════════════════════╗ [00:56:42] ║ Job 47/53: rand-write-qd32-nj16 [00:56:42] ║ Category: rand [00:56:42] ╚══════════════════════════════════════════════════════════╝ [00:56:42] ╔══════════════════════════════════════════════════════════╗ [00:56:42] ║ Benchmark: rand-write-qd32-nj16 [00:56:42] ║ Category: rand [00:56:42] ║ Runs: 3 × cooldown 30s [00:56:42] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [00:56:42] ╚══════════════════════════════════════════════════════════╝ [00:56:42] ──────────────────────────────────────────────────── [00:56:42] Run 1/3: rand-write-qd32-nj16_run1 [00:56:42] ──────────────────────────────────────────────────── [00:56:42] Cleaning fio test files from /tank-mirror/bench … [00:56:42] Test files removed [00:56:42] Dropping Linux page cache + ZFS ARC … [00:56:42] Flushing ZFS ARC (was 1) … [00:56:47] ARC flushed — current size: 23105487152 bytes [00:56:47] NVMe volatile write caches flushed [00:56:49] pmcd healthy (core ✓ pmdazfs ✓) [00:56:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj16_run1.ndjson [00:56:50] Collector running (PID 4034310), settling 5s … [00:56:55] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj16.fio [01:00:05] fio completed in 190s [01:00:05] Post-fio settle 5s … [01:00:10] Stopping collector [01:00:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj16_run1.manifest.json [01:00:12] Cooldown: 30s [01:00:42] ──────────────────────────────────────────────────── [01:00:42] Run 2/3: rand-write-qd32-nj16_run2 [01:00:42] ──────────────────────────────────────────────────── [01:00:42] Cleaning fio test files from /tank-mirror/bench … [01:00:42] Test files removed [01:00:42] Dropping Linux page cache + ZFS ARC … [01:00:43] Flushing ZFS ARC (was 1) … [01:00:48] ARC flushed — current size: 14140258368 bytes [01:00:48] NVMe volatile write caches flushed [01:00:49] pmcd healthy (core ✓ pmdazfs ✓) [01:00:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj16_run2.ndjson [01:00:50] Collector running (PID 4036952), settling 5s … [01:00:55] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj16.fio [01:04:05] fio completed in 190s [01:04:05] Post-fio settle 5s … [01:04:10] Stopping collector [01:04:11] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj16_run2.manifest.json [01:04:11] Cooldown: 30s [01:04:41] ──────────────────────────────────────────────────── [01:04:41] Run 3/3: rand-write-qd32-nj16_run3 [01:04:42] ──────────────────────────────────────────────────── [01:04:42] Cleaning fio test files from /tank-mirror/bench … [01:04:42] Test files removed [01:04:42] Dropping Linux page cache + ZFS ARC … [01:04:42] Flushing ZFS ARC (was 1) … [01:04:47] ARC flushed — current size: 13314424856 bytes [01:04:47] NVMe volatile write caches flushed [01:04:48] pmcd healthy (core ✓ pmdazfs ✓) [01:04:48] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj16_run3.ndjson [01:04:49] Collector running (PID 4039478), settling 5s … [01:04:54] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj16.fio [01:08:05] fio completed in 191s [01:08:05] Post-fio settle 5s … [01:08:10] Stopping collector [01:08:12] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj16_run3.manifest.json [01:08:12] ═══════════════════════════════════════════════════════════ [01:08:12] All 3 runs complete for: rand-write-qd32-nj16 [01:08:12] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [01:08:12] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [01:08:12] ═══════════════════════════════════════════════════════════ [01:08:12] Progress: 47/53 jobs | Elapsed: 37h44m | ETA: ~4h49m [01:08:12] [01:08:12] ╔══════════════════════════════════════════════════════════╗ [01:08:12] ║ Job 48/53: rand-write-qd32-nj32 [01:08:12] ║ Category: rand [01:08:12] ╚══════════════════════════════════════════════════════════╝ [01:08:12] ╔══════════════════════════════════════════════════════════╗ [01:08:12] ║ Benchmark: rand-write-qd32-nj32 [01:08:12] ║ Category: rand [01:08:12] ║ Runs: 3 × cooldown 30s [01:08:12] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [01:08:12] ╚══════════════════════════════════════════════════════════╝ [01:08:12] ──────────────────────────────────────────────────── [01:08:12] Run 1/3: rand-write-qd32-nj32_run1 [01:08:12] ──────────────────────────────────────────────────── [01:08:12] Cleaning fio test files from /tank-mirror/bench … [01:08:12] Test files removed [01:08:12] Dropping Linux page cache + ZFS ARC … [01:08:12] Flushing ZFS ARC (was 1) … [01:08:17] ARC flushed — current size: 13278501200 bytes [01:08:17] NVMe volatile write caches flushed [01:08:19] pmcd healthy (core ✓ pmdazfs ✓) [01:08:19] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj32_run1.ndjson [01:08:20] Collector running (PID 4042005), settling 5s … [01:08:25] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj32.fio [01:11:35] fio completed in 190s [01:11:35] Post-fio settle 5s … [01:11:40] Stopping collector [01:11:42] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj32_run1.manifest.json [01:11:42] Cooldown: 30s [01:12:12] ──────────────────────────────────────────────────── [01:12:12] Run 2/3: rand-write-qd32-nj32_run2 [01:12:12] ──────────────────────────────────────────────────── [01:12:12] Cleaning fio test files from /tank-mirror/bench … [01:12:12] Test files removed [01:12:12] Dropping Linux page cache + ZFS ARC … [01:12:13] Flushing ZFS ARC (was 1) … [01:12:18] ARC flushed — current size: 41405712 bytes [01:12:18] NVMe volatile write caches flushed [01:12:19] pmcd healthy (core ✓ pmdazfs ✓) [01:12:19] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj32_run2.ndjson [01:12:20] Collector running (PID 4044598), settling 5s … [01:12:25] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj32.fio [01:15:36] fio completed in 191s [01:15:36] Post-fio settle 5s … [01:15:41] Stopping collector [01:15:42] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj32_run2.manifest.json [01:15:42] Cooldown: 30s [01:16:12] ──────────────────────────────────────────────────── [01:16:12] Run 3/3: rand-write-qd32-nj32_run3 [01:16:12] ──────────────────────────────────────────────────── [01:16:12] Cleaning fio test files from /tank-mirror/bench … [01:16:12] Test files removed [01:16:12] Dropping Linux page cache + ZFS ARC … [01:16:12] Flushing ZFS ARC (was 1) … [01:16:17] ARC flushed — current size: 41552056 bytes [01:16:17] NVMe volatile write caches flushed [01:16:18] pmcd healthy (core ✓ pmdazfs ✓) [01:16:18] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand/rand-write-qd32-nj32_run3.ndjson [01:16:19] Collector running (PID 4047103), settling 5s … [01:16:24] Starting fio: /scripts_f16_with_data_deletion/jobs/rand-write-qd32-nj32.fio [01:19:35] fio completed in 191s [01:19:35] Post-fio settle 5s … [01:19:40] Stopping collector [01:19:42] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand/rand-write-qd32-nj32_run3.manifest.json [01:19:42] ═══════════════════════════════════════════════════════════ [01:19:42] All 3 runs complete for: rand-write-qd32-nj32 [01:19:42] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/rand [01:19:42] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/rand [01:19:42] ═══════════════════════════════════════════════════════════ [01:19:42] Progress: 48/53 jobs | Elapsed: 37h55m | ETA: ~3h57m [01:19:42] [01:19:42] ╔══════════════════════════════════════════════════════════╗ [01:19:42] ║ Job 49/53: lat-read-qd1-nj1 [01:19:42] ║ Category: latency [01:19:42] ╚══════════════════════════════════════════════════════════╝ [01:19:42] ╔══════════════════════════════════════════════════════════╗ [01:19:42] ║ Benchmark: lat-read-qd1-nj1 [01:19:42] ║ Category: latency [01:19:42] ║ Runs: 3 × cooldown 30s [01:19:42] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [01:19:42] ╚══════════════════════════════════════════════════════════╝ [01:19:42] ──────────────────────────────────────────────────── [01:19:42] Run 1/3: lat-read-qd1-nj1_run1 [01:19:42] ──────────────────────────────────────────────────── [01:19:42] Cleaning fio test files from /tank-mirror/bench … [01:19:42] Test files removed [01:19:42] Dropping Linux page cache + ZFS ARC … [01:19:42] Flushing ZFS ARC (was 1) … [01:19:47] ARC flushed — current size: 42431064 bytes [01:19:47] NVMe volatile write caches flushed [01:19:49] pmcd healthy (core ✓ pmdazfs ✓) [01:19:49] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-read-qd1-nj1_run1.ndjson [01:19:50] Collector running (PID 4049561), settling 5s … [01:19:55] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-read-qd1-nj1.fio [01:25:32] fio completed in 337s [01:25:32] Post-fio settle 5s … [01:25:37] Stopping collector [01:25:39] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-read-qd1-nj1_run1.manifest.json [01:25:39] Cooldown: 30s [01:26:09] ──────────────────────────────────────────────────── [01:26:09] Run 2/3: lat-read-qd1-nj1_run2 [01:26:09] ──────────────────────────────────────────────────── [01:26:09] Cleaning fio test files from /tank-mirror/bench … [01:26:09] Test files removed [01:26:09] Dropping Linux page cache + ZFS ARC … [01:26:10] Flushing ZFS ARC (was 1) … [01:26:15] ARC flushed — current size: 39582816 bytes [01:26:15] NVMe volatile write caches flushed [01:26:16] pmcd healthy (core ✓ pmdazfs ✓) [01:26:16] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-read-qd1-nj1_run2.ndjson [01:26:17] Collector running (PID 4054449), settling 5s … [01:26:22] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-read-qd1-nj1.fio [01:31:57] fio completed in 335s [01:31:57] Post-fio settle 5s … [01:32:02] Stopping collector [01:32:04] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-read-qd1-nj1_run2.manifest.json [01:32:04] Cooldown: 30s [01:32:34] ──────────────────────────────────────────────────── [01:32:34] Run 3/3: lat-read-qd1-nj1_run3 [01:32:34] ──────────────────────────────────────────────────── [01:32:34] Cleaning fio test files from /tank-mirror/bench … [01:32:34] Test files removed [01:32:34] Dropping Linux page cache + ZFS ARC … [01:32:35] Flushing ZFS ARC (was 1) … [01:32:40] ARC flushed — current size: 34665792 bytes [01:32:40] NVMe volatile write caches flushed [01:32:41] pmcd healthy (core ✓ pmdazfs ✓) [01:32:41] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-read-qd1-nj1_run3.ndjson [01:32:42] Collector running (PID 4058622), settling 5s … [01:32:47] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-read-qd1-nj1.fio [01:38:24] fio completed in 337s [01:38:24] Post-fio settle 5s … [01:38:29] Stopping collector [01:38:31] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-read-qd1-nj1_run3.manifest.json [01:38:31] ═══════════════════════════════════════════════════════════ [01:38:31] All 3 runs complete for: lat-read-qd1-nj1 [01:38:31] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency [01:38:31] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency [01:38:31] ═══════════════════════════════════════════════════════════ [01:38:31] Progress: 49/53 jobs | Elapsed: 38h14m | ETA: ~3h7m [01:38:31] [01:38:31] ╔══════════════════════════════════════════════════════════╗ [01:38:31] ║ Job 50/53: lat-write-qd1-nj1 [01:38:31] ║ Category: latency [01:38:31] ╚══════════════════════════════════════════════════════════╝ [01:38:31] ╔══════════════════════════════════════════════════════════╗ [01:38:31] ║ Benchmark: lat-write-qd1-nj1 [01:38:31] ║ Category: latency [01:38:31] ║ Runs: 3 × cooldown 30s [01:38:31] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [01:38:31] ╚══════════════════════════════════════════════════════════╝ [01:38:31] ──────────────────────────────────────────────────── [01:38:31] Run 1/3: lat-write-qd1-nj1_run1 [01:38:31] ──────────────────────────────────────────────────── [01:38:31] Cleaning fio test files from /tank-mirror/bench … [01:38:31] Test files removed [01:38:31] Dropping Linux page cache + ZFS ARC … [01:38:31] Flushing ZFS ARC (was 1) … [01:38:36] ARC flushed — current size: 34249384 bytes [01:38:36] NVMe volatile write caches flushed [01:38:37] pmcd healthy (core ✓ pmdazfs ✓) [01:38:37] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-write-qd1-nj1_run1.ndjson [01:38:38] Collector running (PID 4062789), settling 5s … [01:38:43] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-write-qd1-nj1.fio [01:41:54] fio completed in 191s [01:41:54] Post-fio settle 5s … [01:41:59] Stopping collector [01:42:00] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-write-qd1-nj1_run1.manifest.json [01:42:00] Cooldown: 30s [01:42:30] ──────────────────────────────────────────────────── [01:42:30] Run 2/3: lat-write-qd1-nj1_run2 [01:42:30] ──────────────────────────────────────────────────── [01:42:30] Cleaning fio test files from /tank-mirror/bench … [01:42:30] Test files removed [01:42:30] Dropping Linux page cache + ZFS ARC … [01:42:30] Flushing ZFS ARC (was 1) … [01:42:35] ARC flushed — current size: 43403200552 bytes [01:42:35] NVMe volatile write caches flushed [01:42:36] pmcd healthy (core ✓ pmdazfs ✓) [01:42:36] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-write-qd1-nj1_run2.ndjson [01:42:38] Collector running (PID 4065216), settling 5s … [01:42:43] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-write-qd1-nj1.fio [01:45:53] fio completed in 190s [01:45:53] Post-fio settle 5s … [01:45:58] Stopping collector [01:45:59] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-write-qd1-nj1_run2.manifest.json [01:45:59] Cooldown: 30s [01:46:29] ──────────────────────────────────────────────────── [01:46:29] Run 3/3: lat-write-qd1-nj1_run3 [01:46:29] ──────────────────────────────────────────────────── [01:46:29] Cleaning fio test files from /tank-mirror/bench … [01:46:29] Test files removed [01:46:29] Dropping Linux page cache + ZFS ARC … [01:46:29] Flushing ZFS ARC (was 1) … [01:46:34] ARC flushed — current size: 39432560336 bytes [01:46:34] NVMe volatile write caches flushed [01:46:36] pmcd healthy (core ✓ pmdazfs ✓) [01:46:36] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency/lat-write-qd1-nj1_run3.ndjson [01:46:37] Collector running (PID 4067674), settling 5s … [01:46:42] Starting fio: /scripts_f16_with_data_deletion/jobs/lat-write-qd1-nj1.fio [01:49:52] fio completed in 190s [01:49:52] Post-fio settle 5s … [01:49:57] Stopping collector [01:49:58] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency/lat-write-qd1-nj1_run3.manifest.json [01:49:58] ═══════════════════════════════════════════════════════════ [01:49:58] All 3 runs complete for: lat-write-qd1-nj1 [01:49:58] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/latency [01:49:58] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/latency [01:49:58] ═══════════════════════════════════════════════════════════ [01:49:58] Progress: 50/53 jobs | Elapsed: 38h25m | ETA: ~2h18m [01:49:58] [01:49:58] ╔══════════════════════════════════════════════════════════╗ [01:49:58] ║ Job 51/53: mixed-70r30w-qd16-nj1 [01:49:58] ║ Category: mixed [01:49:58] ╚══════════════════════════════════════════════════════════╝ [01:49:58] ╔══════════════════════════════════════════════════════════╗ [01:49:58] ║ Benchmark: mixed-70r30w-qd16-nj1 [01:49:58] ║ Category: mixed [01:49:58] ║ Runs: 3 × cooldown 30s [01:49:58] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [01:49:58] ╚══════════════════════════════════════════════════════════╝ [01:49:58] ──────────────────────────────────────────────────── [01:49:58] Run 1/3: mixed-70r30w-qd16-nj1_run1 [01:49:58] ──────────────────────────────────────────────────── [01:49:58] Cleaning fio test files from /tank-mirror/bench … [01:49:58] Test files removed [01:49:58] Dropping Linux page cache + ZFS ARC … [01:49:59] Flushing ZFS ARC (was 1) … [01:50:04] ARC flushed — current size: 26473885360 bytes [01:50:04] NVMe volatile write caches flushed [01:50:05] pmcd healthy (core ✓ pmdazfs ✓) [01:50:05] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj1_run1.ndjson [01:50:06] Collector running (PID 4070043), settling 5s … [01:50:11] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj1.fio [01:55:47] fio completed in 336s [01:55:47] Post-fio settle 5s … [01:55:52] Stopping collector [01:55:53] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj1_run1.manifest.json [01:55:53] Cooldown: 30s [01:56:23] ──────────────────────────────────────────────────── [01:56:23] Run 2/3: mixed-70r30w-qd16-nj1_run2 [01:56:23] ──────────────────────────────────────────────────── [01:56:23] Cleaning fio test files from /tank-mirror/bench … [01:56:23] Test files removed [01:56:23] Dropping Linux page cache + ZFS ARC … [01:56:23] Flushing ZFS ARC (was 1) … [01:56:28] ARC flushed — current size: 42762923936 bytes [01:56:28] NVMe volatile write caches flushed [01:56:29] pmcd healthy (core ✓ pmdazfs ✓) [01:56:29] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj1_run2.ndjson [01:56:31] Collector running (PID 4074838), settling 5s … [01:56:36] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj1.fio [02:02:13] fio completed in 337s [02:02:13] Post-fio settle 5s … [02:02:18] Stopping collector [02:02:20] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj1_run2.manifest.json [02:02:20] Cooldown: 30s [02:02:50] ──────────────────────────────────────────────────── [02:02:50] Run 3/3: mixed-70r30w-qd16-nj1_run3 [02:02:50] ──────────────────────────────────────────────────── [02:02:50] Cleaning fio test files from /tank-mirror/bench … [02:02:50] Test files removed [02:02:50] Dropping Linux page cache + ZFS ARC … [02:02:51] Flushing ZFS ARC (was 1) … [02:02:56] ARC flushed — current size: 13487824968 bytes [02:02:56] NVMe volatile write caches flushed [02:02:57] pmcd healthy (core ✓ pmdazfs ✓) [02:02:57] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj1_run3.ndjson [02:02:58] Collector running (PID 4079082), settling 5s … [02:03:03] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj1.fio [02:08:39] fio completed in 336s [02:08:39] Post-fio settle 5s … [02:08:44] Stopping collector [02:08:46] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj1_run3.manifest.json [02:08:46] ═══════════════════════════════════════════════════════════ [02:08:46] All 3 runs complete for: mixed-70r30w-qd16-nj1 [02:08:46] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed [02:08:46] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed [02:08:46] ═══════════════════════════════════════════════════════════ [02:08:46] Progress: 51/53 jobs | Elapsed: 38h44m | ETA: ~1h31m [02:08:46] [02:08:46] ╔══════════════════════════════════════════════════════════╗ [02:08:46] ║ Job 52/53: mixed-70r30w-qd16-nj4 [02:08:46] ║ Category: mixed [02:08:46] ╚══════════════════════════════════════════════════════════╝ [02:08:46] ╔══════════════════════════════════════════════════════════╗ [02:08:46] ║ Benchmark: mixed-70r30w-qd16-nj4 [02:08:46] ║ Category: mixed [02:08:46] ║ Runs: 3 × cooldown 30s [02:08:46] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [02:08:46] ╚══════════════════════════════════════════════════════════╝ [02:08:46] ──────────────────────────────────────────────────── [02:08:46] Run 1/3: mixed-70r30w-qd16-nj4_run1 [02:08:46] ──────────────────────────────────────────────────── [02:08:46] Cleaning fio test files from /tank-mirror/bench … [02:08:46] Test files removed [02:08:46] Dropping Linux page cache + ZFS ARC … [02:08:47] Flushing ZFS ARC (was 1) … [02:08:52] ARC flushed — current size: 67453056 bytes [02:08:52] NVMe volatile write caches flushed [02:08:53] pmcd healthy (core ✓ pmdazfs ✓) [02:08:53] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj4_run1.ndjson [02:08:54] Collector running (PID 4083190), settling 5s … [02:08:59] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj4.fio [02:21:59] fio completed in 780s [02:21:59] Post-fio settle 5s … [02:22:04] Stopping collector [02:22:06] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj4_run1.manifest.json [02:22:06] Cooldown: 30s [02:22:36] ──────────────────────────────────────────────────── [02:22:36] Run 2/3: mixed-70r30w-qd16-nj4_run2 [02:22:36] ──────────────────────────────────────────────────── [02:22:36] Cleaning fio test files from /tank-mirror/bench … [02:22:37] Test files removed [02:22:37] Dropping Linux page cache + ZFS ARC … [02:22:37] Flushing ZFS ARC (was 1) … [02:22:42] ARC flushed — current size: 64820048 bytes [02:22:42] NVMe volatile write caches flushed [02:22:43] pmcd healthy (core ✓ pmdazfs ✓) [02:22:43] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj4_run2.ndjson [02:22:44] Collector running (PID 4092805), settling 5s … [02:22:49] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj4.fio [02:35:49] fio completed in 780s [02:35:49] Post-fio settle 5s … [02:35:54] Stopping collector [02:35:56] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj4_run2.manifest.json [02:35:56] Cooldown: 30s [02:36:26] ──────────────────────────────────────────────────── [02:36:26] Run 3/3: mixed-70r30w-qd16-nj4_run3 [02:36:26] ──────────────────────────────────────────────────── [02:36:26] Cleaning fio test files from /tank-mirror/bench … [02:36:26] Test files removed [02:36:26] Dropping Linux page cache + ZFS ARC … [02:36:27] Flushing ZFS ARC (was 1) … [02:36:32] ARC flushed — current size: 29056362160 bytes [02:36:32] NVMe volatile write caches flushed [02:36:33] pmcd healthy (core ✓ pmdazfs ✓) [02:36:33] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj4_run3.ndjson [02:36:34] Collector running (PID 4103046), settling 5s … [02:36:39] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj4.fio [02:49:38] fio completed in 779s [02:49:38] Post-fio settle 5s … [02:49:43] Stopping collector [02:49:44] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj4_run3.manifest.json [02:49:44] ═══════════════════════════════════════════════════════════ [02:49:44] All 3 runs complete for: mixed-70r30w-qd16-nj4 [02:49:44] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed [02:49:44] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed [02:49:44] ═══════════════════════════════════════════════════════════ [02:49:44] Progress: 52/53 jobs | Elapsed: 39h25m | ETA: ~0h45m [02:49:44] [02:49:44] ╔══════════════════════════════════════════════════════════╗ [02:49:44] ║ Job 53/53: mixed-70r30w-qd16-nj8 [02:49:44] ║ Category: mixed [02:49:44] ╚══════════════════════════════════════════════════════════╝ [02:49:44] ╔══════════════════════════════════════════════════════════╗ [02:49:44] ║ Benchmark: mixed-70r30w-qd16-nj8 [02:49:44] ║ Category: mixed [02:49:44] ║ Runs: 3 × cooldown 30s [02:49:44] ║ Output: /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [02:49:44] ╚══════════════════════════════════════════════════════════╝ [02:49:44] ──────────────────────────────────────────────────── [02:49:44] Run 1/3: mixed-70r30w-qd16-nj8_run1 [02:49:44] ──────────────────────────────────────────────────── [02:49:44] Cleaning fio test files from /tank-mirror/bench … [02:49:44] Test files removed [02:49:44] Dropping Linux page cache + ZFS ARC … [02:49:45] Flushing ZFS ARC (was 1) … [02:49:50] ARC flushed — current size: 42210025928 bytes [02:49:50] NVMe volatile write caches flushed [02:49:51] pmcd healthy (core ✓ pmdazfs ✓) [02:49:51] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj8_run1.ndjson [02:49:52] Collector running (PID 4112559), settling 5s … [02:49:57] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj8.fio [03:12:40] fio completed in 1363s [03:12:40] Post-fio settle 5s … [03:12:45] Stopping collector [03:12:46] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj8_run1.manifest.json [03:12:46] Cooldown: 30s [03:13:16] ──────────────────────────────────────────────────── [03:13:16] Run 2/3: mixed-70r30w-qd16-nj8_run2 [03:13:16] ──────────────────────────────────────────────────── [03:13:16] Cleaning fio test files from /tank-mirror/bench … [03:13:16] Test files removed [03:13:16] Dropping Linux page cache + ZFS ARC … [03:13:16] Flushing ZFS ARC (was 1) … [03:13:21] ARC flushed — current size: 103783153112 bytes [03:13:21] NVMe volatile write caches flushed [03:13:23] pmcd healthy (core ✓ pmdazfs ✓) [03:13:23] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj8_run2.ndjson [03:13:24] Collector running (PID 4129652), settling 5s … [03:13:29] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj8.fio [03:36:21] fio completed in 1372s [03:36:21] Post-fio settle 5s … [03:36:26] Stopping collector [03:36:28] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj8_run2.manifest.json [03:36:28] Cooldown: 30s [03:36:58] ──────────────────────────────────────────────────── [03:36:58] Run 3/3: mixed-70r30w-qd16-nj8_run3 [03:36:58] ──────────────────────────────────────────────────── [03:36:58] Cleaning fio test files from /tank-mirror/bench … [03:36:58] Test files removed [03:36:58] Dropping Linux page cache + ZFS ARC … [03:36:58] Flushing ZFS ARC (was 1) … [03:37:03] ARC flushed — current size: 75087521064 bytes [03:37:03] NVMe volatile write caches flushed [03:37:05] pmcd healthy (core ✓ pmdazfs ✓) [03:37:05] Starting PCP collector → /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed/mixed-70r30w-qd16-nj8_run3.ndjson [03:37:06] Collector running (PID 4146805), settling 5s … [03:37:11] Starting fio: /scripts_f16_with_data_deletion/jobs/mixed-70r30w-qd16-nj8.fio [03:59:40] fio completed in 1349s [03:59:40] Post-fio settle 5s … [03:59:45] Stopping collector [03:59:47] Manifest: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed/mixed-70r30w-qd16-nj8_run3.manifest.json [03:59:47] ═══════════════════════════════════════════════════════════ [03:59:47] All 3 runs complete for: mixed-70r30w-qd16-nj8 [03:59:47] Benchmarks: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/benchmarks/mixed [03:59:47] Metrics: /scripts_f16_with_data_deletion/results/mirror_20260318_112405/metrics/mixed [03:59:47] ═══════════════════════════════════════════════════════════ [03:59:47] Progress: 53/53 jobs | Elapsed: 40h35m | ETA: ~0h0m [03:59:47] [03:59:47] ╔══════════════════════════════════════════════════════════════╗ [03:59:47] ║ ALL BENCHMARKS COMPLETE ║ [03:59:47] ╠══════════════════════════════════════════════════════════════╣ [03:59:47] ║ Pool layout : mirror [03:59:48] ║ Total jobs : 53 [03:59:48] ║ Total runs : 159 [03:59:48] ║ Failed : 0 [03:59:48] ║ Elapsed : 40h 35m 42s [03:59:48] ║ Results : /scripts_f16_with_data_deletion/results/mirror_20260318_112405 [03:59:48] ╚══════════════════════════════════════════════════════════════╝ [03:59:48] [03:59:48] Results tree: