feat: enable JSON output#22
Conversation
|
Here is a sample output: cargo run -- --json{"type":"detected_gpus","detected_gpus":[{"index":0,"name":"Tesla V100-PCIE-16GB","uuid":"ace03226-4b45-24c9-7ceb-2acf6c3bd2ad"},{"index":1,"name":"Tesla V100-PCIE-16GB","uuid":"2d1b21c2-9f4c-3413-6d8c-0fe2780096c9"}]}
{"type":"feature_detection","bf16":false,"fp8":false,"fp32":true,"gpu_memory_used":[{"gpu_idx":0,"memory_gb":13.9140625},{"gpu_idx":1,"memory_gb":13.9140625}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":1022,"flops":1123700883587072,"temp_celsius":55,"throttle_reason":""},{"gpu_idx":1,"nops":1022,"flops":1123700883587072,"temp_celsius":55,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":0,"flops":0,"temp_celsius":50,"throttle_reason":""},{"gpu_idx":1,"nops":0,"flops":0,"temp_celsius":50,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":64,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":64,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":1030,"flops":1132496976609280,"temp_celsius":60,"throttle_reason":""},{"gpu_idx":1,"nops":1030,"flops":1132496976609280,"temp_celsius":60,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":65,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":65,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":63,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":63,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":66,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":66,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":65,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":65,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":67,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":67,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":65,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":65,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":68,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":68,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":66,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":66,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":69,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":69,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":67,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":67,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":69,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":69,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":68,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":68,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":69,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":69,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":69,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":69,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":70,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":71,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":71,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":71,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":72,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":72,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":72,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":73,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":74,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":13,"flops":14293651161088,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""},{"gpu_idx":1,"nops":12,"flops":13194139533312,"temp_celsius":75,"throttle_reason":""}]}
{"type":"progress","gpus":[{"gpu_idx":0,"nops":9,"flops":9895604649984,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"},{"gpu_idx":1,"nops":9,"flops":9895604649984,"temp_celsius":75,"throttle_reason":"SW_POWER_CAP"}]}
{"type":"results","gpus":[{"burn_result":{"gpu_idx":0,"flops_max":14293651161088,"flops_min":13194139533312,"flops_sum":745468883632128,"flops_mean":13311944350573.717,"flops_m2":6.476388319364087e+24,"n_iters":56,"temp_max":75,"temp_sum":4128,"temp_min":68,"throttling_hw":0,"throttling_thermal_sw":0,"throttling_thermal_hw":0},"flops_avg":13311944350573.715},{"burn_result":{"gpu_idx":1,"flops_max":14293651161088,"flops_min":9895604649984,"flops_sum":749866930143232,"flops_mean":13390480895414.86,"flops_m2":2.443757192506717e+25,"n_iters":56,"temp_max":75,"temp_sum":4089,"temp_min":66,"throttling_hw":0,"throttling_thermal_sw":0,"throttling_thermal_hw":0},"flops_avg":13390480895414.857}]}
{"type":"problems","problems":[]} |
|
Hi @Hugoch it's now been more than a month, would you have a bit of time to review that? ππ» |
|
Hi again! gpu-fryer is definitely our go-to tool for this. But to orchestrate it, if we want to collect metrics as they come and follow the state, we definitely need a way to parse the output. We could have parsed "the old way", but it's really annoying to do and inconsistent whereas with the change we can just parse the JSON as-is in any language. (there are also the issues where the output format changes with the update, "oops, we added a line jump there because it was looking better", etc.) |
|
Thank you for the thorough review. I did correct all the suggestions |
Hugoch
left a comment
There was a problem hiding this comment.
Thanks for the edits! A few more remarks and we should be good to merge :)
I rewrote most of the logging system. This allows to have proper JSON output for parsing by external program. There is a few standardized types that will limit the amount of structures required to implement. The original text output is kept mostly as-is, except for the `problem` output, since it was not coherent depending on the problem. Some log lines (`log_raw`) are kept as-is because I was too lazy to implement them as JSON and they probably do not have that much value when used programmatically.
|
I let you merge whenever! |
I rewrote most of the logging system. This allows to have proper JSON output for parsing by external program. There is a few standardized types that will limit the amount of structures required to implement.
The original text output is kept mostly as-is, except for the
problemoutput, since it was not coherent depending on the problem. Some log lines (log_raw) are kept as-is because I was too lazy to implement them as JSON and they probably do not have that much value when used programmatically.Serdecould have been made optional, but it would have made the code more complicated and most of the dependencies are already there anyway.