Adding a total_iodepth option that can be used with librbdfio by harriscr · Pull Request #324 · ceph/cbt

harriscr · 2025-01-30T17:37:35Z

Description

Add an option to the CBT configuration yaml file called total_iodepth. The total_iodepth is then split evenly among the number of volumes used for the test. If the total_iodepth for a run does not divide evenly into the number of volumes, then any remainder will be assigned 1iodepth at a time to the volumes, starting at 0.

For (a simple) example:
For an total_iodepth of 18 and volumes_per_client of 5, the following iodepth allocations would occur:

volume	iodepth
0	4
1	4
2	4
3	3
4	3

If the number of volumes specified is such that there is not enough iodepth for 1 per volume, then the number of volumes for that test will be reduced so that an iodepth of 1 per volume can be achieved.

Example:
For volumes_per_client = 5 and total_iodepth=4, the benchmark would be run with 4 volumes, each of iodepth 1

Testing

Manual testing was done for a number of scenarios, with and without using workloads The below are the output from debug statements added for the purposes of testing.

Regression testing: iodepth=32, volumes_per_client=3

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.1

CHDEBUG: fio_cmd for volume 2 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-2 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-032/randwrite/output.2

total_iodepth=32, volumes_per_client=3

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=11 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=11 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.1

CHDEBUG: fio_cmd for volume 2 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-2 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=10 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-003/iodepth-016/randwrite/output.2

total_iodepth=2, volumes_per_client=3

11:13:28 - WARNING - cbt - The total iodepth requested: 2 is less than 1 per volume.
11:13:28 - WARNING - cbt - Number of volumes per client will be reduced from 3 to 2

CHDEBUG: fio_cmd for volume 0 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.0

CHDEBUG: fio_cmd for volume 1 is /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=300 --numjobs=1 --direct=1 --bs=4096B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1 --log_avg_msec=101 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-016/randwrite/output.1

I also tested with an array of total_iodepths, but the output is too long for here so instead I will attach the file containing the results:
total_iodepth_testing.txt

I'll update with teuthology results once they have run

harriscr · 2025-01-31T08:46:01Z

python unit tests:

============================= slowest 5 durations ==============================
0.01s setup tests/test_bm_kvmrbdfio.py::TestBenchmarkkvmrbdfio::test_valid_archive_dir
0.01s setup tests/test_bm_nullbench.py::TestBenchmarknullbench::test_valid_archive_dir
0.01s setup tests/test_bm_rawfio.py::TestBenchmarkrawfio::test_valid_archive_dir
0.01s setup tests/test_bm_radosbench.py::TestBenchmarkradosbench::test_valid_archive_dir
0.01s setup tests/test_bm_fio.py::TestBenchmarkfio::test_valid_archive_dir
======================== 326 passed, 3 skipped in 0.47s ========================
Finished running tests!

I haven't run black, ruff or mypy against the file to check for pep-8 compliance as the unchanged file gives too many errors currently

harriscr · 2025-02-06T14:00:19Z

Testing with workloads:

precondition: iodepth=[2]

13:45:31 - INFO - cbt - Running rbd fio precondition test, mode randwrite
13:45:31 - WARNING - cbt - CHDEBUG: Using iodepth

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 1 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 2 and vol 7 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-7 --invalidate=0 --rw=randwrite --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.7

The procondition runs on all 8 vlumes with an iodepth of 2, as expected.

total_iodepth=16, volumes=8

13:45:31 - INFO - cbt - Running rbd fio seq32kwrite test, mode write
13:45:31 - WARNING - cbt - CHDEBUG: Using total_iodepth

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 16 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.0

13:45:31 - INFO - cbt - CHDEBUG: fio command for iodepth 16 and vol 1 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-1 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-016/numjobs-001/output.1

total_iodepth=7, volumes=8

13:45:32 - WARNING - cbt - The total iodepth requested: 7 is less than 1 per volume (8)
13:45:32 - WARNING - cbt - Number of volumes per client will be reduced from 8 to 7

13:45:32 - INFO - cbt - CHDEBUG: fio command for iodepth 7 and vol 0 is
sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-hostname -f-0 --invalidate=0 --rw=write --output-format=json --runtime=60 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-hostname -f-file-0 > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-007/numjobs-001/output.0

harriscr · 2025-02-11T09:01:27Z

Pulpito results (finally):
https://pulpito.ceph.com/harriscr-2025-02-06_15:56:31-perf-basic-wip-harriscr-cbt-refactor-2024-10-17-0839-distro-default-smithi/
https://pulpito.ceph.com/harriscr-2025-02-06_15:57:11-rados:perf-wip-harriscr-cbt-refactor-2024-10-17-0839-distro-default-smithi/
https://pulpito.ceph.com/harriscr-2025-02-07_08:01:53-crimson-rados:perf-wip-harriscr-cbt-refactor-2024-10-17-0839-distro-default-smithi/

All passed

lee-j-sanders

I've tested this change out and it is doing what we expect.
Replace iodepth with total_iodepth in the workloads section of the YAML and it just works :)

Functionally this works great, not sure if we need to add unit tests for total_iodepth though?

See comment

lee-j-sanders · 2025-02-12T14:35:23Z

I'm wondering if there should be a test added to test_bm_librbdfio.py to test:

test_valid_total_iodepth()

and I'm wondering why the test didn't with the new total_iodepth ?
Maybe the unit tests don't test the workloads section enough?

perezjosibm · 2025-02-13T11:26:49Z

Probably more flexible if using .fio workload files generated dynamically (as we do with the gnuplot scripts generated by fio_parse_jsons.py), here is a snippet of a .fio file (notice the group option to combine the results from several volumes into a single overall result):

[global]

bs=4K
iodepth=${IO_DEPTH}
ioengine=libaio
group_reporting=1
time_based
runtime=5m
thread=1
ramp_time=30s
rw=randwrite
                            
[nvme0n1p2]
filename=/dev/nvme0n1p2
size=3550G

[nvme3n1p2]
filename=/dev/nvme3n1p2
size=3550G

[nvme4n1p2]
filename=/dev/nvme4n1p2
size=3550G

In this example, the number of jobs are implicit, FIO creates a job (with name indicated by the label enclosed by brackets) per volume/filename.

lee-j-sanders · 2025-02-14T15:32:03Z

After a discussion with Chris about the unit test warnings. I'm happy with this PR now.

lee-j-sanders

I've been using these changes for over a week now, all good.

harriscr · 2025-03-03T10:46:07Z

Made some changes based on comments in PR 327

Farther testing:

Using the following yaml (workload section below:

workloads:
      precondition:
        jobname: 'precond1rw'
        mode: 'randwrite'
        precond: True
        op_size: 65536
        numjobs: [ 1 ]
        iodepth: [ 2 ]
        monitor: False # whether to run the monitors along the test

      seq32kwrite:
        jobname: 'seqwrite'
        mode: 'write'
        op_size: 32768
        numjobs: [ 1 ]
        total_iodepth: [ 1, 2, 24, 36]
        pre_workload_script: "/home/harriscr/test.sh"
      seq32kread:
        jobname: 'seqread'
        mode: 'read'
        op_size: 32768
        numjobs: [ 1 ]
        iodepth: [ 4 ]

gives (extra debug added to show values used:

10:36:37 - INFO     - cbt      - Running rbd fio precondition test, mode randwrite
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is iodepth
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 2, 1: 2, 2: 2, 3: 2, 4: 2, 5: 2, 6: 2, 7: 2}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randwrite --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.0

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randwrite --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=65536B --iodepth=2 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/randwrite_65536/iodepth-002/numjobs-001/output.1


10:36:37 - INFO     - cbt      - Running rbd fio seq32kwrite test, mode write
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is total_iodepth
10:36:37 - WARNING  - cbt      - The total iodepth requested: 1 is less than 1 per volume (8)
10:36:37 - WARNING  - cbt      - Number of volumes per client will be reduced from 8 to 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 1}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-001/numjobs-001/output.0

10:36:37 - WARNING  - cbt      - The total iodepth requested: 2 is less than 1 per volume (8)
10:36:37 - WARNING  - cbt      - Number of volumes per client will be reduced from 8 to 2
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 1, 1: 1}

10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 3, 1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=3 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.0

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=3 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-024/numjobs-001/output.1


10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 5, 1: 5, 2: 5, 3: 5, 4: 4, 5: 4, 6: 4, 7: 4}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=5 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.1

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7


10:36:37 - INFO     - cbt      - Running rbd fio seq32kread test, mode read
10:36:37 - INFO     - cbt      - CHDEBUG: staring on job 1
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth_key is iodepth
10:36:37 - INFO     - cbt      - CHDEBUG: iodepth per volume is: {0: 4, 1: 4, 2: 4, 3: 4, 4: 4, 5: 4, 6: 4, 7: 4}

10:42:17 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=read --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.0

10:36:37 - INFO     - cbt      - == Workloads completed ==

which is as expected

harriscr · 2025-03-04T11:21:02Z

Some farther testing to make sure that the iodepth value used in the results directory structure is the total iodepth value or iodepth value, depending on which one is set.

total_iodepth - no workload
11:12:27 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-3 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.3

11:12:27 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-6 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=1 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.6

11:12:32 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=128 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.0

11:12:32 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=128 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-1024/write/output.7


iodepth - no workload
11:14:43 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=7 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.0

11:14:43 - INFO     - cbt      - CHDEBUG: fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=4194304B --iodepth=7 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-008/iodepth-007/write/output.7


workload - total iodepth
11:16:31 - INFO     - cbt      - CHDEBUG: workload fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-7 --invalidate=0 --rw=write --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/write_32768/iodepth-036/numjobs-001/output.7


workload - iodepth
11:16:31 - INFO     - cbt      - CHDEBUG: workload fio command is sudo /usr/local/bin/fio --ioengine=rbd --clientname=admin --pool=rbd_replicated --rbdname=cbt-librbdfio-`hostname -f`-2 --invalidate=0 --rw=read --output-format=json --runtime=90 --time_based --ramp_time=30 --numjobs=1 --direct=1 --bs=32768B --iodepth=4 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0  > /tmp/cbt/00000000/LibrbdFio/read_32768/iodepth-004/numjobs-001/output.2

harriscr · 2025-03-04T14:08:13Z

Moved to PR 330

harriscr added the enhancement label Jan 30, 2025

harriscr self-assigned this Jan 30, 2025

harriscr force-pushed the ch_wip_total_iodepth branch 2 times, most recently from 1510b6b to dc1a795 Compare February 7, 2025 10:04

harriscr marked this pull request as ready for review February 11, 2025 09:01

harriscr requested review from lee-j-sanders and markhpc February 11, 2025 09:01

lee-j-sanders previously approved these changes Feb 11, 2025

View reviewed changes

lee-j-sanders mentioned this pull request Feb 12, 2025

Add pre_condtime for preconditioning and allow multiple randrw workloads within the same YAML #326

Closed

lee-j-sanders mentioned this pull request Feb 17, 2025

Allow time within a test to override global time setting in YAML and allow multiple mixed read/write workloads within a YAML #327

Open

harriscr requested a review from lee-j-sanders February 18, 2025 10:14

lee-j-sanders approved these changes Feb 18, 2025

View reviewed changes

harriscr mentioned this pull request Feb 24, 2025

Fixes for post_processing code for the total_iodepth feature #328

Merged

harriscr requested a review from perezjosibm March 3, 2025 11:27

harriscr force-pushed the ch_wip_total_iodepth branch 3 times, most recently from a42ab90 to b03f2f9 Compare March 4, 2025 11:17

harriscr force-pushed the ch_wip_total_iodepth branch from b03f2f9 to 3e19d32 Compare March 4, 2025 11:30

harriscr closed this Mar 4, 2025

harriscr force-pushed the ch_wip_total_iodepth branch from 3e19d32 to 724a009 Compare March 4, 2025 11:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a total_iodepth option that can be used with librbdfio#324

Adding a total_iodepth option that can be used with librbdfio#324
harriscr wants to merge 0 commit intoceph:masterfrom
harriscr:ch_wip_total_iodepth

harriscr commented Jan 30, 2025 •

edited

Loading

Uh oh!

harriscr commented Jan 31, 2025 •

edited

Loading

Uh oh!

harriscr commented Feb 6, 2025

Uh oh!

harriscr commented Feb 11, 2025

Uh oh!

lee-j-sanders left a comment •

edited

Loading

Uh oh!

lee-j-sanders commented Feb 12, 2025

Uh oh!

perezjosibm commented Feb 13, 2025

Uh oh!

lee-j-sanders commented Feb 14, 2025

Uh oh!

lee-j-sanders left a comment

Uh oh!

harriscr commented Mar 3, 2025 •

edited

Loading

Uh oh!

harriscr commented Mar 4, 2025

Uh oh!

harriscr commented Mar 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

harriscr commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Regression testing: iodepth=32, volumes_per_client=3

total_iodepth=32, volumes_per_client=3

total_iodepth=2, volumes_per_client=3

Uh oh!

harriscr commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harriscr commented Feb 6, 2025

Testing with workloads:

precondition: iodepth=[2]

total_iodepth=16, volumes=8

total_iodepth=7, volumes=8

Uh oh!

harriscr commented Feb 11, 2025

Uh oh!

lee-j-sanders left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lee-j-sanders commented Feb 12, 2025

Uh oh!

perezjosibm commented Feb 13, 2025

Uh oh!

lee-j-sanders commented Feb 14, 2025

Uh oh!

lee-j-sanders left a comment

Choose a reason for hiding this comment

Uh oh!

harriscr commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harriscr commented Mar 4, 2025

Uh oh!

harriscr commented Mar 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harriscr commented Jan 30, 2025 •

edited

Loading

harriscr commented Jan 31, 2025 •

edited

Loading

lee-j-sanders left a comment •

edited

Loading

harriscr commented Mar 3, 2025 •

edited

Loading