cluster: Addition of new Scrubbing test by amathuria · Pull Request #246 · ceph/cbt

amathuria · 2021-10-12T14:32:45Z

Signed-off-by: Aishwarya Mathuria amathuri@redhat.com

cluster/ceph.py

Add 2 new tests similar to the background Recovery test for running Scrub load with client IO and the other one for running Scrub, Recovery, and Client load. The Scrub and client IO test performs the following steps: - Create a pool and image to populate scrub objects (scrub pool) - Create scrub thread - Populate the scrub pool with objects using radosbench - Initiate deep-scrub on the scrub pool - Create a second pool and an image in it to run client IO - Initiate fio job on the second image at the same time the deep-scrub starts In the second test, we have an additional recovery pool that is populated after an OSD is marked down and out. Once the pool is populated we mark the OSD up and in which starts backfill. At the same time, we begin deep-scrub on the scrub pool and client IO. Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

sseshasa

For the scrubbing test, the name of the states like 'osdout' and 'osdin' can confuse other readers. Can this be changed to more appropriate states like for e.g., 'fillscrubpool', 'initscrub' to make it more intuitive?

cluster/ceph.py

sseshasa

I forgot to add this. The scrub tests can be factored out into a separate file under the same directory to avoid ceph.py from bloating up.

amathuria · 2022-08-23T13:46:35Z

For the scrubbing test, the name of the states like 'osdout' and 'osdin' can confuse other readers. Can this be changed to more appropriate states like for e.g., 'fillscrubpool', 'initscrub' to make it more intuitive?

Makes sense, I'll work on that and shift the scrub tests into a separate file.

sseshasa · 2022-09-07T06:06:50Z

benchmark/librbdfio.py


        common.sync_files('%s/*' % self.run_dir, self.out_dir)

+        if 'scrubbing_test' in self.cluster.config:


Can this just be called 'scrub_test'. I think all instances of 'scrubbing' in function names and other places can be replaced with 'scrub'. This makes the code much neater and consistent to read. What do you think?

sseshasa · 2022-09-07T06:07:08Z

benchmark/librbdfio.py

    def recovery_callback_background(self):
        logger.info('Recovery thread completed!')

+    def scrubbing_callback(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:13Z

client_endpoints/ceph_client_endpoints.py

                rbd_name = '%s-%s' % (self.pool, self.get_rbd_name(node, ep_num))
                self.cluster.mkimage(rbd_name, self.endpoint_size, self.pool, self.data_pool, self.order)

+    def create_rbd_scrubbing(self):


'create_rbd_scrub_pool()' sounds better.

sseshasa · 2022-09-07T06:07:25Z

client_endpoints/client_endpoints.py

    def create_recovery_image(self):
        pass
+
+    def create_scrubbing_image(self):


'scrubbing' to 'scrub' here and in the other *_endpoint.py files.

sseshasa · 2022-09-07T06:07:34Z

cluster/ceph.py

 import json

 from .cluster import Cluster
+from .scrub_tests import ScrubbingTestThreadBackground, ScrubRecoveryThreadBackground


Looks much better now. To be consistent you can consider changing class name from "ScrubbingTestThreadBackground" to "ScrubTestThreadBackground".

sseshasa · 2022-09-07T06:07:43Z

cluster/ceph.py


        return ret

+    def log_scrubbing_stats(self, scrubstatsfile=None, pgid=None):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:51Z

cluster/ceph.py

            self.rt = RecoveryTestThreadBackground(rt_config, self, callback, self.stoprequest, self.haltrequest, self.startiorequest)
        self.rt.start()

+    def create_scrubbing_test(self, run_dir, callback):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:07:58Z

cluster/ceph.py

                break
            self.rt.join(1)

+    def maybe_populate_scrubbing_pool(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:08:04Z

cluster/ceph.py

+            common.pdsh(settings.getnodes('head'), 'sudo %s -p %s bench %s write -b %s --max-objects %s --no-cleanup' % (self.rados_cmd, self.scrub_pool_name, self.prefill_scrub_time, self.prefill_scrub_object_size, self.prefill_scrub_objects)).communicate()
+            #self.check_health()
+
+    def initiate_scrubbing(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:08:10Z

cluster/ceph.py

+        logger.info("Initiating scrub on pool %s" % self.scrub_pool_name)
+        common.pdsh(settings.getnodes('head'), '%s osd pool deep-scrub %s' % (self.ceph_cmd, self.scrub_pool_name)).communicate()
+
+    def wait_scrubbing_done(self):


'scrubbing' to 'scrub'

sseshasa · 2022-09-07T06:09:11Z

benchmark/fio.py

    def recovery_callback_background(self):
        logger.info('Recovery thread completed!')

+    def scrubbing_callback(self):


'scrubbing' to 'scrub'

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

amathuria force-pushed the wip-cbt-addscrubtest branch 3 times, most recently from 2071234 to 52e8dae Compare February 8, 2022 15:06

amathuria marked this pull request as ready for review February 8, 2022 15:07

amathuria requested review from markhpc and sseshasa February 10, 2022 14:11

sseshasa reviewed May 9, 2022

View reviewed changes

cluster/ceph.py Outdated Show resolved Hide resolved

amathuria force-pushed the wip-cbt-addscrubtest branch from 52e8dae to 9b2e21a Compare August 16, 2022 14:23

sseshasa requested changes Aug 17, 2022

View reviewed changes

sseshasa reviewed Aug 17, 2022

View reviewed changes

cluster/ceph.py Outdated Show resolved Hide resolved

sseshasa reviewed Aug 17, 2022

View reviewed changes

amathuria force-pushed the wip-cbt-addscrubtest branch from 1a2c63c to da9ec6d Compare September 6, 2022 14:13

sseshasa reviewed Sep 7, 2022

View reviewed changes

cluster/ceph.py Outdated

return ret

def log_scrubbing_stats(self, scrubstatsfile=None, pgid=None):

Copy link

Contributor

sseshasa Sep 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'scrubbing' to 'scrub'

sseshasa reviewed Sep 7, 2022

View reviewed changes

cluster/ceph.py Outdated

break

self.rt.join(1)

def maybe_populate_scrubbing_pool(self):

Copy link

Contributor

sseshasa Sep 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'scrubbing' to 'scrub'

sseshasa reviewed Sep 7, 2022

View reviewed changes

Moving out scrub tests to a separate file

fd4de2d

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

amathuria force-pushed the wip-cbt-addscrubtest branch from da9ec6d to fd4de2d Compare September 20, 2022 11:30

cluster: Add option for shallow scrub for scrub tests

4291252

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>


		common.sync_files('%s/*' % self.run_dir, self.out_dir)

		if 'scrubbing_test' in self.cluster.config:


		return ret

		def log_scrubbing_stats(self, scrubstatsfile=None, pgid=None):

Conversation

amathuria commented Oct 12, 2021

Uh oh!

Uh oh!

sseshasa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sseshasa left a comment

Choose a reason for hiding this comment

Uh oh!

amathuria commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

amathuria commented Aug 23, 2022 •

edited

Loading