Validations support

README.md

@@ -44,10 +44,20 @@ TBD (PRs welcome!)
 In the coordinator VM, check out this solr-bench repository.
 
 1. `mvn clean compile assembly:single`
-2. `./cleanup.sh && ./stress.sh -c <commit> <config-file>`
+2. `./cleanup.sh && ./stress.sh -c <commit> -f <config-file>`
 
-Example: `./cleanup.sh ./stress.sh -c dfde16a004206cc92e21cc5a6cad9030fbe13c20 suites/stress-facets-local.json`
+Example: `./cleanup.sh ./stress.sh -c dfde16a004206cc92e21cc5a6cad9030fbe13c20 -f suites/stress-facets-local.json`
 
+    Usage: ./stress.sh -c <commit> [-g] [-v] -f <config-file>
+
+| parameter | Required? | Description |
+| ------- | ---------- | --------- |
+|  -f | Required | Configuration file for a suite |
+|  -c | Required | Commit point to run against |
+|  -g | Optional | Generate validations file(s) for all query benchmark tasks |
+|  -v | Optional | Perform validations based on specified validations file(s) |
+
+    Note: Only -g or -v can be specified at a time
 
 #### Available tests
 ```
@@ -66,6 +76,33 @@ Note: This is subject to change
 
 TBD
 
+### Validations
+
+User workflow:
+
+1. Run their suite with `-g` (generate validation) flag. This will generate a file in the `suites/` dir containing a tuple `<query, numFound, facets>`.
+2. Manually verify the generated file (`suites/validations-<testname>-docs-<docs>-queries-<numQueries>.json`).
+3. The validations file can used for validations in subsequent runs.
+
+#### Using validations with a generated file
+
+    a. Add a `"validation": "<file>"` parameter in the query-benchmark definition,
+    b. Run `stress.sh` with a `-v` (validate) flag. It will use the validations file in the query benchmark task and report number of successful and failed queries.
+
+The results would be reported in the query benchmark task, for example (500 validations succeeded, 0 failed):
+
+    {
+       threads=1,
+       50th=8.2426775,
+       90th=18.409747399999993,
+       95th=28.618552849999993,
+       mean=16.281752914583333,
+       total-queries=480,
+       total-time=14085,
+       validations-succeeded=500,
+       validations-failed=0
+    }
+
 ### Visualization
 
 To select/process the test results of specific branch or branch comparisons:

archived-suites/multitenant.json

@@ -123,7 +123,7 @@
       "threadpool": "collection-api-threadpool",
       "wait-for": "task1",
       "mode": "async",
-      "validations": ["all-replicas-active"]
+      "validations": ["all-replicas-active"] // NOCOMMIT: reimplement
     },
     "task2c": {
       "description": "MODIFYCOLLECTION at random",

backtest.sh

@@ -16,8 +16,8 @@ do echo; echo "Running $commit"
         echo "Trying commit: $commit"
         ./cleanup.sh
         ./stress.sh -c $commit -v suites/$testnamefile
-	python3 graph-scripts/generate_graph_json.py -r suites/results/cluster-test -r suites/results/prs-vs-nonprs -r suites/results/stress-facets-local -b branch_9x...branch_9_1
-	cp graph/* /var/www/html
+       	python3 graph-scripts/generate_graph_json.py -r suites/results/cluster-test -r suites/results/prs-vs-nonprs -r suites/results/stress-facets-local -b branch_9x...branch_9_1
+	      cp graph/* /var/www/html
     fi
 
   done

graph-scripts/generate_graph_json.py

@@ -79,10 +79,10 @@ def parse_benchmark_results(meta_props):
         try:
             test_run_dir = meta_prop["test_run_dir"]
             # commit_date = time.gmtime(int(props["committed_date"]))
-            commit_hash = meta_prop["commit"]
-            commit_date = int(meta_prop["commit_date"])
-            commit_msg = meta_prop["message"]
-            test_date = meta_prop["test_date"]
+            commit_hash = meta_prop.get("commit", '')
+            commit_date = int(meta_prop.get("commit_date", '0'))
+            commit_msg = meta_prop.get("message", '')
+            test_date = meta_prop.get("test_date", '')
 
             result_path = os.path.join(test_run_dir, "results.json")
             json_results = json.load(open(result_path))

graph/graph.css

@@ -0,0 +1,40 @@
+div.graph { width: 100%; height: 400px; }
+#sideBar {
+  box-shadow: inset -25px 0px 25px -25px rgba(0, 0, 0, 0.45);
+}
+#sideBar .item {
+    width: 100%;
+    text-align: center;
+    margin: auto;
+    min-height: 50px;
+    padding: 5px;
+    box-sizing: border-box;
+}
+body {
+    margin: 0px;
+    font-family: arial,sans-serif;
+}
+.clickable:hover {
+    cursor: pointer;
+
+}
+.clickable:hover, .selected {
+    color: rgba(95, 158, 160, 0.9);
+}
+.table-row {
+    display: table-row;
+}
+.table .table-row:nth-child(even) {
+    background-color: #f2f2f2;
+}
+.table .table-row:nth-child(odd) {
+    background-color: #ffffff;
+}
+div.section {
+   background-image: var(--section-background-image);
+   box-shadow: inset 0 0 2px white, 0 0 3px rgba(0, 0, 0, 0.5);
+   border-radius: 0.5em 0.5em 0.5em 0.5em;
+   margin: 10px;
+   padding: 10px 10px 20px 10px;
+   box-sizing: border-box;
+}

graph/graph.html

@@ -1,31 +1,6 @@
 <html>
     <head>
-        <style>
-            div.graph { width: 100%; height: 400px; }
-            #sideBar {
-              box-shadow: inset -25px 0px 25px -25px rgba(0, 0, 0, 0.45);
-            }
-            #sideBar .item {
-                width: 100%;
-                text-align: center;
-                margin: auto;
-                min-height: 50px;
-                padding: 5px;
-                box-sizing: border-box;
-            }
-            body {
-                margin: 0px;
-                font-family: arial,sans-serif;
-            }
-            .clickable:hover {
-	            cursor: pointer;
-
-            }
-            .clickable:hover, .selected {
-                color: rgba(95, 158, 160, 0.9);
-            }
-}
-        </style>
+        <link rel="stylesheet" type="text/css" href="graph.css">
         <script type="text/javascript" src="https://www.google.com/jsapi"></script>
         <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.1/jquery.min.js"></script>
         <script type="text/javascript" src="graph-data.js"></script>