Merge upstream #23
Merged
Merge upstream #23
+1,354
−160
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch locks shellcheck severity level to "warning". All "info" level messages will be ignored. Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Explain why SC3043 is ignored in this repository Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
bootrr official upstream repository moved to https://github.com/linux-msm/bootrr. On top of that bootrr now executes tests based on the DTB. It is still possible to overwrite the detected DTB but it's not a recommended way of running the test. On the systems without package manager the bootrr package should be preinstalled before running the test. Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
The test uses alsa-utils tools aplay and arecord to find whether any playback and record devices are available in the system. If the test runs on an OS without package manager and no alsa-utils available it will fall back to look for character devices named "controlCX" where X is a number. This is the same approach that alsa-utils uses. The default path to look for devices is /dev/snd/ Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
distutils package was removed from python3.12. This patch removes distutils from test-runner and replaces it with shutil. Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
instead of the "Auth-Token" way as suggested here:
https://squad.readthedocs.io/en/latest/intro.html#submitting-results
Reported-by: Benjamin Copeland <benjamin.copeland@linaro.org>
Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
When running multiple ltp suites without rebooting, the following error is shown when running kirk the second time: 'kirk: error: JSON report file already exists: /tmp/kirk-report.json' Remove the kirk-report.json file after parsing the file. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
For each test, send-to-lava will look into a results directory for files
named ${test}.log. If that file is found, then its contents will be
pushed to lava as the log snippet corresponding to that test.
The result directory can be passed as the second command line argument.
If omitted, the same directory as the results file will be used.
Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
kirk produces a JSON results file with the logs already collected, what's very convenient. Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
This will make it possible to run multiple iterations to gather multiple min/max/avg values. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
This will make it possible to run multiple iterations to gather multiple min/max/avg values. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
This will make it possible to run multiple iterations to gather multiple min/max/avg values. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
This will make it possible to run multiple iterations to gather multiple min/max/avg values. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
This will make it possible to run multiple iterations to gather multiple min/max/avg values. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Currently kselftest logs for failures aren't displaying full logs. This change creates log files for each failure and tracks the results. Signed-off-by: Theodore Grey <theodore.grey@linaro.org>
If iterations > 1, the tests will be named 'iteration-<number>' where number = iterations that was passed in, so all the other iterations will be overwritten. mening if iterations=3, then it will look like this: iteration-3-t0-min-latency: pass And no iterations-1-* or iterations-2-*. Solve this by manipulating a temporary file and extend the result file with the temporary file. Fixes: 9ebdd88 ("automated: linux: pmqtest: add iterations") Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
If iterations > 1, the tests will be named 'iteration-<number>' where number = iterations that was passed in, so all the other iterations will be overwritten. mening if iterations=3, then it will look like this: iteration-3-t0-min-latency: pass And no iterations-1-* or iterations-2-*. Solve this by manipulating a temporary file and extend the result file with the temporary file. Fixes: 970fdbc ("automated: linux: pi-stress: add iterations") Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
If iterations > 1, the tests will be named 'iteration-<number>' where number = iterations that was passed in, so all the other iterations will be overwritten. mening if iterations=3, then it will look like this: iteration-3-t0-min-latency: pass And no iterations-1-* or iterations-2-*. Solve this by manipulating a temporary file and extend the result file with the temporary file. Fixes: 2c35bc0 ("automated: linux: signaltest: add iterations") Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
automated: linux: pmqtest: parse: fix iteration parsing If iterations > 1, the tests will be named 'iteration-<number>' where number = iterations that was passed in, so all the other iterations will be overwritten. mening if iterations=3, then it will look like this: iteration-3-t0-min-latency: pass And no iterations-1-* or iterations-2-*. Solve this by manipulating a temporary file and extend the result file with the temporary file. Fixes: 8ed13f7 ("automated: linux: rt-migrate-test: add iterations") Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
If iterations > 1, the tests will be named 'iteration-<number>' where number = iterations that was passed in, so all the other iterations will be overwritten. mening if iterations=3, then it will look like this: iteration-3-t0-min-latency: pass And no iterations-1-* or iterations-2-*. Solve this by manipulating a temporary file and extend the result file with the temporary file. Fixes: 24a05ce ("automated: linux: cyclicdeadline: add iterations") Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
It's very unlikely that someone is going to look into passing tests. This helps reduce even more the demand for storage of test results. Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
This is an incredibly basic test, shipped as part of alsa-utils, which plays and records a tone then uses a FFT to verify that the tone appears sufficiently cleanly in the output. It requires that the system under test have previously been set up with a loopback audio path, either within the card or via cables. Signed-off-by: Mark Brown <broonie@kernel.org>
…o#560) After the following lava change: https://gitlab.com/lava/lava/-/merge_requests/2734 the kisscache url is exported by LAVA via the HTTP_CACHE variable, it's better to check and use the varible in the scripts as well, to avoid being passed from the LAVA job definition. Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
This avoids losing signals if they happen to coincide with kernel messages. Signed-off-by: Antonio Terceiro <antonio.terceiro@linaro.org>
This patch fixes the error caused by deprecated setup.py in mkdocs_plugin. The setup.py is rewritten into pyproject.toml according to the docs: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-spec Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
with the tests.zip package provided, which is generated by the normal gki kernel build Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
Add support for parallel test execution with the new LTP test runner, kirk. This update introduces the KIRK_WORKERS variable, which allows you to configure the number of worker processes used for running tests in parallel. This can significantly reduce test runtime on systems with many CPU cores. Suggested-by: Mark Brown <broonie@kernel.org> Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Enhance the cyclicdeadline test to automatically determine a PASS or FAIL result based on collected max-latency measurements. This change: - Adds logic to parse the max-latency values from output. - Calculates the minimum observed latency. - Sets a threshold at 110% of the minimum latency. - Counts the number of runs that exceed this threshold. - Fails the test if more than half the iterations exceed the threshold. This enables automated evaluation of cyclicdeadline results without manual inspection, making the test more suitable for continuous integration environments. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Adds a `pthreshold` field to the cyclicdeadline test configuration. The `pthreshold` parameter allows users to define a performance threshold for the maximum acceptable latency, enabling more flexible validation criteria. If the maximum latency exceeds the configured `pthreshold`, the test will fail, providing an additional mechanism for automated performance evaluation. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Enhance the pi-stress test to automatically determine a PASS or FAIL result based on collected inversion measurements. This change: - Adds logic to parse the inversion values from output. - Calculates the minimum observed inversion. - Sets a threshold at 110% of the minimum inversion. - Counts the number of runs that exceed this threshold. - Fails the test if more than half the iterations exceed the threshold. This enables automated evaluation of pi-stress results without manual inspection, making the test more suitable for continuous integration environments. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Adds a `pthreshold` field to the pi-stress test configuration. The `pthreshold` parameter allows users to define a performance threshold for the maximum acceptable latency, enabling more flexible validation criteria. If the maximum latency exceeds the configured `pthreshold`, the test will fail, providing an additional mechanism for automated performance evaluation. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Add a helper function to scan dmesg after each module insert and remove, looking for common kernel error signatures like "BUG:", "Oops:", "WARNING:", and "Call Trace:". This improves early detection of subtle regressions during module load/unload testing that may otherwise go unnoticed. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Introduce a helper to verify that a module is properly unloaded after modprobe --remove. If a module remains listed in lsmod, it indicates a stuck reference count and a possible kernel bug. This improves the robustness of module load/unload testing by catching incomplete removals. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Add a final memory leak detection pass using kmemleak if it is enabled in the running kernel. Trigger a kmemleak scan after all module load/unload cycles and report any leaked memory objects. This helps detect regressions involving memory leaks caused by module insert/remove operations. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Fix get_modules_list to properly handle the case where both MODULES_LIST and MODULES_SUBDIRS are empty. Previously, an empty MODULES_SUBDIRS resulted in an invalid grep expression, causing no modules to be selected. With this change, the script defaults to using all modules listed in modules.order if no specific list or subdirectory is provided. Also ensure the skiplist is optional and handled safely. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
The pi-stress results shouldn't be based on the minimum value, it should be -X% from the max_inversion. Anything less than threshold is a fail. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Previously, the pi-stress script dynamically computed the baseline (minimum inversion value) across all iterations. This patch adds support for a new `-x` option that lets users manually specify a fixed `user_baseline` value. Using a fixed baseline is preferred over percentage-based thresholds in this context, as it offers more consistent and predictable evaluation—especially in controlled benchmarking environments or when comparing results across different systems and runs. Percentages can be misleading if the dynamic baseline fluctuates due to noise or system conditions. This change improves flexibility and repeatability in post-processing and performance analysis. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Previously, the cyclicdeadline script dynamically computed the minimum latency value across all iterations. This patch introduces a new `-x` option that enables users to specify a fixed `user_baseline` value instead. Using a fixed baseline is preferred over percentage-based thresholds in this context, as it offers more consistent and predictable evaluation—especially in controlled benchmarking environments or when comparing results across different systems and runs. Percentages can be misleading if the dynamic baseline fluctuates due to noise or system conditions. This change improves flexibility and repeatability in post-processing and performance analysis. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
xfail tanslates to "expected failure". This implementation assumes that exit code different than 0 means failure. The patch modifies check_return to accept additional parameter "xfail". If the parameter is set, exit code 0 will be considered "fail" and exit code different than 0 will be considered "pass". Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Remove return that would never be reached after calling error_msg and fix unquoted string variable. Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Add "host" and "none" network options to docker-networking tests. This
allows to cover all simple possibilities when it comes to docker
networking. All tests can be executed in one run by adding -n option to
the script invocation. Example:
docker-networking.sh -n "bridge host"
Test names were appended with ${NETWORK} to indicate which network type
is tested. Default is "bridge".
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Currently the KTAP output parser deletes any '-' it sees in the test name when slugifying it. This means that in a case like the arm64 breakpoints test two distinct tests end up squashed into the same name since the only difference in name is a minus sign: # ok 1 Test size = 1 write offset = 0 watchpoint offset = -1 # ok 3 Test size = 1 write offset = 0 watchpoint offset = 1 becomes: breakpoints_breakpoint_test_arm64_Test_size_1_write_offset_0_watchpoint_offset_1 pass breakpoints_breakpoint_test_arm64_Test_size_1_write_offset_0_watchpoint_offset_1_dup2 pass which isn't ideal. Fix this by converting the '-' to an '_' instead. Signed-off-by: Mark Brown <broonie@kernel.org>
This patch introduces an automated testing setup for tcpreplay-based traffic replay and analysis. The system consists of: 1. `tcpreplay.py`: A test runner that: - Creates and configures a TAP interface - Replays PCAP files using `tcpreplay` - Tracks results and produces a summary for each test - Handles both expected and unexpected results (e.g., xfail -> pass) 2. `generate_pcap.py`: A Scapy-based script to: - Generate a suite of PCAP files for functional and edge case testing - Include both valid packets and malformed/false-positive scenarios - Provide coverage for multiple protocols (TCP, UDP, ICMP, DNS, etc.) - Simulate problematic flows like fragmented packets and invalid flags Highlights: - xfail support: tests expected to fail are counted as 'pass' if they do fail - xpass detection: alerts if a known-broken test unexpectedly succeeds - Easy extensibility for new PCAPs and expectations Example test cases include: - TCP lifecycle (`tcp_basic`, `tcp_full_cycle`) - Bad flag scenarios (`bad_tcp_flags`) - Noise/overlap (`false_positive_overlap`) - Fragmentation and malformed headers Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Add helper functions to log memory usage before and after loading/unloading kernel modules using 'modprobe insert' and 'modprobe remove'. This helps monitor the memory footprint of kernel modules and detect potential memory issues. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
…me (Linaro#573) Currently the number of kirk workers has to be configured explicitly, kirk itself does not support detecting the number of CPUs the system has and test-definitions just passes the value it has straight through to the kirk runner. This is a bit annoying when using templating to share job definitions over multiple device types, requiring explicit per type configuration. Given that a very common case is going to be to use all the CPUs available on the target platform provide support for the magic number of workers "max" which counts the number of CPUs reported by /proc/cpuinfo and uses that as the number of workers. Signed-off-by: Mark Brown <broonie@kernel.org>
Member
Author
|
This is a straight git pull, I didn't make any of the commits. |
By default kirk produces colourised output which results in escape sequences going into the console log. Since the primary use of test-definitions is automated this isn't terribly useful and results in logs that are harder to read. Pass the --no-colors option to disable this output. Signed-off-by: Mark Brown <broonie@kernel.org>
When running a subset of kselftest suites (e.g. kselftest-mm), the existing skipfile logic still applies all entries from skipfile-lkft.yaml, including those unrelated to the selected subsuites. As a result, irrelevant tests (such as those from the breakpoints suite) appear as "skip" in result.json, even though they were not part of the test run. To ensure accurate skip handling, the logic has been updated so that skip entries only apply to tests within the subsuites selected for the current run. Each skip entry is parsed to extract the subsuite name, which is then compared against the entries in TST_CMDFILES. Skip rules are applied only when a match is found, preventing unrelated tests from appearing as "skipped" in the results. This avoids polluting results with unrelated test cases and makes skips more accurate and meaningful. Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
When parsing the JSON output from kirk we capture any logs that were generated for tests that didn't pass. Currently we do this by first invoking jq to generate a list of tests that were run, then for each test invoke jq again to check if the result was a pass or not. If the result wasn't a pass we then invoke jq again to get the log. This is wildly inefficient, we invoke a new copy of jq and parse the entire results file once per test on the test system in order to generate a list of tests that didn't pass (which will hopefully be much shorter than the list of those that fails). Fortunately jq is capable of directly generating the list of non-passing tests so we can do a single parse for the initial filter, update to do that. We still invoke jq again to generate the log files, these are all written into per-test files for reporting, but most of the time most tests should pass. With the existing implementation for the syscalls suite on an Avenger96 the results parsing takes longer than running the actual tests, currently it takes over an hour and 20 minutes for the full job but with this change that time is reduced to a bit under 50 minutes. Approximately 35 minutes of the jobs is taken running the tests so the actual parsing is about twice as fast, there are a lot of failures in these jobs and the benefits should be greater with a higher pass rate. Signed-off-by: Mark Brown <broonie@kernel.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a straight fast forward of the upstream repository which was last synced in January, the main interest for me is in support for parallel execution in LTP and the ALSA BAT test definition.