From 1a948286c4a05b9358ea6f7ab772d99bf2906308 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:18:07 +0100 Subject: [PATCH 1/9] Remove all references to strict --- features/docs/cli/dry_run.feature | 25 +------ features/docs/cli/retry_failing_tests.feature | 20 +----- features/docs/cli/strict_mode.feature | 70 ------------------- .../extending_cucumber/custom_filter.feature | 3 +- .../docs/formatters/junit_formatter.feature | 12 ++-- .../docs/formatters/rerun_formatter.feature | 2 +- .../parameter_types.feature | 2 +- lib/cucumber/cli/configuration.rb | 6 -- lib/cucumber/cli/options.rb | 16 ----- lib/cucumber/configuration.rb | 5 -- lib/cucumber/formatter/console_issues.rb | 4 +- lib/cucumber/formatter/fail_fast.rb | 2 +- lib/cucumber/formatter/junit.rb | 6 +- lib/cucumber/formatter/pretty.rb | 2 +- lib/cucumber/formatter/rerun.rb | 30 +------- lib/cucumber/multiline_argument/data_table.rb | 12 ++-- lib/cucumber/runtime.rb | 2 +- spec/cucumber/cli/configuration_spec.rb | 18 ----- spec/cucumber/formatter/fail_fast_spec.rb | 9 --- spec/cucumber/formatter/rerun_spec.rb | 61 +++++----------- .../multiline_argument/data_table_spec.rb | 6 +- 21 files changed, 50 insertions(+), 263 deletions(-) delete mode 100644 features/docs/cli/strict_mode.feature diff --git a/features/docs/cli/dry_run.feature b/features/docs/cli/dry_run.feature index 8d8f2585e4..c12cfb9e91 100644 --- a/features/docs/cli/dry_run.feature +++ b/features/docs/cli/dry_run.feature @@ -39,35 +39,14 @@ Feature: Dry Run And output should be valid NDJSON And the output should contain NDJSON with key "status" and value "SKIPPED" - Scenario: In strict mode - Given a file named "features/test.feature" with: - """ - Feature: test - Scenario: - Given this step fails - """ - And the standard step definitions - When I run `cucumber --dry-run --strict --publish-quiet` - Then it should pass with exactly: - """ - Feature: test - - Scenario: # features/test.feature:2 - Given this step fails # features/step_definitions/steps.rb:4 - - 1 scenario (1 skipped) - 1 step (1 skipped) - - """ - - Scenario: In strict mode with an undefined step + Scenario: With an undefined step Given a file named "features/test.feature" with: """ Feature: test Scenario: Given this step is undefined """ - When I run `cucumber --dry-run --strict` + When I run `cucumber --dry-run` Then it should fail with: """ Feature: test diff --git a/features/docs/cli/retry_failing_tests.feature b/features/docs/cli/retry_failing_tests.feature index 0e4dd970a8..68b43203ff 100644 --- a/features/docs/cli/retry_failing_tests.feature +++ b/features/docs/cli/retry_failing_tests.feature @@ -67,26 +67,8 @@ Feature: Retry failing tests Solid ✓ """ - Scenario: Flaky scenarios gives exit code zero in non-strict mode + Scenario: Flaky scenarios gives non-zero exit code When I run `cucumber -q --retry 2 --format summary` - Then it should pass with: - """ - - - 3 scenarios (2 flaky, 1 passed) - """ - - Scenario: Flaky scenarios gives exit code zero in non-strict mode even when failing fast - When I run `cucumber -q --retry 2 --fail-fast --format summary` - Then it should pass with: - """ - - - 3 scenarios (2 flaky, 1 passed) - """ - - Scenario: Flaky scenarios gives non-zero exit code in strict mode - When I run `cucumber -q --retry 2 --format summary --strict` Then it should fail with: """ Flaky Scenarios: diff --git a/features/docs/cli/strict_mode.feature b/features/docs/cli/strict_mode.feature deleted file mode 100644 index 407d238795..0000000000 --- a/features/docs/cli/strict_mode.feature +++ /dev/null @@ -1,70 +0,0 @@ -Feature: Strict mode - - Using the `--strict` flag will cause cucumber to fail unless all the - step definitions have been defined. - - Background: - Given a file named "features/missing.feature" with: - """ - Feature: Missing - Scenario: Missing - Given this step passes - """ - And a file named "features/pending.feature" with: - """ - Feature: Pending - Scenario: Pending - Given this step is pending - """ - - Scenario: Fail with --strict due to undefined step - When I run `cucumber -q features/missing.feature --strict` - Then it should fail with: - """ - Feature: Missing - - Scenario: Missing - Given this step passes - Undefined step: "this step passes" (Cucumber::Core::Test::Result::Undefined) - features/missing.feature:3:in `this step passes' - - Undefined Scenarios: - cucumber features/missing.feature:2 - - 1 scenario (1 undefined) - 1 step (1 undefined) - """ - - Scenario: Fail with --strict due to pending step - Given the standard step definitions - When I run `cucumber -q features/pending.feature --strict` - Then it should fail with: - """ - Feature: Pending - - Scenario: Pending - Given this step is pending - TODO (Cucumber::Pending) - ./features/step_definitions/steps.rb:3:in `/^this step is pending$/' - features/pending.feature:3:in `this step is pending' - - Pending Scenarios: - cucumber features/pending.feature:2 - - 1 scenario (1 pending) - 1 step (1 pending) - """ - - Scenario: Succeed with --strict - Given the standard step definitions - When I run `cucumber -q features/missing.feature --strict` - Then it should pass with: - """ - Feature: Missing - - Scenario: Missing - Given this step passes - - 1 scenario (1 passed) - 1 step (1 passed) - """ diff --git a/features/docs/extending_cucumber/custom_filter.feature b/features/docs/extending_cucumber/custom_filter.feature index 6f2091c835..7b3a7497d3 100644 --- a/features/docs/extending_cucumber/custom_filter.feature +++ b/features/docs/extending_cucumber/custom_filter.feature @@ -24,6 +24,5 @@ Feature: Custom filter config.filters << MakeAnythingPass.new end """ - When I run `cucumber --strict` + When I run `cucumber` Then it should pass - diff --git a/features/docs/formatters/junit_formatter.feature b/features/docs/formatters/junit_formatter.feature index 23618713d3..190b93104d 100644 --- a/features/docs/formatters/junit_formatter.feature +++ b/features/docs/formatters/junit_formatter.feature @@ -260,8 +260,8 @@ Feature: JUnit output formatter """ - Scenario: pending and undefined steps with strict option should fail - When I run `cucumber --format junit --out tmp/ features/pending.feature --strict` + Scenario: pending and undefined steps should fail + When I run `cucumber --format junit --out tmp/ features/pending.feature` Then it should fail with: """ @@ -331,8 +331,8 @@ can't convert .* into String \(TypeError\) You *must* specify --out DIR for the junit formatter """ - Scenario: strict mode, one feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined - When I run `cucumber --strict --format junit --out tmp/ features/scenario_outline.feature` + Scenario: One feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined + When I run `cucumber --format junit --out tmp/ features/scenario_outline.feature` Then it should fail with: """ @@ -409,8 +409,8 @@ You *must* specify --out DIR for the junit formatter """ - Scenario: strict mode with --expand option, one feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined - When I run `cucumber --strict --expand --format junit --out tmp/ features/scenario_outline.feature --publish-quiet` + Scenario: Testing with --expand option, one feature, one scenario outline, four examples: one passing, one failing, one pending, one undefined + When I run `cucumber --expand --format junit --out tmp/ features/scenario_outline.feature --publish-quiet` Then it should fail with exactly: """ diff --git a/features/docs/formatters/rerun_formatter.feature b/features/docs/formatters/rerun_formatter.feature index 8ad54fc3d2..d8598e1549 100644 --- a/features/docs/formatters/rerun_formatter.feature +++ b/features/docs/formatters/rerun_formatter.feature @@ -96,7 +96,7 @@ Feature: Rerun formatter Given this step passes """ - When I run `cucumber --publish-quiet -f rerun --strict` + When I run `cucumber --publish-quiet -f rerun` Then it should fail with exactly: """ features/mixed.feature:3:6:9 diff --git a/features/docs/writing_support_code/parameter_types.feature b/features/docs/writing_support_code/parameter_types.feature index 9e03c9cd30..fc694cd4a8 100644 --- a/features/docs/writing_support_code/parameter_types.feature +++ b/features/docs/writing_support_code/parameter_types.feature @@ -108,5 +108,5 @@ Feature: Parameter Types expect(name).to eq(@name) end """ - When I run `cucumber features/employees.feature --strict` + When I run `cucumber features/employees.feature` Then it should pass diff --git a/lib/cucumber/cli/configuration.rb b/lib/cucumber/cli/configuration.rb index ba2693218c..2017284de5 100644 --- a/lib/cucumber/cli/configuration.rb +++ b/lib/cucumber/cli/configuration.rb @@ -27,8 +27,6 @@ def parse!(args) @args = args @options.parse!(args) arrange_formats - raise("You can't use both --strict and --wip") if strict.strict? && wip? - set_environment_variables end @@ -44,10 +42,6 @@ def seed Integer(@options[:seed] || rand(0xFFFF)) end - def strict - @options[:strict] - end - def wip? @options[:wip] end diff --git a/lib/cucumber/cli/options.rb b/lib/cucumber/cli/options.rb index ff278a1868..e3969568d6 100644 --- a/lib/cucumber/cli/options.rb +++ b/lib/cucumber/cli/options.rb @@ -132,10 +132,6 @@ def parse!(args) opts.on('-q', '--quiet', 'Alias for --no-snippets --no-source --no-duration --publish-quiet.') { shut_up } opts.on('--no-duration', "Don't print the duration at the end of the summary") { set_option :duration, false } opts.on('-b', '--backtrace', 'Show full backtrace for all errors.') { Cucumber.use_full_backtrace = true } - opts.on('-S', '--[no-]strict', *strict_msg) { |setting| set_strict(setting) } - opts.on('--[no-]strict-undefined', 'Fail if there are any undefined results.') { |setting| set_strict(setting, :undefined) } - opts.on('--[no-]strict-pending', 'Fail if there are any pending results.') { |setting| set_strict(setting, :pending) } - opts.on('--[no-]strict-flaky', 'Fail if there are any flaky results.') { |setting| set_strict(setting, :flaky) } opts.on('-w', '--wip', 'Fail if there are any passing scenarios.') { set_option :wip } opts.on('-v', '--verbose', 'Show the files and features loaded.') { set_option :verbose } opts.on('-g', '--guess', 'Guess best match for Ambiguous steps.') { set_option :guess } @@ -289,13 +285,6 @@ def name_msg ] end - def strict_msg - [ - 'Fail if there are any strict affected results ', - '(that is undefined, pending or flaky results).' - ] - end - def parse_formats(value) formatter, *formatter_options = value.split(',') options_hash = Hash[formatter_options.map { |string| string.split('=') }] @@ -467,10 +456,6 @@ def shut_up @options[:duration] = false end - def set_strict(setting, type = nil) - @options[:strict].set_strict(setting, type) - end - def stdout_formats @options[:formats].select { |_, _, output| output == @out_stream } end @@ -528,7 +513,6 @@ def reverse_merge(other_options) @options[:source] &= other_options[:source] @options[:snippets] &= other_options[:snippets] @options[:duration] &= other_options[:duration] - @options[:strict] = other_options[:strict].merge!(@options[:strict]) @options[:dry_run] |= other_options[:dry_run] @profiles += other_options.profiles diff --git a/lib/cucumber/configuration.rb b/lib/cucumber/configuration.rb index f460ee872d..ab512979c2 100644 --- a/lib/cucumber/configuration.rb +++ b/lib/cucumber/configuration.rb @@ -23,7 +23,6 @@ def self.default_options { autoload_code_paths: %w[features/support features/step_definitions], filters: [], - strict: Cucumber::Core::Test::Result::StrictConfiguration.new, require: [], dry_run: false, fail_fast: false, @@ -114,10 +113,6 @@ def guess? @options[:guess] end - def strict - @options[:strict] - end - def wip? @options[:wip] end diff --git a/lib/cucumber/formatter/console_issues.rb b/lib/cucumber/formatter/console_issues.rb index e93fee8929..9e52978905 100644 --- a/lib/cucumber/formatter/console_issues.rb +++ b/lib/cucumber/formatter/console_issues.rb @@ -14,9 +14,9 @@ def initialize(config, ast_lookup = AstLookup.new(config)) @config.on_event(:test_case_finished) do |event| if event.test_case != @previous_test_case @previous_test_case = event.test_case - @issues[event.result.to_sym] << event.test_case unless event.result.ok?(strict: @config.strict) + @issues[event.result.to_sym] << event.test_case unless event.result.ok? elsif event.result.passed? - @issues[:flaky] << event.test_case unless Core::Test::Result::Flaky.ok?(strict: @config.strict.strict?(:flaky)) + @issues[:flaky] << event.test_case unless Core::Test::Result::Flaky.ok? @issues[:failed].delete(event.test_case) end end diff --git a/lib/cucumber/formatter/fail_fast.rb b/lib/cucumber/formatter/fail_fast.rb index 3ebac8acf7..fab38fd447 100644 --- a/lib/cucumber/formatter/fail_fast.rb +++ b/lib/cucumber/formatter/fail_fast.rb @@ -12,7 +12,7 @@ def initialize(configuration) test_case, result = *event.attributes if test_case != @previous_test_case @previous_test_case = event.test_case - Cucumber.wants_to_quit = true unless result.ok?(strict: configuration.strict) + Cucumber.wants_to_quit = true unless result.ok? elsif result.passed? Cucumber.wants_to_quit = false end diff --git a/lib/cucumber/formatter/junit.rb b/lib/cucumber/formatter/junit.rb index f4f9dba47d..3b314e962c 100644 --- a/lib/cucumber/formatter/junit.rb +++ b/lib/cucumber/formatter/junit.rb @@ -54,7 +54,7 @@ def on_test_step_finished(event) test_step, result = *event.attributes return if @failing_test_step - @failing_test_step = test_step unless result.ok?(strict: @config.strict) + @failing_test_step = test_step unless result.ok? end def on_test_case_finished(event) @@ -111,7 +111,7 @@ def create_output_string(test_case, scenario, result, row_name) scenario_source = @ast_lookup.scenario_source(test_case) keyword = scenario_source.type == :Scenario ? scenario_source.scenario.keyword : scenario_source.scenario_outline.keyword output = "#{keyword}: #{scenario}\n\n" - return output if result.ok?(strict: @config.strict) + return output if result.ok? if scenario_source.type == :Scenario if @failing_test_step @@ -140,7 +140,7 @@ def build_testcase(result, scenario_designation, output) testcase_attributes = get_testcase_attributes(classname, name, duration, filename) @current_feature_data[:builder].testcase(testcase_attributes) do - if !result.passed? && result.ok?(strict: @config.strict) + if !result.passed? && result.ok? @current_feature_data[:builder].skipped @current_feature_data[:skipped] += 1 elsif !result.passed? diff --git a/lib/cucumber/formatter/pretty.rb b/lib/cucumber/formatter/pretty.rb index 97ba8d796d..f47a9418d6 100644 --- a/lib/cucumber/formatter/pretty.rb +++ b/lib/cucumber/formatter/pretty.rb @@ -153,7 +153,7 @@ def attach(src, media_type, filename) private def find_exception_to_be_printed(result) - return nil if result.ok?(strict: options[:strict]) + return nil if result.ok? result = result.with_filtered_backtrace(Cucumber::Formatter::BacktraceFilter) exception = result.failed? ? result.exception : result diff --git a/lib/cucumber/formatter/rerun.rb b/lib/cucumber/formatter/rerun.rb index bbf7b30818..39193c156c 100644 --- a/lib/cucumber/formatter/rerun.rb +++ b/lib/cucumber/formatter/rerun.rb @@ -21,37 +21,17 @@ def output_envelope(envelope) def finish_report @query.find_all_test_case_started.each do |test_case| status = @query.find_most_severe_test_step_result_by(test_case).status + # RULE: Don't log test cases without a pickle (Unsure what these could be?) pickle = @query.find_pickle_by(test_case) next if pickle.nil? - # RULE: (Configuration specific) - # -> If the test case has already been logged (And so we're retrying), we remove prior references of failures - if passing?(test_case) && !rerun_flaky_tests? + # RULE: If the test case has already been logged as a failure (And we're retrying), remove prior reference of failure + if passing?(test_case) uri_and_location_hash[pickle.uri].delete(pickle.location.line) next end - # RULE: (Configuration specific - to be amended once CCK conformance is finalised) - # -> If the strict configuration permits the result - handle it accordingly - if status == 'UNDEFINED' && !@config.strict.strict?(:undefined) - Cucumber.deprecate( - 'The strict configuration in cucumber is going away and moving towards a standardised set of behaviours', - '--strict=undefined', - '12.0.0' - ) - next - end - - if status == 'PENDING' && !@config.strict.strict?(:pending) - Cucumber.deprecate( - 'The strict configuration in cucumber is going away and moving towards a standardised set of behaviours', - '--strict=pending', - '12.0.0' - ) - next - end - # RULE: Passing test cases are not considered failures (Don't log these) next if passing?(test_case) @@ -79,10 +59,6 @@ def uri_and_location_hash @uri_and_location_hash ||= Hash.new { |hash, key| hash[key] = Set.new } end - def rerun_flaky_tests? - @config.strict.strict?(:flaky) - end - def passing?(test_case_started) most_severe_test_step_result = @query.find_most_severe_test_step_result_by(test_case_started) most_severe_test_step_result.status == Cucumber::Messages::TestStepResultStatus::PASSED diff --git a/lib/cucumber/multiline_argument/data_table.rb b/lib/cucumber/multiline_argument/data_table.rb index c01a1624ca..8fcbb5c436 100644 --- a/lib/cucumber/multiline_argument/data_table.rb +++ b/lib/cucumber/multiline_argument/data_table.rb @@ -77,7 +77,7 @@ def eof; end # This is a Hash being initialized with a default value of a Hash # DO NOT REFORMAT TO REMOVE {} - Ruby 3.4+ will interpret these as keywords and cucumber will not work - NULL_CONVERSIONS = Hash.new({ strict: false, proc: ->(cell_value) { cell_value } }).freeze + NULL_CONVERSIONS = Hash.new({ proc: ->(cell_value) { cell_value } }).freeze # @param data [Core::Test::DataTable] the data for the table # @param conversion_procs [Hash] see map_column @@ -265,9 +265,7 @@ def map_headers(mappings = {}, &block) # Returns a new Table with an additional column mapping. # # Change how #hashes converts column values. The +column_name+ argument identifies the column - # and +conversion_proc+ performs the conversion for each cell in that column. If +strict+ is - # true, an error will be raised if the column named +column_name+ is not found. If +strict+ - # is false, no error will be raised. Example: + # and +conversion_proc+ performs the conversion for each cell in that column. # # Given /^an expense report for (.*) with the following posts:$/ do |table| # posts_table = posts_table.map_column('amount') { |a| a.to_i } @@ -276,9 +274,9 @@ def map_headers(mappings = {}, &block) # end # end # - def map_column(column_name, strict: true, &conversion_proc) + def map_column(column_name, &conversion_proc) conversion_procs = @conversion_procs.dup - conversion_procs[column_name.to_s] = { strict: strict, proc: conversion_proc } + conversion_procs[column_name.to_s] = { proc: conversion_proc } self.class.new(Core::Test::DataTable.new(raw), conversion_procs, @header_mappings.dup, @header_conversion_proc) end @@ -467,7 +465,7 @@ def create_cell_matrix(ast_table) def convert_columns! @conversion_procs.each do |column_name, conversion_proc| - verify_column(column_name) if conversion_proc[:strict] + verify_column(column_name) end cell_matrix.transpose.each do |col| diff --git a/lib/cucumber/runtime.rb b/lib/cucumber/runtime.rb index 9a43289450..63ac2c64c7 100644 --- a/lib/cucumber/runtime.rb +++ b/lib/cucumber/runtime.rb @@ -88,7 +88,7 @@ def failure? if @configuration.wip? summary_report.test_cases.total_passed.positive? else - !summary_report.ok?(strict: @configuration.strict) + !summary_report.ok? end end diff --git a/spec/cucumber/cli/configuration_spec.rb b/spec/cucumber/cli/configuration_spec.rb index 22bd934e18..021dbadbd1 100644 --- a/spec/cucumber/cli/configuration_spec.rb +++ b/spec/cucumber/cli/configuration_spec.rb @@ -53,24 +53,6 @@ def reset_config expect(config.options[:require]).to eq ['from/yml'] end - it 'allows --strict to be set by a profile' do - given_cucumber_yml_defined_as('bongo' => '--strict') - config.parse!(%w[--profile bongo]) - - expect(config.options[:strict].strict?(:undefined)).to be true - expect(config.options[:strict].strict?(:pending)).to be true - expect(config.options[:strict].strict?(:flaky)).to be true - end - - it 'allows --strict from a profile to be selectively overridden' do - given_cucumber_yml_defined_as('bongo' => '--strict') - config.parse!(%w[--profile bongo --no-strict-flaky]) - - expect(config.options[:strict].strict?(:undefined)).to be true - expect(config.options[:strict].strict?(:pending)).to be true - expect(config.options[:strict].strict?(:flaky)).to be false - end - it 'parses ERB syntax in the cucumber.yml file' do given_cucumber_yml_defined_as("---\ndefault: \"<%=\"--require some_file\"%>\"\n") config.parse!([]) diff --git a/spec/cucumber/formatter/fail_fast_spec.rb b/spec/cucumber/formatter/fail_fast_spec.rb index e740721d69..9aa744a2d3 100644 --- a/spec/cucumber/formatter/fail_fast_spec.rb +++ b/spec/cucumber/formatter/fail_fast_spec.rb @@ -77,15 +77,6 @@ module Formatter expect(Cucumber.wants_to_quit).to be_falsey end - - context 'when in strict mode' do - let(:configuration) { Cucumber::Configuration.new strict: Cucumber::Core::Test::Result::StrictConfiguration.new([:undefined]) } - - it 'sets Cucumber.wants_to_quit' do - execute [@gherkin], [StandardStepActions.new], configuration.event_bus - expect(Cucumber.wants_to_quit).to be_truthy - end - end end end end diff --git a/spec/cucumber/formatter/rerun_spec.rb b/spec/cucumber/formatter/rerun_spec.rb index ae4385701e..00b79581d8 100644 --- a/spec/cucumber/formatter/rerun_spec.rb +++ b/spec/cucumber/formatter/rerun_spec.rb @@ -96,59 +96,36 @@ module Formatter end context 'with a flaky scenario' do - context 'with option --no-strict-flaky' do - it 'prints nothing' do - gherkin = gherkin('foo.feature') do - feature do - scenario do - step 'flaky' - end + it 'prints nothing' do + gherkin = gherkin('foo.feature') do + feature do + scenario do + step 'flaky' end end + end - described_class.new(config) - execute [gherkin], [FakeObjects::FlakyStepActions.new, Filters::BroadcastTestRunStartedEvent.new(config), Filters::BroadcastTestCaseReadyEvent.new(config)], config.event_bus - - config.event_bus.test_run_finished + described_class.new(config) + execute [gherkin], [FakeObjects::FlakyStepActions.new, Filters::BroadcastTestRunStartedEvent.new(config), Filters::BroadcastTestCaseReadyEvent.new(config)], config.event_bus + config.event_bus.test_run_finished - expect(io.string).to eq('') - end + expect(io.string).to eq('') end - context 'with option --strict-flaky' do - let(:config) { Configuration.new(out_stream: io, strict: Core::Test::Result::StrictConfiguration.new([:flaky])) } - - it 'prints the location of the flaky scenario' do - gherkin = gherkin('foo.feature') do - feature do - scenario do - step 'flaky' - end + it 'does not include the same failing scenario more than once' do + gherkin = gherkin('foo.feature') do + feature do + scenario do + step 'failing' end end - - described_class.new(config) - execute [gherkin], [FakeObjects::FlakyStepActions.new, Filters::BroadcastTestRunStartedEvent.new(config), Filters::BroadcastTestCaseReadyEvent.new(config)], config.event_bus - config.event_bus.test_run_finished - - expect(io.string).to eq('foo.feature:3') end - it 'does not include the same failing scenario more than once' do - gherkin = gherkin('foo.feature') do - feature do - scenario do - step 'failing' - end - end - end - - described_class.new(config) - execute [gherkin, gherkin], [StandardStepActions.new, Filters::BroadcastTestRunStartedEvent.new(config), Filters::BroadcastTestCaseReadyEvent.new(config)], config.event_bus - config.event_bus.test_run_finished + described_class.new(config) + execute [gherkin, gherkin], [StandardStepActions.new, Filters::BroadcastTestRunStartedEvent.new(config), Filters::BroadcastTestCaseReadyEvent.new(config)], config.event_bus + config.event_bus.test_run_finished - expect(io.string).to eq('foo.feature:3') - end + expect(io.string).to eq('foo.feature:3') end end end diff --git a/spec/cucumber/multiline_argument/data_table_spec.rb b/spec/cucumber/multiline_argument/data_table_spec.rb index 652638aca8..3927be2190 100644 --- a/spec/cucumber/multiline_argument/data_table_spec.rb +++ b/spec/cucumber/multiline_argument/data_table_spec.rb @@ -84,14 +84,14 @@ module MultilineArgument it 'passes silently once #hashes are interrogated if a mapped column does not exist in non-strict mode' do expect do - new_table = table.map_column('two', strict: false, &:to_i) + new_table = table.map_column('two', &:to_i) new_table.hashes end.not_to raise_error end it 'fails once #hashes are interrogated if a mapped column does not exist in strict mode' do expect do - new_table = table.map_column('two', strict: true, &:to_i) + new_table = table.map_column('two', &:to_i) new_table.hashes end.to raise_error('The column named "two" does not exist') end @@ -136,7 +136,7 @@ module MultilineArgument it 'supports header and column mapping' do table = described_class.from([%w[one 1111], %w[two 22222]]) - t2 = table.map_headers({ 'two' => 'Two' }, &:upcase).map_column('two', strict: false, &:to_i) + t2 = table.map_headers({ 'two' => 'Two' }, &:upcase).map_column('two', &:to_i) expect(t2.rows_hash).to eq('ONE' => '1111', 'Two' => 22_222) end From b5da4e4473fcbac0137d8b007bcb0ca082e93ecb Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:19:16 +0100 Subject: [PATCH 2/9] Fix up missing spec issue from strict mode --- spec/cucumber/multiline_argument/data_table_spec.rb | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/spec/cucumber/multiline_argument/data_table_spec.rb b/spec/cucumber/multiline_argument/data_table_spec.rb index 3927be2190..808246fa07 100644 --- a/spec/cucumber/multiline_argument/data_table_spec.rb +++ b/spec/cucumber/multiline_argument/data_table_spec.rb @@ -82,14 +82,7 @@ module MultilineArgument expect(new_table.rows.first).not_to include('4444') end - it 'passes silently once #hashes are interrogated if a mapped column does not exist in non-strict mode' do - expect do - new_table = table.map_column('two', &:to_i) - new_table.hashes - end.not_to raise_error - end - - it 'fails once #hashes are interrogated if a mapped column does not exist in strict mode' do + it 'fails once `#hashes` are interrogated if a mapped column does not exist' do expect do new_table = table.map_column('two', &:to_i) new_table.hashes From baca846bd693489e27ef75179b1535eff0525ba3 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:21:29 +0100 Subject: [PATCH 3/9] Remove strict from default options --- cucumber.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cucumber.yml b/cucumber.yml index 6ae40822bc..f91fca0c45 100644 --- a/cucumber.yml +++ b/cucumber.yml @@ -1,5 +1,5 @@ <% -std_opts = "--format progress features -r features --strict --publish-quiet".dup +std_opts = "--format progress features -r features --publish-quiet".dup std_opts << " --tags 'not @wip'" std_opts << " --tags 'not @wip-jruby'" if defined?(JRUBY_VERSION) From b67bdaa399bc7db67fbe5b07891458356a4da6e1 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:24:13 +0100 Subject: [PATCH 4/9] Dry run with undefined scenarios will still pass as undefined is a permissible item --- features/docs/cli/dry_run.feature | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/features/docs/cli/dry_run.feature b/features/docs/cli/dry_run.feature index c12cfb9e91..f43a6e7db0 100644 --- a/features/docs/cli/dry_run.feature +++ b/features/docs/cli/dry_run.feature @@ -23,7 +23,6 @@ Feature: Dry Run 1 scenario (1 skipped) 1 step (1 skipped) - """ Scenario: With message formatter @@ -47,21 +46,21 @@ Feature: Dry Run Given this step is undefined """ When I run `cucumber --dry-run` - Then it should fail with: + Then it should pass with: """ Feature: test Scenario: # features/test.feature:2 Given this step is undefined # features/test.feature:3 - Undefined step: "this step is undefined" (Cucumber::Core::Test::Result::Undefined) - features/test.feature:3:in `this step is undefined' - - Undefined Scenarios: - cucumber features/test.feature:2 # Scenario: 1 scenario (1 undefined) 1 step (1 undefined) + You can implement step definitions for undefined steps with these snippets: + + Given('this step is undefined') do + pending # Write code here that turns the phrase above into concrete actions + end """ Scenario: With BeforeAll and AfterAll hooks @@ -92,5 +91,4 @@ Feature: Dry Run 1 scenario (1 skipped) 1 step (1 skipped) - """ From aaca9fef0b718ff7badc68f88635f8a7a9a9cf88 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:26:30 +0100 Subject: [PATCH 5/9] Add warning for retry_failing_tests invalid failure --- features/docs/cli/retry_failing_tests.feature | 1 + 1 file changed, 1 insertion(+) diff --git a/features/docs/cli/retry_failing_tests.feature b/features/docs/cli/retry_failing_tests.feature index 68b43203ff..bcd28d030a 100644 --- a/features/docs/cli/retry_failing_tests.feature +++ b/features/docs/cli/retry_failing_tests.feature @@ -18,6 +18,7 @@ Feature: Retry failing tests Scenario: Retry once, so Fails-once starts to pass Given a scenario "Fails-forever" that fails When I run `cucumber -q --retry 1 --format summary` + # These split assertions are due to an issue with aruba treating the () as regex and it failing erroneously Then it should fail with: """ 4 scenarios (2 failed, 1 flaky, 1 passed) From 0018ebb2c4772a8a70246693a25b5f9bccdb3581 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 17:30:29 +0100 Subject: [PATCH 6/9] Fix up retry_failing_tests to be more conformant now --- features/docs/cli/retry_failing_tests.feature | 30 +++---------------- 1 file changed, 4 insertions(+), 26 deletions(-) diff --git a/features/docs/cli/retry_failing_tests.feature b/features/docs/cli/retry_failing_tests.feature index bcd28d030a..531a124a4c 100644 --- a/features/docs/cli/retry_failing_tests.feature +++ b/features/docs/cli/retry_failing_tests.feature @@ -16,19 +16,14 @@ Feature: Retry failing tests And a scenario "Solid" that passes Scenario: Retry once, so Fails-once starts to pass - Given a scenario "Fails-forever" that fails When I run `cucumber -q --retry 1 --format summary` # These split assertions are due to an issue with aruba treating the () as regex and it failing erroneously Then it should fail with: """ - 4 scenarios (2 failed, 1 flaky, 1 passed) + 3 scenarios (1 failed, 1 flaky, 1 passed) """ And it should fail with: """ - Fails-forever - Fails-forever ✗ - Fails-forever ✗ - Fails-once feature Fails-once ✗ Fails-once ✓ @@ -42,19 +37,13 @@ Feature: Retry failing tests """ Scenario: Retry twice, so Fails-twice starts to pass too - Given a scenario "Fails-forever" that fails When I run `cucumber -q --retry 2 --format summary` - Then it should fail with: + Then it should pass with: """ - 4 scenarios (1 failed, 2 flaky, 1 passed) + 3 scenarios (2 flaky, 1 passed) """ - And it should fail with: + And it should pass with: """ - Fails-forever - Fails-forever ✗ - Fails-forever ✗ - Fails-forever ✗ - Fails-once feature Fails-once ✗ Fails-once ✓ @@ -68,17 +57,6 @@ Feature: Retry failing tests Solid ✓ """ - Scenario: Flaky scenarios gives non-zero exit code - When I run `cucumber -q --retry 2 --format summary` - Then it should fail with: - """ - Flaky Scenarios: - cucumber features/fails-once_feature.feature:2 - cucumber features/fails-twice_feature.feature:2 - - 3 scenarios (2 flaky, 1 passed) - """ - Scenario: Retry each test but suspend retrying after 2 failing tests, so later tests are not retried Given a scenario "Fails-forever-1" that fails And a scenario "Fails-forever-2" that fails From 09051ae289a1344d80700373e3985d455d41df38 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 18:01:48 +0100 Subject: [PATCH 7/9] Clean up rerun formatter --- .../docs/formatters/rerun_formatter.feature | 5 ---- lib/cucumber/formatter/rerun.rb | 28 ++++++++----------- 2 files changed, 12 insertions(+), 21 deletions(-) diff --git a/features/docs/formatters/rerun_formatter.feature b/features/docs/formatters/rerun_formatter.feature index d8598e1549..be06f90e1c 100644 --- a/features/docs/formatters/rerun_formatter.feature +++ b/features/docs/formatters/rerun_formatter.feature @@ -31,7 +31,6 @@ Feature: Rerun formatter Scenario: Given this step passes - """ When I run `cucumber --publish-quiet -f rerun` @@ -55,7 +54,6 @@ Feature: Rerun formatter Scenario: Given this step passes - """ And a file named "features/all_good.feature" with: """ @@ -86,7 +84,6 @@ Feature: Rerun formatter Scenario: Given this step passes - """ And a file named "features/all_good.feature" with: """ @@ -115,7 +112,6 @@ Feature: Rerun formatter | status | | passes | | fails | - """ When I run `cucumber -f rerun` Then it should fail with: @@ -179,7 +175,6 @@ Feature: Rerun formatter | status | | passes | | fails | - """ When I run `cucumber --expand -f rerun` Then it should fail with: diff --git a/lib/cucumber/formatter/rerun.rb b/lib/cucumber/formatter/rerun.rb index 39193c156c..52deb10f19 100644 --- a/lib/cucumber/formatter/rerun.rb +++ b/lib/cucumber/formatter/rerun.rb @@ -22,21 +22,19 @@ def finish_report @query.find_all_test_case_started.each do |test_case| status = @query.find_most_severe_test_step_result_by(test_case).status - # RULE: Don't log test cases without a pickle (Unsure what these could be?) + # RULE: Don't log test cases without a pickle (We cannot query their location data to log them) pickle = @query.find_pickle_by(test_case) next if pickle.nil? - # RULE: If the test case has already been logged as a failure (And we're retrying), remove prior reference of failure - if passing?(test_case) + # RULE: Test cases with the worst result as Passing is not considered a failure (Don't log these) + if status == Cucumber::Messages::TestStepResultStatus::PASSED + # If the test case in question had already been logged as a failure (And we're retrying), remove the prior reference of failure uri_and_location_hash[pickle.uri].delete(pickle.location.line) next end - # RULE: Passing test cases are not considered failures (Don't log these) - next if passing?(test_case) - - # RULE: Skipped test cases are not considered failures (on their own, don't log these) - next if skipped?(test_case) + # RULE: Test cases with the worst result as Skipped/Pending/Undefined are not considered failures (don't log these) + next if non_rerunnable_status?(status) # RULE: Before logging a failure, ensure we are not on a retried test case (Don't log a retry multiple times) next if test_case.attempt > 1 @@ -59,14 +57,12 @@ def uri_and_location_hash @uri_and_location_hash ||= Hash.new { |hash, key| hash[key] = Set.new } end - def passing?(test_case_started) - most_severe_test_step_result = @query.find_most_severe_test_step_result_by(test_case_started) - most_severe_test_step_result.status == Cucumber::Messages::TestStepResultStatus::PASSED - end - - def skipped?(test_case_started) - most_severe_test_step_result = @query.find_most_severe_test_step_result_by(test_case_started) - most_severe_test_step_result.status == Cucumber::Messages::TestStepResultStatus::SKIPPED + def non_rerunnable_status?(status) + [ + Cucumber::Messages::TestStepResultStatus::PASSED, + Cucumber::Messages::TestStepResultStatus::PENDING, + Cucumber::Messages::TestStepResultStatus::UNDEFINED + ].include?(status) end end end From bcf950802501f6dd8e7358a8e1a197b69c965089 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 18:20:52 +0100 Subject: [PATCH 8/9] Fix erroneous message class --- lib/cucumber/formatter/rerun.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/cucumber/formatter/rerun.rb b/lib/cucumber/formatter/rerun.rb index 52deb10f19..305a389a63 100644 --- a/lib/cucumber/formatter/rerun.rb +++ b/lib/cucumber/formatter/rerun.rb @@ -59,7 +59,7 @@ def uri_and_location_hash def non_rerunnable_status?(status) [ - Cucumber::Messages::TestStepResultStatus::PASSED, + Cucumber::Messages::TestStepResultStatus::SKIPPED, Cucumber::Messages::TestStepResultStatus::PENDING, Cucumber::Messages::TestStepResultStatus::UNDEFINED ].include?(status) From d7e6d6319a9f112f85fa2dffa0c4c019d7e88418 Mon Sep 17 00:00:00 2001 From: Luke Hill Date: Thu, 21 May 2026 18:22:07 +0100 Subject: [PATCH 9/9] Remove erroneous scenario --- .../docs/formatters/rerun_formatter.feature | 31 ------------------- 1 file changed, 31 deletions(-) diff --git a/features/docs/formatters/rerun_formatter.feature b/features/docs/formatters/rerun_formatter.feature index be06f90e1c..c079086ad4 100644 --- a/features/docs/formatters/rerun_formatter.feature +++ b/features/docs/formatters/rerun_formatter.feature @@ -68,37 +68,6 @@ Feature: Rerun formatter """ """ - Scenario: Exit code is not zero, regular scenario - Given a file named "features/mixed.feature" with: - """ - Feature: Mixed - - Scenario: - Given this step fails - - Scenario: - Given this step is undefined - - Scenario: - Given this step is pending - - Scenario: - Given this step passes - """ - And a file named "features/all_good.feature" with: - """ - Feature: All good - - Scenario: - Given this step passes - """ - - When I run `cucumber --publish-quiet -f rerun` - Then it should fail with exactly: - """ - features/mixed.feature:3:6:9 - """ - Scenario: Exit code is not zero, scenario outlines For details see https://github.com/cucumber/cucumber/issues/57 Given a file named "features/one_passing_one_failing.feature" with: