class Minitest::Bisect

Minitest::Bisect helps you isolate and debug random test failures.

Constants

RUBY: Borrowed from rake

Attributes

culprits[RW]

An array of tests seen so far. NOT cleared by reset.

failures[RW]

Failures seen in this run. Shape:

{"file.rb"=>{"Class"=>["test_method1", "test_method2"] ...} ...}

tainted[RW]

True if this run has seen a failure.

tainted?[RW]

True if this run has seen a failure.

Public Class Methods

new() click to toggle source

Instantiate a new Bisect.

# File lib/minitest/bisect.rb, line 93
def initialize
  self.culprits = []
  self.failures = Hash.new { |h, k| h[k] = Hash.new { |h2, k2| h2[k2] = [] } }
end

run(files) click to toggle source

Top-level runner. Instantiate and call run, handling exceptions.

# File lib/minitest/bisect.rb, line 82
def self.run files
  new.run files
rescue => e
  warn e.message
  warn "Try running with MTB_VERBOSE=2 to verify."
  exit 1
end

Public Instance Methods

bisect_methods(files, rb_flags, mt_flags) click to toggle source

Normal: find “what is the minimal combination of tests to run to

make X fail?"

Run with: minitest_bisect … –seed=N

Verify the failure running normally with the seed.
1. If no failure, punt.
2. If no passing tests before failure, punt. (No culprits == no debug)
Verify the failure doesn’t fail in isolation.
1. If it still fails by itself, warn that it might not be an ordering issue.
Cull all tests after the failure, they’re not involved.
Bisect the culprits + bad until you find a minimal combo that fails.
Display minimal combo by running one last time.

Inverted: find “what is the minimal combination of tests to run to

make this test pass?"

Run with: minitest_bisect … –seed=N -n=“/failing_test_name_regexp/”

Verify the failure by running normally w/ the seed and -n=/…/
1. If no failure, punt.
Verify the passing case by running everything.
1. If failure, punt. This is not a false positive.
Cull all tests after the bad test from #1, they’re not involved.
Bisect the culprits + bad until you find a minimal combo that passes.
Display minimal combo by running one last time.

# File lib/minitest/bisect.rb, line 163
def bisect_methods files, rb_flags, mt_flags
  bad_names, mt_flags = mt_flags.partition { |s| s =~ /^(?:-n|--name)/ }
  normal   = bad_names.empty?
  inverted = !normal

  if inverted then
    time_it "reproducing w/ scoped failure (inverted run!)...", build_methods_cmd(build_files_cmd(files, rb_flags, mt_flags + bad_names))
    raise "No failures. Probably not a false positive. Aborting." if failures.empty?
    bad = map_failures
  end

  cmd = build_files_cmd(files, rb_flags, mt_flags)

  msg = normal ? "reproducing..." : "reproducing false positive..."
  time_it msg, build_methods_cmd(cmd)

  if normal then
    raise "Reproduction run passed? Aborting." unless tainted?
    raise "Verification failed. No culprits? Aborting." if culprits.empty? && seen_bad
  else
    raise "Reproduction failed? Not false positive. Aborting." if tainted?
    raise "Verification failed. No culprits? Aborting." if culprits.empty? || seen_bad
  end

  if normal then
    bad = map_failures

    time_it "verifying...", build_methods_cmd(cmd, [], bad)

    new_bad = map_failures

    if bad == new_bad then
      warn "Tests fail by themselves. This may not be an ordering issue."
    end
  end

  idx = culprits.index bad.first
  self.culprits = culprits.take idx+1 if idx # cull tests after bad

  # culprits populated by initial reproduction via minitest/server
  found, count = culprits.find_minimal_combination_and_count do |test|
    prompt = "# of culprit methods: #{test.size}"

    time_it prompt, build_methods_cmd(cmd, test, bad)

    normal == tainted? # either normal and failed, or inverse and passed
  end

  puts
  puts "Minimal methods found in #{count} steps:"
  puts
  puts "Culprit methods: %p" % [found + bad]
  puts
  cmd = build_methods_cmd cmd, found, bad
  puts cmd.sub(/--server \d+/, "")
  puts
  cmd
end

reset() click to toggle source

Reset per-bisect-run variables.

# File lib/minitest/bisect.rb, line 101
def reset
  self.seen_bad = false
  self.tainted  = false
  failures.clear
  # not clearing culprits on purpose
end

run(args) click to toggle source

Instance-level runner. Handles Minitest::Server, argument processing, and invoking bisect_methods.

# File lib/minitest/bisect.rb, line 112
def run args
  Minitest::Server.run self

  cmd = nil

  mt_flags = args.dup
  expander = Minitest::Bisect::PathExpander.new mt_flags

  files = expander.process
  rb_flags = expander.rb_flags
  mt_flags += ["--server", $$.to_s]

  cmd = bisect_methods files, rb_flags, mt_flags

  puts "Final reproduction:"
  puts

  system cmd.sub(/--server \d+/, "")
ensure
  Minitest::Server.stop
end