MOI.Test: make it easy to understand what tests are run

[odow]

In discussions with @kdheepak and @jd-lara:

Calling unittest(solver, config) (etc) is a little opaque with respect to what tests are actually run.

• Add verbose flag that prints out the name of the tests
• Make it easy to obtain a list of tests that will be run so you can add them to exclude when developing the wrapper of a new solver
• Or document that the function calls the dictionary of the same name, e.g.,:

julia> using MathOptInterface

julia> MathOptInterface.Test.unittests
Dict{String,Function} with 44 entries:
  "number_threads"        => number_threads
  "solve_qcp_edge_cases"  => solve_qcp_edge_cases
  "solve_zero_one_with_b… => solve_zero_one_with_bounds_1
  "solve_with_lowerbound" => solve_with_lowerbound
  "delete_nonnegative_va… => delete_nonnegative_variables
  "solve_affine_greatert… => solve_affine_greaterthan
  "add_variable"          => add_variable
  "solve_singlevariable_… => solve_singlevariable_obj
  "solve_duplicate_terms… => solve_duplicate_terms_vector_affine
  "solve_unbounded_model" => solve_unbounded_model
  "solve_constant_obj"    => solve_constant_obj
  "solve_single_variable… => solve_single_variable_dual_max
  "solve_single_variable… => solve_single_variable_dual_min
  "solver_name"           => solver_name
  "delete_variable"       => delete_variable
  "solve_time"            => solve_time
  "solve_duplicate_terms… => solve_duplicate_terms_obj
  "delete_variables"      => delete_variable
  "raw_status_string"     => raw_status_string
  "add_variables"         => add_variables
  "feasibility_sense"     => feasibility_sense
  "solve_affine_equalto"  => solve_affine_equalto
  "max_sense"             => max_sense
  "variablenames"         => variablenames
  "solve_integer_edge_ca… => solve_integer_edge_cases
  "silent"                => silent
  "update_dimension_nonn… => update_dimension_nonnegative_variables
  "solve_blank_obj"       => solve_blank_obj
  "min_sense"             => min_sense
  "solve_zero_one_with_b… => solve_zero_one_with_bounds_3
  "solve_affine_interval" => solve_affine_interval
  "solve_with_upperbound" => solve_with_upperbound
  "solve_duplicate_terms… => solve_duplicate_terms_scalar_affine
  "time_limit_sec"        => time_limit_sec
  "delete_soc_variables"  => delete_soc_variables
  "getvariable"           => getvariable
  "solve_zero_one_with_b… => solve_zero_one_with_bounds_2
  "solve_qp_edge_cases"   => solve_qp_edge_cases
  "getconstraint"         => getconstraint
  "solve_result_index"    => solve_result_index
  "get_objective_functio… => get_objective_function
  "solve_affine_lessthan" => solve_affine_lessthan
  "solve_objbound_edge_c… => solve_objbound_edge_cases
  "solve_affine_deletion… => solve_affine_deletion_edge_cases

[yashvardhan747]

I think rather than just test name it will be more good if it prints out the one-line description of the test.

[odow]

We can to the conclusion that this was just a documentation issue. Adding and maintaining the verbose flag and print statements is a lot of work.

We just need to add a paragraph to
https://www.juliaopt.org/MathOptInterface.jl/dev/apimanual/#Testing-guideline-1
explaining that unittest(model, config) runs all of the tests in the MOI.Test.unittests dictionary. Use keys(MOI.Test.unittests) to obtain a list of keys that can be passed to the exclude keyword argument of unittest.

[yashvardhan747]

But keyword argument exclude does not exist for unittest, we have to implement it right?

[odow]

I guess I meant the third positional argument.

[yashvardhan747]

I cant find out function unittest in Test folder. Please help!!

[yashvardhan747]

Ohh cool I got it.

[yashvardhan747]

I think we should make exclude as a keyword argument
in line

[matbesancon]

The fact that moitestset is a macro make it impossible to figure out what test was run from the stack trace too

[odow]

Having a debug or verbose flag would help in cases like this, where SDPA segfaults somewhere in MOI.Test.unittest. jump-dev/SDPA.jl#29

[odow]

Closing this as mostly done by #1408