Add documentation for Qnit, split implementation files

Author: MichaelRFairhurst

README.md

@@ -397,7 +397,50 @@ This module takes a set of starting points, ending points, and edges in a graph,
 
 For displaying the discovered paths to users, see the `CustomPathProblem` module above.
 
-### Testing
+### Testing with Qnit
+
+While codeql's `test run` subcommand is a great way to test queries, it can be better in some cases
+to write a more traditional unit test for CodeQL libraries. Rather than selecting a set of outputs
+in a query and then inspecting that the query result (in the `.expectations` file) makes sense, qtil
+provides a library called "Qnit" for writing direct test cases with expectations, so that there's
+better cohesion between a test case and its expected output.
+
+To use Qnit, import the `qtil.testing.Qnit` module, and create a test class that extends
+`Test, Case`. Inside the class override the `run(Qnit test)` member predicate, and conditionally
+call `test.pass(name)` or `test.fail(description)` as appropriate.
+
+```ql
+import qtil.testing.Qnit
+
+class MyTest extends Test, Case {
+  override predicate run(Qnit test) {
+    if 1 = 1
+    then test.pass("1 equals 1")
+    else test.fail("1 does not equal 1")
+  }
+}
+```
+
+You may define as many test classes as you like, and they will all be run when you run the command
+`codeql test run $TESTDIR`. If all tests pass, the test will output "{n} tests passed." If any test
+fails, the result of each test will be selected (including failing and passing tests).
+
+For correct use, ensure that each test class passes with a unique name, and that tests always hold
+for some result, whether its a pass or a fail.
+
+```
+  override predicate run(Qnit test) {
+    if 1 = 1
+    then test.pass("1 equals 1") // Ensure this is unique to the test
+    else none() // This would be valid CodeQL, but it would not fail.
+  }
+```
+
+It is particularly risky, albeit useful, to write `test.fail("..." + somePredicate().toString())`,
+as this test will **not** fail if `somePredicate()` does not hold. This is a risky pattern, and so
+should only be applied with some caution.
+
+See the README in the `qtil.testing` directory for more information on how to use Qnit.
 
 ### Parameterization
 

src/qtil/testing/Qnit.qll

@@ -27,139 +27,6 @@
  * Be careful in deviating from this pattern, but do so at your own risk, so long as `pass()` and
  * `fail()` hold as in the example above.
  */
-
-private import codeql.util.Unit
-private import qtil.strings.Plural
-private import qtil.inheritance.UnderlyingString
-import QnitImpl::Public
-
-private module QnitImpl {
-  module Public {
-    /**
-     * A string that is re-typedef'd to Qnit for the sake of a pretty API.
-     *
-     * This class is used to define the predicate `run(Qnit test)`, which is prettier than
-     * `run(string test)`. See also the `pass()` and `fail()` methods.
-     */
-    bindingset[this]
-    class Qnit extends UnderlyingString {
-      bindingset[this]
-      predicate isFailing() { str().matches("FAILURE: %") }
-
-      bindingset[this]
-      predicate isPassing() { str().matches("PASS: %") }
-
-      /**
-       * Call this method inside of `Test.run(Qnit test)` to report a failing test case.
-       *
-       * It is recommended to use unique strings for each test case, as this will allow you to
-       * uniquely identify which tests failed, due to the way QL works.
-       */
-      bindingset[description]
-      predicate fail(string description) { this = "FAILURE: " + description }
-
-      /**
-       * Call this method inside of `Test.run(Qnit test)` to report a passing test case.
-       *
-       * It is recommended to use unique strings for each test case, as this will allow Qnit to
-       * properly count the number of tests that passed, due to the way QL works.
-       */
-      bindingset[name]
-      predicate pass(string name) { this = "PASS: " + name }
-    }
-
-    /**
-     * A test case that can be run by Qnit.
-     *
-     * This class is used to define the predicate `run(Qnit test)`, which is used to run the test
-     * case. The `pass()` and `fail()` methods are used to report the result of the test case, as
-     * follows:
-     *
-     * ```ql
-     * class MyTest extends Test, Case {
-     *  override predicate run(Qnit test) {
-     *    if (someCondition)
-     *    then test.pass("some condition passed")
-     *    else test.fail("some condition failed")
-     *   }
-     * }
-     * ```
-     *
-     * This is simply an abstract class extending the CodeQL's empty `Unit` type, as its subclasses
-     * are singletons with no underlying data.
-     */
-    abstract class Case extends Unit {
-      /**
-       * Overridable method to define the behavior of the test case, which should generally follow
-       * the pattern of:
-       *
-       * ```ql
-       * if (someCondition)
-       * then test.pass("some condition passed")
-       * else test.fail("some condition failed")
-       * ```
-       *
-       * It is best to use `pass()` and `fail()` with unique strings, as this will allow Qnit to
-       * properly count the number of tests that passed, and uniquely identify which tests failed,
-       * due to the way QL works.
-       *
-       * This is designed this way because we cannot execute an abstract method in QL while knowing
-       * which concrete class it belongs to. Rather, `Case.run(x)` holds for all `x` defined in all
-       * test cases. Making `x` a field does not solve this problem. We also must work around the
-       * limitation of not having anonymous functions. Etc.
-       */
-      abstract predicate run(Qnit test);
-    }
-
-    /**
-     * Extend this class to suppress a warning that is generated by QL when you override an
-     * abstract class without implementing a characteristic predicate.
-     *
-     * ```ql
-     * class MyTest extends Test, Case {
-     *   ...
-     * }
-     * ```
-     *
-     * Extends CodeQL's `Unit` type to match the `Test` class and support multiple inheritance.
-     */
-    class Test extends Unit { }
-
-    query predicate test(string report) {
-      if count(Qnit test | isFailing(test)) = 0
-      then
-        exists(int passed |
-          passed = count(Qnit test | isPassing(test)) and
-          report = plural("1 test", "All " + passed + " tests", passed) + " passed."
-        )
-      else
-        exists(Qnit test |
-          (isFailing(test) or isPassing(test)) and
-          report = test
-        )
-    }
-  }
-
-  /**
-   * A base class that defines `toString()` in order to enable the API `extends Test, Case`.
-   *
-   * Extends CodeQL's `Unit` type becaue it has no underlying data.
-   *
-   * Ordinarily, the QL compiler will issue a warning if you override an abstract class without
-   * implementing a characteristic predicate, as in `class MyTest extends Case {...}`. In order to
-   * suppress this warning, we can extend an additional non-abstract class, and therefore our
-   * recommended pattern is to extend `Test, Case` in order to define a test case.
-   *
-   * In QL, `Test` and `Case` must both extend the same base type. If that base type were `Unit`,
-   * then both `Test` and `Case` would need to implement `toString()`. At this point, QL would
-   * issue a warning for `class MyTest extends Test, Case {...}` because QL cannot disambiguate
-   * which `toString()` method to inherit. Therefore, we define `CaseBase` as a subclass of `TCase`
-   * which defines an unambiguous `toString()` method.
-   *
-   * With this workaround, we can define `class MyTest extends Test, Case {...}` without any
-   * warnings from the QL compiler, and without having to implement `toString()` in every test.
-   */
-  private predicate isFailing(Qnit test) { exists(Case c | c.run(test) and test.isFailing()) }
-
-  private predicate isPassing(Qnit test) { exists(Case c | c.run(test) and test.isPassing()) }
-}
+import qtil.testing.impl.Qnit
+import qtil.testing.impl.Test
+import qtil.testing.impl.Runner