Merge pull request #282 from jgray-19/test_ptc

Further Improvements to PTC tests

Author: ldeniau

tests/tests/test-ptc-maps/element_tests.mad

@@ -1,35 +1,61 @@
 Building a test for an element
 ------------------------------
 
-To create a test for a single element, there are two main components that need to be created; the sequence and the configuration specific to the element.
+To create a test for a single element, there is a single component that is required; an object containing the configuration of the test. This object must hold the following information: 
+
+* A string, named `elm`, containing the MAD-X element definition, with the parameters to be tested as part of the string, ready to be formatted with the configuration table, see below.
+* An array part, named `alist` for attribute list, within the object, containing the names of the parameters to be tested.
+* The hash part of the object, containing an *iterable* for each parameter, see below.
+
+Since the array part and the hash part also need to contain the reference parameters, the configuration object specific to your test must inherit from ref_cfg, which is a table that contains the reference parameters. To take from reference `alist`, you must use the `tbl_cat` function, see below.  
 
 The sequence is a string that contains the MAD-X script to create the element that you desire to test, with all the necessary parameters, the parameters can be assigned through formatting with the configuration table. For example, a quadrupole with the parameters `k1`, `k1s` `fringe` and `tilt` can be created with the following string: 
 ```
-local elm_str = "QUADRUPOLE, at=0.75, l=1.5, k1=${k1}, k1s=${k1s}, tilt=${tilt}*pi/8, fringe=${fringe}"
+"QUADRUPOLE, at=0.75, l=1.5, k1=${k1}, k1s=${k1s}, tilt=${tilt}*pi/8, fringe=${fringe}"
 ```
 
-The configuration table is a table that contains the names of the parameters and therefore keys in the array part of the table, and the values attached to the parameter name in the hash part of the table, see the example below;
-
+Below is an example of a configuration object for a quadrupole, with the parameters `k1`, `k1s` `fringe` and `tilt`:
 ```
-quad_cfg = {
-    "tilt", "fringe", "k1", "k1s",
+local cfg = ref_cfg "quadrupole" {
+    elm = " QUADRUPOLE, at=0.75, l=1.5, k1=${k1}, k1s=${k1s}, tilt=${tilt}*pi/8, fringe=${fringe}",
+    tol = 1000,
+
+    alist = tblcat(ref_cfg.alist, {"tilt", "fringe", "k1", "k1s"}),
     tilt   = 0  ..4,
     fringe = 0  ..3 ..3,
     k1     = -0.2..0.2..0.2,
     k1s    = -0.2..0.2..0.2,
 }
+run_test(cfg)
 ```
 
-Then this can be combined with the reference table using ``tbl_cat(ref_cfg, quad_cfg)`` to create a table that contains all the parameters for the element. The reference table contains the parameters that are common to all elements, such as the energy, the number of slices, the initial conditions, etc. To change the reference table, you must copy the table using ``tblcopy`` and then change the values in the copy, as the reference table is used by all the tests. 
+Above shows the required parts of the object, the element string, tolerance, an extended attribute list and the parameters with their ranges. The tolerance is the number that the difference between the maps is compared to, it is multiplied by your machine epsilon (`eps`) to get the number to compare to. The tolerance is optional as the default is 1000. If the tolerance is set to a string then the test will try to read the tolerance from a file with the same name as the tolerance string
+
+Finally, the test is run by calling ``run_test(cfg)``. 
+
+How the tests work
+------------------
+
+The entry function to running the tests is inventively called `run_test`. This function takes a configuration object as the only argument. Within this function the following steps are taken:
+
+1. Checks if cfg.dorun is not nil or false, if it is, then the function prints the previous results from the previous run and returns.
+2. An mtable with 2 columns is created, the first column will store the configuration snapshots, while the second column will store the results of each configuration snapshot.
+3. The configuration snapshots are created using a recursive function that takes the configuration object and creates the snapshots and runs the function `run_cfg` on each snapshot.  
+    The function `run_cfg` takes a configuration snapshot and does the following:
+    1. Calls `do_trck`, which runs track and ptc to get the difference between MAD and PTC, then stores the results in a table and returns it.
+    2. Stores the configuration snapshot and the results into the mtable.
+    3. If cfg.doprnt then the results are printed to the console in a single line per config.
+    4. If cfg.dodbg then the results are compared to the tolerance, if the results are outside the tolerance then the test stops and the results are printed to the console.
+4. Now the mtable has been filled with the results of each configuration snapshot, we create the generator columns for the mtable, which link the tables of each configuration snapshot and the results to a column in the mtable.
+5. If cfg.dosave or the file does not exist then the mtable is saved to a file with the name of the element appended with `_cfg` or `_res` for the configuration snapshots and the results respectively.
+6. If cfg.doprnt then a summary of the results is printed to the console. (See below for an example of the summary)
+7. If the test has not been stopped by cfg.dodbg then several excess files are removed.
 
-Finally, the test is run by calling ``run_test(element, elm_str, cfg, tol_)``, where `element` is the element object that would like to be tested, `elm_str` is the string that contains the MAD-X script to create the element, `cfg` is the total configuration table, and `tol_` is the (optional) tolerance for the test. This test will first create the config mtable then loop through the config, creating a file called ``element.seq`` that is called by MAD-X and MAD-NG. The final map is then gathered from the mflow in MAD-NG and from the debug output of MAD-X-PTC to be compared and stored into a max dif mtable, which is then saved to a file called ``element_name_max_err.tfs`` at the end of the test.
 
 Running the tests
 -----------------
 
-To run the test, just call the element function with a optionally a tolerance as the only argument. The tolerance should be an integer, as the number that the difference between the maps is compared to is this tolerance multiplied by your machine epsilon (`eps`). 
-
-For example, ``testQUAD(1000)`` will run the test `testQUAD` (the quadrupole) with a tolerance of `1000*eps`.
+To run the test, just call the element function For example, ``testQUAD()`` will run the test `testQUAD` (the quadrupole).
 
 There are four flags that can be used to control how your tests run and the reported results:
 
@@ -44,11 +70,14 @@ There are four flags that can be used to control how your tests run and the repo
     >require "madl_dbgmap".cmpmdump("element_name")
     >```
 
-With the `doprnt=true`, you will also receive a final output. This output is the sum of all the failures for each order, for each column of your config. The idea of this sum is to give you an idea of what is causing the failure. For example, see the output below for a quadrupole (truncated at order 0), run on linux with ``testQUAD(1000)`` as the input command.
+With the `doprnt=true`, you will also receive a final output. This output is the sum of all the failures for each order, for each column of your config. The idea of this sum is to give you an idea of what is causing the failure. For example, see the output below for a quadrupole (truncated at order 0), run on linux with ``testQUAD()`` as the input command.
 
 ```
 cfg = {
-    "model", "energy", "method", "nslice", "x0i", "order", "icase", "tilt", "fringe", "k1", "k1s",
+    elm = " QUADRUPOLE, at=0.75, l=1.5, k1=${k1}, k1s=${k1s}, tilt=${tilt}*pi/8, fringe=${fringe}",
+    tol = 1000,
+    
+    alist = {"model", "energy", "method", "nslice", "x0i", "order", "icase", "tilt", "fringe", "k1", "k1s",},
     model = 1..2,
     energy = {1, 6500}, 
     method = 2..2..2,