add fail fast option for cli/generate

[huntergregory]

Fixes #129

Example Output

Running the following command:

./cyclonus generate --fail-fast --server-protocol=TCP,UDP --include=target,delete-policy,delete-pod

Test Cases Run

Supposed to run 9 test cases.

test cases to run by tag:
- all-namespaces: 0
- policy-stack: 0
- peer-ipblock: 0
- target-namespace: 3
- delete-pod: 1
- ip-block-no-except: 0
- namespaces-by-label: 0
- ip-block-with-except: 0
- deny-all: 0
- pathological: 0
- target-pod-selector: 3
- pods-by-label: 0
- action: 3
- delete-policy: 1
- named-port: 0
- multi-peer: 0
- set-pod-labels: 0
- any-port-protocol: 0
- any-peer: 0
- udp: 0
- example: 0
- update-policy: 0
- miscellaneous: 0
- peer-pods: 0
- upstream-e2e: 0
- policy-namespace: 0
- ingress: 0
- create-pod: 1
- direction: 0
- conflict: 0
- create-namespace: 1
- rule: 0
- any-port: 0
- all-pods: 0
- numbered-port: 0
- protocol: 0
- namespaces-by-default-label: 0
- port: 0
- egress: 0
- sctp: 0
- target: 6
- tcp: 0
- allow-all: 0
- create-policy: 1
- end-port: 0
- set-namespace-labels: 0
- delete-namespace: 1
- multi-port/protocol: 0
testing 9 cases

test #1: set namespace to x
 - tags: target, target-namespace
test #2: set namespace to y
 - tags: target, target-namespace
test #3: set namespace to z
 - tags: target, target-namespace
test #4: set pod selector to ["MatchLabels",[],"MatchExpression",null]
 - tags: target, target-pod-selector
test #5: set pod selector to ["MatchLabels",["pod: a"],"MatchExpression",null]
 - tags: target, target-pod-selector
test #6: set pod selector to ["MatchLabels",[],"MatchExpression",[{"key":"pod","operator":"In","values":["a","b"]}]]
 - tags: target, target-pod-selector
test #7: Create/delete policy
 - tags: action, create-policy, delete-policy
test #8: Create/delete namespace
 - tags: action, create-namespace, delete-namespace
test #9: Create/delete pod
 - tags: action, create-pod, delete-pod

Output on Failure

Quit the run and print the summary table for the tests that ran.

finished policy #4
WARN[2023-03-08T17:26:22-08:00] failing fast due to failure                  
SummaryTable:
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                   TEST                    | RESULT |   STEP/TRY    | WRONG | RIGHT | IGNORED |       TCP        | SCTP |       UDP        |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 1: set namespace to x                     | passed |               |       |       |         |                  |      |                  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 1 |     2 |    70 |       9 | 144 / 144 (100%) | -    | 142 / 144 (98%)  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 2 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 2: set namespace to y                     | passed |               |       |       |         |                  |      |                  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 1 |     3 |    69 |       9 | 143 / 144 (99%)  | -    | 140 / 144 (97%)  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 2 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 3: set namespace to z                     | passed |               |       |       |         |                  |      |                  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 1 |     8 |    64 |       9 | 133 / 144 (92%)  | -    | 128 / 144 (88%)  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 2 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 4: set pod selector to                    | failed |               |       |       |         |                  |      |                  |
| ["MatchLabels",[],"MatchExpression",null] |        |               |       |       |         |                  |      |                  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 1 |    19 |    53 |       9 | 114 / 144 (79%)  | -    | 106 / 144 (73%)  |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 2 |    42 |    30 |       9 | 78 / 144 (54%)   | -    | 60 / 144 (41%)   |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 3 |    42 |    30 |       9 | 78 / 144 (54%)   | -    | 61 / 144 (42%)   |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 4 |    42 |    30 |       9 | 78 / 144 (54%)   | -    | 60 / 144 (41%)   |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 5 |    43 |    29 |       9 | 77 / 144 (53%)   | -    | 60 / 144 (41%)   |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                           |        | Step 1, try 6 |    42 |    30 |       9 | 78 / 144 (54%)   | -    | 60 / 144 (41%)   |
+-------------------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+

target counts:
+---------------------+--------+--------+----------+
|       FEATURE       | PASSED | FAILED | PASSED % |
+---------------------+--------+--------+----------+
| target-pod-selector |      0 |      1 |        0 |
| target-namespace    |      3 |      0 |      100 |
+---------------------+--------+--------+----------+

Pass/Fail for probes on protocols:
+---------------+--------+--------+----------+
|   PROTOCOL    | PASSED | FAILED | PASSED % |
+---------------+--------+--------+----------+
| probe on TCP  |   1355 |    373 |       78 |
| probe on SCTP |      0 |      0 |        0 |
| probe on UDP  |   1249 |    479 |       72 |
+---------------+--------+--------+----------+

Feature results:
| Tag | Result |
| --- | --- |
| action | 3 / 4 = 75% ❌ |
|  - action: create policy | 3 / 4 = 75% ❌ |
| egress | 3 / 4 = 75% ❌ |
|  - 0 peers | 3 / 4 = 75% ❌ |
|  - 1 peer | 3 / 4 = 75% ❌ |
|  - 1 port/protocol | 3 / 4 = 75% ❌ |
|  - 2+ port/protocols | 3 / 4 = 75% ❌ |
|  - 2+ rules | 3 / 4 = 75% ❌ |
|  - numbered port | 3 / 4 = 75% ❌ |
|  - peer namespace selector match expression | 3 / 4 = 75% ❌ |
|  - peer pod selector match expression | 3 / 4 = 75% ❌ |
|  - policy on TCP | 3 / 4 = 75% ❌ |
|  - policy on UDP | 3 / 4 = 75% ❌ |
| general | 3 / 4 = 75% ❌ |
|  - policy with both ingress and egress | 3 / 4 = 75% ❌ |
|  - policy with egress | 3 / 4 = 75% ❌ |
|  - policy with ingress | 3 / 4 = 75% ❌ |
|  - target: empty pod selector | 0 / 1 = 0% ❌ |
|  - target: pod selector match labels | 3 / 3 = 100% ✅ |
|  - target: specific namespace | 3 / 4 = 75% ❌ |
| ingress | 3 / 4 = 75% ❌ |
|  - 1 peer | 3 / 4 = 75% ❌ |
|  - 1 port/protocol | 3 / 4 = 75% ❌ |
|  - 1 rule | 3 / 4 = 75% ❌ |
|  - numbered port | 3 / 4 = 75% ❌ |
|  - peer namespace selector match expression | 3 / 4 = 75% ❌ |
|  - peer pod selector match expression | 3 / 4 = 75% ❌ |
|  - policy on TCP | 3 / 4 = 75% ❌ |

Tag results:
| Tag | Result |
| --- | --- |
| target | 3 / 4 = 75% ❌ |
|  - target-namespace | 3 / 3 = 100% ✅ |
|  - target-pod-selector | 0 / 1 = 0% ❌ |

[mattfenwick]

thanks @huntergregory ! looks good , will merge soon