[WIP] Check Results API #997

awstlaur · 2017-04-01T18:15:43Z

This PR adds a --run-full-report command-line flag whose usage is as follows:

node build/phaseA/pyret.jarr --run-full-report file.arr

This will parse, compile, and run file.arr, and write JSON data to standard out.

No matter what happens in file.arr (and any Pyret dependencies it has), I claim
this API is robust enough to not allow stray messages to be written to stdout.
This is important, because this API will be used by grading scripts to collect
data: a grading script may want to redirect stdout to a file.arr.json file,
and then do its remaining "business logic" outside of Pyret, in the language
of its author's choice.

For examples of input programs and their resulting JSON data, see
TM030 Check Results API

Some notes about the PR and its changeset:

This is marked work-in-progress (WIP) because I haven't added tests of the
proposed functionality. Before I put effort into writing said tests, I would
like us to agree upon the new interface and expected behavior.
The diff on checker.arr is a bit messy: the gist there is that the
functionallity of results-summary was expanded and moved into the new results-report
function; now, results-summary depends on results-report.
The added toJSON in ffi.js has been robust enough for this use case, but
I do not claim it can cleanly convert any Pyret data structure into JSON.
Its cases mirror those in json-structs.arr:tojson, but it does not handle
string-dicts.
load-lib.js has some repeated logic: renderCheckResults and renderCheckReport
are similar, and renderErrorMessage and renderErrorReport are also similar.
In theory, the renderCheckResults and renderErrorMessage may be able to depend
on the render*Report functions, but (I believe) it would require messy stack
management and thus might not be worth the trouble.

it will not exit with EXIT_ERROR_CHECK_FAILURES when needed. however, it the main function in pyret.arr will return an appropriate error code, if we can leverage that also: this commit fails on no-diff-standalone

doing this mainly to investigate timeouts on travis ci

This reverts commit b9a6e04.

This reverts commit 5b020fa.

each of the 4 phaseX scripts is a symlink to run-phase, which uses the basename command to get the name of the symlink file and go from there

(I was going to just use the json lib, but there were annoying module cycles arising)

Manually resolved conflicts in pyret.arr. --run-full-report will not work as of this commit

(at least in spirit)

notes: - maybe-surface-parse has output type Either<{exn: ParseError, message: String}, AST> - surface-parse either returns the ast, or console.error's the message and raises the exn - test-parse.arr was mostly auto-generated from tests/parse/parse.js

This reverts commit c1236f6. Reverting on this branch to instead opt for merging in the maybe-surface-parse branch, which has an open PR into horizon

notes: - make phaseA fails at this commit. - this should make merging easier

With manual conflict resolution for the following: Conflicts: src/arr/compiler/cli-module-loader.arr src/arr/compiler/pyret.arr src/js/base/handalone.js src/js/trove/load-lib.js

- adds a 'capture-output' field to the options - render(Check|Error)Report will propagate the captured output to the report json

There were technically merge conflicts, but they were very easy to resolve.

jpolitz · 2018-04-08T17:58:39Z

@jswrenn I'm curious if this design is useful to you, or if your recent work on doing lots of testing automation supersedes this in useful ways.

awstlaur · 2019-03-10T14:05:50Z

I'm not coming back to this

awstlaur added 30 commits December 27, 2016 19:00

begin working on checker api (see TM030)

c2d6dbf

set up reason/fancy-reason differentiator

efd3350

Merge branch 'horizon' into checker-api

51be9a8

make --run work, modulu exit codes

f05dd21

it will not exit with EXIT_ERROR_CHECK_FAILURES when needed. however, it the main function in pyret.arr will return an appropriate error code, if we can leverage that also: this commit fails on no-diff-standalone

revert unnecessary diff

f305fc0

propogate exit code correctly

3deb191

render error and exit correctly on runtime errors

4fbe82c

defer all work to results-report

7ba2c4b

update broken load-lib test

1f781f6

add print to type-check main.arr

b9a6e04

doing this mainly to investigate timeouts on travis ci

Revert "add print to type-check main.arr"

12fda50

This reverts commit b9a6e04.

investigate timeout on travis ci#fx-run

5b020fa

okay now that test works

e3f93d6

fix invalid option: s/compiled-dir/compiled-cache/

c4573ba

Revert "investigate timeout on travis ci#fx-run"

d1fb5bc

This reverts commit 5b020fa.

clean up load-lib

caa3f9d

add phaseX scripts

ac1f136

each of the 4 phaseX scripts is a symlink to run-phase, which uses the basename command to get the name of the symlink file and go from there

re-add raise for compilation error

0445c70

revert to raise semantics for compilation errors

45e1837

disallow run-phase to be run directly - must be run via a phaseX symlink

798965c

fix throwInternalError call

9f84117

switch to command-line interface

29fdb98

make ffi.toJSON behave more like json-struct's to-json

9d22cb7

(I was going to just use the json lib, but there were annoying module cycles arising)

make message optional - use in results-summary

52129d5

Merge branch 'fix-run' into checker-api

debf728

Manually resolved conflicts in pyret.arr. --run-full-report will not work as of this commit

make resumeWith.message a js string

8b4f598

(at least in spirit)

Merge branch 'fix-run' into checker-api

2332319

make compile/run error report output json as well

e66ccfb

add maybe-surface-parse to capture parse errors in an Either

c1236f6

awstlaur added 24 commits January 14, 2017 13:28

Revert "add maybe-surface-parse to capture parse errors in an Either"

c1f09b7

This reverts commit c1236f6. Reverting on this branch to instead opt for merging in the maybe-surface-parse branch, which has an open PR into horizon

Merge branch 'horizon' into checker-api

30b7c0d

Merge branch 'maybe-surface-parse' into checker-api

4310b1a

leverage maybe-surface-parse

fc18dfa

Merge branch 'horizon' into fix-run

4eee725

fix get-stack for rendering check results

c5fc30e

Merge branch 'fix-run' into checker-api

d654aaa

fix get-stack for renderCheckReport

c57233c

Merge branch 'horizon' into checker-api

cfc8b9d

Merge branch 'horizon' into fix-run

d4ec4ea

checkout horizon's pyret.arr and handalone.js

75ed2c9

notes: - make phaseA fails at this commit. - this should make merging easier

Merge branch 'horizon' into fix-run

0658df1

re-instate the propogation of information from CLI.run

32a7d90

propagate exit while using the --run flag

99aa480

document run-phase scripts in README

30d8e0d

Merge branch 'fix-run' into checker-api

236b4a6

With manual conflict resolution for the following: Conflicts: src/arr/compiler/cli-module-loader.arr src/arr/compiler/pyret.arr src/js/base/handalone.js src/js/trove/load-lib.js

re-add run-full-report flag

b885758

add timestamp to checker api report

b0828e3

capture output to include in final report

117cccc

- adds a 'capture-output' field to the options - render(Check|Error)Report will propagate the captured output to the report json

add note about exit code (vis a vis error report)

e08ecf6

change captureOutput to be a separate argument

6c3ecaf

remove stray line

83fa9a7

Merge branch 'horizon' into checker-api

e027035

There were technically merge conflicts, but they were very easy to resolve.

remove stray comments

f338f58

awstlaur closed this Mar 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Check Results API #997

[WIP] Check Results API #997

Uh oh!

awstlaur commented Apr 1, 2017

Uh oh!

jpolitz commented Apr 8, 2018

Uh oh!

awstlaur commented Mar 10, 2019

Uh oh!

Uh oh!

[WIP] Check Results API #997

[WIP] Check Results API #997

Uh oh!

Conversation

awstlaur commented Apr 1, 2017

Uh oh!

jpolitz commented Apr 8, 2018

Uh oh!

awstlaur commented Mar 10, 2019

Uh oh!

Uh oh!