Output error messages to a file?

[xiaozhour]

Can we output the super long message that appears on the REPL when running the model to a text file so that the actual key error message will not be overwhelmed by the other irrelevant stuff? Something like the default option in MITgcm with the STDOUT/STDERR files. For those who run on a terminal, this is probably fine because they can scroll up (assuming they don't miss it!) but in VScode there's a limit on how much you can output.

[glwagner]

This is a great question, though more of a Julia issue than something specific to Oceananigans! However, there is an off-chance that there is some feature we could implement to make this easier for people.

Note that terminals also have line limits and its a problem for people working in the terminal too. I use the silly but practical solution of reducing my font size and re-executing code until l can see the top of the error message. However, there are probably other solutions.

I wrote up a little example to experiment:

using Oceananigans
grid = RectilinearGrid(size=(1, 1, 1), extent=(1, 1, 1))
model = HydrostaticFreeSurfaceModel(; grid)
simulation = Simulation(model, Δt=1.0)
bad_callback(sim) = error("Why did you write such a bad callback?")
simulation.callbacks[:error] = Callback(bad_callback)
run!(simulation)

Some hunting on google returned this thread: https://discourse.julialang.org/t/redirect-output-and-error-to-a-file/58908/3

One option is to pipe to a file with the terminal utility tee:

$ julia --project | tee out.txt

All output generated thereafter is written to out.txt. I can't really see myself using this solution though. Changing the text size is easier...

It looks it may also be possible to use the Julia module StackTraces:

https://docs.julialang.org/en/v1/manual/stacktraces/

With the above example I found

julia> err, st = try
       run!(simulation)
       catch err
       err, stacktrace()
       end
[ Info: Initializing simulation...
(ErrorException("Why did you write such a bad callback?"), Base.StackTraces.StackFrame[top-level scope at REPL[2]:4, eval at boot.jl:360 [inlined], eval_user_input(ast::Any, backend::REPL.REPLBackend) at REPL.jl:139, repl_backend_loop(backend::REPL.REPLBackend) at REPL.jl:200, start_repl_backend(backend::REPL.REPLBackend, consumer::Any) at REPL.jl:185, run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool) at REPL.jl:317, run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:305, (::Base.var"#875#877"{Bool, Bool, Bool})(REPL::Module) at client.jl:387, #invokelatest#2 at essentials.jl:708 [inlined], invokelatest at essentials.jl:706 [inlined], run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:372, exec_options(opts::Base.JLOptions) at client.jl:302, _start() at client.jl:485])

julia> st
13-element Vector{Base.StackTraces.StackFrame}:
 top-level scope at REPL[2]:4
 eval at boot.jl:360 [inlined]
 eval_user_input(ast::Any, backend::REPL.REPLBackend) at REPL.jl:139
 repl_backend_loop(backend::REPL.REPLBackend) at REPL.jl:200
 start_repl_backend(backend::REPL.REPLBackend, consumer::Any) at REPL.jl:185
 run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool) at REPL.jl:317
 run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:305
 (::Base.var"#875#877"{Bool, Bool, Bool})(REPL::Module) at client.jl:387
 #invokelatest#2 at essentials.jl:708 [inlined]
 invokelatest at essentials.jl:706 [inlined]
 run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:372
 exec_options(opts::Base.JLOptions) at client.jl:302
 _start() at client.jl:485julia> err, st = try
       run!(simulation)
       catch err
       err, stacktrace()
       end
[ Info: Initializing simulation...
(ErrorException("Why did you write such a bad callback?"), Base.StackTraces.StackFrame[top-level scope at REPL[2]:4, eval at boot.jl:360 [inlined], eval_user_input(ast::Any, backend::REPL.REPLBackend) at REPL.jl:139, repl_backend_loop(backend::REPL.REPLBackend) at REPL.jl:200, start_repl_backend(backend::REPL.REPLBackend, consumer::Any) at REPL.jl:185, run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool) at REPL.jl:317, run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:305, (::Base.var"#875#877"{Bool, Bool, Bool})(REPL::Module) at client.jl:387, #invokelatest#2 at essentials.jl:708 [inlined], invokelatest at essentials.jl:706 [inlined], run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:372, exec_options(opts::Base.JLOptions) at client.jl:302, _start() at client.jl:485])

julia> st
13-element Vector{Base.StackTraces.StackFrame}:
 top-level scope at REPL[2]:4
 eval at boot.jl:360 [inlined]
 eval_user_input(ast::Any, backend::REPL.REPLBackend) at REPL.jl:139
 repl_backend_loop(backend::REPL.REPLBackend) at REPL.jl:200
 start_repl_backend(backend::REPL.REPLBackend, consumer::Any) at REPL.jl:185
 run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool) at REPL.jl:317
 run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:305
 (::Base.var"#875#877"{Bool, Bool, Bool})(REPL::Module) at client.jl:387
 #invokelatest#2 at essentials.jl:708 [inlined]
 invokelatest at essentials.jl:706 [inlined]
 run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:372
 exec_options(opts::Base.JLOptions) at client.jl:302
 _start() at client.jl:485

I'm not sure how this can be turned into something useful, but perhaps its a starting point. It may even be possible (though probably a bit futuristic) to implement a kwarg in run!, or to provide some debugging features for users that isolate errors due to user-defined functions.

[xiaozhour]

This is helpful - thanks Greg. It's good to have some options when unable to find the error messages during debugging!

[glwagner]

Probably the best solution right now is to re-run in a terminal. But there is still some improvement to be had since that option requires re-compilation, so will slow development down.

If you find any VSCode-specific information please post here. I wonder if VSCode can pipe stdout to file?

Also we may want to convert this to a discussion, what do you think?

[glwagner]

Well, the tee solution isn't great for REPL work because

$ julia | tee out.txt

then

julia> @info "hi tim"
[ Info: hi tim

julia> error("what the heck")
ERROR: what the heck
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] top-level scope
   @ REPL[2]:1

yields the file

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.5 (2021-12-19)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

^[[?2004h^M^[[0Kjulia> ^M^[[7C^M^[[7C@info "hi tim"^M^[[0Kjulia> ^M^[[7C@info "hi tim"^M^[[21C
^[[?2004l
^M^[[0K^M^[[0Kjulia> ^M^[[7C^M^[[7C^[[?2004h^M^[[0Kjulia> ^M^[[7C^M^[[7Cerror("what the heck")^M^[[0Kjulia> ^M^[[7Cerror("what the heck")^M^[[29C
^[[?2004lERROR: what the heck
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] top-level scope
   @ REPL[2]:1

^M^[[0K^M^[[0Kjulia> ^M^[[7C^M^[[7C^[[?2004h^M^[[0Kjulia> ^M^[[7C^M^[[7C
^[[?2004l

There's probably some setting to produce simple i/o even at the REPL though.

[glwagner]

@TimConnelly

[aramirezreyes]

Would something like https://github.com/BioTurboNick/AbbreviatedStackTraces.jl be a good approach to abbreviate the kernelabstraction stacktraces?

[pjrusello]

VS Code and most other terminals have an setting for the output buffer size. I set mine to 10000 (default is 1000) because I was getting some extra-long errors recently and couldn't see the important part of the stacktrace.

For VS Code, open File:Settings then search for "buffer" in the settings document. The specific setting is called Terminal>Integrated:Scrollback.

[glwagner]

Apple's iTerm2 has an option for unlimited scrollback: https://stackoverflow.com/questions/27616474/how-can-i-scroll-up-more-increase-the-scroll-buffer-in-iterm2

[jagoosw]

To get around the terminal not being long enough I use InteractiveErrors.jl which gives you a much shorter initial message and lets you copy the whole thing to the clipboard instead of printing it out. So I usually do that and then paste it into a file.

[ali-ramadhan]

For interactive debugging AbbreviatedStackTraces.jl and InteractiveErrors.jl look great.

For file output LoggingExtras.jl lets you keep terminal logging with a custom logger like OceananigansLogger plus writes the log to a file using FileLogger.

Example: https://github.com/ali-ramadhan/NeuralFreeConvection.jl/blob/7a440564a881b22a602f0a375ca5fb09b619ee92/train_free_convection_nde.jl#L114-L120

log_filepath = joinpath(output_dir, "training.log")
TeeLogger(
    OceananigansLogger(),
    MinLevelLogger(FileLogger(log_filepath), Logging.Info)
) |> global_logger

Could be used to provide stdout and stderr files.