Run ble.sh (2022)

[andychu]

Continuation of #653

At the very least we should see if we can reproduce the results of running ble.sh unit tests in #762 on our new container-based continuous build

I'm also curious what functionality is missing

[andychu]

Just ran it again: http://travis-ci.oilshell.org/github-jobs/2022-01-07__05-34-01.wwz/_tmp/soil/logs/ble-test.txt

[section] util: 895/1011 (116 fail, 0 crash, 0 skip)
osh-0.9.6$ ^D
DONE

[andychu]

Hm from #762 we were at

[section] util: 913/1011 (98 fail, 0 crash, 0 skip)

Hm maybe we're on the wrong branch? I think there is a branch that is maybe not maintained

But still there is a regression

@akinomyoga Can you tell me if this osh branch is still maintained? Or should I try running the tests on master?

https://github.com/oilshell/oil/blob/master/test/ble.sh#L18

I revived this build we did in 2020! I haven't touched it since then but it still kinda works. Just curious if there have been any changes

[andychu]

Also continuing the thread from #257 , here is a page I drafted about help I would like to raise money to pay for

https://github.com/oilshell/oil/wiki/Compiler-Engineer-Job

Although now that I read it over it looks too detailed. The short summary is: Write a 10K line Python compiler in Python, and 10K line C++ runtime, and get a free shell :-)

(and get paid, although I'm still working on raising the funds)

I think this is a very fun project for the right person. It's fun but I think will take full time effort. (And remember it is over half done, but the code is messy because of the way we used MyPy, and probably needs a rewrite)

---

I made this web page with line counts, although for some reason it makes things look bigger than they are ... it is a small amount of code, at least compared with GNU bash itself:

https://www.oilshell.org/release/0.9.6/pub/metrics.wwz/line-counts/for-translation.html

[akinomyoga]

#257 (comment) by @andychu

So, actually, I guess, after a few hooks for the user input are provided by Oil, it is not so hard for me to adjust ble.sh (and probably to submit additional small fixes to Oil) so that it works as a line editor of Oil.

Oh really that would be very interesting -- do you have an idea of the missing features? Maybe we can discuss more on #653 or start a new issue for "ble.sh 2022 :)"

I don't remember the details after two years, but basically

1. some alternative mechanism for bind -x, which is also related to the signal handling,
2. read -t 0 or poll/select
3. some mechanism of eval -g (Provide an API to evaluate code in a subinterpreter #704) or some API for the save/restore the state of options, signal handling, etc.

I think there can be several approaches to 1 and 2:

• One way is to emulate Bash's behavior perfectly including the behavior of bind -x and signal handling. This is the easiest way from the perspective of ble.sh, but I think it will take time to implement the perfect emulation of the signal handling of Bash. Also, this approach that ble.sh takes for Bash is not so clean.
• Also, the design will largely depend on who controls the main loop of the interactive shell. If the line editor written in shell (ble.sh) can control the main loop of the shell, the line editor (ble.sh) can behave in the pull style (it is easy to write a pull adapter for push implementation such as while read-byte; do process-byte; done). In that case, hooks for the user input are not needed because the line editor (ble.sh) can directly read a byte from stdin using read.
• Also we need to consider the special treatment of the signal handling for ble.sh (or a line editor written in shell). For example, SIGINT (C-c) should not terminate the line editor (ble.sh) nor should not trigger the trap handler set by the user in the middle of the processing of the line editor (ble.sh).

---

oil-native is making slow progress -- as of the latest release there are 1131 tests passing, out of 1900 or so.

Hmm, OK. If we use ble.sh with the Python osh/oil, it might be a good demonstration of osh/oil, but I have to say that the response is too slow to be used as a line editor of daily use. Also, in my vague memory, the footprint of osh with ble.sh was of the order of a few or several hundred megabytes (but it might be different).

[akinomyoga]

@akinomyoga Can you tell me if this osh branch is still maintained? Or should I try running the tests on master?

Not maintained, meaning that it doesn't track the changes in the latest master. I think the osh branch is basically based on ble.sh in about the middle of 2020. Actually, the latest ble.sh has many changes since May, 2020 and relies on other Bash's behavior that ble.sh didn't use on May, 2020 and Oil doesn't implement. For example, saved_func=$(declare -f func) / eval "$saved_func" #647 are now used in important places, but I guess it will require much efforts to support declare -f func in osh.

[andychu]

Ah OK thanks for the info. Off the top of my head I think we have read -t 0 already, which does select().

I don't think declare -f is too hard but I would have to think about it. (We might just print the source code rather than pretty-printing the AST like bash does.)

oil-native should make things faster and smaller! It should be almost as fast as bash to start, and it will use less memory, although I don't know exactly how much. And as mentioned I think we can easily optimize speed by 2x.

There is a rough idea of memory usage here:

https://www.oilshell.org/release/0.9.6/benchmarks.wwz/mycpp-examples/

e.g. on small examples the Python version uses 7.3 MB while the C++ version uses 3.4 MB. That seems like it's going in the right direction but probably not good enough. (hence why I am looking for help!)

How much memory does ble.sh use when run under bash?

I agree oil-native is the important part, and that's why I'm seeking help for it. In the meantime, if you have any feedback about Oil let me know!

[andychu]

Hm actually now I realize a good memory benchmark would be source ble.sh in bash and OSH, and compare memory usage. This will even work in oil-native now I think, since it just has to parse and load all the function bodies.

I suspect it will be a lot higher because Oil has a very detailed representation of the code, and it does eager parsing inside $(( )) and ${}, unlike bash. But it will be interesting to see how much. Also I'm sure it can be reduced 2x with optimization, although maybe it's 10x higher already ...

[akinomyoga]

Ah OK thanks for the info. Off the top of my head I think we have read -t 0 already, which does select().

Oh, OK, I have now confirmed that. I think I need to update the wiki page Running ble.sh With Oil · oilshell/oil Wiki.

I don't think declare -f is too hard but I would have to think about it.

That's nice to hear.

(We might just print the source code rather than pretty-printing the AST like bash does.)

Yeah, that is also one option.

I remember that the source of the functions in JavaScript which is obtained by .toString() is also the actual source code (instead of the one reconstructed from AST), where the code comments are also preserved. There was an interesting trick that one writes the documentation of JavaScript functions in the first comment in the function body so that it can be extracted from func.toString(). If one writes sufficient information in the comments in the function body, that could be used for some kind of reflection.

How much memory does ble.sh use when run under bash?

Actually, it depends on the Bash version. The memory use of ble.sh is larger for recent Bash. For example, the latest ble.sh uses about 50 MB in the latest Bash. Actually, this is one of the recent problems of ble.sh.

Hm actually now I realize a good memory benchmark would be source ble.sh in bash and OSH, and compare memory usage.

I have tried it.

Bash version	plain	ble.sh (2020-07) only source / +module+attach	ble.sh (2022-01) only source / +module+attach
Bash-3.2	3.6 MB	14 MB / 17 MB	15 MB / 19 MB
Bash-4.3	3.8 MB	17 MB / 29 MB	23 MB / 37 MB
Bash-5.1	4.3 MB	21 MB / 37 MB	27 MB / 47 MB
osh-0.9.6	15 MB	76 MB / n/a	90 MB / n/a

In the previous reply, I wrote a few hundred megabytes used by osh in my memory, but that was wrong. (Actually, I have slightly modified ble.sh for osh to suppress the memory use. I think I remember the footprint before that modification.) The memory increase of osh is roughly about (76 - 15) / (21 - 4.3) = 3.653x bash. If ble.sh is fully loaded and activated, I think the footprint of osh will surpass 100 MB: 15 + 3.653 * (47 - 4.3) = 171 MB with a simple extrapolation.

[andychu]

Great, thank you for trying it! This is very helpful. I filed #1070 to automate this -- it will help with the C++ translation, which is the focus of the upcoming year.

Actually it's better than I thought -- osh-0.9.6 creates a huge number of Python classes, which you can see with osh -n myscript.

Python objects are very large:

$ python2

class C(object):
  pass

>>> x = C()

>>> sys.getsizeof(x)
64

>>> sys.getsizeof(x.__dict__)
280

So the translation to C++ should reduce that by a lot -- a pointer will be 8 bytes, an integer will be 4 bytes, etc. It's statically laid out rather than dynamic with a __dict__.

I think we can get within 2x of bash on the first try, and then optimize it with enough effort.

[andychu]

Also Oil has something like the JavaScript feature you're talking about. It uses ### for doc comments:

http://www.oilshell.org/blog/2021/09/multiline.html#reminder-doc-comments-with

This works with traditional shell functions as well as Oil proc:

f() {
  ### my comment
  echo
}

pp proc f  # pretty print a table

You could use these in ble.sh right now because the syntax is compatible with bash. That is, you can use Oil as a documentation extractor if that proves useful.

Although now I see you use a ## as a prefix, and it has multiple lines. Hm let me think about this.

https://github.com/akinomyoga/ble.sh/blob/master/lib/core-complete.sh

What kind of metadata would you want to put in there?

---

Anyway, I welcome more feedback on Oil. I hope it will converge with ble.sh at some point, and we both agree C++ translation is important.

Also, I realized that compat_array which you implemented, to make ${a} equal to ${a[0]} should be the default (we should change it to strict_array and opt in). Even though that behavior is confusing, multiple shells agree on it (I think), and Oil has a separate array syntax that behaves more conventionally (const x = a[i], etc.).

This is #967

Also I remember you know quite a bit about C++ too, and I think the translation task could be pretty fun in that respect. Here is a listing of the generated code:

https://www.oilshell.org/release/0.9.6/pub/metrics.wwz/line-counts/oil-cpp.txt

   1366 _tmp/native-tar-test/oil-native-0.9.6/mycpp/gc_heap.h
   2321 _tmp/native-tar-test/oil-native-0.9.6/_build/cpp/runtime_asdl.cc
   4221 _tmp/native-tar-test/oil-native-0.9.6/_build/cpp/syntax_asdl.h
   9999 _tmp/native-tar-test/oil-native-0.9.6/_build/cpp/syntax_asdl.cc
  29060 _tmp/native-tar-test/oil-native-0.9.6/_devbuild/gen/osh-lex.h
  32805 _tmp/native-tar-test/oil-native-0.9.6/_build/cpp/osh_eval.cc
  93228 total

And this already passes 1131 out of 1900 spec tests! It can run those fibonacci and bubble_sort examples, etc. I think it should be able to source ble.sh, or at least the old copy. I will try that out.

---

Actually what is the reason for the n/a in the table? Is ble.sh now using a feature that OSH cannot source? Or was there a regression?

(I didn't look at why the unit test numbers went down, but I was a little surprised by that, since I don't know of any OSH regressions. It is very highly tested. I may dig into it more)

[andychu]

Thinking about the doc comments a little more, I think it wouldn't be that hard to support ## as well as ### and then add another column for pre_doc_comment in addition to post_doc_comment

osh$ f() {
> ### hi
> echo
> }
osh$ pp proc f
proc_name       doc_comment
f       hi

The strings are encoded in QSN format, so this can be parsed and used by a doc generator. I'm not sure if this is important for ble.sh, but just mentioning it in case. I guess it is probably not that hard to make a custom parser but I think it is nice if the language can do it for you.

[akinomyoga]

Also Oil has something like the JavaScript feature you're talking about. It uses ### for doc comments:

Oh, pp proc f looks very useful although I currently don't have a plan to use them from ble.sh itself. I just remember some use cases in JavaScript. The purpose of document comments in ble.sh (starting from ##) is purely notes for me (at least currently).

What kind of metadata would you want to put in there?

I usually put 1) what arguments the shell function accepts, 2) what variable the shell function accesses through the dynamic scoping, and 3) the description of exit status, along with the details of each data format, etc. I actually don't think the short description of the function is so important because I feel that that information should be usually represented by the function name,

Actually what is the reason for the n/a in the table? Is ble.sh now using a feature that OSH cannot source? Or was there a regression?

"n/a" for +module+activate: Yes, there are still some syntaxes that OSH fails to parse in modules. It's not a regression because I was actually aware back then. These were related to or relying on the module for the syntax analysis that is designed for Bash syntax but not for Osh syntax, so I didn't make much effort to find out the causes and to report it here at that time.

"n/a" for ble.sh (2022-01): Just because I haven't tried it yet. First of all, ble.sh cannot be loaded in osh without modifications and there were already many modifications to the osh branch of ble.sh in 2020-07. I first need to rebase these changes to the latest version of ble.sh. In addition, the codebase is largely changed so I guess there will be again many new problems (just as in #653). I did know that there are still many errors in parsing other modules of ble.sh (including syntax analysis), so I don't think all the incompatibility of osh is solved by the discussion in #653. I haven't tried it, but I anticipate that we need efforts to make the latest version of ble.sh loadable in osh.

(I didn't look at why the unit test numbers went down, but I was a little surprised by that, since I don't know of any OSH regressions. It is very highly tested. I may dig into it more)

I'm not sure what you mean by regression, but any behavioral changes could affect the results. It's not surprising to me that the changes to osh behavior affect the result of tests.

Thinking about the doc comments a little more, I think it wouldn't be that hard to support ## as well as ### and then add another column for pre_doc_comment in addition to post_doc_comment

I actually don't use the document comments from inside the script. Maybe I do some static analysis in the future, but I don't think I will do it by the script but would rather write a program in C++ or other languages.

[akinomyoga]

ble.sh (2022-01) for osh

I have finished the rebasing of the branch osh on the latest master.

• read -e (edit line using ble.sh), which had worked with osh-0.8.3, does not work now with osh-0.9.6.
• The footprint after sourcing ble.osh (2022-01) is 90 MB (I have updated the table in the above comment).

Test results

Test summary in osh-0.9.6. The full log is here: blesh-test.txt [ Note: this text file contains escape sequences for color codes, so I recommend opening it with less -r in a terminal ].

[section] ble/main: 16/19 (3 fail, 0 crash, 0 skip)
[section] ble/util: 985/1193 (208 fail, 0 crash, 0 skip)
[section] ble/canvas/trace (relative:confine:measure-bbox): 0/0 (0 fail, 0 crash, 17 skip)
[section] ble/decode: 33/33 (0 fail, 0 crash, 0 skip)

• Many errors are caused by missing declare -f for the function definition.

For comparison, this is the expected result generated by Bash:

[section] ble/main: 19/19 (0 fail, 0 crash, 0 skip)
[section] ble/util: 1193/1193 (0 fail, 0 crash, 0 skip)
[section] ble/canvas/trace (relative:confine:measure-bbox): 17/17 (0 fail, 0 crash, 0 skip)
[section] ble/canvas/trace (cfuncs): 18/18 (0 fail, 0 crash, 0 skip)
[section] ble/canvas/trace (justify): 24/24 (0 fail, 0 crash, 0 skip)
[section] ble/canvas/trace-text: 11/11 (0 fail, 0 crash, 0 skip)
[section] ble/textmap#update: 5/5 (0 fail, 0 crash, 0 skip)
[section] ble/unicode/GraphemeCluster/c2break: 72/72 (0 fail, 0 crash, 0 skip)
[section] ble/unicode/GraphemeCluster/c2break (GraphemeBreakTest.txt): 3251/3251 (0 fail, 0 crash, 0 skip)
[section] ble/decode: 33/33 (0 fail, 0 crash, 0 skip)

[andychu]

Great thank you for testing it! It's good to see some progress with 985/1193 vs. 895/1011.

I'm not sure what happened with read -e, but I think we can easily implement it.

90 MB doesn't seem too bad considering how large the Python objects are. I'm looking forward to testing out oil-native in #1070. I'll let you know when we do that.

---

Although I also wonder what method you use to get 3.8 MB "plain" for bash 4.3 ?

I'm trying to compare plain bash vs. OSH here too. The way I do it is $sh -c 'sleep 0.001; cat /proc/$$/status' and then look at VmRSS and VmPeak.

https://github.com/oilshell/oil/blob/master/benchmarks/vm-baseline.sh#L39

Actually bash's peak seems to be higher than OSH which seems wrong because of the Python issue, but I can't see any bug in the benchmark. (This chart doesn't include oil-native.)

https://www.oilshell.org/release/0.9.0/benchmarks.wwz/vm-baseline/

This seems reasonable but maybe there is a better way, and a way to make our measurements more consistent.