Try to parse and run ble.sh

[andychu]

Why?

• it's a very substantial program using bash as a programming language, but it's not a batch program
• it's probably one of the hardest scripts to run (probably won't get it to run in totality, but maybe some parts will)
• It could be faster. maybe that will help optimize Oil.
  • I want Oil to be fast at char-by-char processing
• Maybe it will help us figure out what kind of API we need to let other people build other interactive UIs
  • $LINES and $COLUMNS, etc.

https://github.com/akinomyoga/ble.sh

[andychu]

#620 was the original issue, already addressed return 2>&1 etc.

[andychu]

Two issues blocking parsing:

• several cases of arith assignment where var name is dynamic doesn't work #640
• the \$file one

~/git/wild/ble.sh$ for name in */*.sh; do echo $name; ~/git/oilshell/oil/bin/osh -n $name | wc -l; done

...

memo/D1259.readbind.sh
84
out/ble.sh
            ((_ble_hook_c_$name=0))
                               ^
out/ble.sh:1644: Left-hand side of this assignment is invalid
0
src/benchmark.sh
2342
src/canvas.sh
    ((${_prefix}x=_pos[0]))
                 ^
src/canvas.sh:1521: Left-hand side of this assignment is invalid
0
src/color.sh
    (($_var=_ret))
           ^
src/color.sh:670: Left-hand side of this assignment is invalid
0
src/decode.sh
22515
src/def.sh
133
src/edit.sh
45409
src/history.sh
        ((_ble_builtin_history_rskip_dict[\$file]+=$2))
                                          ^
src/history.sh:701: Unexpected token in arithmetic context
0
src/util.sh
            ((_ble_hook_c_$name=0))
                               ^
src/util.sh:917: Left-hand side of this assignment is invalid
0

[andychu]

Note: can parse lib/core-syntax.sh (6651 lines, one of the biggest) by commenting out 5 instances of dynamic LHS in arith

• src/history.sh (1671 lines) parses with a few edits

[andychu]

• src/util.sh needs many patches (or relaxing parsing). This pattern is used all over the place.
  • there is at least one instance of it in our WILD corpus too
  • might want to take another look at arith assignment where var name is dynamic doesn't work #640 , or just allow parsing as a half-measure, and do
    evaluation later

Rules:

• arith_expr.Word:

  • statically a var name -> treat as VarRef -- but do this at evaluation time?
  • otherwise evaluate as word, and then treat that as an integer constant

• or just add Word on LHS as valid? And that must be an integer constant on RHS.

  • or on LHS it can be a var name?

[andychu]

@akinomygoa

On #640 I added a temporary hack shopt -s parse_unimplemented that parses dynamic arithmetic LHS but doesn't implement it. So at least this idiom won't prevent you from using osh -n. Now I can do:

for name in */*.sh; do echo $name; ~/git/oilshell/oil/bin/osh -O parse_unimplemented -n $name | wc -l; done

and there's only one remaining error:

src/history.sh
        ((_ble_builtin_history_rskip_dict[\$file]+=$2))
                                          ^
src/history.sh:701: Unexpected token in arithmetic context

If you want to use osh -n, maybe rewrite this without (( )) or with eval? I think this is particularly confusing because as we discussed \$file in this context does not mean '$file' -- it means something closer to "$file", correct?

---

And it should take 30 seconds to build osh from head on most Linux machines, although you can also wait until the next release:

https://github.com/oilshell/oil/wiki/Contributing

[akinomyoga]

Oh, I didn't notice your comment (but my user name is akinomyoga). I tried the latest commit which worked nicely! In testing I replaced the \$file line with eval '((...))', but I think I'll leave it as it is in the master branch for now because ble.sh is currently unlikely to run on Oil, so ble.sh is still only targeting Bash. Also, probably I'm too used to Bash syntaxes, but the \$file trick is natural to me. Nevertheless, I'll add some code comments above the line.

[akinomyoga]

I'm currently trying to source ble.sh in osh. ble.sh can be loaded without activation by passing the option --attach=none. Even if it doesn't fully work, I think it may be possible to just source ble.sh with minimal initialization. I'm currently using the following script to load ble.sh (with some modifications such as \$file)

# ble.osh
shopt -s parse_unimplemented
BASH_VERSION='5.0.11(1)-release'
BASH_VERSINFO=(5 0 11 1 release x86_64-redhat-linux-gnu)
set -o emacs
source "$(dirname "${BASH_SOURCE[0]}")/ble.sh" --attach=none

1. No line-editing mode by default ([[ -o emacs/vi ]])

The first one that I encountered is the line-editing mode detection. In Bash, the line editing mode can be tested with [[ -o emacs ]] and [[ -o vi ]]. When both of the tests fail, it implies that the line editing is turned off in Bash. ble.sh rejects the session with the line-editing turned off. I tested Oil and found that it implements the option, but both emacs and vi option is unset by default even though the twos cannot be unset at the same time by set -o/+o emacs/vi. In Oil, set +o vi implies set -o emacs, and set +o emacs implies set -o vi, and vice versa. What is the conceptual model for the state of the line-editing mode in Oil? I created a test for this.

test1.sh

#!/usr/bin/env bash

function print-line-editing-mode {
  local emacs=disabled vi=disabled
  [[ -o emacs ]] && emacs=enabled
  [[ -o vi ]] && vi=enabled
  echo "emacs $emacs, vi $vi"
}

echo -n "[default] "
print-line-editing-mode

echo -n "[set +o emacs] "
set +o emacs
print-line-editing-mode

echo -n "[set -o emacs] "
set -o emacs
print-line-editing-mode

echo -n "[set -o vi] "
set -o vi
print-line-editing-mode

echo -n "[set +o vi] "
set +o vi
print-line-editing-mode

echo -n "[set +o emacs] "
set +o emacs
print-line-editing-mode

set -o emacs

bash$ source test1.sh
[default] emacs enabled, vi disabled
[set +o emacs] emacs disabled, vi disabled
[set -o emacs] emacs enabled, vi disabled
[set -o vi] emacs disabled, vi enabled
[set +o vi] emacs disabled, vi disabled
[set +o emacs] emacs disabled, vi disabled

osh$ source test1.sh
[default] emacs disabled, vi disabled
[set +o emacs] emacs disabled, vi enabled
[set -o emacs] emacs enabled, vi disabled
[set -o vi] emacs disabled, vi enabled
[set +o vi] emacs enabled, vi disabled
[set +o emacs] emacs disabled, vi enabled

2. Comment: Array/scalar $FUNCNAME

The second one was $FUNCNAME which is actually the first array element ${FUNCNAME[0]} intentionally unsupported in Oil. I think the current Oil design is reasonable, but let me make a comment. The variable FUNCNAME was initially a scalar variable but extended to be an array in Bash 3.0, so it is a scalar variable and an array at the same time in Bash. This is the historical reason why the name is not FUNCNAMES but FUNCNAME even though it is an array. [ Note: Even for pure array variables, ble.sh uses this kind of parameter expansions in many places where the first element of an array contains the main result and the others contain additional information. ]

3. BUG: Start of ${@:offset} and ${*:offset} is shifted

The third one is that the results of ${@:offset} and ${*:offset} is shifted in Oil. I think this is a bug.

test2.sh

#!/usr/bin/env bash

set X Y Z W
echo "\$1=$1 \$2=$2 \$3=$3 \$4=$4"
echo "\${*:2}=(${*:2}) \${@:2}=(${@:2})"

osh$ source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Z W) ${@:2}=(Z W)

bash$ source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

zsh% source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

ksh$ source ./test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

4. BUG: ${!prefix_@} in a function fails in the second invocation

See the following test case. Even though it succeeds for the first time, it fails for the second time.

test3.sh

#!/usr/bin/env bash

abc_hello=1
function hello1 { echo "${!abc_@}"; }
hello1
hello1

$ osh test3.sh
abc_hello
  function hello1 { echo "${!abc_@}"; }
                             ^~~~
test3.sh:4: fatal: Bad indirect expansion: ''

[andychu]

This is great, thanks for all the patches! About the first one, I just changed something there, and I agree the behavior is odd, but I see bash and OSH agree (at HEAD):

$ bin/osh --rcfile /dev/null -c 'test -o emacs; echo $?; test -o vi; echo $?'
1
1
$ bash  --rcfile /dev/null -c 'test -o emacs; echo $?; test -o vi; echo $?'
1
1

$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

Do you get a different behavior?

I agree they are mutually exclusive but they both start off OFF. bash has a weird conceptual model but I think OSH copies it.

[akinomyoga]

Thank you for your comment!

I agree they are mutually exclusive but they both start off OFF. bash has a weird conceptual model but I think OSH copies it.

The default value in Bash depends on whether Bash is started in a non-interactive session or an interactive session. When you use the form sh -c '...', the shell starts in a non-interactive session while ble.sh works in an interactive session. In non-interactive sessions, test -o emacs/vi neither succeeds in Bash, which reflects the fact that the line editing is not activated in non-interactive sessions. This is reasonable. While, in interactive sessions, the emacs line-editing is activated by default, so it is reasonable that test -o emacs succeeds.

On the other hand, test -o emacs/vi neither succeeds in Oil by default even in the interactive session where the line-editing is enabled. The behavior of Oil in which test -o emacs/vi fails in both interactive and non-interactive sessions is strange.

By the way, the line-editing can be turned off in interactive sessions of Bash. The setting is set +o emacs. This is the reason why set +o emacs doesn't imply set -o vi in Bash unlike in Oil.