Skip to content

Syntax highlighting for shell heredoc is backward due to interference with rules for quotes #4715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks done
dotysan opened this issue May 26, 2025 · 7 comments · May be fixed by #4718
Open
2 tasks done

Syntax highlighting for shell heredoc is backward due to interference with rules for quotes #4715

dotysan opened this issue May 26, 2025 · 7 comments · May be fixed by #4718
Assignees
@zyv
Labels
area: mcedit mcedit, the built-in text editor prio: medium Has the potential to affect progress state: approved If not using PRs: associated branch has been approved
Milestone
4.8.34

Comments

@dotysan
Copy link

dotysan commented May 26, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Midnight Commander version and build configuration

GNU Midnight Commander 4.8.30-494-g589227e13
Built with GLib 2.80.3
Built with S-Lang 2.3.3 with terminfo database
Built with libssh2 1.11.0
With builtin editor
With subshell support as default
With support for background operations
With mouse support on xterm
With internationalization support
With multiple codepages support
Virtual File Systems:
 cpiofs, tarfs, sfs, extfs, ftpfs, sftpfs, shell
Data types:
 char: 8; int: 32; long: 64; void *: 64; size_t: 64; off_t: 64; uintmax_t: 64;

Operating system

Linux localhost 6.10.2-x86_64-linode165 #1 SMP PREEMPT_DYNAMIC Tue Jul 30 15:03:21 EDT 2024 x86_64 GNU/Linux

Is this issue reproducible using the latest version of Midnight Commander?

  • I confirm the issue is still reproducible with the latest version of Midnight Commander

How to reproduce

Use mcedit to open a file with this:

cat <<EOF
$HOME
EOF

And you won't see the variable highlighted. This implies they aren't expanded. Which is false.

However open a file with this:

cat <<'EOF'
$HOME
EOF

And you will see syntax highlighting. This implies they are expanded. But they aren't.

This is backwards!

Expected behavior

The opposite.

Actual behavior

See above.

Additional context

No response
@dotysan dotysan added prio: medium Has the potential to affect progress needs triage Needs triage by maintainers labels May 26, 2025
@github-actions github-actions bot added this to the Future Releases milestone May 26, 2025
@zyv
Copy link
Member

zyv commented May 27, 2025

And you won't see the variable highlighted. This implies they aren't expanded. Which is false.

I don't think this implies anything other than that the syntax highlighting in mcedit is not to be taken too seriously :)

If you look at the source, you will find the following:

https://github.com/MidnightCommander/mc/blob/master/misc/syntax/sh.syntax

context exclusive whole <<\[\s\]\[-\]\[\s\]\[\\\]EOF EOF green
context exclusive whole <<\[\s\]\[-\]\[\s\]\[\\\]END END green

This basically means that anything beginning with <<EOF and ending with EOF should be highlighted in green, regardless of what's inside.

For reference, see how string literals ("") are handled below, and why the variables are highlighted there:

context " " green
    ...

Writing <<'EOF' instead makes all other rules apply. This means that the inside of a heredoc is not seen as a string literal but as normal shell code. As a result, variables are highlighted, but also everything else.

The syntax definition language of mcedit is very restricted and is only meant to provide basic coloring, not proper language support, especially for complex languages.

I don't think it's possible to express rules such as "something starting with '<<' and then letters, but eventually closing with the same letters" because, as far as I know, it doesn't have back references.

Having said that, I think there are two ways to deal with the situation, other than you taking over maintenance of mcedit and rewriting its syntax engine.

  1. We can remove the "fake" heredoc rules to make things less confusing. These rules have been around since the beginning of time and were added by the original author of CoolEdit.
  2. We can add support for single quotes in delimiters and copy & paste the variable highlighting rules from double quotes.

I don't have a strong opinion either way.

Obviously, Option 1 feels like a downgrade, especially since nobody has complained in 30 years, and does not necessarily make it any better.

However, Option 2 has the limitation of still not supporting arbitrary heredoc delimiter tokens (which may be confusing if the common ones are handled more correctly).

@zyv zyv added area: mcedit mcedit, the built-in text editor and removed needs triage Needs triage by maintainers labels May 27, 2025
@zyv zyv modified the milestones: Future Releases, 4.8.34 May 27, 2025
@zyv
Copy link
Member

zyv commented May 27, 2025

Image

@ossilator
Copy link
Contributor

deja vu 😁

@zyv
Copy link
Member

zyv commented May 27, 2025

... and #2931 :(

@zyv zyv changed the title syntax highlighting for shell here doc is backward Syntax highlighting for shell heredoc is backward due to quotes May 27, 2025
@zyv zyv changed the title Syntax highlighting for shell heredoc is backward due to quotes Syntax highlighting for shell heredoc is backward due to interference with rules for quotes May 27, 2025
zyv added a commit that referenced this issue May 27, 2025
@zyv
Ticket #4715: fix basic heredoc highlighting for shell syntax
c3adff6 
Signed-off-by: Yury V. Zaytsev <[email protected]>
@zyv zyv linked a pull request May 27, 2025 that will close this issue
Open
@zyv
Copy link
Member

zyv commented May 27, 2025

I can't believe it myself, but it seems that I have found a hack to make it work in #4718! Could you guys please test it?

@mc-worker mc-worker added the state: approved If not using PRs: associated branch has been approved label May 29, 2025
@github-actions GitHub Actions
Copy link

The changes in the branch associated with this issue have been approved.

mc-butler pushed a commit that referenced this issue May 30, 2025
@zyv @github-actions
Ticket #4715: fix basic heredoc highlighting for shell syntax
028f2b1 
Signed-off-by: Yury V. Zaytsev <[email protected]>
zyv added a commit that referenced this issue May 30, 2025
@zyv
fixup! Ticket #4715: fix basic heredoc highlighting for shell syntax
fe8cc63 
Signed-off-by: Yury V. Zaytsev <[email protected]>
@zyv
Copy link
Member

zyv commented May 30, 2025

FYI, here are my tests and what it looks like now:

mc-4715-heredoc.sh.zip

Image

The keyword exclusive is important for the hack to work (something with multi-line matching?!). I re-added it where it was missing and clarified the comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zyv zyv

Labels
area: mcedit mcedit, the built-in text editor prio: medium Has the potential to affect progress state: approved If not using PRs: associated branch has been approved
Milestone
4.8.34
Development

Successfully merging a pull request may close this issue.

Ticket #4715: fix basic heredoc highlighting for shell syntax MidnightCommander/mc
4 participants
@zyv @ossilator @dotysan @mc-worker