Trailing newlines not getting stripped from items

[alex-huff]

Checklist

• I have read through the manual page (man fzf)
• I have searched through the existing issues
• For bug reports, I have checked if the bug is reproducible in the latest version of fzf

Output of fzf --version

0.60 (devel)

OS

• Linux
• macOS
• Windows
• Etc.

Shell

• bash
• zsh
• fish

Problem / Steps to reproduce

Since commit 84e2262#diff-37113d7d967fa0fd21c0250890d9cda61c7d2591eeedd43017e9e7bc471b432aL138, newlines are no longer stripped from the end of each item when there are extra bytes (such as ANSI escape sequences) after the newline.

Before 84e2262:

fzfbefore-an.mp4

After:

fzfafter-an.mp4

Also, if you look at the after video I am moving back and forth between the first and second item and when I am focused on the second item the first item is no longer shown.

Here is a simple reproducing command:

printf "foo\n\e[0m\0bar\n\e[0m\0baz\n\e[0m\0" | ./fzf --read0 --ansi --with-nth ..

Before 84e2262:

After:

I will also try and find a reproducing command for the first item not showing.

[junegunn]

There was a regression. Does e4489dc fix the problem?

[alex-huff]

There was a regression. Does e4489dc fix the problem?

No, I am currently on e4489dc

[alex-huff]

fzfghost-an.mp4

Here is a reproducing command for the first item being truncated:

for _ in $(seq 19); do printf "foo\nbar\0"; done | ./fzf --read0

Change 19 so that the items no longer fit on screen.

[junegunn]

Since commit 84e2262#diff-37113d7d967fa0fd21c0250890d9cda61c7d2591eeedd43017e9e7bc471b432aL138, newlines are no longer stripped from the end of each item when there are extra bytes (such as ANSI escape sequences) after the newline.

I think the new behavior is correct here. fzf shouldn't trim the trailing new line without user's consent.

The second issue needs to be fixed, I'll look into it , but I don't think your patch is the correct way to approach the problem.

for _ in $(seq 5); do printf "foo\nbar\0"; done | fzf --read0 --height 11 --bind load:last

EDIT: Let me correct myself. I'll review your patch.

[junegunn]

Since commit 84e2262#diff-37113d7d967fa0fd21c0250890d9cda61c7d2591eeedd43017e9e7bc471b432aL138, newlines are no longer stripped from the end of each item when there are extra bytes (such as ANSI escape sequences) after the newline.

Can you show me the command you used here?

[alex-huff]

Can you show me the command you used here?

The command is massive because I wanted to support a bunch of weird edge-cases.
The full command that was used in those videos can be found here:
https://github.com/alex-huff/dotfiles/blob/main/dotfiles/.local/bin/rg-fzf

It relies on:
https://github.com/alex-huff/dotfiles/blob/main/dotfiles/.local/bin/rg-fzf-rg-wrapper.py

[alex-huff]

I think the new behavior is correct here. fzf shouldn't trim the trailing new line without user's consent.

I don't disagree with this and I can easily just update my script to omit trailing newlines. However, I think the following commands should produce visibly similar output.
Edit: the script seems to work fine when just removing \e[0m entirely. (similar to the second command below)

printf "foo\n\e[0m\0bar\n\e[0m\0baz\n\e[0m\0" | ./fzf --read0 --ansi --with-nth ..

printf "foo\n\0bar\n\0baz\n\0" | ./fzf --read0 --with-nth ..

printf "foo\n\0bar\n\0baz\n\0" | ./fzf --read0

It seems weird to me that a trailing newline will be stripped as long as ANSI escape sequences like \e[0m don't exist after it. It also seems weird that --with-nth .. causes newlines to be stripped but without it they are kept.

[junegunn]

--with-nth is an option that commands fzf to transform the input and it removes the trailing whitespaces after transforming it. It has been so for quite some time, and this is indeed a desirable behavior in many common scenarios. Multi-line support is a very recent addition to fzf, so this new-line trimming case was definitely not something we considered back then.

Two things.

• Maybe --with-nth .. should be a no-op. OR, if the user knows that it removes the trailing whitespaces, they can take advantage of it to just remove trailing whitespaces.
• The first case where --with-nth .. failed to remove \n is not an expected behavior. We should fix it, but it can be quite tricky to get it right. When tokenizing the fields for transforming the input, fzf doesn't trim ANSI escape sequences, because the we want to keep the colors that are stateful, and the "trimming" we mentioned above happens here, and because there is \e[0m, fzf cannot remove new line before it.

[alex-huff]

• The first case where --with-nth .. failed to remove \n is not an expected behavior. We should fix it, but it can be quite tricky to get it right. When tokenizing the fields for transforming the input, fzf doesn't trim ANSI escape sequences, because the we want to keep the colors that are stateful, and the "trimming" we mentioned above happens here, and because there is \e[0m, fzf cannot remove new line before it.

It worked before by calling the now removed item.text.TrimTrailingWhitespaces() after processing ANSI stuff. Was this removed for a reason?

[junegunn]

Good point. It was removed while implementing --accept-nth and factoring out the transformation logic. I'll see what I can do.

[junegunn]

So TrimTrailingWhitespaces is back. Although the question of whether we should remove the trailing new lines or not in multi-line mode remains, we're back to the old behavior.