[discarded] Add append_file to CodeActAgent, incl. tests; some markdown fixes

[tobitege]

This PR adds a new method append_file(content: str) to the agent skills.

In testing OD with some file operations, it is a noticable issue for repeated errors by the model (used Gemini Pro 1.5) that adding content to a file often causes exceptions due to an invalid "start" line number for the edit_file command.

It seems very hard for the model to identify or keep track the total number of lines in a file or it is just not good enough in counting (ask an LLM to count the words of its answer and you know what I mean).
This issue can cause extra cost when it shouldn't.

Such an error then looks like this:

An example prompt to demonstrate the use of it could be like:
"Write the numbers 5 to 10 into a new file named test.txt"
A followup prompt could then just say:
"Append the numbers 20 to 25 to the same file."
If it were to use the edit_file command, chances would be high that it would use the wrong line number(s) again.

Also: fixed some more markdown lints in readme's, added several test methods for append_file.
Pre-commit was clean.

I need help with this PR! I am not a Python developer and not a pro-Linux user! /shamebell
How to run these tests - I haven't figured that out yet.
It isn't clear to me, whether I'm to produce some of the log outputs for demonstration
or I missed some place to add the new method name.
Would be great if someone could assist me or provide advice, thanks a lot in advance! :)

[li-boxuan]

(this is not a review) Thank you! I think this could potentially be helpful! Given that we are running a few benchmarks on the current CodeActAgent, I propose we don't merge this (or other PRs attempting to tweak CodeActAgent or edit tools) in a week.

[tobitege]

(this is not a review) Thank you! I think this could potentially be helpful! Given that we are running a few benchmarks on the current CodeActAgent, I propose we don't merge this (or other PRs attempting to tweak CodeActAgent or edit tools) in a week.

Absolutely fine with me. Good luck with the benching! :D

[li-boxuan]

A counter-argument: agent skills should only accept skills that are non-trivial to implement. Appending could be easily done via python or bash commands, and we probably shouldn't prompt LLM to use it.

[tobitege]

A counter-argument: agent skills should only accept skills that are non-trivial to implement. Appending could be easily done via python or bash commands, and we probably shouldn't prompt LLM to use it.

There are several logistical, technical and economic advantages of an explicit append_file command:

• An "append" is easier to understand and does not require the LLM to first determine the last line number of a file.
• We cannot require or presume, that the targeted file has a trailing blank line (unlike linted Python source files) that could easily be "replaced". But even that blank line might be intended and needed to be preserved.
• The LLM may then intransparently decide to replace that last content line ("empty" or not) or wrongly come up with a line number higher than the current line count (as in above screenshot).
• It may itself need to create an extra Python script to determine the correct line number, requiring extra turns, raising time and cost. And that potentially for any subsequent part of a repetitive task (e.g. iterate files in folders for summarizations).

Above is "trivial" for us, but with the current implementation it is not a cost-effective and seemingly "harder" way for the LLM to achieve results.

Major question:
Alternatively, could edit_file's behavior be slightly changed, that IF the LLM's start line number is higher then the actual file's line count do an append automatically?
Is the behavior more determined due to CodeActAgent needing Python file generation being strict?

Potential PR changes
When I started this PR, I intentionally stayed away from changing the core edit_file, just to be safe.
However, I could easily imagine the edit_file and potential append_file methods use a protected common method with "append" just being a boolean parameter, which certainly would reduce the agentskills codebase.

[li-boxuan]

There are several logistical, technical and economic advantages of an explicit append_file command:

Your points are valid and I agree it is hard to LLM to use edit_file correctly to realize append, but... it could simply do echo "xxx" >> file, right?

[tobitege]

There are several logistical, technical and economic advantages of an explicit append_file command:

Your points are valid and I agree it is hard to LLM to use edit_file correctly to realize append, but... it could simply do echo "xxx" >> file, right?

I'll look into it, but we do have the optional linting around it, so not sure if we can cut it down to a simple echo

[neubig]

I think this is something that's good to experiment with. @tobitege when you're happy with the code we can try to run a benchmark to see the effect on accuracy/cost. Please ping me then!

(blocked by #2085 as well)

[tobitege]

Thanks, I'll try to get the code cleaned up a little more first and let you know when ready. 👍

[tobitege]

@neubig got my branch updated, make lint'ed, agentskills tests successfull, good for review.

Note: I did add append_file to prompt.py but not with specific examples.

[tobitege]

Need to check why integration tests fail, might be merge issue because of outdated mock log files.

[tobitege]

My branch is in a bad state. Will re-issue this PR with fresh branch off of main.