Store hard drive diffs in some way

[vadosnaprimer]

In cases when the game has to be installed onto a hard drive image, it may be legally iffy to allow freely sharing the movie file containing the diff as a SaveRAM anchor.

And it may be too huge to even upload to tasvideos (up to hundreds of MBs if we only take Amiga, but also some MAME games need installation, and if we ever emulate IBM PC then we're talking gigs).

My initial thought was to keep SaveRAM for those movies that depend on prior installation separately from the movie, and pull from disk as needed. There's some issue with it that I'd like CPP to explain.

But if diff is actually written to the file it applies to (as libTAS does it), that also sounds like a great solution, even tho ideally it'd be kept separately and applied on the fly in memory.

Also ideally there'd be a libTAS like option to not touch the file and to only keep the diff in memory, and to kill it once the game is closed, instead of flushing.

[nattthebear]

Based on some offline discussions with @vadosnaprimer, I'm marking this as waterbox because that's the only place we see any immediate need for this.

Suppose we have a waterbox core that uses a writable disk image. During initial load (before seal), it calls

wbx_mount_file(..., writable: true)

with a many-megabyte file. That file then stays around for the entire life of the guest core, and the guest uses IO syscalls to write some data to the file, but probably doesn't overwrite every byte in the file.

The requirements on waterbox are then:

1. Faster savestates. As of right now, the full disk image is stored in every savestate. There's no dirty detection for these. Ideally, if only 10KB of data has changed, we'd only write that changed 10KB. For the savestates to be coherent, Bizhawk would have the obligation to pass the exact same starting image file to wbx_mount_file every run, which should be no surprise; it's the same sort of restriction we already have with syncsettings and romfiles; you must pass the exact same sequence of data and commands to the guest pre-seal in order to have coherent savestates across sessions post-seal.
2. Interrogate the file's current contents without unmounting it.* As of right now, when a file is loaded in the guest in this way, you can get back the full content of the file, including any changes the guest made to it, by calling wbx_unmount_file. But that can only be done if the file isn't currently loaded in the guest, which makes it not workable here. We need some way to get the current state of the data out so it can be stored in external diffs without interrupting it. Since Bizhawk knows what it passed to wbx_mount_file, we could just export the diff that we'll be generating in bullet 1.
3. Pass a diff when mounting. Once we have bullet 2, it makes sense to allow passing that diff back to wbx_mount_file, so that it will compute the initial state for you.

[CasualPokePlayer]

Keep in mind the current wbx_mount_file with writable is bugged, possibly causing crashes with savestate usage. It hasn't been an actual issue since the use case it was intended for (DSi NAND) doesn't use it, instead just putting it all in memory and has some abstract file i/o core apis handling it instead of C file i/o apis (also particularly important since state size grows drastically).

Also, the issue I had with this originally was less with waterbox details, but more with the idea that you want to create SaveRAM but without actually storing it in movies. This should not be done, SaveRAM should always be stored in movies. If something SaveRAM like is wanted without actually being SaveRAM per se, the core has the option to use the "User folder" (well, this is more a hack for Encore since it dumps NAND and shit there, perhaps this would be a good approach here). It could also possibly be some file sent as a "ROM" with multidisk bundler.

[vadosnaprimer]

Documenting a test case for Amiga.

• Put amiga-os-310-a4000.rom/Kickstart v3.1 r40.068 (1993-12)(Commodore)(A4000)[!].rom in Firmwares.
• Load Workbench v3.1 rev 40.42 (1994)(Commodore)(M10)(Disk 2 of 6)(Workbench)[!].adf (filename may slightly vary, disk 2 is required).
• Set Machine configuration to A4000.
• Wait for workbench screen to appear.
• Double-click Workbench3.1 icon, then Utilities icon.
• Left-click MultiView icon, hold right mouse button, hover on the top Icons menu, highlight Copy and release RMB.
• That will make Copy_of_MultiView appear in the same folder.

This operation calls zfile_fwrite()->fwrite() inside the core, and when done in libretro-uae (which is our upstream), it writes the changes to the ADF file upon core exit.

[vadosnaprimer]

Also, the issue I had with this originally was less with waterbox details, but more with the idea that you want to create SaveRAM but without actually storing it in movies. This should not be done, SaveRAM should always be stored in movies. If something SaveRAM like is wanted without actually being SaveRAM per se, the core has the option to use the "User folder" (well, this is more a hack for Encore since it dumps NAND and shit there, perhaps this would be a good approach here). It could also possibly be some file sent as a "ROM" with multidisk bundler.

While there's no solution for this on the wbx end, I kept thinking of a workaround on the core side. My plan is mounting the writable file just like we currently mount read-only ones, and having the core load it into memory and use it like that.

At first it'd write to it in a way that only keeps all the changes in savestates, because its initial use is writable amiga floppies, which are tiny and need no extra fixes for the core to even do it.

Then I might turn whatever is written to the file into a SaveRAM kinda diff and store it on the hawk side, just like normal nvram, and read from it when loading the session again.

Real heavy-weighting starts when those writable files need to be huge and can't be put into a savestate or a movie (hundreds of megs). Maybe the diff should only exist on the hawk side, with some hook in the core that substitutes data in core's memory with hawk-side nvram. And when we flush nvram, it'd go to that User folder and become an anchor for the new movie we start from this nvram. And in the new session during wbx init, that diff would get superimposed on the buffer that holds the original file's data and THEN we'd seal wbx, so the diff itself doesn't go to any new savestates, only changes to it.