file_io: Zip file support

[dholm]

Support for loading files stored inside a zip file.

Treats zipped files as folders and allows loading files from inside the
zip file. Archived folder structures are browsable just as if they were
decompressed.

Compressed files are loaded on-demand from the archive.

Nested zip-files are currently not supported.

[dholm]

I've tried to address all the issues now.

I have tested it with the NES core as well as Minimig. I would like to test it with some really large image, like a harddrive image for ao486, but I don't have one set up at the moment.

[sorgelig]

Hard drive images are supposed to be writable, so forget about such usage.
Zip is only for read-only files. Some cores use read-only flag, so you can hardcode this flag for files from zip.

[sorgelig]

By quick look at diff file i've noticed you've removed some earlier returns on error in sharpmz. Hope you didn't brake that code.

[sorgelig]

oh, you again put these std:: types. I really hate them. code is harder to read. And additionally VS code highlight feature doesn't like std types. It seems the tool agrees with me - std:: is evil :)

[sorgelig]

btw, something wrong with file lister. It doesn't let me enter any folder.

[smokku]

Is there a reason you are going against this recommendation?
You could just put two files miniz.c and miniz.h to MiSTer sources, not include the whole source tree of unneeded files.

[dholm]

Hmm, I did download the file from the relese, the contents should be the same as you will find in this tarball:

• https://github.com/richgel999/miniz/archive/2.0.8.tar.gz

[smokku]

This is just an archived bundle of source tag.
The actual release is here: https://github.com/richgel999/miniz/releases/download/2.0.8/miniz-2.0.8.zip

[dholm]

@sorgelig Hmm, you are of course right about the hard drive images and I already deny opening files anything other than read-only so it should work as expected.

Regarding the change to sharpmz, it comes from the commit that changes from file descriptors til file streams. Since I changed how fileTYPE interacts with miniz I don't think it is needed any longer. If you want me to I can submit a revert of that change. It should still work but I haven't tested sharpmz specifically.

Sorry regarding std::. I thought you were against using them only in C-types like uint32_t and size_t. If you throw in a using namespace std; you can remove all of them.

Is MiSTer printing any helpful error messages when you try to enter folders? I tested both browsing normal folders as well as browsing zipped folders extensively before I pushed this change so I'm not sure what I could have missed. If you can't see anything I will run some more tests after work and perhaps throw in some more debug printouts.

[sorgelig]

Since i was busy by some cores, i didn't trace the problem and simply rolled back to a working version.
If it works for you then i will debug the problem on my side.

[sorgelig]

It's not like i totally against std. Dynamic arrays are useful feature and hard to replace without similar solution. But std::min is just a one line define. String usage also barely necessary. I see only one not obvious place to rework is splitting the path to zip file and file itself. Other string usages have equivalents in libc

[sorgelig]

problem was here:

	return (stat64(path, &st) < 0) ? 0 : st.st_mode;

which must be

 	return (stat64(full_path, &st) < 0) ? 0 : st.st_mode;

[sorgelig]

btw, i know what will be next request from users: support for 7z ))))

[dholm]

@sorgelig Regarding the strings I know. I considered doing it with C strings but out of laziness I used std::string, also because I would have had to add two length arguments as well. My thinking was that if you complain about it I can easily fix it and if not I won't have to obsess about those pesky length arguments I would have to add to the string functions. ;)
If you want it fixed let me know and I will do it.

Thanks for tracking down the problem! I was a bit confused about why the code sometimes uses "full_path" and why it sometimes only uses the partial one and I didn't fully understand when to use one or the other so I tried looking at the surrounding code to see which one was used.

Regarding 7z support I already figured that out as well so I already looked at it a little. The best thing would have been if there was a library that supported both formats but I couldn't find any such thing. In fact, only the LZMA SDK appears to provide 7z support in library form. The SDK is quite big, even if you only take the C parts and I also found it somewhat lacking in terms of documentation. MiSTer is very nice and small in size so I put the 7z work on hold until someone convinces you that it must be supported. :)

[sorgelig]

It's tempting to use std::string instead of char as it's easier to write a = b + c instead of strcpy/strcat/sprintf :)
But it's easy to make a mistake and get memory leak or corruption.
I will see if it will be good to use char*
zip path can be split in-place by replacing '/' to 0.
I will evaluate it myself and by the way I will be more familiar with your addition.

i've also noticed that OSD doesn't remember last selected ROM and always open list at the first entry.

Strange that 7z has no lightweight library. Anyway, zip-only is enough for time being.

[smokku]

Other option of minimizing errors is using C-native string handling library.
I can recommend https://github.com/antirez/sds (which is also a single .h + single .c file library)

[sorgelig]

i've made conversion to char. It wasn't hard. some places are even more compact now

[sorgelig]

With stream file i/o functions it's unclear how to operate with *.sav files.
Originally if sav file is absent then it will be created upon first save and will grow up to the last write. It seems doesn't work with stream functions.

[sorgelig]

It seems problem is much more serious that i've thought. O_SYNC option is not applied to files opened for writing! This is not acceptable behavior - all writing should be done immediately with update in catalog without holding any buffers.
Update: It seems fflush() does the sync