feat: Unify PublicArchive and PrivateArchive in a single type (for discussion)

[happybeing]

FYI: @traktion @loziniak

Background

I've been modifying dweb to display websites and directories from not just a PublicArchive but either that or a PrivateArchive. To achieve this I made a wrapper that holds one of each, can deserialise either just from an address, and provides a simple but unified API.

While doing this I saw that there is very little difference between these types and having anything in posession of either struct, has full access to the files (whether the struct is a PublicArchive or PrivateArchive).

So it seems unnecessary, wasteful and a bit misleading to have separate structs. In fact for many use cases (e.g. serving websites, sharing directories) it will be best to use a PrivateArchive - even for public data - because of efficiencies in storage (and therefore cost) and speed - because you save one chunk per file by storing the datamap in the archive.

Proposal for discussion

Create a single Archive type that has two maps, one for private files and one for public. These correspond to the slightly different map in each of PrivateArchive and PublicArchive. Then provide APIs for accessing each map separately and some additional features on top of that such as make_file_public().

This also simplifies sharing, so an app can show the private files and a use select which to share. As their datamaps are put on the network then can be moved from the private map to the public map.

Also, I think all files, public and private should have, or to least be able to include their datamaps in the Archive. One way to do this is to just store the datamap, but have an API for getting the content address for any file in the public files map.

The client APIs related to archives could remain unchanged apart from returning the new unified Archive type which keeps the semantics/understanding of what is happening clear and minimises work updating old code for the change. So dir_content_upload(), dir_upload() and dir_upload_public() etc all work with the same underlying Archive type as now. All changes are hidden under the hood until someone wants to access the public or private files in the archive.

Related, on deserialisation of archives: #2892

[happybeing]

As part of this reworking of Archive I propose that Archives be able to store the content of small files in the archive, in a similar way to then storing the datamap in PrivateArchive.

See description: #2271 (comment):

So an Archive can reference content in three ways:

• network address of datamap
• embedded file datamap
• embedded file content

Ideally the API will give control over these options with a reasonable set of defaults based on the file size and other relevant factors (eg private v public).

This approach could dramatically reduce the cost of storage for some applications, and reduce the load on the network where applications store many small files in batches. Including for example, very common applications such as a website which contains many small files.

More on these benefits is in the OP: #2271