GUID database storage rework

[vit9696]

There are numerous problems with the way we handle GUID identification in the firmwares:

• Our GUID database is hard bundled with UEFITool and is not visible to other tools (UEFIExtract, UEFIFind).
• Its contents only cover a pair of GUID and name, whereas different GUIDs have different names and they also have attributes like vendor origin.
• Custom GUID databases cannot be used across different runs without manually loading the file after starting UEFITool.
• GUID database imported filename search does not work.

I believe this problem should be addressed in the following approach:

1. Make UEFITool export its database on mismatch at launch to <settings>/default_guids.csv.

2. Make UEFITool use <settings>/user_guids.csv database by default and fallback to <settings>/default_guids.csv if it is missing.

3. Add "Install/Uninstall GUID database" menu entries to UEFITool, which will create/remove <settings>/default_guids.csv based on user supplied .csv file.

4. Make UEFIFind and UEFIReplace use first found database from the list: ./guids.csv, <settings>/user_guids.csv, <settings>/default_guids.csv.

5. Create a separate repository with GUID database in some different format, that will support exporting to .csv UEFITool understands. I am currently thinking of some yaml files located at <Vendor>/<Type>.yaml. For example, we could have Tianocore/Files.yaml:

   06A2CB7F-6A18-4E2F-B43B-9920A733700A:
     Name:
       - DxeCore
       - DxeCoreAlternative 
   ...
   

6. Remove all GUIDs from UEFITool and pick either of the three:

   • download latest exported GUID database from that repository at launch
   • download latest exported GUID database from that repository during build
   • have some minimal version of GUID database with e.g. tianocore-only GUIDs within UEFITool periodically exported from the main repository.

7. Make UEFITool export all discovered GUID names instead of just not existent ones into .csv. Previously it was done to deal with duplicates, but at this step I believe it should be done at GUID database import stage with different names accounted in the yaml files.

8. Fix search of names from the database :)

Notes:

• <settings should expand in platform-specific way. I expect Linux to get ~/.config/LongSoft and macOS to get ~/Library/Application Support/LongSoft. I personally have no idea what Windows prefers, but likely %AppData%/Roaming/LongSoft.
• I am not positive whether GUID repository should reside in LongSoft, but I do not mind if so. Perhaps it will be more convenient to have it in ours Acidanthera, as we have more people with commit access there.
• For exported .csv files I believe the first name should be chosen.
• In the future I would like to not only have Files.yaml but also protocols and other guids present in files themselves. That will be useful for clever searching.

@NikolajSchlej, will be great if you could handle some parts of this.

[NikolajSchlej]

• our GUID database is hard bundled with UEFITool.

This is by design and comes from the fact that UEFITool aims to be a fully-standalone application that requires no supplemental files or installation. This means a built-in GUID database is a must.

• and is not visible to other tools (UEFIExtract, UEFIFind).

It is unclear what kind of benefit a GUID database support will provide to UEFIFind, but I agree that adding it to UEFIExtract makes sense.

• UEFIExtract: add GUID database support

• Its contents only cover a pair of GUID and name, whereas different GUIDs have different names and they also have attributes like vendor origin.

Vendor-specific variants should be handled by multiple files, different names is not a problem (use the most generic one).

• Custom GUID databases cannot be used across different runs without manually loading the file after starting UEFITool.

Storing the currently-used DB in settings makes sense, but we also need some indication of what DB is used right now.

• UEFITool: add support for persistent external GUID DB and UI for it

• GUID database imported filename search does not work.

It's unclear what you mean here, please elaborate.

[vit9696]

Vendor-specific variants should be handled by multiple files, different names is not a problem (use the most generic one).

They are a problem when you need to merge two databases, as you do not know which previous names were already used. It is also a problem when you need to look the filename up in the codebase, as there may be completely different and unrelated modules :x All in all, this one is not a problem of UEFITool, as we want a separate database, but I believe it should be made aware of.

Storing the currently-used DB in settings makes sense, but we also need some indication of what DB is used right now.

Could we add some logging to the output window?

It's unclear what you mean here, please elaborate.

Currently, when the loaded firmware does not contain the file name in its UI section, the files may still get identified based on the GUID database. However, the user cannot find the files by these identification names (e.g. DxeCore) with any means of search, as the search only looks for data in ROM contents and is unaware of GUID names assigned to the files.

I believe that a separate radio button entry should be added to look for GUID database assigned names entries in the search panel.

[NikolajSchlej]

All in all, this one is not a problem of UEFITool.

Thats the main reason why it should be somewhere else.
For UEFITool, the database is just a convenience to make RE slightly easier for firmwares with removed UI sections, prioritizing such nice-to-have functionality over parser improvement and builder development makes little sense to me. (Also, all new code and new features have to be supported nearly forever, so any additional nice-to-have needs to be justified).

Could we add some logging to the output window?

Yes, probably to a new "Status" tab where all info on current run, currently opened file, preformed actions, etc. can be shown.

• Add "Status" tab for misc. information coming from the program itself, not it's components.

I believe that a separate radio button entry should be added to look for GUID database assigned names entries in the search panel.

Please provide a mockup of that UI.

[vit9696]

Sorry for a delay, was a bit busy.

Please provide a mockup of that UI.

Either like this, or as a separate tab DataBase with Search scopes and Text search options hidden.

[skochinsky]

I would do it like this:

• in GUID tab, [...] button next to GUID field
• when you click it, a list pops up with the list of predefined GUIDs and a text search box
• after selecting, insert GUID into the search field and print the name below it
  Maybe also:
• whenever a GUID is printed in GUI (e.g. in info tab), add a context menu entry "Find GUID" which would pop up the search dialog with it pre-filled