feat: Add a component for parsing external partition tables (IEC-333)

[adokitkat]

Checklist

• Component contains License
• Component contains README.md
• Component contains idf_component.yml file with url field defined
• Component was added to upload job
• Component was added to build job
• Optional: Component contains unit tests
• CI passing

Description

This MR adds a component which can parse and generate external partition tables. Right now it only supports MBR. Adding support for another partition table like GPT is possible but not planned for now.

esp_err_t esp_mbr_parse(void* mbr_buf, esp_ext_part_list_t* part_list, esp_mbr_parse_extra_args_t* extra_args);

esp_err_t esp_mbr_generate(mbr_t* mbr, esp_ext_part_list_t* part_list, esp_mbr_generate_extra_args_t* extra_args);

Parsing a partition table outputs esp_ext_part_list_t structure filled with esp_ext_part_list_item_t items.

typedef struct {
    uint32_t data[1];
    esp_ext_part_signature_type_t type;
} esp_ext_part_list_signature_t;

typedef struct {
    uint64_t address; /*!< Start address in bytes */
    uint64_t size; /*!< Size in bytes */
    uint32_t extra; /*!< Extra information (e.g. LittleFS block size stored in CHS hack, etc.) */
    char* label;
    esp_ext_part_flags_t flags; /*!< Flags for the partition */
    esp_ext_part_type_known_t type; /*!< Known partition type for this component */
} esp_ext_part_t;

typedef struct esp_ext_part_list_item_ {
    esp_ext_part_t info;
    SLIST_ENTRY(esp_ext_part_list_item_) next;
} esp_ext_part_list_item_t;

typedef struct {
    esp_ext_part_list_signature_t signature; /*!< Disk signature or identifier */
    SLIST_HEAD(esp_ext_part_list_head_, esp_ext_part_list_item_) head; /*!< Head of the partition list */
    esp_ext_part_list_flags_t flags; /*!< Flags for the partition list */
    esp_ext_part_sector_size_t sector_size; /*!< Sector size (storage medium property) */
} esp_ext_part_list_t;

When generating MBR, user can decide on sector size (512B, 4KiB) and partition alignment (4KiB, 1MiB). The defaults are 512B sector size and 1MiB partition alignment.

Note: Generating MBR doesn't mean formatting any partition on given sector span, however helper function could be created which would combine this behavior - each partition type has it's own formatting function (FATFS formats differently than LittleFS, etc.).

These are recognized types:

typedef enum {
    ESP_EXT_PART_TYPE_UNKNOWN = 0,
    ESP_EXT_PART_TYPE_FAT_ANY, /*!< FAT partition (any type) */
    ESP_EXT_PART_TYPE_FAT12,
    ESP_EXT_PART_TYPE_FAT16, /*!< FAT16 with LBA addressing */
    ESP_EXT_PART_TYPE_FAT32, /*!< FAT32 with LBA addressing */
    ESP_EXT_PART_TYPE_LITTLEFS, /*!< Possibly LittleFS (MBR CHS field => LittleFS block size hack) */
// Note: The following types are not supported, but we can return a type for them
    ESP_EXT_PART_TYPE_LINUX_ANY, /*!< Linux partition (any type) */
    ESP_EXT_PART_TYPE_EXFAT_OR_NTFS, /*!< Not supported, but we can return a type for it */
    ESP_EXT_PART_TYPE_GPT_PROTECTIVE_MBR, /*!< Not supported, but we can return a type for it */
} esp_ext_part_type_known_t;

Future TODOs and could haves:

• FATFS and this component is right now not coupled - it would be nice to add a function to FATFS or VFS, which could return FAT type of a partition, so it ca be used in this component.
• Add support for BDL and generic BDL partition components, also probably PR to esp_littlefs repo
• Support formatting along with partitioning?

[CLAassistant]

All committers have signed the CLA.

[adokitkat]

@igrr @pacucha42 please take a look

[pacucha42]

Please, provide an explanation why you fill the CHS fields in for LBA file-system types (as discussed 1:1, this is given mostly by the FatFS support of FDISK functionality). Tbh, it would be great to know the reasons behind, otherwise this operation looks useless

[adokitkat]

I do it for FAT partitions to stay as close to FATFS library behavior ESP-IDF is using. FATFS library can format partitions and generate MBR as well and it does fill CHS fields. However technically it isn't needed because modern systems should ignore it in favor of LBA.

[pacucha42]

Why do we enforce 1MB partition alignment? Isn't this option a good candidate for a parameter setting? If not, why don't we use 4kB which would fit much better into usual IoT storage device requirements?

Sorry, I have missed ESP_EXT_PART_ALIGN_1MiB is just a default. Other questions are valid. Thanks

[adokitkat]

SD cards present themselves with minimal read/write size of 512B, which is 1 sector. However internally NAND page sizes can be from 4kB to 64kB. If the accessed data is misaligned on those boundaries, it can lead to slower performance. This sector and page sizes are also different for different storage media.

I used 1MiB as a default because it is dividable by all of those sizes and should be always properly aligned, plus this component is to be used with SD cards or maybe USB drives whose capacities are in from tens to thousands of gigabytes range. If this component is to be used with a media of smaller capacity, 4kiB still can be chosen.

[pacucha42]

The code base looks good to me, however, I left few questions here and there. Thanks for you efforts @adokitkat !

Generally, it would be great to provide more explanations and in-code comments.

[pacucha42]

Just wondering: what would be use-case for such a function?

[adokitkat]

To be honest I thought I need it for something in an example code but then I resolved the problem another way and the function stayed here. I just thought copying the structure is non-trivial due to pointers and malloc'd memory so I created this function (i.e. the logic is hidden here and if there were changes in the part list struct users wouldn't need to change anything in their code). However right now I am not sure if this functionality needs to be in the public API (if and why it is something users would need often), so I can remove it.

[adokitkat]

One use case may be if the user wanted to load the MBR and make changes to it, while also keeping the original part list created from the MBR as "a reference" / a backup, so the changes would be made to a copy.

[pacucha42]

Is this line necessary?

[adokitkat]

I used memcpy in case void* signature points to to an array or something for some reason and out is just a temporary variable so the code is cleaner (will probably be optimized out anyway?).

[pacucha42]

LGTM, though the code still needs polishing. I left few comments, mostly minor things. Thanks @adokitkat