Model classification api

[jazzhaiku]

Summary

The goal of this PR is to make it easier to add an new config type.
This scope of this PR is to integrate the API and does not include adding new configs (outside tests) or porting existing ones.

One of the glaring issues of the existing legacy probe is that the logic for each type is spread across multiple classes and intertwined with the other configs. This means that adding a new config type (or modifying an existing one) is complex and error prone.

This PR attempts to remedy this by providing a new API for adding configs that:

• Is backwards compatible with the existing probe.
• Encapsulates fields and logic in a single class, keeping things self-contained and easy to modify safely.

Below is a minimal toy example illustrating the proposed new structure:

class MinimalConfigExample(ModelConfigBase):
    type: ModelType = ModelType.Main
    format: ModelFormat = ModelFormat.Checkpoint
    fun_quote: str

    @classmethod
    def matches(cls, mod: ModelOnDisk) -> bool:
        return mod.path.suffix == ".json"

    @classmethod
    def parse(cls, mod: ModelOnDisk) -> dict[str, Any]:
        with open(mod.path, "r") as f:
            contents = json.load(f)

        return {
            "fun_quote": contents["quote"],
            "base": BaseModelType.Any,
        }

To create a new config type, one needs to inherit from ModelConfigBase and implement its interface.

The code falls back to the legacy model probe for existing models using the old API.
This allows us to incrementally port the configs one by one.

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[jazzhaiku]

Having a review of my own code here: for simplicity, the legacy_probe decorator and concrete_subclasses() function could be replaced with 2 sets: UsingLegacyProbe and UsingModelClassify.

[RyanJDick]

Just one minor comment, otherwise looks good to me. I ran a few manual smoke tests; no issues.

[psychedelicious]

Nice work! Let's wait until v5.8.0 is tagged and released to merge.