Release v1.0.0 of `ion-schema-rust`

[popematt]

As the Ion team and other users have used ion-schema-rust since its initial release in 2022, we have learned more about what customers need from ion-schema-rust and discovered bugs and other sharp edges.

In order to continue developing tools that depend on ion-schema-rust we need to have a stable release of the library. (If we don't, then there will eventually end up being a critical mass of things depending on the current state of ion-schema-rust and that will become the de facto stable release.)

Items that need to be addressed:

• Tech Debt—there are several correctness issues that need to be fixed
  • ViolationCode should be marked as non-exhaustive #209
  • Cyclical "is-a" relationships between types causes stack overflow #205
  • Use an enum instead of a boolean to indicate the closed-ness of a fields constraint in function parameters #224
  • Add bag type for storing constraints and type arguments #199
  • Make ISL model constructor method naming consistent #197
  • Create single resuable method to resolve type references for Constraint implementations  #44
• Correctness
  • Equivalence for two ISL models shoudn't consider order of types, imports and constraints #184
  • Change ISL model to retain order of non-ISL(open content) and ISL values #147
  • Allow self-referencing types only in the scenarios where a type is composed of other instances of the same type #27
• Architectural Challenges
  • The use of IonSchemaSystem makes certain combinations of operations impossible. You cannot load a schema and all its dependencies and then try to manipulate or inspect those schemas. It's impossible to load schemas, add a new type and then use for validation.
  • The codebase exhibits tight coupling between validation and type resolution processes
  • Schema manipulation capabilities are limited
  • The violation reporting system lacks flexibility
  • The maintenance of two parallel models introduces complexity, as changes often need to be implemented and synchronized across both representations.
• API Limitations
  • Validation doesn’t produce violations in a deterministic order #206
  • Add a function to expose the ISL model for Schema and Type #138
  • Use version marker trait for ISL model #187
  • Add new_type API for Schema #207
  • Make DocumentAuthority related APIs public from SchemaSystem #212

[popematt]

To address the architectural challenges, we will make the following changes from the current design.

1. Validation Output: ViolationRecorder Trait

Rather than maintaining a violation accumulator in the validation logic, we can decouple it with a ViolationRecorder trait that is essentially a visitor over all violations. The visitor function would be called once for each constraint/value combination that is violated. The trait implementation can be provided by the user, and the user can decide what to do when a violation is encountered. They could store as little or as much information about the violation as they choose, in whatever data structure they choose. The trait-based approach makes it easy to support fast-fail validation because it may return Continue(()) to continue validation, or Break(()) to end validation prematurely.

pub trait ViolationRecorder {
    /// Accepts a violation.
    fn accept(&mut self, violation: &ViolationInfo) -> ControlFlow<()>;
}


impl ResolvedType {
    pub fn validate<'a, V, R, F>(&self, value: V, recorder: F) -> Result<(), R>
        where V: AsRef<IonSchemaElement<'a>>, 
              R: ViolationRecorder,
              F: FnOnce() -> R
    {
        // Internally, wrap the recorder in something *like* LazyCell
        // We can't use LazyCell until this is part of stable Rust:
        // https://doc.rust-lang.org/std/cell/struct.LazyCell.html#method.into_inner
        
        // Then run the validation, passing in &mut LazyCell<R>
        
        // Once validation is done, if the LazyCell is initialized, then there was
        // at least one violation, so return Err(R). Otherwise return Ok(()).
        todo!()
    }
    
    pub fn is_valid<a, V: AsRef<IonSchemaElement<a>>>(&self, value: V) -> bool {
        let noop_fastfail = |_| { ControlFlow::Break(()) }
        self.validate(value, || noop_fastfail).is_ok()
    }
}

• Maximum flexibility for consumers
• Can implement either flat or tree structure—we can even provide one or both of these with the library, but users are not limited to these.
• Enables early termination of validation
• Can filter violations on the fly
• Can support streaming validation (hopefully possible in future) in a memory efficient way
• Extensible interface

2. Schema Model: Single, un-versioned model with versioned builders

We would like to prevent users from programmatically constructing schemas that combine Ion Schema 1.0, Ion Schema 2.0, and future versions in ways that are not valid, and we would like to enforce that at compile time. For example, it should never be possible to create a type definition that has both a scale and exponent constraint.

Having a single, un-versioned model is preferred because it confines version-specific complexity to parts of the codebase where it's relevant, while keeping the runtime model simple and version-agnostic. This approach provides compile-time version safety during schema construction without polluting the core API signatures with version parameters, resulting in better ergonomics and cleaner error messages. Additionally, it leaves room for future extensions like constraint-type-safety for type definitions while maintaining straightforward validation and serialization interfaces.

Here is an example of how builder functions would be created for only certain versions of ISL.

// ISL 1.0
impl TypeDefinitionBuilder<ISL_1_0> {
    pub fn scale(mut self, range: Range<isize>) -> Self {
        let c = Scale { range } ;
        self.constraints.push(AnyConstraint::Scale(c));
        self
    }
}

// ISL 2.x
impl <V: SINCE_ISL_2_0> TypeDefinitionBuilder<V> {
    pub fn exponent(mut self, range: Range<isize>) -> Self {
        let c = Exponent { range } ;
        self.constraints.push(AnyConstraint::Exponent(c));
        self
    }
}

// All ISL Versions
impl <V: ISLVersion> TypeDefinitionBuilder<V> {
    pub fn precision(mut self, range: Range<usize>) -> Self {
        let c = Precision { range } ;
        self.constraints.push(AnyConstraint::Precision(c));
        self
    }
}

3. Resolving Type References: Resolved-Type Facade

The Resolved-Type Facade provides a clean separation between unresolved and resolved schemas while maintaining efficient access to type definitions. When schemas are resolved, they are stored in a hidden schema store (InvisibleSchemaStore) that maintains all schemas and their relationships, while ResolvedSchema and ResolvedType act as facade objects that provide controlled access to this store through index-based lookups. Like the current implementation, there are different types for resolved and unresolved Schemas and Types. However, unlike the current implementation, the different types are just a wrapper around the original data model, so there is no need to maintain two separate data models.

This approach offers several advantages: it maintains a single source of truth in the InvisibleSchemaStore, provides efficient access through index-based lookups, and clearly separates the concerns of schema resolution from schema usage. The facade objects implement AsRef for their respective types, allowing immutable access to the original data model while maintaining the integrity of the resolved state. By using Arc for the schema store reference, the design ensures efficient sharing of the resolved schemas across multiple facades while preventing invalid modifications to the resolved state

/// This is the hidden struct that is the backing behind any 
/// ResolvedSchema and ResolvedType.
struct InvisibleSchemaStore {
    schemas: Vec<Schema>,
    schema_index_by_id: HashMap<String, usize>,
}

struct ResolvedSchema {
    schema_store: Arc<InvisibleSchemaStore>,
    schema_index: usize,
}

struct ResolvedTypeDef {
    schema_store: Arc<InvisibleSchemaStore>,
    schema_index: usize,
    type_index: usize,
}

pub struct TypeReference {
    schema_id: Option<String>,
    type_name: String,
    // Not exposed in public API.
    // Initially `None`, but changed to `Some(_)` when resolving the schemas
    resolved: Cell<Option<ResolvedTypeReference>>,
}

pub(crate) struct ResolvedTypeReference {
    /// TODO: If we later change `InvisibleSchemaStore` to hold things in an arena, 
    ///   then these fields can be replaced with an immutable borrow of the TypeDef.
    schema_index: usize,
    type_index: usize,
}

[popematt]

Next Steps

• Define Ion Schema version trait
• Define cross-cutting internal traits
• Implement the un-versioned model including internal trait impls
  • TypeDefinition
  • AllOf Constraint
  • AnnotationsV1 Constraint
  • AnnotationsV2Simple Constraint
  • AnnotationsV2Standard Constraint
  • AnyOf Constraint
  • ByteLength Constraint
  • CodepointLength Constraint
  • ContainerLength Constraint
  • Contains Constraint
  • Element Constraint
  • Exponent Constraint
  • Fields Constraint
  • FieldNames Constraint
  • Ieee754Float Constraint
  • Not Constraint
  • OneOf Constraint
  • OrderedElements Constraint
  • Precision Constraint
  • Regex Constraint
  • Scale Constraint
  • TimestampOffset Constraint
  • TimestampPrecision Constraint
  • Type Constraint
  • Utf8ByteLength Constraint
  • ValidValues Constraint
• Define the ViolationRecorder trait
• Define DocumentAuthority trait
• Add versioned builder for the model
• Add parsing and serialization implementations for the model
• Implement the ViolationRecorder trait basic implementations
• Implement the DocumentAuthority trait basic implementations
• Create the InvisibleSchemaStore and associated facade objects
• Implement new APIs for schema loading, resolution, and un-resolving
• Beta release and community feedback
• Final adjustments and GA release

All components should have tightly focused unit tests. The existing Ion Schema test runner will continue to work for end-to-end tests.

Please add any additional tasks, considerations, or milestones as needed. This issue will be updated as progress is made towards the GA release.

[popematt]

Organization

Project structure:

• lib.rs – Contains docs and re-exports only
• model/ – Contains the "unresolved" data model
• builder/ – Contains the builder types (and traits?)
• resolved/ – Resolved-type facades, invisible schema store, and resolution logic
• validate/– contains internal validate trait and implementations
• parse/ – contains internal parse trait and implementations
• serialize/ – contains internal serialize trait and implementations
• authority.rs – contains DocumentAuthority trait and implementations for Path, HashMap<String, String>, and possibly others
• ion_schema_element.rs – IonSchemaElement
• ion_schema_version.rs – ISL version marker types
• ion_extensions.rs – traits that add helper functions to Element, Struct, etc.
• result.rs – IonSchemaResult, etc.
• violation_recorder.rs – ViolationInfo struct, ViolationRecorder trait, implementations for (), Vec<ViolationInfo>, BTreeSet<ViolationInfo>

Tentative public module structure:

crate {
    ResolvedType
    ResolvedSchema
    ViolationInfo
    // public functions for loading and resolving schemas
    // public traits
    // ISL Version
    // Ion Schema Element
    pub mod model {
        // "Unresolved" data model and helper types
        pub mod builder {
            // Builder types/traits
        }
    }
}