Skip to content

Provide a way to gradually introduce the new Schema interpreters (v2.2) #464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leighman opened this issue May 8, 2020 · 14 comments
Closed
Labels
experimental something related to the experimental features

Comments

@leighman
Copy link

leighman commented May 8, 2020

🚀 Feature request

As mentioned on Slack.
If a codebase is heavily using io-ts it could be a lot of work to update to the new Schema based system. Is it possible to provide a Compat interpreter for Schema that outputs a Type to allow gradual migration?

@leighman leighman changed the title Provide a way to gradually introduce the new Schema interpreters Provide a way to gradually introduce the new Schema interpreters (v2.2) May 8, 2020
@gcanti
Copy link
Owner

gcanti commented May 8, 2020

@leighman this is very interesting. Is there an open source, non-trivial domain model written with io-ts's Type that I can use in a POC? (you can change the field names and / or the codec names, it doesn't matter, I'm just interested in a real world example)

@gcanti gcanti added the experimental something related to the experimental features label May 8, 2020
@leighman
Copy link
Author

leighman commented May 8, 2020

Something like this? (Definitely improved with Decoder)

export const Nullable = <C extends t.Mixed>(codec: C) =>
  t.union([codec, t.null])

const CStatus = t.union([
  t.literal('status1'),
  t.literal('status2'),
  t.literal('status3'),
  t.literal('status4'),
])
export type CStatus = t.TypeOf<typeof CStatus>

const BaseEvent = t.type({
  a: Nullable(t.string),
  b: Nullable(t.number),
  c: Nullable(t.string),
  d: Nullable(t.string),
  e: t.number,
  f: t.string,
  g: t.number,
  h: Nullable(t.string),
  i: t.union([t.boolean, t.undefined]),
  j: t.union([t.boolean, t.undefined]),
  k: Nullable(t.number),
  l: t.any,
  m: t.string,
  n: t.string,
})

const OEvent = t.intersection([
  BaseEvent,
  t.type({
    event_type: t.literal('o'),
  }),
])

const AEvent = t.intersection([
  BaseEvent,
  t.type({
    event_type: t.literal('a'),
    aa: t.type({
      a: t.string,
      b: CStatus,
    }),
  }),
])

const BEvent = t.intersection([
  BaseEvent,
  t.type({
    event_type: t.union([t.literal('b'), t.literal('c')]),
    ca: t.type({
      a: t.string,
      b: CStatus,
    }),
  }),
])

export const Event = t.union([AEvent, BEvent, OEvent])
export type Event = t.TypeOf<typeof Event>

@leighman
Copy link
Author

leighman commented May 8, 2020

Other stuff uses lots of brands (Anything missing should be from io-ts-types)

interface DBrand {
  readonly D: unique symbol
}

export const DThing = t.brand(
  NonEmptyString,
  (id): id is t.Branded<NonEmptyString, DBrand> => true,
  'D'
)
export type DThing = t.TypeOf<typeof DThing>

interface IsoDateBrand {
  readonly IsoDate: unique symbol
}

export const IsoDateString = t.brand(
  t.string,
  (s): s is t.Branded<string, IsoDateBrand> => isValidIso(s),
  'IsoDate'
)
export type IsoDateString = t.TypeOf<typeof IsoDateString>

const PostType = t.keyof({
  A: null,
  B: null,
  C: null,
  D: null,
})

export type PostType = t.TypeOf<typeof PostType>

const PostC = t.exact(
  t.type({
    a: t.string,
    b: t.number,
    c: t.number,
    d: t.string,
    e: t.string,
  })
)

const PostH = t.union([
  t.type({
    p: t.literal('a'),
    a: t.string,
  }),
  t.type({
    p: t.literal('b'),
    b: t.string,
  }),
])

export type PostH = t.TypeOf<typeof PostH>

export const PostInput = t.exact(
  t.type({
    a: nonEmptyArray(t.string),
    b: NonEmptyString,
    type: PostType,
    c: PostC,
    d: t.union([DThing, t.null]),

    e: t.array(t.string),
    f: t.array(t.string),
    g: t.number,
    h: t.union([PostH, t.null]),

    i: IsoDateString,
  })
)

export type PostInput = t.TypeOf<typeof PostInput>

@steida
Copy link

steida commented May 8, 2020

I have a similar slightly related use-case. I am using io-ts to describe the application domain model, which I recently moved to the Fauna database entirely to be able to call the Fauna directly from the browser (from 340 to 45 ms, no roundtrips). Fauna is so powerful than I don't need intermediate serverless functions API anymore - that's where I validated io-ts types. Fauna uses its own simple functional expressions based language - FQL. It's basically functional programming inside the database (which can ensure transactions and ACID).

My question (or I should say desire), is to automatically map io-ts Coded to FQL.

I will probably have to reimplement io-ts somehow. Not sure yet.

The model I am using:

export const Api = t.type({
  signUp: endpoint(
    t.type({ email: t.union([Email, UniqueEmail]), password: Password }),
    UserDoc,
    'server',
  ),

  login: endpoint(
    t.type({ email: Email, password: t.union([Password, VerifiedPassword]) }),
    UserDoc,
    'server',
  ),

  user: endpoint(t.type({}), UserDoc),

  logout: endpoint(t.type({}), t.boolean),

  changePassword: endpoint(
    t.type({
      oldPassword: t.union([Password, VerifiedPassword]),
      newPassword: Password,
    }),
    t.literal(true),
  ),

  importWeb: endpoint(
    t.type({
      url: t.union([Url, ExistingHttpOrHttpsUrl]),
      windowSize: WindowSize,
    }),
    t.type({ webDoc: WebDoc, pageDoc: PageDoc }),
  ),

  deleteWeb: endpoint(t.type({ id: FaunaID }), t.literal(true)),

  account: endpoint(t.type({}), t.type({ userDoc: UserDoc })),

  edit: endpoint(
    t.type({ webID: FaunaID, pageID: option(FaunaID) }),
    t.union([
      t.type({
        type: t.literal('webNotFound'),
      }),
      t.type({
        type: t.literal('pageNotFound'),
        webDoc: WebDoc,
      }),
      t.type({
        type: t.literal('web'),
        webDoc: WebDoc,
        // pages: t.array(PageListItem),
      }),
      t.type({
        type: t.literal('page'),
        webDoc: WebDoc,
        pageDoc: PageDoc,
      }),
    ]),
  ),

  webs: endpoint(t.type({}), t.array(WebDoc)),
});

@gcanti
Copy link
Owner

gcanti commented May 9, 2020

@leighman POC on branch 464 (contains a new experimental Type.ts module). You can play with it by running npm i gcanti/io-ts#464.

Migration path

The migration path should be something like:

  1. define a "prelude"
import { Kind, URIS } from 'fp-ts/lib/HKT'
import { memoize, Schemable, WithUnion } from 'io-ts/lib/Schemable'

interface MySchemable<S extends URIS> extends Schemable<S>, WithUnion<S> {
  readonly undefined: Kind<S, undefined>
  readonly any: Kind<S, any>
}

interface Schema<A> {
  <S extends URIS>(S: MySchemable<S>): Kind<S, A>
}

type TypeOf<S> = S extends Schema<infer A> ? A : never

function make<A>(f: Schema<A>): Schema<A> {
  return memoize(f)
}

where MySchemable will contain any additional operation required by your domain model.

  1. convert a type (starting from the leafs, i.e. types with no dependencies)

So let's start from CStatus

const CStatus = t.union([t.literal('status1'), t.literal('status2'), t.literal('status3'), t.literal('status4')])
export type CStatus = t.TypeOf<typeof CStatus>

becomes

const CStatusSchema = make((S) => S.literal('status1', 'status2', 'status3', 'status4'))

export type CStatus = TypeOf<typeof CStatus>

Now to produce the old CStatus (which is a t.Type) we need to provide a suitable instance to CStatusSchema

import * as T from 'io-ts/lib/Type' // npm i gcanti/io-ts#464

const instance: MySchemable<'Type'> = {
  ...T.instance, // <= this is an instance of Schemable<S> & WithUnion<S>
  undefined: t.undefined, // <= I must implement the missing operations
  any: t.any
}

const CStatus = CStatusSchema(instance)
// ^--- this is equivalent to the old `Ctatus`
  1. Goto 2)

Here's the complete migration:

Snippet 1

import * as t from 'io-ts' // npm i gcanti/io-ts#464
import { Kind, URIS } from 'fp-ts/lib/HKT'
import { memoize, Schemable, WithUnion } from 'io-ts/lib/Schemable'
import * as T from 'io-ts/lib/Type'

// ---------------------------------------------------
// prelude
// ---------------------------------------------------

interface MySchemable<S extends URIS> extends Schemable<S>, WithUnion<S> {
  readonly undefined: Kind<S, undefined>
  readonly any: Kind<S, any>
}

interface Schema<A> {
  <S extends URIS>(S: MySchemable<S>): Kind<S, A>
}

type TypeOf<S> = S extends Schema<infer A> ? A : never

function make<A>(f: Schema<A>): Schema<A> {
  return memoize(f)
}

// ---------------------------------------------------
// schemas
// ---------------------------------------------------

const CStatus = make((S) => S.literal('status1', 'status2', 'status3', 'status4'))

const BaseEvent = make((S) =>
  S.type({
    a: S.nullable(S.string),
    b: S.nullable(S.number),
    c: S.nullable(S.string),
    d: S.nullable(S.string),
    e: S.number,
    f: S.string,
    g: S.number,
    h: S.nullable(S.string),
    i: S.union(S.boolean, S.undefined),
    j: S.union(S.boolean, S.undefined),
    k: S.nullable(S.number),
    l: S.any,
    m: S.string,
    n: S.string
  })
)

const OEvent = make((S) =>
  S.intersection(
    BaseEvent(S),
    S.type({
      event_type: S.literal('o')
    })
  )
)

const AEvent = make((S) =>
  S.intersection(
    BaseEvent(S),
    S.type({
      event_type: S.literal('a'),
      aa: S.type({
        a: S.string,
        b: CStatus(S)
      })
    })
  )
)

const BEvent = make((S) =>
  S.intersection(
    BaseEvent(S),
    S.type({
      event_type: S.literal('b', 'c'),
      ca: S.type({
        a: S.string,
        b: CStatus(S)
      })
    })
  )
)

const Event = make((S) => S.union(AEvent(S), BEvent(S), OEvent(S)))

export type Event = TypeOf<typeof Event>

// ---------------------------------------------------
// instances
// ---------------------------------------------------

const typeInstance: MySchemable<'Type'> = {
  ...T.instance,
  undefined: t.undefined,
  // tslint:disable-next-line: deprecation
  any: t.any
}

export const typeEvent = Event(typeInstance)

import * as D from 'io-ts/lib/Decoder'
import { draw } from 'io-ts/lib/Tree'
import { pipe } from 'fp-ts/lib/pipeable'
import * as E from 'fp-ts/lib/Either'

const decoderInstance: MySchemable<'Decoder'> = {
  ...D.decoder,
  undefined: D.fromGuard({ is: (u: unknown): u is undefined => u === undefined }, 'undefined'),
  any: D.fromGuard({ is: (u: unknown): u is any => true }, 'any')
}

export const decoderEvent = Event(decoderInstance)

const input = {}

console.log(pipe(decoderEvent.decode(input), E.mapLeft(draw), E.fold(String, String)))
/*
member 0
├─ required property "a"
│  ├─ member 0
│  │  └─ cannot decode undefined, should be null
│  └─ member 1
│     └─ cannot decode undefined, should be string
├─ required property "b"

...

*/

Snippet 2

import { Kind, URIS } from 'fp-ts/lib/HKT'
import { memoize, Schemable, WithUnion, WithRefinement } from 'io-ts/lib/Schemable'
import { NonEmptyArray } from 'fp-ts/lib/NonEmptyArray'

// ---------------------------------------------------
// prelude
// ---------------------------------------------------

interface NonEmptyStringBrand {
  readonly NonEmptyString: unique symbol
}

type NonEmptyString = string & NonEmptyStringBrand

interface MySchemable<S extends URIS> extends Schemable<S>, WithUnion<S>, WithRefinement<S> {
  readonly NonEmptyString: Kind<S, NonEmptyString>
  readonly nonEmptyArray: <A>(schema: Kind<S, A>) => Kind<S, NonEmptyArray<A>>
}

interface Schema<A> {
  <S extends URIS>(S: MySchemable<S>): Kind<S, A>
}

type TypeOf<S> = S extends Schema<infer A> ? A : never

function make<A>(f: Schema<A>): Schema<A> {
  return memoize(f)
}

// ---------------------------------------------------
// schemas
// ---------------------------------------------------

interface DBrand {
  readonly D: unique symbol
}

type DThing = NonEmptyString & DBrand

const DThing = make((S) => S.refinement(S.NonEmptyString, (id): id is DThing => true, 'D'))

interface IsoDateBrand {
  readonly IsoDate: unique symbol
}

type IsoDateString = string & IsoDateBrand

const IsoDateString = make((S) => S.refinement(S.string, (s): s is IsoDateString => true, 'IsoDate'))

const PostType = make((S) => S.literal('A', 'B', 'C', 'D'))

const PostC = make((S) =>
  S.type({
    a: S.string,
    b: S.number,
    c: S.number,
    d: S.string,
    e: S.string
  })
)

const PostH = make((S) =>
  S.sum('p')({
    a: S.type({
      p: S.literal('a'),
      a: S.string
    }),
    b: S.type({
      p: S.literal('b'),
      b: S.string
    })
  })
)

export type PostH = TypeOf<typeof PostH>

export const PostInput = make((S) =>
  S.type({
    a: S.nonEmptyArray(S.string),
    b: S.NonEmptyString,
    type: PostType(S),
    c: PostC(S),
    d: S.nullable(DThing(S)),
    e: S.array(S.string),
    f: S.array(S.string),
    g: S.number,
    h: S.nullable(PostH(S)),
    i: IsoDateString(S)
  })
)

export type PostInput = TypeOf<typeof PostInput>

// ---------------------------------------------------
// instances
// ---------------------------------------------------

import * as D from 'io-ts/lib/Decoder'
import { draw } from 'io-ts/lib/Tree'
import { pipe } from 'fp-ts/lib/pipeable'
import * as E from 'fp-ts/lib/Either'

const decoderInstance: MySchemable<'Decoder'> = {
  ...D.decoder,
  NonEmptyString: D.refinement(D.string, (s): s is NonEmptyString => s.length > 0, 'NonEmptyString'),
  nonEmptyArray: <A>(decoder: D.Decoder<A>) =>
    D.refinement(D.array(decoder), (as): as is NonEmptyArray<A> => as.length > 0, 'NonEmptyArray')
}

const decoderPostInput = PostInput(decoderInstance)

const input = {
  a: [],
  b: '',
  e: [],
  f: [],
  g: 0,
  h: null,
  i: ''
}

console.log(pipe(decoderPostInput.decode(input), E.mapLeft(draw), E.fold(String, String)))
/*
required property "a"
└─ cannot refine [], should be NonEmptyArray
required property "b"
└─ cannot refine "", should be NonEmptyString
required property "type"
└─ cannot decode undefined, should be "A" | "B" | "C" | "D"
required property "c"
└─ cannot decode undefined, should be Record<string, unknown>
required property "d"
├─ member 0
│  └─ cannot decode undefined, should be null
└─ member 1
   └─ cannot decode undefined, should be string
*/

@leighman
Copy link
Author

Sweet. Look really useful, will give it a try this week.

  • We would need parse too. Does is make sense to expose a WithParse<S> and implement on Type (is that possible)?
  • Type.md file is now potentially confusing

@gcanti
Copy link
Owner

gcanti commented May 11, 2020

Does is make sense to expose a WithParse<S> and implement on Type (is that possible)?

@leighman that's not possible because you can't implement neither is nor encode

import * as t from 'io-ts'
import { Kind, URIS } from 'fp-ts/lib/HKT'
import { Schemable, WithUnion } from 'io-ts/lib/Schemable'
import * as E from 'fp-ts/lib/Either'

// ---------------------------------------------------
// prelude
// ---------------------------------------------------

interface MySchemable<S extends URIS> extends Schemable<S> {
  readonly parse: <A, B>(from: Kind<S, A>, parser: (a: A) => E.Either<string, B>) => Kind<S, B>
}

// ---------------------------------------------------
// instances
// ---------------------------------------------------

import * as T from 'io-ts/lib/Type'

const I_CANT_IMPLEMENT_THIS: any = null as any

const myschemableType: MySchemable<T.URI> = {
  ...T.instance,
  parse: <A, B>(from: t.Type<A>, parser: (a: A) => E.Either<string, B>) =>
    new t.Type<B>(
      `parse(${from.name})`,
      (u): u is B => I_CANT_IMPLEMENT_THIS,
      (u, c) =>
        E.either.chain(from.decode(u), (a) => {
          const e = parser(a)
          return E.isLeft(e) ? t.failure(e.left, c) : t.success(e.right)
        }),
      (b) => I_CANT_IMPLEMENT_THIS
    )
}

You have to enrich MySchemable instead, let's see an example with NumberFromString

import { Kind, URIS } from 'fp-ts/lib/HKT'
import { Schemable } from 'io-ts/lib/Schemable'

// ---------------------------------------------------
// prelude
// ---------------------------------------------------

interface MySchemable<S extends URIS> extends Schemable<S> {
  readonly NumberFromString: Kind<S, number>
}

// ---------------------------------------------------
// instances
// ---------------------------------------------------

import * as T from 'io-ts/lib/Type'
import { NumberFromString } from 'io-ts-types/lib/NumberFromString'

export const type: MySchemable<T.URI> = {
  ...T.instance,
  NumberFromString
}

import * as D from 'io-ts/lib/Decoder'
import { left, right } from 'fp-ts/lib/Either'

export const decoder: MySchemable<D.URI> = {
  ...D.decoder,
  NumberFromString: D.parse(D.string, (s) => {
    const n = parseFloat(s)
    return isNaN(n) ? left(`cannot parse ${s} to number`) : right(n)
  })
}

// or even an `Eq` instance...

import * as E from 'io-ts/lib/Eq'

export const eq: MySchemable<E.URI> = {
  ...E.eq,
  NumberFromString: E.number
}

@leighman
Copy link
Author

Nice. Thanks for the explanation. I like how it forces you to pull together what operations are allowed in your schema.

@gcanti
Copy link
Owner

gcanti commented May 19, 2020

@leighman ts3.9 broke Schema, see #467

I have a possible fix but unfortunately the prelude will become a bit more verbose

Snippet 2

import { Kind, URIS, HKT } from 'fp-ts/lib/HKT'
import { memoize, Schemable, WithRefinement, Schemable1, WithRefinement1 } from '../src/Schemable'
import { NonEmptyArray } from 'fp-ts/lib/NonEmptyArray'

// ---------------------------------------------------
// prelude
// ---------------------------------------------------

interface MySchemable<S> extends Schemable<S>, WithRefinement<S> {
  readonly nonEmptyArray: <A>(schema: HKT<S, A>) => HKT<S, NonEmptyArray<A>>
}

interface MySchemable1<S extends URIS> extends Schemable1<S>, WithRefinement1<S> {
  readonly nonEmptyArray: <A>(schema: Kind<S, A>) => Kind<S, NonEmptyArray<A>>
}

interface Schema<A> {
  <S>(S: MySchemable<S>): HKT<S, A>
}

type TypeOf<S> = S extends Schema<infer A> ? A : never

function make<A>(f: Schema<A>): Schema<A> {
  return memoize(f)
}

export function interpreter<S extends URIS>(S: MySchemable1<S>): <A>(schema: Schema<A>) => Kind<S, A> {
  return (schema: any) => schema(S)
}

// ---------------------------------------------------
// schemas
// ---------------------------------------------------

interface NonEmptyStringBrand {
  readonly NonEmptyString: unique symbol
}
type NonEmptyString = string & NonEmptyStringBrand
const NonEmptyString = make((S) => S.refinement(S.string, (s): s is NonEmptyString => s.length > 0, 'D'))

interface DBrand {
  readonly D: unique symbol
}
type DThing = NonEmptyString & DBrand
const DThing = make((S) => S.refinement(NonEmptyString(S), (_): _ is DThing => true, 'D'))

interface IsoDateBrand {
  readonly IsoDate: unique symbol
}
type IsoDateString = string & IsoDateBrand
const IsoDateString = make((S) => S.refinement(S.string, (_): _ is IsoDateString => true, 'IsoDate'))

const PostType = make((S) => S.literal('A', 'B', 'C', 'D'))

const PostC = make((S) =>
  S.type({
    a: S.string,
    b: S.number,
    c: S.number,
    d: S.string,
    e: S.string
  })
)

const PostH = make((S) =>
  S.sum('p')({
    a: S.type({
      p: S.literal('a'),
      a: S.string
    }),
    b: S.type({
      p: S.literal('b'),
      b: S.string
    })
  })
)

export type PostH = TypeOf<typeof PostH>

export const PostInput = make((S) =>
  S.type({
    a: S.nonEmptyArray(S.string),
    b: NonEmptyString(S),
    type: PostType(S),
    c: PostC(S),
    d: S.nullable(DThing(S)),
    e: S.array(S.string),
    f: S.array(S.string),
    g: S.number,
    h: S.nullable(PostH(S)),
    i: IsoDateString(S)
  })
)

export type PostInput = TypeOf<typeof PostInput>

// ---------------------------------------------------
// instances
// ---------------------------------------------------

import * as D from '../src/Decoder'
import { draw } from '../src/Tree'
import { pipe } from 'fp-ts/lib/pipeable'
import * as E from 'fp-ts/lib/Either'

const decoder: MySchemable1<D.URI> = {
  ...D.decoder,
  nonEmptyArray: <A>(decoder: D.Decoder<A>) =>
    D.refinement(D.array(decoder), (as): as is NonEmptyArray<A> => as.length > 0, 'NonEmptyArray')
}

const decoderPostInput = interpreter(decoder)(PostInput)

const input = {
  a: [],
  b: '',
  e: [],
  f: [],
  g: 0,
  h: null,
  i: ''
}

console.log(pipe(decoderPostInput.decode(input), E.mapLeft(draw), E.fold(String, String)))
/*
required property "a"
└─ cannot refine [], should be NonEmptyArray
required property "b"
└─ cannot refine "", should be NonEmptyString
required property "type"
└─ cannot decode undefined, should be "A" | "B" | "C" | "D"
required property "c"
└─ cannot decode undefined, should be Record<string, unknown>
required property "d"
├─ member 0
│  └─ cannot decode undefined, should be null
└─ member 1
   └─ cannot decode undefined, should be string
*/

@gcanti
Copy link
Owner

gcanti commented May 21, 2020

Closed by #471

@gcanti gcanti closed this as completed May 21, 2020
@leighman
Copy link
Author

@gcanti Thanks for the update. Any simple explanation on why the 1 stuff is now required?
Makes it a bit of a harder sell 😅

@gcanti
Copy link
Owner

gcanti commented May 25, 2020

@leighman I need MySchemable to correctly infer the types, and MySchemable1 to define the instances

@MoSheikh
Copy link

Apologies for opening this back up, but I wasn't sure where would be the right place to ask this. I'm confused about the use-case for the Schema system - couldn't decoders more easily be derived from extending the base type like @leighman was doing in his example?

Some clarification would be greatly appreciated. I'm still wrapping my head around this library and I am thoroughly impressed - having some insight into what kind of problems the Schema module intends to solve that are difficult with the current implementation would be very helpful.

@gcanti
Copy link
Owner

gcanti commented Jan 17, 2021

having some insight into what kind of problems the Schema module intends to solve that are difficult with the current implementation would be very helpful

@MoSheikh

Schema allows to define (through the make constructor) a generic schema once and then derive multiple concrete instances

see https://github.com/gcanti/io-ts/blob/f13a10ae9d405f3fb8564cf3b7917d31aa7c0e27/Schema.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experimental something related to the experimental features
Projects
None yet
Development

No branches or pull requests

4 participants