Skip to content

dft street manager odp 0024 migration

Jump to bottom
Phil Allen edited this page Apr 27, 2018 · 3 revisions

Migration on demand

Why migration matters

If Street Manager (SM) provides no support for migration at all, then LHAs and promoters ("organisations") face a long period of:

  1. Work within a time window being split between systems. This affects inspection planning for LHAs. For all organisations, it affects periodic reporting and reconciliation of finances.
  2. Parallel running of different systems. This affects both IT (e.g. routing data to the right system, licence costs) and people (looking in the right system, learning two UIs). Owing to case law about shoddy workmanship (no time limit on warranty obligations), organisations might need to deal with historic works that closed several years previously.

But migration is hard

The difficulty of migrating data in bulk between old and new systems is well-established: late discovery of data quality problems, the risk of unintended disclosure, the problem of "exactly one" loading and reconciliation.

There are three additional factors that make migration into SM distinctively hard:

  • SM will have a tighter data model than EToN version 6. Differences between SM and EToN data are more than syntactic, so complete automation of migration is impossible.
  • There is no standard export format for EToN data from EToN tools. Moreover, there are differences in interpretation of the EToN specification between EToN tools that cause inter-operability difficulties, and could lead to data mapping difficulties in migration.
  • EToN works are really conversations of messages between an LHA and a promoter. Having each organisation replay its messages in order is clearly impractical. So one organisation (e.g. the LHA) would have to clean and load data about each work and the other (e.g. the promoter) would have to trust / validate / reconcile.

Related functionality

Support for migration is related to three other areas of SM, in the sense that development work that is already in scope for MVP potentially overlaps with some support for migration.

  1. Plan for offline working. We will build API endpoints that allow retrospective uploads of data by the LHA.
  2. Highways England. SM will master works data for NSG streets but not for the HE network. SM will need to import a slave copy of HE works data that SM will not
  3. Owing to assumptions built into organisations' systems and business processes, SM will have to support the same structure of organisation-generated entity IDs that EToN does (EToN organisation ID followed by ID unique within organisation, which may itself be an intelligent key). This enables upload of data for historic and in-flight data without breaking uniqueness constraints.

Proposed limited support for migration

The Migration Q&A (OneDrive) is written from an industry stand-point. It assumes the following functionality is provided by SM.

  • API endpoint that supports upload of individual works, following SM schema, to be called by LHA, only. SM schema means LHA will have to clean and transform works data before upload. Note that the upload API will call for different validation from the normal (OLTP) API with respect to at least access control and dates, and possibly other factors.
  • API endpoint supports both SM as master and SM as slave. In case of slave, the SM UI and (normal OLTP) API will not support updates. This case will support using SM as a single view of works data, without committing to manage in-flight and historical works going forward. (This is what Highways England support needs.)

The Migration Q&A explains possible scenarios of use for this functionality. By way of example, an LHA may decide to migrate only warranty works onto SM.

Possibilities for greater support

Practically, automated bulk upload of historic and in-flight data would be possible only if SM supports the logical data model underlying EToN v6 as its first API version. Tightening the data model would then be handled as an API version change. (There are bound to be further API changes, so this would mean adding just one more.)

Support for EToN v6 logically goes with the Transition Hub idea, and simplifies its implementation somewhat, assuming that the hub only need support the first major API version of SM.

The overall shape of adoption would therefore be:

  1. EToN vendors build importers.
  2. EToN vendors and customers redirect their installs to route via the Hub.
  3. Each LHA migrates in-flight data and as much historic data (v6 only) as it really needs to.
  4. Transition, one LHA at a time.
  5. Retire the Hub.
  6. Change the SM API as outlined in the API Versioning Policy.
Clone this wiki locally