Skip to content

Commit 9170adb

Browse files
ageiduschekHelper Bot
authored and
Helper Bot
committed
[US_OZ] Disable some old ingest views that no longer have data (Recidiviz/recidiviz-data#29765)
## Description of the change The US_OZ ingest pipeline is crashing because we started enforcing that all raw data inputs to pipeline ingest views have data in them. At some point we cleared out some of the raw data in the us_oz_raw_data dataset (related to raw data pruning I think?). It's not actually important that we run these views, so this disables them so they only run in tests. Sandbox pipeline run [here](https://console.cloud.google.com/dataflow/jobs/us-west1/2024-05-14_15_47_49-6402012709265618349;mainTab=JOB_GRAPH;graphView=0?project=recidiviz-staging&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22))). ## Type of change > All pull requests must have at least one of the following labels applied (otherwise the PR will fail): | Label | Description | |----------------------------- |----------------------------------------------------------------------------------------------------------- | | Type: Bug | non-breaking change that fixes an issue | | Type: Feature | non-breaking change that adds functionality | | Type: Breaking Change | fix or feature that would cause existing functionality to not work as expected | | Type: Non-breaking refactor | change addresses some tech debt item or prepares for a later change, but does not change functionality | | Type: Configuration Change | adjusts configuration to achieve some end related to functionality, development, performance, or security | | Type: Dependency Upgrade | upgrades a project dependency - these changes are not included in release notes | ## Related issues See https://recidiviz.pagerduty.com/incidents/Q18BUAX0WII2P0 ## Checklists ### Development **This box MUST be checked by the submitter prior to merging**: - [x] **Double- and triple-checked that there is no Personally Identifiable Information (PII) being mistakenly added in this pull request** These boxes should be checked by the submitter prior to merging: - [ ] Tests have been written to cover the code changed/added as part of this pull request ### Code review These boxes should be checked by reviewers prior to merging: - [x] This pull request has a descriptive title and information useful to a reviewer - [x] Potential security implications or infrastructural changes have been considered, if relevant GitOrigin-RevId: ed9607b24b951645ecc26ed6bdf1efacbe3919e2
1 parent c86b502 commit 9170adb

8 files changed

+24
-0
lines changed

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_eg_person.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_eg_person` to corresponding
33
# Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- Person_ID
710
- FirstName

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_gloFS_periods.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_gloFS_periods` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- ID
710
- PersonID

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_internDS_initialtable.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_internDS_initialtable` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- ID
710
- FirstName

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_internDS_table.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_internDS_table` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- ID
710
- incarceration_type

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_nd_periods.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_nd_periods` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- entry_id
710
- person_id

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_nd_person_details.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_nd_person_details` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- person_id
710
unused_columns: []

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_vfds_periods.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_vfds_periods` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- ID
710
- PersonID

recidiviz/ingest/direct/regions/us_oz/ingest_mappings/us_oz_vfds_person.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
# This file associates the columns outputted from the query in `view_vfds_person` to
33
# corresponding Ingest Object fields.
44
manifest_language: 1.0.0
5+
# The raw data for this view no longer exists in staging so we don't run this view
6+
launch_env:
7+
$env: is_local
58
input_columns:
69
- ID
710
- FirstName

0 commit comments

Comments
 (0)