Skip to content

publish pipeline: send pager duty alert on worfklow failure #43420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 12, 2024
Merged

publish pipeline: send pager duty alert on worfklow failure #43420

merged 1 commit into from
Aug 12, 2024

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Aug 9, 2024 •
edited
Loading

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/9128
Closes https://github.com/airbytehq/airbyte-internal-issues/issues/8998
We want to send a pager duty alerts to a service when a publish workflow fails.

How

  • Create a reusable action to send alerts via the PagerDuty Events API V2.
  • Send a "critical" alerts on publish workflow failure
  • Remove the instatus integration as its not relevant to use Instatus for this use case (a failure does not mean our whole publish flow is down).

Bonus

  • Tag the person who merged the triggering PR in the slack message on failure

@vercel Vercel
Copy link

vercel bot commented Aug 9, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Aug 9, 2024 8:39am

@alafanechere Graphite App
Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite Graphite

@alafanechere alafanechere changed the title publish pipeline: sent pager duty alert on worfklow failure publish pipeline: send pager duty alert on worfklow failure Aug 9, 2024
@alafanechere alafanechere marked this pull request as ready for review August 9, 2024 07:54
@alafanechere alafanechere requested review from a team and natikgadzhi August 9, 2024 07:54
@alafanechere alafanechere force-pushed the augustin/08-09-publish_pipeline_sent_pager_duty_alert_on_worfklow_failure branch from 6e8667f to c04bd98 Compare August 9, 2024 07:56
@alafanechere alafanechere force-pushed the augustin/08-09-publish_pipeline_sent_pager_duty_alert_on_worfklow_failure branch from c04bd98 to fe4a9f6 Compare August 9, 2024 08:39
@alafanechere alafanechere enabled auto-merge (squash) August 9, 2024 14:07
@alafanechere alafanechere requested a review from clnoll August 9, 2024 14:07
clnoll
clnoll approved these changes Aug 12, 2024
View reviewed changes
Copy link
Contributor

@clnoll clnoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere the code change LGTM, just one minor comment.

Question - is the Pagerduty integration key for our team? Does this mean we'll be paged any time a connector fails to publish? (Or is there some other place where we control the propagation of the GHA notification from Pagerduty to us?)

.github/workflows/publish_connectors.yml
id: pager-duty
uses: ./.github/actions/send-pager-duty-event
with:
integration_key: ${{ secrets.PAGER_DUTY_PUBLISH_FAILURES_INTEGRATION_KEY }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put a comment in here saying which team's integration key this is? (I assume it's tooling's?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration keys are not per team, they are per serivce.
We have a Connector Publish Pipeline service.
I set up an "Events API V2" integration (basically a webhook) which our GH workflow POST to.
This creates an alert. Then how this alert is managed is handled in PD settings with escalation policies etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay cool thanks for clarifying. I think it would be nice if the secret name had the service name in it, or if there was a comment saying exactly which service it's attached to. (I can basically tell by the name but would be nice not to have to guess!)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clnoll I'll add a comment with an URL to the integration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#43846

@alafanechere alafanechere merged commit 3f5e0f2 into master Aug 12, 2024
29 checks passed
@alafanechere alafanechere deleted the augustin/08-09-publish_pipeline_sent_pager_duty_alert_on_worfklow_failure branch August 12, 2024 13:59
LouisAuneau pushed a commit to LouisAuneau/airbyte that referenced this pull request Aug 13, 2024
@alafanechere @LouisAuneau
publish pipeline: send pager duty alert on worfklow failure (airbyteh…
e5ede6e 
…q#43420)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clnoll clnoll clnoll approved these changes

@natikgadzhi natikgadzhi Awaiting requested review from natikgadzhi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alafanechere @clnoll