how to construct an analytics-focused incident system

[dbeatty10]

What topic would you like to see discussed more in-depth?

As an analytics engineer, I want to know successful patterns for responding to data incidents

Why do you think this is important?

Due to the nature of the way data moves and flows, we can expect unexpected breakages within any analytics pipeline.

In an ideal world, there's lots of people using the data that we produce as analytics engineers, which increases the surface area of the impact during incidents.

Software engineers know there plan and procedures for dealing with their incidents -- we should know our plans too!

Optional:

Are there any existing resources that you’ve seen explore this topic?

I haven't seen any write-ups focused on analytics engineering on the topic of incident management. (I also don't have any go-to resources for incident management for software engineers.)

What do you think they’re missing?

N/A

[gwenwindflower]

👏🏻this 👏🏻 is 👏🏻 YES 👏🏻 -- this is much yes.

honestly this is i think, the highest ratio of importance to lack of coverage i've seen in topic ideas to date.

this is so critical and i don't think i've ever seen a thorough article about this beyond product-type content that's like 'you can get alerts!'. what do you do with those alerts?? that's the hard part.

anyway, i love this.

[gwenwindflower]

to kick things off - i can outline the baseline that i see implemented usually:

1. team configures Slack alerts that go to a channel that only the data team monitors
2. when things break they get the alerts and fix them

that's it.

what i would like to see in the wild is not just the data team getting alerts, but folks using metadata panels in BI tools so that stakeholders can see the problem and contact the right owner. dbt exposures allow for this but i think are under-utilized.

i'd also love more pager duty style ownership rotations, SWEs now that there will be incidents, and those incidents don't care about day or time -- but most data teams still seem to operate as if they expect nothing to ever break on a Saturday.

[Victoriapm]

I really like this topic and I feel like in my last team we had a really nice set up. When I first joined, we had errors on daily basis and it took many of our mornings. One year after the errors were drastically reduced and resolving them was also taking less time from us in average.

On top of the slack alerts that @gwenwindflower mentioned, which are a must do, we also had some other stuff set up:

• Weekly Ad-hoc person from the data platform team to monitor the alerts channel, to make sure our response times to errors were under 30 minutes during working hours, and after working hours it was more relaxed and usually anyone who was online would jump in.
• We had a detailed document where we had the adhoc person responsibilities plus the planning, the assigned person would get extra adhoc time in our bi-weekly sprint planning. This person main responsibility was around the alerts channel, identify if the problem was platform related otherwise tag and support the owner. Other responsibility was around PR reviews, not super mandatory, but mainly keep an eye on new PRs from a platform standpoint, check they respect guidelines, good practices, etc.
• Every model had two custom meta fields that helped us identify who to tag every time: owner and group. These were mandatory fields and to enforce that, the existence of such field was checked in our CI job.
• The team had SLOs around % of scheduled jobs that were failing and data downtime, for this reason, if the owner was not resolving the alerts we would either fix it ourselves or remove the models from affected runs and alert stakeholders.
• I would check such SLOs, plus what models to refactor, general state of our meta fields mentioned above in a looker dashboard from the dbt APIs. (model owners could also take these dashboard and have their own, filtered by their name to monitor their models run times, errors, etc)
• During the sprint planning, we had an update session on our tools, including dbt and the SLOs and we would do a small "post-mortem" like thinking on how could we avoid the errors next time. For example, customs tests to alert us on what could later derive on run failures, more CI checks, etc.
• It also helped that we had "strict" guidelines on schedules, it could only go with the tags we had defined and we would try to avoid to add a new model to a scheduled job without having checked that it was running fine and for how long. We used a PR template for that. Any new schedules requests had to be discussed with the platform team.

[ernestoongaro]

This is so good @Victoriapm!

@gwenwindflower I wonder if this is better for Dev Blog than docs since it's "best advice for what dbt Cloud can do today" and docs is more about "what's available" ?

[ernestoongaro]

Winnie just explained that this isn't a docs issue so ignore my Blog comment above!

[dbeatty10]

Gloria Lin shared this great article describing what engineering teams can learn from the handling of the April 2022 Atlassian outage:
The Scoop: Inside the Longest Atlassian Outage of All Time

[glin7]

Gergely (the author of the post above) also shared this link as an example of incident comms handled well.