Covering the challenges of managing a self-hosted observability stack and Dan’s unique on-call experiences, like troubleshooting an incident from an airplane! Later we dive into the Production Engineering initiative at HashiCorp designed to immerse software engineers in proactively exploring observability data to gain a deeper understanding of system behavior. Of course we cover the significance of understanding roles in incident management, like incident leads and scribes, and how to identify your personal style of leading incidents.

Check out Dan's course Leading Incidents and catch up on his Monitorama talk "No Observability Without Theory" or read the blog version. Connect with him on Mastodon

Follow Off-Call

• Mastodon
• X (Twitter)
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

• [00:30] Matthew's First Monitoring Tool was.....
• [02:08] 11 years (and counting) on-call!
• [07:40] That Time Storing A Password In Cleartext Saved The Day
• [12:29] Unwinding Off Call & Signs of Burnout
• [17:13] Responding To A Customer's Malicious Insider Incident
• [19:32] What app devs need to know about monitoring databases
• [28:32] Has database monitoring gotten better over time?
• [32:15] Operating DBs in the time of Kubernetes
• [34:44] Wrap Up

Paige chats with Matthew Sanabria, Staff SRE at Cockroach Labs, about memorable moments from his 11 years on-call, the warning signs of burnout, and what developers and operators can do for each other to improve database observability for everyone!

Find Matthew

• Personal Site https://matthewsanabria.dev/
• Twitter https://twitter.com/sudomateo
• Mastodon https://mastodon.online/@sudomateo@mastodon.online
• Cockroach Labs https://www.cockroachlabs.com/
• Platform Engineering New York (PENY) Meetup https://www.meetup.com/platform-engineering-new-york/

Follow Off-Call

• Mastodon
• X (Twitter)
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io

Overview

• 00:45 | Network Observability FTW!
• 07:09 | How has networking changed in the last 5-10 years?
• 10:28 | Has Kubernetes made network engineers' lives easier?

Paige and Leon cover need to know networking concepts for developers and answer the very important questions “How much has networking really changed in the last 5-10 years?” AND “Has Kubernetes made any of this stuff easier?!”

Leon’s Links

• Adatosystems.com
• LinkedIn
• Mastodon
• Bluesky

Recommended Resources

• Alerts don't suck YOUR alerts suck!
• Technically Religious
• Imperfect Genius - Rachel Foster
• Screaming In The Cloud - Corey Quinn
• www.paigerduty.com
• Telemetry Now
• What's New at Kentik
• Kentik Close-Up

Follow Off-Call

• Mastodon
• X (Twitter)
• https://offcall.simplecast.com/

Check out Chronosphere at www.chronosphere.io