How to build in Observability at Petabyte Scale
- Forfattere
- Afsnit
- 279
- Udgivet
- 7. sep. 2025
- Forlag
- 0 Anmeldelser
- 0
- Afsnit
- 279 of 276
- Længde
- 45M
- Sprog
- Engelsk
- Format
- Kategori
- Økonomi & Business
Share Episode
We welcome guest Ang Li and dive into the immense challenge of observability at scale, where some customers are generating petabytes of data per day. Ang explains that instead of building a database from scratch—a decision he says went "against all the instincts" of a founding engineer—Observe chose to build its platform on top of Snowflake, leveraging its separation of compute and storage on EC2 and S3.
The discussion delves into the technical stack and architectural decisions, including the use of Kafka to absorb large bursts of incoming customer data and smooth it out for Snowflake's batch-based engine. Ang notes this choice was also strategic for avoiding tight coupling with a single cloud provider like AWS Kinesis, which would hinder future multi-cloud deployments on GCP or Azure. The discussion also covers their unique pricing model, which avoids surprising customers with high bills by providing a lower cost for data ingestion and then using a usage-based model for queries. This is contrasted with Warren's experience with his company's user-based pricing, which can lead to negative customer experiences when limits are exceeded.
The episode also explores Observe's "love-hate relationship" with Snowflake, as Observe's usage accounts for over 2% of Snowflake's compute, which has helped them discover a lot of bugs but also caused sleepless nights for Snowflake's on-call engineers. Ang discusses hedging their bets for the future by leveraging open data formats like Iceberg, which can be stored directly in customer S3 buckets to enable true data ownership and portability. The episode concludes with a deep dive into the security challenges of providing multi-account access to customer data using IAM trust policies, and a look at the personal picks from the hosts. 💡 Notable Links: • Fact - Passkeys: Phishing on Google's own domain • and It isn't even newEpisode: All About OTELEpisode: Self Healing Systems🎯 Picks: • Warren - The Shadow (1994 film) • Ang - XREAL Pro AR Glasses
Other podcasts you might like ...
- The Journal.The Wall Street Journal & Spotify Studios
- The Can Do WayTheCanDoWay
- HW News Business Tit-BitsHW News Network
- 1,5 graderAndreas Bäckäng
- Self-Compassionate ProfessorPhD
- Networth and Chill with Your Rich BFFVivian Tu
- Don't Be SourMaxx Chewning
- Mark My Words PodcastMark Homer
- EGO NetCastMartin Lindeskog
- The Pathless Path with Paul MillerdPaul Millerd
- The Journal.The Wall Street Journal & Spotify Studios
- The Can Do WayTheCanDoWay
- HW News Business Tit-BitsHW News Network
- 1,5 graderAndreas Bäckäng
- Self-Compassionate ProfessorPhD
- Networth and Chill with Your Rich BFFVivian Tu
- Don't Be SourMaxx Chewning
- Mark My Words PodcastMark Homer
- EGO NetCastMartin Lindeskog
- The Pathless Path with Paul MillerdPaul Millerd
