Kafka Streams, State & Windowing | The Duchess & The Doctor #2 ft. Anna McDonald & Matthias J. Sax
Kafka Streams, State & Windowing | The Duchess & The Doctor #2 ft. Anna McDonald & Matthias J. Sax
Apache Kafka® experts Anna McDonald and Matthias J. Sax analyze the suppress function, emit strategies in Kafka Streams, and the architectural trade-offs between in-memory buffers and RocksDB-based storage for stateful operations. They also talk state during blue-green deployments and the semantic implications of changing window types in production.
Learn more fundamentals of the Apache Kafka and Flink ecosystems at: https://developer.confluent.io/courses/#fundamentals
CHAPTERS
0:00 - Introduction
0:14 - Internals of the Suppress Function
1:14 - New Emit Strategies for Windowing
2:00 - Emit on Change vs. Emit on Close
4:28 - Watermark-based Systems and Apache Flink
5:22 - Use Cases for Final Emit Strategies
6:10 - Intermediate Results in Sliding and Hopping Windows
7:17 - Disabling Batching for Testing and Visibility
7:48 - Blue-Green Deployments in Stateful Microservices
8:45 - Challenges of Changing Window Types
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Kafka experts explain Events | The Duchess & The Doctor #1 ft. Anna McDonald & Matthias J. Sax
Two of Confluent’s most beloved Apache Kafka® experts, The Duchess (Anna McDonald) and The Doctor (Matthias J. Sax), along with their extinct friend, Phil, discuss all things eventing in this remastered upload. They cover why events are a mindset, why immutability is relaxing, and sliding and hopping windows in stream processing.
Learn more fundamentals of the Apache Kafka and Flink ecosystems at: https://developer.confluent.io/courses/#fundamentals
CHAPTERS
0:16 - Eventing as a Mindset
2:03 - Organizing Events Around Business Logic
2:30 - Eventing vs. Streaming Analytics and ETL
3:33 - Handling Missing Events and Timeouts
8:23 - Aggregating Windowed Events
11:00 - Sliding vs. Hopping Windows
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Updates and Deletes | Apache Iceberg + Tableflow
Course site coming soon! | In this episode, Tim Berglund explains how Apache Iceberg handles real-world updates and deletes with full ACID guarantees. He walks through how to copy-on-write and merge-on-read work looks under the hood, how delete files (equality and position deletes) and compaction tame write amplification and small-file sprawl, and how to choose the right strategy for read-heavy vs. write-heavy, streaming-first workloads in your data lake.
Timecodes:
00:00:05 Why updates & deletes
00:01:38 Customer table example
00:02:27 Copy-on-write
00:06:39 Merge-on-read
00:11:30 Choosing a strategy
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
ACID Transactions | Apache Iceberg + Tableflow
Course site coming soon! | Tim is back and is here to explain how Apache Iceberg brings real ACID transactions to files sitting in object stores so your data lake tables can finally behave like a database. You’ll see how immutable files, atomic metadata swaps, and snapshot isolation work together to guarantee consistency even under heavy concurrent writes. Tim details where Iceberg’s transactional model shines, where multi-table transactions are still evolving, and how modern catalogs can extend what’s possible on your lakehouse.
Timecodes:
00:00 ACID on S3
00:01:18 ACID Basics
00:02:57 Iceberg’s ACID Design
00:05:33 Concurrent Writes
00:10:51 Multi-Table Limits
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Data Types and Schema Evolution | Apache Iceberg + Tableflow
Course site coming soon! | Tim Berglund explores how Apache Iceberg turns schema evolution from a painful migration into a fast, metadata-only update. This episode details how Iceberg tracks columns by ID to safely add, drop, rename, widen, and reorder fields without rewriting data or breaking historical queries. He also introduces Iceberg v3’s new types: variant for semi-structured JSON, native geometry/geography for geospatial analytics, and nanosecond-precision timestamps which enables analytics scenarios that are awkward or impractical in traditional data lakes.
Timecodes:
00:00 - Schema evolution
00:00:38 - Iceberg data types
00:03:38 - Safe schema changes
00:06:01 - User events demo
00:09:38 - Widening & governance
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Killing Clusters & Orchestrating Chaos with Colt McNealy | Ep. 20 | Confluent Developer Podcast
Listen: https://confluent.buzzsprout.com | Tim Berglund talks to Colt McNealy (LittleHorse Enterprises) about his career in distributed systems. Colt’s first job: software engineer at a real estate company. His challenge: working in a complex microservices environment and turning that pain into Little Horse.
Colt's Current 2024 talk: https://current.confluent.io/2024-sessions/kafka-streams-as-a-data-store-for-a-workflow-engine
Gunnar Morling's blog: https://www.morling.dev/blog/
Jack Vanlightly's blog: https://jack-vanlightly.com/
🎧 Subscribe to Confluent Developer wherever you listen to podcasts: https://confluent.buzzsprout.com
▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes: https://youtube.com/@ConfluentDeveloper?sub_confirmation=1
👍 If you enjoyed this, please like it and leave us a comment
🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais: https://www.confluent.io/life-is-but-a-stream-show
CREDITS
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
ABOUT THE PODCAST
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#confluentdeveloper #podcast #confluent
Architecture & Concepts | Apache Iceberg + Tableflow
Course site coming soon! | In this episode, Tim Berglund traces the evolution from fragile, file-based data lakes to Apache Iceberg, the open table format that makes your lake behave like a transactional database. You’ll learn how Iceberg’s layered architecture enables reliable time travel, safe concurrent writes, and fast scans at scale. With this foundation, you’ll be ready to dive into the magic of the Iceberg catalog in the next episode.
Timecodes:
0:00 – From data lakes to Parquet and partitions
1:18 – Hive
2:21 – Limitations of Hive
2:41 – Introducing Apache Iceberg
3:31 – Inside Iceberg
7:58 – How catalogs name Iceberg tables
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Catalog | Apache Iceberg + Tableflow
Course site coming soon! | Today, Tim Berglund breaks down the Iceberg catalog: the layer that maps table names to metadata files so engines like Spark and Trino can find your data, guarantee atomic commits, and handle concurrent writes safely. You’ll see how the Iceberg REST Catalog spec powers portability across implementations like Polaris, Unity Catalog, and others, and why your catalog choice shapes what’s possible in your data lake before we dive into ACID transactions next.
Timecodes:
0:00 – Iceberg Data vs Metadata Layers
0:42 – Why Iceberg Needs a Catalog
1:46 – Atomic Commits & Concurrent Writers
2:35 – Iceberg REST Catalogs & Governance
4:19 – Cloud, Legacy Options & How to Choose
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Welcome to Apache Iceberg + Tableflow
Course site coming soon! | Tim Berglund hosts our newest course: Apache Iceberg and Tableflow. This 14-module course covers how Iceberg brings transactions, time travel, and painless schema evolution to petabyte-scale data lakes. You’ll also see how leading platforms like AWS, Google, Azure, Snowflake, Databricks, and Confluent Cloud are standardizing on it.
Timecodes:
00:00:00 Intro: Tim Berglund and Apache Iceberg
00:00:09 Why Iceberg? Time travel, rollbacks, and schema evolution
00:00:26 What Iceberg Is: Open table format for modern data lakes
00:00:47 Industry Adoption: AWS, Snowflake, Databricks & more
00:00:58 Course Overview: Architecture, features, and Tableflow in Confluent Cloud
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Apache Kafka 4.2 | 38 KIPs, Kafka Streams, Observability Improvements, Rebalance Protocol, and more
Blog: https://kafka.apache.org/blog/2026/01/14/apache-kafka-4.2.0-release-announcement/ | Sandon Jacobs (Senior Developer Advocate, Confluent) highlights the key features in Apache Kafka 4.2, a cloud-native release powered by 38 new KIPs. These highlights include GA Queues for Kafka (KIP-932) with support for long-running processing and new share consumer lag metrics (KIPs 1222, 1226), broker-driven Kafka Streams rebalancing (KIP-1071, building on KIP-848), dead letter topic support (KIP-1034), and enhancements to CloseOptions for Kafka Streams(KIP-1153). Plus broader improvements to efficiency, operations, and observability across the platform. For a comprehensive overview, view the full release notes: https://kafka.apache.org/downloads
LEARN MORE
► Queues for Kafka Lightboard: https://youtu.be/Wb0xyqgaIqw
► Apache Release notes: https://kafka.apache.org/downloads
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent
Deleting Architecture for Better Systems ft. Daniel Doubrovkine | Ep. 19 | Confluent Developer Pod
Listen: https://confluent.buzzsprout.com | Adi Polak talks to Daniel Doubrovkine (Shopify) about his career building data‑intensive systems. Daniel’s first job: delivering pharmacy medications by bike. His challenge: building Artsy’s Art Genome and auctions as simple as possible.
🎧 Subscribe to Confluent Developer wherever you listen to podcasts: https://confluent.buzzsprout.com
▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes: https://youtube.com/@ConfluentDeveloper?sub_confirmation=1
👍 If you enjoyed this, please like it and leave us a comment
🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais: https://www.confluent.io/life-is-but-a-stream-show
CREDITS
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
ABOUT THE PODCAST
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#confluentdeveloper #podcast #confluent
Prompt Engineering is dead.
Tim Berglund and the lightboard are back with an overview of Context Engineering in agentic AI. He breaks down the components that make up context (including system prompts, tools, and resources) and provides a framework for managing these limited resources to build more efficient agents, helping you prepare for the long-horizon future.
You can try Confluent Intelligence at https://www.confluent.io/product/confluent-intelligence
Promo code coming soon...
LEARN MORE
► Confluent Intelligence: https://www.confluent.io/product/confluent-intelligence
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1
Community Slack: https://confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: https://www.linkedin.com/company/confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#contextengineering #promptengineering #confluent
Error 400: API key not valid. Please pass a valid API key..
Domain code: global
Reason code: badRequest