Jeff Meyerson talks with Frances Perry about Apache Beam, a unified batch and stream processing model. Topics include a history of batch and stream processing, from MapReduce to the Lambda Architecture to the more recent Dataflow model, originally defined in a Google paper. Dataflow overcomes the problem of event time skew by using watermarks and other methods discussed between Jeff and Frances. Apache Beam defines a way for users to define their pipelines in a way that is agnostic of the underlying execution engine, similar to how SQL provides a unified language for databases. This seeks to solve the churn and repeated work that has occurred in the rapidly evolving stream processing ecosystem.
Frances Perry on Apache Beam
Find Your Interest
- Codable synthesis for Swift enums
- Danilo Krummrich on Rust in the Linux Kernel
- Combining opaque return types with primary associated types
- Tips and tricks for when using SwiftUI’s ViewBuilder
- Donny Nadolny on Debugging Distributed Systems
- Florian Gilcher on Rust
- Steve McConnell on Software Estimation
- Kief Morris on Infrastructure as Code
- Florian Gilcher on Rust for Safety-Critical Systems
- Decoding Swift types that require additional data
