Tag archive

#big-data

Topic-specific entry points for internal linking, discovery, and deeper subject exploration.

3 articles

The 'Small Files' Problem in Data Lakes: Why Your Kafka Sink is Slow

The 'Small Files' Problem: The Data Lake Killer Streaming data from Kafka into a Data Lake (like Amazon S3 or Azure Blob Storage) seems simple. However, if you write data as soon as it arrives, you will quickly hit the S…

Apr 20, 20262 min read

Deep Dive

#data-lake#s3#kafka-connect

AWSExpert

S3 Express One Zone: When to Use it for Stateful Workloads

S3 Express One Zone Amazon S3 Express One Zone stores data in a single AZ, reducing network hops and latency. It's not a general-purpose storage; it's a specialized tool. 1. Use Case: Transient Data Perfect for Spark Shu…

Apr 20, 20261 min read

Deep DiveBackend Systems Mastery

#aws#s3#storage

System DesignAdvanced

System Design: Designing a Distributed File System (HDFS/GCS Style)

System Design: Designing a Distributed File System (HDFS) A Distributed File System (like HDFS or GFS) is designed to store massive datasets across a cluster of commodity servers. It handles the complexity of breaking la…

Apr 20, 20262 min read

Deep Dive

#system-design#hdfs#distributed-storage