Tag archive

#s3

Topic-specific entry points for internal linking, discovery, and deeper subject exploration.

8 articles

The 'Small Files' Problem in Data Lakes: Why Your Kafka Sink is Slow

The 'Small Files' Problem: The Data Lake Killer Streaming data from Kafka into a Data Lake (like Amazon S3 or Azure Blob Storage) seems simple. However, if you write data as soon as it arrives, you will quickly hit the S…

Apr 20, 20262 min read

Deep Dive

#data-lake#s3#kafka-connect

AWSExpert

S3 Express One Zone: When to Use it for Stateful Workloads

S3 Express One Zone Amazon S3 Express One Zone stores data in a single AZ, reducing network hops and latency. It's not a general-purpose storage; it's a specialized tool. 1. Use Case: Transient Data Perfect for Spark Shu…

Apr 20, 20261 min read

Deep DiveBackend Systems Mastery

#aws#s3#storage

AWSAdvanced

S3 Express One Zone: When to use it

S3 Express One Zone For stateful data processing (like Spark shuffle files), standard S3 latency is too high. S3 Express One Zone offers sub-millisecond access for transient data.

Apr 20, 20261 min read

Deep DiveBackend Systems Mastery

#aws#s3#storage

System DesignAdvanced

System Design: Designing Google Drive (Distributed File Storage)

System Design: Designing Google Drive Designing a distributed file storage system like Google Drive or Dropbox requires more than just uploading files to S3. You must handle large files efficiently, synchronize state acr…

Apr 20, 20263 min read

Case Study

#system-design#google-drive#distributed-storage

System DesignAdvanced

System Design: Designing an Object Store (Amazon S3 Internals)

System Design: Designing an Object Store (Amazon S3) An Object Store is a distributed storage system designed to store massive amounts of unstructured data (Photos, Videos, Backups). Unlike a file system, it provides a s…

Apr 20, 20263 min read

Case Study

#system-design#s3#object-store

System DesignAdvanced

System Design: Building a File Upload Platform

File upload looks easy until production traffic arrives. Users upload huge files, mobile networks drop halfway through, browsers retry, malicious files appear, thumbnails fail, metadata gets out of sync, signed URLs expi…

Apr 8, 202610 min read

Deep Dive

#system design#file upload#object storage

AWSAdvanced

Terraform Infrastructure as Code: Production Patterns and Pitfalls

Terraform is the industry-standard tool for Infrastructure as Code (IaC) — defining cloud infrastructure as declarative HCL configuration that can be version-controlled, reviewed, and applied reproducibly. The value prop…

May 14, 20257 min read

Deep Dive

#terraform#infrastructure as code#aws

AWSAdvanced

Cloud Cost Optimization: Engineering Practices That Cut AWS Bills by 50%

Cloud bills scale with usage — but they also scale with inattention. Most teams that haven't deliberately optimized their AWS spend are 30-50% over what they need to pay for the same workload. The savings come from a pre…

Apr 29, 20257 min read

Deep Dive

#aws#cost optimization#ec2