Andy Warfield recounts how his team at Amazon tackled a recurring pain point: moving massive genomic datasets between local disks and cloud storage. While helping botanist Loren Rieseberg’s lab run burst‑parallel analyses on S3, researchers spent hours copying files back and forth, a workflow that slowed discovery and wasted compute cycles.

The experiment revealed a broader pattern across scientific, ML, and media workloads—tools expect POSIX‑style filesystems, yet data lives in object stores. To bridge that gap, the S3 group built S3 Files, a thin abstraction that presents object data as a native filesystem, eliminating manual copy scripts and improving repeatability. Early tests with GATK4 and Spark showed faster job startup and lower storage costs.

Parallel to this effort, the team observed that structured formats like Apache Iceberg were gaining traction, offering mutable tables atop S3’s object API. By surfacing richer table semantics, Iceberg reduces friction for analytics pipelines. Together, S3 Files and Iceberg illustrate a shift toward storage that adapts to application needs rather than forcing developers to reshape their code.