How CSIRO built scalable, cost-optimized genomic variant querying on AWS

This article describes how CSIRO built Serverless Beacon (sBeacon), a production-ready serverless implementation of the GA4GH Beacon protocol for securely querying genomic variant data on AWS.

• Scales to support hundreds of millions of individuals and billions of genomic locations for mega-biobank datasets
• Operates at approximately USD 0.40 per month for 1000 Genomes-scale datasets with low ingestion and query costs
• Returns real-world queries in seconds (~5 seconds) with near real-time data lookups
• Directly consumes standard VCF files without heavy data ingestion or transformation
• Supports rapid onboarding of new data with generation-to-use lifecycle in ~18 seconds
• Implements zero-trust security with explicit authentication, least-privilege access, ephemeral compute isolation, and cloud-native boundary controls
• Enables decentralized data ownership where genomic data stays with original holders and doesn't require central databases
• Uses modular Lambda-based architecture with separate functions for metadata and variant querying

sBeacon demonstrates how serverless AWS services enable affordable, scalable, and secure genomic data sharing for research and clinical applications across institutions.