Scalable and Observable E2E Genomics Pipelines using AWS Step Functions
A reference architecture for getting sequencer output into the cloud and through secondary analysis without fragile file watchers and manual checks. Each pipeline phase is a reusable, decoupled Step Functions state machine with a consistent input and output shape; DataSync moves data to S3, AWS Batch provides compute, and Nextflow runs the analysis. It scales by running workflows in parallel, keeps cost tied to active runs with serverless components, and makes every step visible, with ChatOps alerts when something fails.
- AWS Step Functions
- AWS Batch
- DataSync
- Nextflow
- Genomics