High Throughput Scheduling for Financial Services with YellowDog HTS on AWS

This article evaluates YellowDog's High Throughput Scheduling (HTS) solution on AWS for financial services workloads, demonstrating significant performance improvements in task scheduling and execution.

• YellowDog HTS achieves sustained throughput of 40,000 tasks per second, a 13x improvement over previous scheduler
• Benchmark tested 100 million tasks across 200 EC2 instances with 40,000 workers
• Task submission rate sustained at ~44,600 TPS with negligible queuing delays
• Worker utilization exceeded 99% across the cluster
• 100-million-task workload completes in under 45 minutes versus 9+ hours previously
• Adaptive batching model eliminates per-task coordination overhead and scheduler bottlenecks
• Handles Spot Instance interruptions without creating backlogs or exceeding batch windows
• Enables overnight batch runs (VaR, FRTB, RWA) to complete before trading day starts

YellowDog HTS on AWS enables financial institutions to process massive parallel workloads efficiently, supporting both intraday latency-sensitive tasks and large-scale overnight batch processing at scale.