Architecting offline-first generative AI applications for edge deployments using AWS services

This article presents a reference architecture for deploying offline-first generative AI applications at the edge using AWS services, enabling AI inference in locations with unreliable or unavailable cloud connectivity.

• Use Amazon Bedrock and SageMaker AI to customize small language models (SLMs) for domain-specific tasks in the cloud
• Deploy fine-tuned models to edge devices using AWS IoT Greengrass for orchestration and lifecycle management
• Implement hybrid approach combining fine-tuning with Retrieval Augmented Generation (RAG) using ChromaDB for accurate, up-to-date responses
• Run local inference using Ollama with Strands Agents orchestration for extensible tool integration without cloud dependency
• Create feedback loops that synchronize usage data back to cloud when connectivity permits for continuous model improvement
• Apply security controls including encryption at rest/transit, authentication, input validation, network segmentation, and centralized logging
• Demonstrated 17% performance improvement with fine-tuned gpt-oss-20b + RAG versus base model across multiple LLM evaluators

This architecture enables AI-powered decision support in manufacturing, offshore energy, remote agriculture, and other environments with connectivity constraints or strict latency requirements.