
Patsnap optimized GPT-2 inference for patent search autocomplete using NVIDIA TensorRT on Amazon SageMaker, reducing latency by 50% and improving throughput by 200%.


<div><p>This article describes how Patsnap optimized GPT-2 inference for patent search autocomplete using NVIDIA TensorRT and Amazon SageMaker.</p><ul><li>Patsnap needed GPT-2 inference latency under 600ms for real-time patent search suggestions</li><li>NVIDIA TensorRT optimization reduced average latency from 1,172ms to 531ms (55% improvement)</li><li>Queries per second increased from 3.4 to 7.5 at maximum concurrency (120% improvement)</li><li>PyTorch model converted to TensorRT via ONNX intermediate format with no accuracy loss</li><li>Model deployed on SageMaker using bring-your-own-container with custom Docker image</li><li>Achieved 2.9x acceleration on NVIDIA P3.2xlarge GPU instances</li></ul><p>The solution demonstrates how TensorRT optimization enables cost-effective, low-latency deployment of large language models in production environments using SageMaker.</p></div>


Related articles