Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo. You can now serve models that are too large for a single instance, and keep token generation fast when prompts are long. You don’t have to build or operate distributed serving infrastructure yourself. HyperPod manages Dynamo for you through the Inference Operator add-on.
As models grow, two problems limit inference. A large model can need more GPU memory than one instance has. Long prompts also slow token generation, because the prefill and decode stages compete for the same GPUs. Until now, customers had to build and run their own distributed serving stack to work around both.
With Dynamo on HyperPod, you can run a single model across multiple nodes. You can also split prefill and decode into separate GPU pools that scale independently, so a long prompt no longer stalls token generation for other requests. KV-aware routing sends each request to a worker that already holds the relevant cached context. This cuts recomputation and raises throughput. HyperPod runs the underlying infrastructure: EFA networking, GPU capacity management, node replacement, endpoint registration, autoscaling, and observability. Your team can spend its time on the models instead.
To get started, bring your existing Dynamo manifests if you want full control. You can also use a simplified configuration that HyperPod translates into the underlying infrastructure.
Amazon SageMaker HyperPod Inference with NVIDIA Dynamo support is available in US East (N. Virginia, Ohio), US West (N. California, Oregon), Europe (Frankfurt, Stockholm, Ireland, London, Spain), Asia Pacific (Mumbai, Hyderabad, Singapore, Sydney, Jakarta, Melbourne, Tokyo), and South America (São Paulo). To learn more, see the Amazon SageMaker HyperPod documentation.
Categories: marketing:marchitecture/artificial-intelligence
Source: Amazon Web Services


