AWS announces support for NVIDIA Inference Xfer Library (NIXL) with Elastic Fabric Adapter (EFA) to accelerate disaggregated large language model (LLM) inference on Amazon EC2. This integration enhances disaggregated inference serving through three key improvements: increased KV-cache throughput, reduced inter-token latency, and optimized KV-cache memory utilization.
NIXL with EFA enables high throughput, low-latency KV-cache transfer between prefill and decode nodes, and it enables efficient KV-cache movement between various storage layers. NIXL is interoperable with all EFA-enabled EC2 instances and integrates natively with frameworks including NVIDIA Dynamo, SGLang, and vLLM. Combined, NIXL with EFA enables flexible integration with your EC2 instance and framework of choice, providing performant disaggregated inference at scale.
AWS supports NIXL version 1.0.0 or higher with EFA installer version 1.47.0 or higher on all EFA-enabled EC2 instance types in all AWS regions at no additional cost. For more information, visit the EFA documentation.
Categories: marketing:marchitecture/networking-and-content-delivery
Source: Amazon Web Services
Latest Posts
- Amazon RDS for MySQL now supports new minor version 8.4.11
- Amazon Redshift now supports concurrency scaling of streaming ingestion workloads from Amazon Kinesis data streams
- MC1459141: Microsoft Teams Adds Admin Policy to Automatically Block External Meeting Bots
- MC1411574: Microsoft Entra ID Applies System-Preferred Authentication to First-Factor Sign-In
