AWS announces support for NVIDIA Inference Xfer Library (NIXL) with Elastic Fabric Adapter (EFA) to accelerate disaggregated large language model (LLM) inference on Amazon EC2. This integration enhances disaggregated inference serving through three key improvements: increased KV-cache throughput, reduced inter-token latency, and optimized KV-cache memory utilization.
NIXL with EFA enables high throughput, low-latency KV-cache transfer between prefill and decode nodes, and it enables efficient KV-cache movement between various storage layers. NIXL is interoperable with all EFA-enabled EC2 instances and integrates natively with frameworks including NVIDIA Dynamo, SGLang, and vLLM. Combined, NIXL with EFA enables flexible integration with your EC2 instance and framework of choice, providing performant disaggregated inference at scale.
AWS supports NIXL version 1.0.0 or higher with EFA installer version 1.47.0 or higher on all EFA-enabled EC2 instance types in all AWS regions at no additional cost. For more information, visit the EFA documentation.
Categories: marketing:marchitecture/networking-and-content-delivery
Source: Amazon Web Services
Latest Posts
- R2 – R2 Data Catalog now supports read-only API tokens
- MC1425968: Dynamics 365 Project Operations Adds Configurable Maximum Daily Time Entry Duration
- MC1425995: Dynamics 365 Project Operations Adds Time Zone Independent Time Entry Grid
- MC1422074: Microsoft 365 Copilot Adds Option for OpenAI-Operated Models via New Admin Setting
