Amazon SageMaker JumpStart now offers a model leaderboard in Amazon SageMaker Studio. You can use it to compare foundation models side by side on throughput, latency, and cost, and then deploy or customize the model that fits your workload.
Comparing models has usually meant reading model cards one at a time and relying on benchmark scores that different sources published under different conditions. The JumpStart leaderboard puts consistent measurements in one view. AWS evaluates every model on three reasoning benchmarks, BIG-Bench Hard, GPQA Diamond, and MMLU-Pro, using the same evaluation setup for each model. AWS also measures time to first token, throughput, and instance cost on SageMaker for each JumpStart deployment configuration. You can use all of the filters already available for JumpStart models, such as model provider, license, and input and output modality, to narrow the list. You can also choose a use case profile and an optimization goal (balanced, cost, latency, or throughput) to see the metrics for the configuration you plan to deploy. At launch, the leaderboard covers more than 70 LLMs from providers including Meta, Google, Qwen, IBM, and Arcee.
The JumpStart model leaderboard is available at no additional charge in all AWS commercial Regions where Amazon SageMaker Studio is available. To get started, open SageMaker JumpStart in SageMaker Studio and choose the Leaderboard view. To learn more, see JumpStart leaderboard methodology and read the launch blog post.
Categories: general:products/aiml,marketing:marchitecture/artificial-intelligence,general:products/amazon-sagemaker-jumpstart
Source: Amazon Web Services


