Fireworks Adds 31 Partner Evals to Specialized Intelligence Index

When you're choosing a model, rank on a general leaderboard often doesn't help. You need the one that reviews contracts or builds financial models well, not the one with the highest overall score. Fireworks' Specialized Intelligence Index (SII) is meant to fill that gap, and it has just added evals from 31 more partners.

Let the people doing the work write the tests

When SII launched on September 22, it covered seven domains: healthcare, legal, cybersecurity, finance, customer support, productivity and software. As of the site's October 28 update it lists 66 benchmarks across 14 domains. The new partners include Zapier, Upwork, Grafana and Hebbia, with tasks drawn from their real workloads. Harvey, Mercor and Sierra were already on the board.

Fireworks acts as the standardized runner, using one serving configuration and one harness per benchmark across open, closed and custom models. The benchmarks are designed by the partners; Fireworks says it doesn't design them. Partners can also keep their eval sets private and publish scores only.

No composite score, on purpose

The most unusual thing about the index is that it deliberately has no overall score. It doesn't average ranks or weight quality against cost and duration. It reports score, cost per task and duration within each benchmark and domain, and leaves the trade-offs to the reader. The methodology also says it doesn't measure general capability and is no substitute for a generalist index.

The data shows plenty. On Hebbia's financial services benchmark, Claude Opus 5.5 leads at 74.6% for about $0.73 per task, while open-weight DeepSeek V4.1 Flash is fourth at 63.1% for about $0.08. On CyberGym, open models take the top five places, led by DeepSeek V4.1 Flash at 95.36%. On Harvey's legal benchmark LAB, open models GLM 5.3 and Kimi K3 rank second and third, ahead of every OpenAI and Anthropic model.

One thing to keep in mind

SII is run by Fireworks, an inference platform, and its own Ember-1 model appears on the boards, so it is both the organizer and one of the participants. Benchmarks are designed by third parties, each is labeled with who can reproduce it, and the methodology is versioned. But eval sets are generally kept private, which limits how much outsiders can check.

For teams putting AI to work, the more useful move is to find the benchmark closest to your own business and read score and cost per task together. You will often find that a second-tier model at a tenth of the price is the better deal. The data is also available as JSON and Markdown, so you can pull it down for your own comparisons.