PlatPhorm News PUBLIC DMZ
Root Network HN Source Priority Newest First MCP Ready
Platform
NetworkdirectoryToolscapabilitiesAPIspublic docsMCPregistryDocsopenapi
Trust
StatushealthSupportpacketsTrustpolicyHealthjson
Machine
GraphtopologyMCPjson-rpcOpenAPIschemaRSSfeedLLMSagentsSitemapcrawlSearchOpssourceTrustOpssourceAgentOpssource

comments (10)

  • Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...

    From Terminal-bench-2.1 details,

    > We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.

    This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.

    For reference, in tbench-2.1,

    1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)

    2. 8 out of 89 tasks allow 8GB RAM

    This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1

    As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.

    GodelNumbering

  • I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too)

    - Chinese models

    - Grok

    - Meta

    - Google

    - OpenAI

    - Anthropic

    I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.

    kilroy123

  • Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead and now with xAI and Meta at least delivered something that's competitive with useful models and cheap too. Granted, the narrative that the two leading labs are ahead still holds with Fable (and perhaps an upcoming GPT6), but it's not as over as common knowledge by the opinion leaders would have us believe.

    Sol-

  • The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input!

    https://dev.meta.ai/docs/getting-started/pricing-rate-limits

    Tiberium

  • My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?

    phillipcarter

  • It seems to trade blows with GPT 5.5 and Opus 4.8 in performance while being cheaper than GLM 5.2.

    bel8

  • > Model API is not available in your region.

    :(

    Well, Vietnam is not in the list of restricted territories.

    Anyway, what is "your region" ?

    Is this where I am now, or is it where I activated my Oculus 2 five years ago ?

    eugene3306

  • How is every company able to show itself at the top of every benchmark?

    paxys

  • Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.

    What kind of use case would be best for that shape?

    EgregiousCube

  • Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.

    chvid