Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
satvikpendem
Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
ashivkum
I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.
Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
mchusma
One tangible thing this doesn't touch on: SpeechAnalyzer supports streaming, so you can see what it's hearing from you as you talk. A massive UX improvement. Many of the other models force you to record, then it transcribes the audio to text as a single job, then returns the entire blob of text. It's slow, and frustrating if you're talking, only to realize it stopped listening after 2 minutes.
mvkel
Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
summarity
Just this week I built a live subtitles app for my mother in law who is hard of hearing (she has a hearing aid but still has trouble decoding, but can still read faster than me)
So I ended up organically testing and ending up with SpeechAnalyzer because it was not only fast and accurate enough, but you also see live results as you talk. It also has speaker identification and people can register their voices. And it does all processing on device.
It also had the best model for Indian accented English, given she lives in India.
So I was quite impressed, but the holy grail to me is transcription that does speaker identification but also works in a standard family conversation, where multiple people interrupt each other all the time.
I will say though, I'm really curious as to what Claude Code Desktop uses for their voice mode, because it seems even better than Apple's, and it provides realtime feedback. Maybe they're using apple's model?
atonse
Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
modeless
Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.
MBCook
Finally. I‘d be delighted though if they actually implemented language autodetection (like everywhere else) though. There’s little more frustrating in my day to day than having dictated half a page to find that it‘s complete gibberish because Apple forces you to select the right language first…
endymi0n
I took a swing at bringing this into Handy.computer if anybody's interested: https://github.com/cjpais/Handy/discussions/1031 . Looks like there has been past demand for someone to implement it, but no proposed PRs. This article was inspiring.
comments (10)
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
satvikpendem
ashivkum
Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
mchusma
mvkel
summarity
So I ended up organically testing and ending up with SpeechAnalyzer because it was not only fast and accurate enough, but you also see live results as you talk. It also has speaker identification and people can register their voices. And it does all processing on device.
It also had the best model for Indian accented English, given she lives in India.
So I was quite impressed, but the holy grail to me is transcription that does speaker identification but also works in a standard family conversation, where multiple people interrupt each other all the time.
I will say though, I'm really curious as to what Claude Code Desktop uses for their voice mode, because it seems even better than Apple's, and it provides realtime feedback. Maybe they're using apple's model?
atonse
modeless
MBCook
endymi0n
akurilin