WebLLM: high-performance in-browser LLM inference engine

comments (10)

• This seems to be the demo:

  https://chat.webllm.ai/

  I am getting:

      WebGPUNotAvailableError: WebGPU is not supported in
      your current environment, but it is necessary to
      run the WebLLM engine.
  

  On both, FireFox and Chromium on Linux.

  TekMol 20d ago

• Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.

  refulgentis 20d ago

• It is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.

  mandeepj 20d ago

• I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.

  MarioMan 20d ago

• Oh nice, this actually works in Safari on my Mac! Prefill: 181.2 tok/s, Decode: 58.5 tok/s on my M4 Max -- not too impressive for 1B, but definitely impressive that it runs in my browser!

  LoganDark 19d ago

• I love it! It works fine for me on Google Chrome on my MacBook Pro 14-inch M5 16GB RAM and 10 Cores CPU and 10 Cores GPU, with Metal 4 support

  karuppiah7890 19d ago

• Is there any use case/demo of lighter models like using embeddings for full text search, STT, OCR etc?

  aitchnyu 17d ago

• You might like webml-kit https://npm.im/webml-kit

  init0 20d ago

• Bake me a cake

  responds with

  > Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.

  conceptme 20d ago

• A WebX technology that actually involves browsers!

  adastra22 20d ago