>The browser swarm from earlier this year peaked at roughly 1,000 commits per hour on Git. The new system peaks at around 1,000 commits per second.
>To facilitate this rate of activity, we built a new version control system (VCS) from scratch. Throughput was not the only reason to own this layer. Every change in the system passes through the VCS, so it is where collisions first become visible, and several of the coordination mechanisms in the next section are implemented directly inside of it.
Talk about inventing the universe to make a button.
htrp
Love to see these crazy kinds of experiments going on. Even if this doesn't 100% work or is prohibitively expensive for now, these are glimpses into the future in the same way people were talking about coding agents in 2023 when we just had tab complete.
anthonypasq
Am I reading this right? Opus + Composer did a comparable job to Fable, at ~1/19th the price, and half the LoC?
dctwin
> To test that progress, we returned to a task the old swarm had struggled with: building SQLite from scratch, in Rust, from nothing but its documentation.
Isn't SQLite's source code in its training data?
handfuloflight
How do we know if these models weren’t trained on Turso’s rewrite of SQLite in Rust?
It seems both likely that they were and impossible to remove that code from pretraining. Doesn’t that make this just about LLM memorization of the training set? What am I missing?
shay_ker
I would have loved to see more of the harness engineering shared as code. Instead we are left with only the outcome.
I guess that makes sense since the harness is the product in the case of Cursor.
whinvik
Today it is SQLite, tomorrow Linux?
I'm not sure of the long-term viability of these brute force swarm approaches, but clearly the AI companies dig the marketing they get out of it. But the fact that they get so far at this stage suggests we have a very interesting 2 years coming up.
stillpointlab
These tests are interesting but I can't help feeling that these are benchmaxxing results. "We made our harness make sqlite in rust" sounds like a much easier probelm statement than having to recreate an application like facebook. I think the concept of having to integrate is the real challenge. Re-writing code in new languages based on decent docs is a difficult but less useful measure of how AI can help replace engineers.
Imagine this: you need to build a new CRUD app but your AI agent spends 20M toks on just writing it's own custom login function, that's a waste of time and money. I would not like to have a new system developed for me in such cases. I think we need a test where the agents are more specifically tasked to find technologies worth integrating and hence write the fewest lines of code.
notahan
I initially thought that getting agents to work for longer and in large groups was the future, but I'm increasingly thinking that, at least for engineering, just one thread makes more sense. The agent pulls things into context as needed. One thing that I've been experimenting with is also letting the agent remove things from context, such as files. But just adding to the context and compacting when it's full seems like it might beat a lot of more advanced options. Because the model is good, it knows what to put in the summary; just enough for the model to be able to rebuild the context from that seed (e.g. pulling in relevant files/data into the context).
edg5000
This is almost a year behind Steve Yegge's first post on beads. Gas Town and Gas City provide orchestration for the swarm. So far I haven't seen a perfect implementation but this idea isn't new.
comments (10)
>To facilitate this rate of activity, we built a new version control system (VCS) from scratch. Throughput was not the only reason to own this layer. Every change in the system passes through the VCS, so it is where collisions first become visible, and several of the coordination mechanisms in the next section are implemented directly inside of it.
Talk about inventing the universe to make a button.
htrp
anthonypasq
dctwin
Isn't SQLite's source code in its training data?
handfuloflight
It seems both likely that they were and impossible to remove that code from pretraining. Doesn’t that make this just about LLM memorization of the training set? What am I missing?
shay_ker
I guess that makes sense since the harness is the product in the case of Cursor.
whinvik
I'm not sure of the long-term viability of these brute force swarm approaches, but clearly the AI companies dig the marketing they get out of it. But the fact that they get so far at this stage suggests we have a very interesting 2 years coming up.
stillpointlab
Imagine this: you need to build a new CRUD app but your AI agent spends 20M toks on just writing it's own custom login function, that's a waste of time and money. I would not like to have a new system developed for me in such cases. I think we need a test where the agents are more specifically tasked to find technologies worth integrating and hence write the fewest lines of code.
notahan
edg5000
smoyer