Maybe this is what the author is getting at, but if LP solvers are useful tools for editing weights, then Astra access to this methodology might make it more effective at weight editing.
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
jsw97
Very interesting analysis. It would be great to see the LP comeback and optimize. I always have the impression that we have settled to suboptimal solutions in many areas just because we have become too attached to LLMs, wary of numerical analysis.
rbnafo
Nice analysis. The simplicity of this indicates that it should be happening during training via regularization.
dansitu
What exactly is the difference between weight-space programming and something like back propagation? I guess AI can reason deeper about the weights somehow and take into account more parameters than a simple feedback from back propagation?
daemonk
Not strictly on topic but related to this -- I wonder if there is any effort/research being done with LLMs when it comes to finding low dimensional weight space representations/manifolds. If you could drastically lower the dimensionality with some algortihm X, you could efficiently use global optimization algorithms to find sparse but accurate networks (Lottery Ticket Hypothesis etc etc.)
deerstalker
Good article! The initial headlines from a few days ago made it seem like frontier models were becoming able to read, understand, and manipulate weights directly. This seemed unlikely to me since they aren’t trained on weights, they’re trained on tokens and task completion (RL). This analysis makes me think a better description is that Astra did some interesting model training research that included running small experiments with altering weights and found a nice edit that fixes some problems. Cool, also cool that this analysis finds a faster approach. But a far cry from the headlines when the paper first came out.
ultimatefan1
So we are talking about linear programming a la operations research / industrial research / e g simplex? That would be wild if 1940s-50s OR fills in a key step to p̶o̶s̶s̶i̶b̶l̶y̶ ̶e̶n̶d̶ ̶o̶f̶ ̶t̶h̶e̶ ̶w̶o̶r̶l̶d̶ RSI (if that’s the subtext on the table here). Because then I guess if the world does end, it’s just one more world-ending technology we can blame on the 1940s-50s.
ricksunny
annnd we can just 'reinvent'(*) the same techniques used to uncensor models for some time.. one step at a time.
(*) scarequotes because the LLMs that suggested these approaches have presumably been exposed to the refusal direction suppression work
nullc
Does this describe a very powerful LLM, like Astra, editing smaller models?
comments (9)
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
jsw97
rbnafo
dansitu
daemonk
deerstalker
ultimatefan1
ricksunny
(*) scarequotes because the LLMs that suggested these approaches have presumably been exposed to the refusal direction suppression work
nullc
Could Astra edit itself?
FrustratedMonky