This is, basically, text-to-LoRA with some extra stuff.
I.e. it basically takes text, computes and embedding and makes a LoRA adapter out of this embedding.
Note that it is equivalent to a recurrent module attached to a transformer. Dynamically generated weights (proposed in the article) are computationally equivalent to multiplicative-gating network with fixed weights. Basically just a beefier variant of GLU operating on a slightly larger state.
I.e. it basically takes text, computes and embedding and makes a LoRA adapter out of this embedding.
Note that it is equivalent to a recurrent module attached to a transformer. Dynamically generated weights (proposed in the article) are computationally equivalent to multiplicative-gating network with fixed weights. Basically just a beefier variant of GLU operating on a slightly larger state.