Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
sigmoid.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A social space for people researching, working with, or just interested in AI!

Administered by:

Server stats:

640
active users

sigmoid.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.8

Leshem Choshen

Parallel generation from auto regressive LMs
Para ll el!
Well not exactly, use a fast LM to propose the next words first
arxiv.org/abs/2302.01318

Feb 07, 2023, 08:01 AM·Public·Web
2boosts·0favorites
Quiet public
Leshem Choshen

The story is very simple
Auto regressive models predict the next word given the last, annoying and - with a strong model - slow
Instead, they propose to use a fast model to predict the next words
Then check on all of those words whether the strong model agrees about them

Quiet public
Leshem Choshen

For a bit more details:
q - strong model
p - poor model
p generated x1 .. xn words
q then calculates their probabilities (did I say on parallel?)
We accept them if q gives high probability (eq in fig)

Quiet public
Leshem Choshen

What if we reject?
We just pick another with some other probability
and while not explicit, I guess lose the future predictions by the poor model?

Great speedups, simple and clean.

Quiet public
Leshem Choshen

If you noted, they did choose a rather odd way of rechoosing when they did not agree, this is contrastive sampling, right?
It is assumed to be better, but if it is not, there is no reason not to sample in any other way at this point or am I missing something

Quiet public
Leshem Choshen

Apparently something quite similar was proposed in November, getting hot in here
arxiv.org/abs/2211.17192

arXiv.orgFast Inference from Transformers via Speculative DecodingInference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel. At the heart of our approach lie the observations that (1) hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models, and (2) using speculative execution and a novel sampling method, we can make exact decoding from the large models faster, by running them in parallel on the outputs of the approximation models, potentially generating several tokens concurrently, and without changing the distribution. Our method can accelerate existing off-the-shelf models without retraining or architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.
ExploreLive feeds
About