Meet Medusa: An Efficient Machine Learning Framework for Accelerating Large Language Models (LLMs) Inference with Multiple Decoding Heads

The most up-to-date development within the subject of Artificial Intelligence (AI), i.e., Large Language Models (LLMs), has demonstrated some nice enchancment in language manufacturing. With mannequin sizes reaching billions of parameters, these fashions are getting into each area, starting from healthcare and finance to schooling.

Though these fashions have proven wonderful capabilities, the event of the mannequin’s measurement has led to an elevated inference latency, which poses an issue for real-world purposes. Memory-bound operations signify the principle bottleneck in LLM inference, as it’s inefficient to move all mannequin parameters from High Bandwidth Memory (HBM) to the accelerator’s cache throughout auto-regressive decoding.

Researchers have been placing in efforts to discover a resolution to those limitations, certainly one of which is to lower the variety of decoding steps and enhance the arithmetic depth of the decoding course of. Using a smaller draft mannequin for speculative decoding, which produces a collection of tokens which can be then improved upon by the larger authentic mannequin, has been steered. However, there are difficulties with incorporating a draft mannequin right into a distributed system.

To overcome these challenges, a staff of researchers in a latest research has introduced MEDUSA, an environment friendly strategy that enhances LLM inference by incorporating further decoding heads to foretell a number of subsequent tokens in parallel. It makes use of the spine mannequin’s quite a few decoding heads to hurry up inference. These heads overcome the difficulties of speculative decoding by concurrently predicting quite a few tokens. 

MEDUSA doesn’t require a separate draft mannequin like speculative decoding requires, which makes it able to getting simply built-in into present LLM methods, even in dispersed conditions. The staff has shared that MEDUSA builds a number of candidate continuations in every decoding part and verifies them concurrently utilizing a tree-based consideration mechanism. By using parallel processing, MEDUSA lowers the variety of essential decoding steps whereas introducing little or no overhead by way of single-step latency.

Two new insights have been added to MEDUSA. First, quite a few candidate continuations have been generated utilizing MEDUSA heads, they usually have been verified concurrently. Secondly, an acceptance process has been used to decide on appropriate candidates. The staff has shared the rejection sampling technique utilized in speculative decoding, which a temperature-based threshold can successfully substitute to deal with deviations.

The research has steered two strategies for fine-tuning LLMs’ predictive MEDUSA heads, that are as follows.

1. MEDUSA-1: This permits lossless inference acceleration by instantly fine-tuning MEDUSA on prime of a frozen spine LLM. MEDUSA-1 has been steered for use when incorporating MEDUSA into an present mannequin or in settings with restricted computational assets. It makes use of much less reminiscence and might be made much more environment friendly by making use of quantization methods.

2. MEDUSA-2: This methodology adjusts MEDUSA and the principle LLM concurrently. While it presents a higher speedup and improved prediction accuracy for MEDUSA heads, it necessitates a singular coaching recipe to keep up the spine mannequin’s performance. MEDUSA-2 is suitable when assets are plentiful and permits simultaneous coaching of MEDUSA heads and the spine mannequin with out sacrificing output high quality or next-token prediction capacity.

The analysis has additionally steered a number of additions to boost or broaden using MEDUSA. These embody a traditional acceptance scheme to extend the acceptance charge with out sacrificing technology high quality and a self-distillation methodology within the absence of coaching information. The staff has shared that the analysis strategy of MEDUSA included testing on fashions of various sizes and coaching protocols. The outcomes have demonstrated that MEDUSA-1 can speed up information by greater than 2.2 occasions with out sacrificing technology high quality. Moreover, the acceleration is improved to 2.3-3.6× utilizing MEDUSA-2. 

Check out the Paper and Github. All credit score for this analysis goes to the researchers of this challenge. Also, don’t overlook to observe us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you want our work, you’ll love our e-newsletter..

Don’t Forget to affix our Telegram Channel