Researchers from NVIDIA and the University of Maryland Propose ODIN: A Reward Disentangling Technique that Mitigates Hacking in Reinforcement Learning from Human Feedback (RLHF)

The well-known Artificial Intelligence (AI)-based chatbot, i.e., ChatGPT, which has been constructed on high of GPT’s transformer structure, makes use of the approach of Reinforcement Learning from Human Feedback (RLHF). RLHF is an more and more essential technique for using the potential of pre-trained Large Language Models (LLMs) to generate extra useful, truthful responses that are in line with human preferences.

In RLHF, a language mannequin is skilled to supply responses that maximize the realized reward by means of reinforcement studying, after which a reward mannequin is skilled primarily based on human preferences for specific prompts. Since gathering human rankings is usually simpler than gathering demos for supervised fine-tuning, this strategy streamlines the course of of gathering knowledge. 

However, reward hacking is a delicate drawback with RLHF, the place the coverage will get a big reward with out assembly the actual targets. This occurs because of this of the reward mannequin’s restricted Out-Of-Distribution (OOD) generalization and potential imperfections in representing human preferences. Being a powerful LLM, the language mannequin can present OOD examples to take benefit of flaws in the reward mannequin. 

The state of affairs is additional difficult by human choice knowledge, which is continuously skewed and inconsistent because of job complexity and subjectivity, defects in ranking requirements, and the low caliber of raters. Verbosity is a well-liked instance of reward hacking, in which fashions produce extra tokens to look extra thorough or higher formatted in responses, however there isn’t any actual enchancment in high quality.

In order to deal with these points, current analysis from NVIDIA and the University of Maryland has aimed to mitigate reward hacking by inspecting how RL algorithms and incentive fashions have an effect on verbosity and efficiency. The group has introduced an analysis approach to check varied coaching setups and account for biases in model-based evaluations. The approach has offered a complete information of varied response durations by evaluating efficiency on the Pareto entrance of analysis rating vs. size. 

This course of is meant to research the trade-off between the LLM’s evaluation rating and response period, permitting for a scientific comparability of totally different coaching settings. By various the coaching hyperparameters, it may be evaluated how these modifications have an effect on the ratio of verbosity to reply high quality.

The research appears to be like at RL hyperparameters and methods, akin to reward clipping and size penalty, to minimize reward hacking on size. The main objective is to take away the spurious size sign from the reward, although varied tuning procedures can yield higher outcomes. To accomplish this, the group has recommended a two-head reward mannequin that separates representations for size from true preferences. The size head is deleted throughout RL. 

The recommended reward disentangling approach, ODIN, has been used with the assist of which, even with a extra expensive tuning price range, the coverage was capable of attain a bigger Pareto entrance than prior outcomes. Proximal Policy Optimisation (PPO) and ReMax each profit from ODIN’s effectiveness, indicating that it may be used to boost different RL-tuning strategies and reduce size hacking.

In conclusion, this technique’s experimental outcomes have proven a noteworthy lower in the reward mannequin’s affiliation with response period. The derived technique performs considerably higher when the high quality of the info is prioritized over verbosity. This technique efficiently reduces the drawback of response length-related reward hacking, enhancing the dependability and utility of LLMs skilled utilizing the RLHF paradigm.

Check out the Paper. All credit score for this analysis goes to the researchers of this undertaking. Also, don’t overlook to observe us on Twitter and Google News. Join our 37k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you want our work, you’ll love our e-newsletter..

Don’t Forget to hitch our Telegram Channel