STA 290 Seminar: Lifeng Lai

Statistics Seminar

Event Date

Location
Mathematical Sciences Building 1147

Speaker: Lifeng Lai, Professor, Department of Electrical and Computer Engineering, UC Davis

Title: Reinforcement learning based fine-tuning methods: Vulnerability and robust algorithms

Abstract: In recent years, many reinforcement learning (RL) based fine-tuning methods have been proposed for large language models (LLMs) post-training. Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) are some of the popular methods, each offering distinct advantages. The widespread applications of these fine-tuning methods raise important questions: are these RL based fine-tuning methods robust to adversarial attacks? If not, how can we modify them so that they are robust to adversarial attacks? Some recent works have empirically shown that RLHF faces substantial security issues related to the reliability of the human preference data, i.e., when there are adversarial attacks on preferences in human preferences. The human preference data can be manipulated by adversaries to mislead the RLHF algorithms, which can lead to catastrophic consequences in the real world. These important empirical studies motivate us to theoretically understand the impact and limitations of adversarial attacks on RL based fine-tuning frameworks, and to design improved RL based fine-tuning algorithms to ensure their robustness, which could maintain their safety and reliability in real-world applications. In this talk, we will discuss our recent works that aim to address these questions.

Faculty Website (ECE, UCD)

Tags