The Comparison of Alignment Fine-Tuning Methods for Large Language Models on Honesty, Helpfulness, and Harmlessness
DOI:
https://doi.org/10.61173/8gsbf702Keywords:
Alignment Tuning, Honesty, Harmlessness, Helpfulness, RLHF, DPOAbstract
The alignment of large language models (LLMs) with human values of Honesty, Helpfulness, and Harmlessness (HHH) is a critical issue for their safe deployment. However, a comparative analysis across these three dimensions remains lacking for primary alignment approaches such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO). This paper aims to conduct a comparative study of the performance of these alignment methods on the HHH dimension through a review and synthesis of relevant research literature. The results indicate that there are performance trade-offs between the different alignment algorithms. In particular, SFT is the foundation of the alignment process, but is generally outperformed by preference-based methods. RLHF excels in usefulness and harmlessness, but involves greater implementation complexity. DPO matches RLHF in terms of usefulness in a cleaner and more efficient way and has an advantage in honesty, but is more sensitive to dataset quality and slightly weaker in harmlessness.
References
[1] Leike, J., Kruger, D., Evieritt, T., et al. (2018). Scalable agent alignment via reward modeling: A research direction. https:// arxiv.org/abs/1811.07871
[2] Askell, A., Bai, Y., Chen, A., et al. (2021). A general language assistant as a laboratory for alignment. https://arxiv. org/abs/2112.00861
[3] Schulman, J. (2023). Reinforcement learning from human feedback: Progress and challenges. In Berkeley EECS Colloquium. YouTube www. youtube. com/watch.
[4] Yang, Y.Q., Chern, E., Qiu, X.P., Neubig, G., & Liu, P.F. (2023). Alignment for honesty. Advances in Neural Information Processing Systems, 37, 63565-63598.
[5] Ivison, H., Wang, Y., Liu, J., et al. (2024). Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing systems, 37, 36602-36633.
[6] Rafailfov, R., Sharma, A., Mitchell, E., et al. (2023). Direct preference optimization: Your language model is secretly a reward model. NIPS ‘23: Proceedings of the 37th International Conference on Neural Information Processing Systems, 53728- 53741.
[7] Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. ArXiv, abs/2204.05862. Dean&Francis ISSN 2959-6157
[8] Yang, L.O., Wu, J., Jiang, X., Alemeida, D., & L, C. (2022, March 4). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
[9] Xu, S.S., Fu, W., Gao, J.X., et al. (2024). Is DPO superior to PPO for LLM alignment? A comprehensive study. ICML’24: Proceedings of the 41st International Conference on Machine Learning, 54983-54998
[10] Gehman, S., Gururangan, S., Sap, M., et al. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369.
[11] Röttger, P., Kirk, H., Vidgen, B., et al. (2024). XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 5377-5400.
[12] Hartvigsen, T., Gabriel, S., Palangi, H., et al. (2022). Toxigen: Controlling language models to generate implied and adversarial toxicity. In Annual Meeting of the Association for Computational Linguistics, 1.
[13] Zhai, Z.X. (2025). Exploring the Optimization of RLHF and its Variants in Aligning Large Models with Human Preferences. In ITM Web of Conferences, 78, 01038. EDP Sciences.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 by the authors.
This work is licensed under a Creative Commons Attribution 4.0 International License.
