Optimization in foundation models often encounters challenges like deceptive landscapes and mode collapse, necessitating diversity in the solution space for improved generalization and performance. This thesis investigates intrinsic mechanisms for integrating diversity within optimization processes, eliminating the need for manually defined diversity metrics and enhancing the efficiency of optimization.
Our first strategy employs Pareto optimality to navigate the objective space by selecting diverse solutions. We introduce 'gradient lexicase selection', a method that incorporates lexicase selection within gradient-based algorithms to promote diversity by prioritizing solutions at the boundaries of the Pareto front. Empirical results show that gradient lexicase selection enhances the generalization of prevalent deep network architectures. Subsequent work presents probabilistic and approximation techniques that achieve significant speedups without compromising on problem-solving performance.
The second strategy infers diversity that aligns with human intuition and uses it to drive optimization. We propose 'Quality Diversity through Human Feedback' (QDHF), which models diversity through preference-based learning and utilizes it in a diversity-driven optimization framework. QDHF showcases superior capabilities in autonomous diversity discovery and improving the variation and satisfaction of generative model responses. Additionally, we present ongoing efforts aiming to enhance preference-based learning using Pareto optimality in human preferences.
This thesis underscores the significance of intrinsic diversity in optimization, offering a nuanced understanding of its benefits and limitations. We demonstrate the potential of the proposed methods to tackle challenging optimization problems and pave the way for further research in this area.
Advisor: Lee Spector