Neural networks research has largely focused on understanding a single model or training on a single dataset. However, little is known about the characteristics and relationships between different models, especially those trained or tested on different datasets. To address this gap, this talk delves into the weight and loss spaces and how they are interconnected through the mapping between models’ weights and performance.

Different fine-tuned language models based on their weights and performance were compared, with the result being that models trained in similar ways have similar weights, which fall in the same region of the weight space. Therefore, models trained on the same dataset form a tight cluster, and models trained on the same task form larger clusters.

The inverse is also shown: weights in certain regions represent models with high performance. Therefore, it is possible to traverse from one model to another in the same region and reach new models that perform comparably or even better, even on tasks that the original models were not fine-tuned for.

The findings provide insight into the relationships between models, showing that a model located between two similar models can gain the knowledge of both. This finding can improve efficient fine-tuning by choosing a model from the center of a region.