Deepak Mishra, a researcher at Microsoft’s Research Lab, shared insights at the company’s research forum in January about how LASER (Layerwise Selective Rank Reduction) technology can boost the accuracy of large language models, Microsoft’s researchers successfully demonstrated the replacement of a large weight matrix in these models with a smaller one using LASER technology.
Understanding Weights in Neural Networks
Weights play a crucial role in artificial neural networks (ANNs), mirroring the function of synaptic connections in biological neural networks, they are essential for the network’s ability to learn and make predictions. The larger the weight in a large language model, the more it relies on that weight for accuracy.
Maintaining Accuracy with Smaller Weights
Contrary to expectations that reducing weights with LASER would lead to a loss in model accuracy, Microsoft found that the model’s performance did not decline, in fact, Mishra noted that the loss decreased with the right kind of intervention using LASER technology.
Successful Implementation in Open-Source Models
Microsoft successfully applied LASER technology to three different open-source large language models: RoBERTa, Llama 2, and GPT-J, this resulted in significant improvements, with some models experiencing up to a 30% increase in performance, for instance, the GPT-J model’s accuracy in predicting gender based on bios improved from 70.9% to 97.5% after LASER intervention.
Addressing AI Model Accuracy Concerns
While AI models are prone to realistic errors, the accuracy of large language models remains a significant concern, this includes issues of hallucination, where models don’t just misunderstand but fabricate information, inaccurate AI models and hallucinations can cause considerable harm, underscoring the need for technologies like LASER to enhance model accuracy and reliability.
