GNNAVI: Navigating the Information Flow in Large Language Models by Graph Neural Network

Research output: Contribution to book/Conference proceedings/Anthology/ReportConference contributionContributedpeer-review

Contributors

Abstract

Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient finetuning (PEFT) approach. GNNAVI leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNAVI employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 show GNNAVI surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNAVI with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNAVI enhances information flow and ensures a clear aggregation process.

Details

Original languageEnglish
Title of host publication62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Proceedings of the Conference
EditorsLun-Wei Ku, Andre Martins, Vivek Srikumar
PublisherAssociation for Computational Linguistics (ACL)
Pages3987-4001
Number of pages15
ISBN (electronic)979-889176099-8
Publication statusPublished - 16 Aug 2024
Peer-reviewedYes

Conference

Title62nd Annual Meeting of the Association for Computational Linguistics
Abbreviated titleACL 2024
Conference number62
Duration11 - 16 August 2024
Website
LocationCentara Grand and Bangkok Convention Centre
CityBangkok
CountryThailand

External IDs

Scopus 85205293526

Keywords