GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanism

Research output: Contribution to book/Conference proceedings/Anthology/ReportConference contributionContributedpeer-review

Abstract

Pretrained Language Models (PLMs) benefit from external knowledge stored in graph structures for various downstream tasks. However, bridging the modality gap between graph structures and text remains a significant challenge. Traditional methods like linearizing graphs for PLMs lose vital graph connectivity, whereas Graph Neural Networks (GNNs) require cumbersome processes for integration into PLMs. In this work, we propose a novel graph-guided self-attention mechanism, GraSAME. GraSAME seamlessly incorporates token-level structural information into PLMs without necessitating additional alignment or concatenation efforts. As an end-to-end, lightweight multimodal module, GraSAME follows a multi-task learning strategy and effectively bridges the gap between graph and textual modalities, facilitating dynamic interactions between GNNs and PLMs. Our experiments on the graph-to-text generation task demonstrate that GraSAME outperforms baseline models and achieves results comparable to state-of-the-art (SOTA) models on WebNLG datasets. Furthermore, compared to SOTA models, GraSAME eliminates the need for extra pre-training tasks to adjust graph inputs and reduces the number of trainable parameters by over 100 million.

Details

Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics
EditorsKevin Duh, Helena Gomez, Steven Bethard
Place of PublicationMexico City, Mexico
PublisherAssociation for Computational Linguistics (ACL)
Pages920–933
Number of pages14
ISBN (electronic)979-889176119-3
Publication statusPublished - Jun 2024
Peer-reviewedYes

Conference

Title2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Abbreviated titleNAACL 2024
Duration16 - 21 June 2024
Website
LocationHilton Reforma Mexico City
CityMexico City
CountryMexico

External IDs

Scopus 85197895877

Keywords

Research priority areas of TU Dresden