Contrastive fine-tuning to improve generalization in deep NER

Research output: Contribution to journalArticlepeer-review

Abstract

A novel algorithm of two-stage fine-tuning of a BERT-based language model for more effective named entity recognition is proposed. The first stage is based on training BERT as a Siamese network using a special contrastive loss function, and the second stage consists of fine-tuning the NER as a "traditional" sequence tagger. Inclusion of the contrastive first stage makes it possible to construct a high-level feature space at the output of BERT with more compact representations of different named entity classes. Experiments have shown that this fine-tuning scheme improves the generalization ability of named entity recognition models fine tuned from various pre-trained BERT models. The source code is available under an Apache 2.0 license and hosted on GitHub https://github.com/bond005/runne_contrastive_ner
Translated title of the contributionСопоставительное дообучение для повышения обобщающей способности нейросетевого распознавателя именованных сущностей
Original languageEnglish
Article number8
Pages (from-to)70-80
Number of pages11
JournalKomp'juternaja Lingvistika i Intellektual'nye Tehnologii
Issue number21
DOIs
Publication statusPublished - 15 Jun 2022

Keywords

  • named entity recognition
  • contrastive learning
  • Siamese neural networks
  • BERT
  • NER

OECD FOS+WOS

  • 1.02.EP COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE
  • 6.02.OT LINGUISTICS

Fingerprint

Dive into the research topics of 'Contrastive fine-tuning to improve generalization in deep NER'. Together they form a unique fingerprint.

Cite this