The BERT model was proposed in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. It's a bidirectional transformer pretrained using a combination of masked language modeling objective and next sentence prediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.
The abstract from the paper is the following:
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
Tips:
-
BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.
-
BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.
-
Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:
- a special mask token with probability 0.8
- a random token different from the one masked with probability 0.1
- the same token with probability 0.1
-
The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.
This model was contributed by thomwolf. The original code can be found here.
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
- A blog post on BERT Text Classification in a different language.
- A notebook for Finetuning BERT (and friends) for multi-label text classification.
- A notebook on how to Finetune BERT for multi-label classification using PyTorch. 🌎
- A notebook on how to warm-start an EncoderDecoder model with BERT for summarization.
- [
BertForSequenceClassification] is supported by this example script and notebook. - [
TFBertForSequenceClassification] is supported by this example script and notebook. - [
FlaxBertForSequenceClassification] is supported by this example script and notebook. - Text classification task guide
- A blog post on how to use Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition.
- A notebook for Finetuning BERT for named-entity recognition using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this version of the notebook instead.
- [
BertForTokenClassification] is supported by this example script and notebook. - [
TFBertForTokenClassification] is supported by this example script and notebook. - [
FlaxBertForTokenClassification] is supported by this example script. - Token classification chapter of the 🤗 Hugging Face Course.
- Token classification task guide
- [
BertForMaskedLM] is supported by this example script and notebook. - [
TFBertForMaskedLM] is supported by this example script and notebook. - [
FlaxBertForMaskedLM] is supported by this example script and notebook. - Masked language modeling chapter of the 🤗 Hugging Face Course.
- Masked language modeling task guide
- [
BertForQuestionAnswering] is supported by this example script and notebook. - [
TFBertForQuestionAnswering] is supported by this example script and notebook. - [
FlaxBertForQuestionAnswering] is supported by this example script. - Question answering chapter of the 🤗 Hugging Face Course.
- Question answering task guide
Multiple choice
- [
BertForMultipleChoice] is supported by this example script and notebook. - [
TFBertForMultipleChoice] is supported by this example script and notebook. - Multiple choice task guide
⚡️ Inference
- A blog post on how to Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia.
- A blog post on how to Accelerate BERT inference with DeepSpeed-Inference on GPUs.
⚙️ Pretraining
🚀 Deploy
- A blog post on how to Convert Transformers to ONNX with Hugging Face Optimum.
- A blog post on how to Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS.
- A blog post on Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module.
- A blog post on Serverless BERT with HuggingFace, AWS Lambda, and Docker.
- A blog post on Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler.
- A blog post on Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker.
[[autodoc]] BertConfig - all
[[autodoc]] BertTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary
[[autodoc]] BertTokenizerFast
[[autodoc]] TFBertTokenizer
[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput
[[autodoc]] models.bert.modeling_tf_bert.TFBertForPreTrainingOutput
[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput
[[autodoc]] BertModel - forward
[[autodoc]] BertForPreTraining - forward
[[autodoc]] BertLMHeadModel - forward
[[autodoc]] BertForMaskedLM - forward
[[autodoc]] BertForNextSentencePrediction - forward
[[autodoc]] BertForSequenceClassification - forward
[[autodoc]] BertForMultipleChoice - forward
[[autodoc]] BertForTokenClassification - forward
[[autodoc]] BertForQuestionAnswering - forward
[[autodoc]] TFBertModel - call
[[autodoc]] TFBertForPreTraining - call
[[autodoc]] TFBertLMHeadModel - call
[[autodoc]] TFBertForMaskedLM - call
[[autodoc]] TFBertForNextSentencePrediction - call
[[autodoc]] TFBertForSequenceClassification - call
[[autodoc]] TFBertForMultipleChoice - call
[[autodoc]] TFBertForTokenClassification - call
[[autodoc]] TFBertForQuestionAnswering - call
[[autodoc]] FlaxBertModel - call
[[autodoc]] FlaxBertForPreTraining - call
[[autodoc]] FlaxBertForCausalLM - call
[[autodoc]] FlaxBertForMaskedLM - call
[[autodoc]] FlaxBertForNextSentencePrediction - call
[[autodoc]] FlaxBertForSequenceClassification - call
[[autodoc]] FlaxBertForMultipleChoice - call
[[autodoc]] FlaxBertForTokenClassification - call
[[autodoc]] FlaxBertForQuestionAnswering - call