Beyond Word Embeddings

how-to-understand-bert 🤗

This is a collection of Bert Learning resources created by Roman Kryvokhyzha. The tokenization in context of BERT provides great details of the embeddings used in Bert.

---

Theory 🙌

So, I have started my long way in BERT with blog posts and videos produced by Chris McCormick. After watching the first video, I clearly understood that it is necessary to read more about Attention and Transformer.

1. Attention

   • [en] Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) ❤️

   • [en] Attention is all you need - original paper

   • [en] Attention is all you need - short video about this paper

   • [en] Attention is all you need - short video about this paper with excellent explanation ❤️

   • [ru] Attention, attention! - seminar organized by JetBrains Research

2. Transformer

   • [en] Awesome illustrations was created by Jay Alammar ❤️

   • [en] The Narrated Transformer Language Model - explanation by Jay Alammar ❤️

   • [en] Transformer (Attention is all you need) - short video with great visualizations

   • [ru] Self-Attention. Transformer overview ❤️

   • [en] Just a short recap video

   • also, in transformer folder you can find different implementations of this architecture ❤️‍🔥

---

Next, let’s talk about tokenization in context of BERT.

• [en] WordPiece Embeddings - II part of BERT Research by Chris McCormick ❤️

• [en] Tokenizers: How machines read ❤️

• also, in vocabulary folder you can find different examples ❤️‍🔥

---

And finally, let’s read more about BERT.

• [en] The Illustrated BERT blog post ❤️

• [en] A Visual Guide to Using BERT for the First Time

• [en] Google AI Blog post about open sourcing BERT

• [en] One more blog post explaining BERT

• [ru] Context based models. BERT overview ❤️

---

Examples 💪

• [en] BERT Research - Fine Tuning - Chris McCormick tutorial ❤️

• [en] BERT classifier fine-tuning with PyTorch, HuggingFace, and Catalyst - Yury Kashnitsky tutorial ❤️

• in sequence-classification folder you can find different examples ❤️‍🔥

• in document-classification folder you can find different examples ❤️‍🔥

• in bert-finetuning-catalyst folder you can find great example based on Yury Kashnitsky tutorial ❤️‍🔥

---

Additional sources 📚

• [en] Check this repository if you wanna more papers

• [en] Transformer encoder - visualized

• [en] Firing a cannon at sparrows: BERT vs. logreg

• [en] DistillBERT overview (distillation will be covered later in our course) blog post

• [en] OpenAI blog post Better Language Models and Their Implications (GPT-2)

• [en] Great PyTorch library: pytorch-transformers

• [en] Post about GPT-2 in OpenAI blog (by 04.10.2019)

• [en] OpenAI API request