Reading Discussion 6

Key Word(s): Transfer Learning

Selected Readings

• Expository

  • Sebastian Ruder: NLP's ImageNet moment has arrived.
  • Jay Alammar: The Illustrated Word2Vec. Visual introduction to word embeddings, language models, bags of words and skip grams.
  • Chris McCormick's tutorial on the Word2Vec skip-gram model and article on its application to recommenders.

  • Jurafsky and Martin (2019), 'N-gram Language Models', §3.1 (Ngrams), §3.2 (Evaluating Language Models), and §3.3 (Generalization and Zeros). (The other sections are interesting too, but tangential to this course.)

  • Mihail Eric: Deep Contextualized Word Representations with ELMo

  • FastAI's 'code-first' lectures on language modelling and transfer learning for NLP (2019 course).

  • Evaluation Metrics for Language Modelling. An introduction to perplexity, bits-per-character, and cross entropy.
  • Sebastian Ruder: The State of Transfer Learning in NLP (blog post) and Transfer Learning in Open-Source Natural Language Processing (conference presentation).
  • Tracking Progress in Natural Language Processing.

• Use Cases

  • TensorFlow Embedding Projector. An excellent tool to visualise the Word2Vec or your own embeddings in 2-3 dimensions. Projection is done using either PCA, t-SNE, or UMAP.
  • Using Word2vec for Music Recommendations.

• Research

  • Mikolov et al (2013a), 'Efficient Estimation of Word Representations in Vector Space'.
  • Mikolov et al (2013b), 'Distributed Representations of Words and Phrases and their Compositionality'.
  • Pennington et al (2014), 'GLoVe: Global Vectors for Word Representation'.
  • Y. Kim et al 2015, Character-Aware Neural Language Models
  • K. W. Zhang and S. Bowman (2019), More than Syntax
  • J. Howard and S. Ruder, 2018, Universal Language Model Fine-tuning for Text Classification
  • Peters et al (2018), 'Deep contextualized word representations'.
  • Yang et al (2019), 'XLNet: Generalized Autoregressive Pretraining for Language Understanding'.
  • McCann et al, 2018, Learned in Translation: Contextualized Word Vectors

* Next presentations, select from Research or Use Case