07 Topic Models and Bayesian Methods

In this block we cover:

• Simple topic models
  • The Bag Of Words
  • term frequency, inverse document frequency (tf-idf) representation
  • N-grams
• Navigating Bayesian Methodology
• Latent Dirichlet Allocation
  • Perplexity and coherence
  • Text cleaning
• Practical use of LDA
  • Regexp (regular expressions)
  • Pipelines for data processing
  • Example application

Lectures:

A lightning tour of Bayesian statistics:

• 7.1 A lightning tour of Bayesian Methods
• 7.1.1 Neural Nets and the Perceptron Discussion

Topic Models:

• 7.2 Bags of Words and Topic Models
• 7.2.1 CNNs and Transformers Discussion

Workshop:

• 7.3.1 Workshop on Topic Modelling (Part 1, Setup)
• 7.3.2 Workshop on Topic Modelling (Part 2, Main Content)

Assessments:

• The main long form portfolio Portfolio 7.
• Block07 on Noteable via Blackboard (2026-27 link)

References

Bag of Words

• Python Bag of Words: p259 Python Machine Learning (Raschka & Mirjalili, 2nd ed 2017).
• Topic Modeling and Latent Dirichlet Allocation: An Overview (Weifeng Li, Sagar Samtani and Hsinchun Chen)
• Stephen Robinson, Microsoft Research Understanding Inverse Document Frequency: On theoretical arguments for IDF

  Bayesian Methodology

• Bayesian Programming languages: Software which allows you to specify the model without requiring you to specify the method:
  • OpenBUGS
  • JAGS
  • STAN
• There is a super useful list of conjugate priors and interpretations on the Conjugate Prior Wikipedia page!
• Monte Carlo:
  • Gamerman and Hedibert. Markov chain Monte Carlo: stochastic simulation for Bayesian inference.
  • Doucet, Godsill, and Andrieu. “On sequential Monte Carlo sampling methods for Bayesian filtering” Statistics and computing 10.3 (2000): 197-208.
  • Andrieu, Doucet, and Holenstein Particle Markov chain Monte Carlo methods
• ABC:
  • Beaumont, Zhang, and Balding. “Approximate Bayesian computation in population genetics.” Genetics 162.4 (2002): 2025-2035.
  • Murray, Ghahramani, and MacKay. “MCMC for doubly-intractable distributions” Proceedings of the 22nd Annual Conference on Uncertainty in Artificial Intelligence (UAI) (2006).
• Variational Inference: * Blei, Kucukelbir and McAuliffe. “ Variational Inference: A Review for Statisticians”, JASA (2017): 859-877. * Blei and Jordan. “Variational inference for Dirichlet process mixtures”, Bayesian analysis 1.1 (2006): 121-143. * A Beginner’s Guide to Variational Methods, by Eric Jang.

Latent Dirichlet Allocation

• B. Barde and A. Bainwad. “An overview of topic modeling methods and tools” (2017) ICICCS 745-750.
• Blei, David M., Andrew Y. Ng, and Michael I. Jordan. “Latent dirichlet allocation”, Journal of machine Learning research 3.Jan (2003): 993-1022.
• Neural Networks approaches to document models:
  • Mathgen generates random papers
  • Topic-RNN infers a topic model using a Neural Network

Data science topic modelling

• Preparing Data for Topic Modelling
• NLP for legal documents
• Machine-Learning-In-Law github repo

Judging topic models

• Chang, Jonathan, Jordan Boyd-Graber, Sean Gerrish, Chong Wang and David M. Blei. 2009. Reading Tea Leaves: How Humans Interpret Topic Models. NIPS.
• Stevens, Kegelmeyer, Andrzejewsk and Buttler Exploring Topic Coherence over many models and many topics

Data sources

• Kaggle dataset for fake news
• Intelligence and Security Informatics Data Sets
• Vizsec security data collection
• Threatminer cyber data with NLP
• Phishing data corpus

Previous: Block 06. Next: Block 08.