Probability and Statistics for Information Theory

A course that connects probability, estimation, and stochastic modeling to information and coding theory.

Synopsis

This course begins by establishing probability foundations, random variables, and dependence, then moves to inequalities, limit theorems. These tools are presented with an eye toward information theory, entropy as an expectation, mutual information as dependence, KL divergence via Jensen’s inequality, and decoding as a hypothesis test. The second part introduces information measures, the source and channel coding theorems, and the connection between statistical inference and reliable communication.

Prerequisites

Basic probability, calculus, and familiarity with random variables.

Format

Lectures focus on theory and fundamental principles.

Assessment

  • Problem sets and proof-based exercises.
  • Coding sessions for the Information Theory part.

Topics covered

  • Probability axioms, sample spaces, and Bayes’ theorem
  • Random variables, PMF/PDF/CDF, and common distributions
  • Expectation, variance, and higher moments
  • Joint distributions, independence, conditional independence, and correlation
  • Conditional expectation and the law of iterated expectation
  • Markov, Chebyshev, and Jensen inequalities
  • Entropy, mutual information, KL divergence, and differential entropy
  • Source coding theorem, asymptotic equipartition property, and channel capacity

References

  • David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.
  • Morris H. DeGroot and Mark J. Schervish, Probability and Statistics, 4th edition, Pearson, 2012.

Link: Link to the course (Moodle key available upon request)

Machine Learning

An introductory course that situates modern AI, explores statistical learning foundations, and presents the main neural network architectures.

Synopsis

This course starts from the historical context of AI, from Dartmouth to large language models, and positions symbolic AI relative to deep learning. It covers supervised, unsupervised, and reinforcement learning, and introduces key tasks such as classification, regression, and generation. The course then reviews mathematical foundations for regression and classification, before presenting statistical learning methods, decision trees, KNN, clustering, and dimensionality reduction. Finally, it introduces neural networks, backpropagation, optimization algorithms, and modern architectures including CNNs, autoencoders, GANs, and Transformers, with a discussion of frugal AI and environmental impact.

Prerequisites

Calculus, linear algebra, Probability and Statistics, Python programming.

Format

Lectures that introduce the theory of Machine Learning, and practical algorithmic examples.

Assessment

  • Practical sessions on regression, classification, and neural network concepts.
  • Mini projects exploring model architectures and evaluation metrics.

Topics covered

  • Supervised, unsupervised, and reinforcement learning
  • Tasks: classification, regression, and generation
  • Architectures: MLP, CNN, Transformers, and autoencoders
  • Probability, statistics, algebraic notation, and matrix operations
  • Linear regression, logistic regression, and classification fundamentals
  • KNN, clustering, decision trees, random forests, and ensemble methods
  • Optimization, loss functions, generalization, overfitting, and bias/variance tradeoff
  • Parametric vs nonparametric models and dimensionality reduction (PCA)
  • Neural networks: perceptron, MLP, backpropagation, SGD, and Adam
  • Modern architectures: CNNs, GANs, Transformers, ViT, LLMs, and LVMs
  • Environmental impact and frugal AI: pruning, quantization, and distillation

References

  • Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, 2016.
  • Simon Prince, Understanding Deep Learning, MIT Press, 2023.
  • Chloé-Agathe Azencott, Introduction au Machine Learning, DUNOD, 2025.

Link: Link to the course (Moodle key available upon request)

Formal Languages, Complexity, and Computability

A course introducing the theoretical foundations of computer science through languages, computability, and complexity.

Synopsis

The course introduces the theoretical foundations of computer science through three complementary axes: formal languages and recognition by automata, the limits of computability, and the classification of problems by algorithmic difficulty. Students will learn to define main models of computation, distinguish language classes, identify decidable and undecidable problems, and manipulate complexity classes such as P, NP, coNP, and PSPACE.

Prerequisites

Basic computer science concepts, discrete mathematics, and programming fundamentals.

Format

Lectures on theory, models, and proofs, with examples of automata and algorithms.

Assessment

  • Problem sets on automata, decidability, and complexity.
  • Implementation of automata and decision algorithms.

Topics covered

  • Finite automata, regular expressions, pushdown automata, context-free grammars, and pumping lemmas
  • Turing machines, Church-Turing thesis, decidability, halting problem, reducibility, and recursion theorem
  • Complexity measures in time and space, classes P, NP, L, NL, PSPACE, BPP, and IP
  • Complete problems, P vs NP conjecture, quantifiers and games, hierarchy theorems, relativized computation, and interactive proof systems

References

  • Christos Papadimitriou, Computational Complexity, Addison-Wesley, 1994.
  • John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman, Introduction to Automata Theory, Languages and Computation, Pearson, 3rd edition, 2006.
  • Michael Sipser, Introduction to the Theory of Computation, Cengage Learning, 3rd edition, 2013.

Link: Link to the course (Moodle key available upon request)