Outline of the Tutorial | Combined slides

0. Big Picture of Bayesian Optimization (BO) | Slides

  • Motivation from real-world applications
  • Challenges of optimizing expensive black-box functions
  • Overview of Bayesian optimization
  • Three key elements: statistical model, acquisition function (AF), and acquisition function optimizer (AFO)

1. Foundations of BO | Slides

  • Background on Gaussian Processes (GPs)

    • Two views of GPs: weight space and function space
    • Learning and inference with GPs
    • Scalability challenges and solutions

  • Background on Acquisition Functions (AFs)

    • Exploration vs. Exploitation trade-off
    • Example AFs: Expected improvement (EI), Upper confidence bound (UCB), Thompson Sampling (TS), and Information-theoretic methods
    • Optimizers: DIRECT, gradient-based methods, and evolutionary search

2. BO over Combinatorial Spaces | Slides

  • Motivating Applications

    • Manycore systems design
    • Biological sequence design
    • Drug/Vaccine/Molecule design

  • Key Challenges

    • Effective surrogate models for combinatorial structures (e.g., sequences, trees, graphs)
    • Trading-off complexity of statistical model and tractability of AFO

  • BO Methods over Original Space

    • Simple models and tractable AFO
    • Complex models and heuristic search
    • Complex models and tractable search
    • Complex models and effective search via learning-to-search

  • BO Methods over Latent Continuous Space

    • Advantages of reduction to BO over continuous space
    • Deep generative models for learning latent space
    • Challenges of BO over latent space: valid structures, high-dimensionality, imperfect decoder
    • Recent algorithmic advances: weighted re-training, LADDER, and high-dimensional BO

3. BO over Hybrid Spaces | Slides

  • Motivating Applications

    • Material design
    • Microbiome design
    • Auto ML tasks

  • Key Challenges

    • Effective surrogate models for hybrid structures
    • Trading-off complexity of statistical model and tractability of AFO

  • BO Methods over Original Space

    • Simple models and tractable AFO
    • Complex models and heuristic search

4. Multi-fidelity BO | Slides

  • Background on Multi-fidelity optimization

    • Function approximations (fidelities) with varying cost and accuracy of evaluation
    • Discrete fidelity vs. Continuous fidelity
    • Example real-world applications (e.g., Auto ML)

  • Key Challenges

    • Surrogate modeling of multiple fidelity functions and information sharing
    • Selection of input and fidelity pair in each BO iteration

  • BO Algorithms

    • Simple and Information-theoretic methods

5. Constrained BO | Slides

  • Background on Constrained Optimization

    • Valid vs. Invalid inputs via constraints
    • Black-box constraints
    • Example real-world applications (e.g., design of safe and effective vaccines/drugs)

  • Key Challenges

    • Modeling black-box constraints
    • Reasoning with learned model over constraints

  • BO Algorithms

    • Simple and Information-theoretic methods

6. Multi-objective BO | Slides

  • Motivating Applications

    • Hardware design to trade-off power and performance
    • Drug/Vaccine design to trade-off efficacy, safety, and cost

  • Background on Multi-Objective Optimization

    • Optimal Pareto set
    • Optimal Pareto front

  • BO Algorithms

    • Simple methods
    • Information-theoretic methods
    • Differentiable expected hypervolume improvement

  • Multi-Fidelity BO Algorithms

  • Constrained BO Algorithms

7. Outstanding Challenges and Frontiers of BO | Slides