Shuhei Watanabe (渡邊修平)

I have worked on optimization both in academia and industry. Although my publications strongly lean toward Bayesian optimization, classical optimization knowledge, such as (quasi-) Newton method, and (mixed-integer) linear programming, is also essential for the acquisition function optimization.

I was also intensively working on real-world applications of Bayesian optimization, including the extensions of TPE to multi-objective optimization, constrained optimization, and meta-learning setups, and materials discovery and Sim2Real transfer applications. Importantly, these applications require significant engineering efforts and dirty work to enhance the practical performance through trial and error. The dirty work necessitates convenient tools for analysis, and one of the tools inspired during these works is PED-ANOVA accepted to IJCAI'23.

I have a strong interest in statistical pattern recognition and measure theory. Statistical pattern recognition is essential for Gaussian process-based Bayesian optimization algorithms. I developed a notebook for statistical pattern recognition as self-study and consolidated knowledge about the multi-task Gaussian process in an arXiv paper. Regarding measure theory, I read through a textbook to acquire graduate-level knowledge, which led to the meta-learning TPE paper accepted to IJCAI'23. This paper required a robust understanding of measure theory, such as Schaffe's Lemma and the continuous mapping theorem to prove the almost sure convergence of the total variation distance. I also summarized a compact notebook for the knowledge required to write this paper.

I am familiar with both the parallel computation usage, especially on clusters, and job scheduling for optimization. The former includes large-scale experiments using thousands of communicating CPUs, and experiment data management using a database. Works for the latter are published at ACPR'19 and AutoML'24, which is a short version of my master thesis. My master thesis is a work on a general job scheduling wrapper for parallel black-box optimization. Since this wrapper needs to be able to handle various parallel processing methods, such as multi-processing, multi-threading, and file (or server)-based synchronization (and of course, I need to understand all of them), the implementation correctness must be guaranteed carefully, where mathematics and unit tests played a pivotal role. Furthermore, the wrapper was verified using 52 benchmark problems and 8 optimization methods over 30 different random seeds with 4 different parallelisms, overall 49920 setups. The wrapper successfully finished the whole experiment without any errors in 43M CPU seconds, which would have taken 58B CPU seconds naively.

Although my OSS activities are not limited to Optuna, my significant works are visible in Optuna:

• Development of GPSampler and its extensions (multi-objective, constrained, robust optimizations, and 60x speedup),
• Speedup of TPESampler (300x speedup),
• Enhance the robustness of the Optuna backend,
• c-TPE implementation available in OptunaHub (the paper is accepted to IJCAI'23),
• PED-ANOVA implementation available in Optuna, which is now the default algorithm in Optuna Dashboard (the paper is accepted to IJCAI'23),
• Multi-mbjective TPE implementation available in Optuna (the paper is accepted to JAIR),
• Warm-starting CMA-ES implementation available in Optuna (the paper is accepted to AAAI'21), and
• Community activity to bridge AutoML research groups and the Optuna team.

I defer the technical details to individual pages, but the development involves a deep understanding of numerical computations such as numerical optimization, numerical stability, discrete algorithms, and Monte-Carlo integrations.

I am personally interested in Game & Physics-based simulations during my Bachelor. The first example shows the MineSweeper solver in C++ using constraint satisfaction, depth-first search, and maximum likelihood estimation. The second example shows the simulation of the time evolving temperature on 2D plane. Besides these simulators, I also worked on an agent to play Puyo Puyo, and a large-scale simulation using the lattice Boltzmann method. The lattice Boltzmann implementation was tested on a cluster using 2,000 CPUs. I reported the scaling test results here.