Multi-Task Gaussian Process Implementation Details: What paper should I check?

[nabenabe0928]

Hi, I am now working on Bayesian optimization using multi-task Gaussian process (MTGP).
Basically, I am wondering if the citation for MultiTaskGP is correct.

BoTorch says here that it uses MTGP regression from this paper.

I read through this paper and I think this paper discusses the MTGP for the Kronecker structure, meaning that every data point must be evaluated at every task.
In principle, this paper claims that they first parameterized the marginal log-likelihood with the Cholesky decomposed task correlation $L$ and directly optimized the matrix elements via the maximization of the marginal log-likelihood.
BoTorch is using IndexKernel according to these lines and I am not sure if it does the same because this module is only taking the task indices if I understand correctly.

So the questions are:

1. Does IndexKernel approximate the task correlation in the same fashion as the linked paper does?
2. How is the merge of kernel matrices in an LCM fashion work? Are there any papers on that?

I highly appreciate any feedback:)

[Balandat]

I read through this paper and I think this paper discusses the MTGP for the Kronecker structure, meaning that every data point must be evaluated at every task.

That is indeed the case.

BoTorch is using IndexKernel according to these lines and I am not sure if it does the same because this module is only taking the task indices if I understand correctly.

Those lines just define the index kernel (aka the cross-task covariance aka the "coregionalization matrix"), the actual evaluation and generation of the full covariance matrix happens here" https://github.com/pytorch/botorch/blob/7cfff9c3ce6a83eb976f744a2fcc736b4e7f91e5/botorch/models/multitask.py#L308-L312

This medium post explains the basic ICM setup: https://medium.com/@ruglioni07/multi-task-gaussian-processes-04d3ea2878fb. The model is in fact the same whether or not there is a "block design" of evaluating every task at every input location - so the answer to your first question is "yes" - it just so happens that in case of a block design the resulting covariance can be expressed as a Kronecker product, which affords some computational benefits (for some shameless self-promotion plug, take a look at this paper that discusses how some of these benefits can be realized even if the full covariance matrix is just some submatrix of a full Kronecker product: https://arxiv.org/abs/2410.09239)

How is the merge of kernel matrices in an LCM fashion work? Are there any papers on that?

This is simply a Hadamard product of two covariance matrices that are already of the full (across tasks and data point) size. Basically, covar_i = self.task_covar_module(task_idcs) takes in the task_idcs - which is a tensor of indices, one for each data point - and generates the full cross-task covariance matrix, which is then multiplied with the elements of the data covariance matrix element-wise.

[nabenabe0928]

Thank you so much for the detailed response 🙏

I ran MultiTaskGP with some debugging messages and confirmed the LCM behavior!

Plus, I also confirmed the task kernel was learned appropriately depending on functions provided (e.g., sin x and sin x --> strong correlation, sin x and cos x --> much weaker correlation)

Also thank you for the guidance to your paper about the benefit of the Kronecker structure!

Although I haven't read your paper, I confirmed that we can calculate the gradient w.r.t. hyperparameters, the cholesky matrix of the task kernel, and the variance of prior.

I will check out your paper to know more about the benefit!