[GENERAL SUPPORT]: How is standard error incorporated into the Gaussian Process?

[claysmyth]

Question

Dear Ax Community,

Thank you for this wonderful tool. I am currently using Ax to optimize field experiments for invasive neuromodulatory procedures. I'm writing to ask how the standard errors are incorporated into the Gaussian Process surrogate model.

I am interested in incorporating variable observation noise around my measurements. My understanding of how this heteroskedastic noise is incorporated comes from Garnett 2023 - Bayesian Optimization:

Where:

I noticed that when calling ax_client.complete_trial() the user can input a standard error around each observation. I want to confirm that the SEM around each observation corresponds the diagonal elements of N matrix in the second screenshot above?

Additionally, how is the baseline noise around measurements typically inferred? I want to use the noise around measurements as a means of implementing 'confidence' in a specific measurement, and would love guidance on choosing the right scale for including this metadata.

Thank you for your time!

Best,
Clay Smyth

Please provide any relevant code snippet if applicable.

Code of Conduct

• I agree to follow this Ax's Code of Conduct

[Balandat]

I want to confirm that the SEM around each observation corresponds the diagonal elements of N matrix in the second screenshot above?

Up to a square, yes - SEM denotes the standard error of the measurement*, but N contains the variances.

*SEM is really a misnomer here, it should be standard error. The reason it's called SEM is for historical in that Ax use cases initially focused on parameter tuning for online experiments, where the observed outcome is the mean of some population (and the associated standard error is the standard error of that mean, hence sem).

This asterisk above also partially answers your other question - in some applications we actually do observe (at least an estimate) of the variance of the observation. In other cases you may want to run some metastudy and evaluate some measurements repeatedly to estimate the variance from those repeated observations (this will be most scalable if your noise is homoskedastic).

I want to use the noise around measurements as a means of implementing 'confidence' in a specific measurement, and would love guidance on choosing the right scale for including this metadata

Do you have the variance of these measurements / can you estimate it? Or is this more about trying to somehow incorporate some domain knowledge / user prior without having a well-defined notion of measurement noise? This paper is quite relevant to your question, I recommend you take a look at it and the references therein to get an idea of what people have done in these cases: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7124771cf8c9b0f6e9c7fec0a66c5866-Abstract-Conference.html