[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [IPTA-cs] Bayesian upper limits

Hi Ken / All,

(Andrea, Siyuan, see my comments below.)

Thanks for your message. My impression is that (please correct me if I am wrong) many of the points you are raising boil down to the fact that we will eventually be constructing Bayesian credible intervals rather than frequentist confidence intervals.

Just quoting from Wikipedia [ https://en.wikipedia.org/wiki/Credible_interval ] (sorry, I don't have a better reference at hand at the moment), Bayesian credible intervals are indeed (a) prior-dependent and (b) not unique. In particular, the underlying philosophy differs from the frequentist situation, which is why we must be careful using formulations such as "this and that happens 95% of the time", which sounds like as if we were able to do repeated measurements as in a frequentist setting.

So what we can do in the Bayesian case is to state a possible (but not unique) 95% credible interval, which reflects our subjective belief that the true value of the model parameter (Gmu) is contained inside this interval with a probability of 95%. But that does not mean that values of the model parameter outside the 95% credible interval are necessarily highly disfavored. Some of these values may be contained in our 95% credible interval after all if only we decide to choose a different approach to construct the interval.

Wikipedia discusses several approaches to constructing Bayesian credible intervals (highest posterior density intervals, equal-tailed intervals, etc.) and states that, from a decision theory perspective, the highest posterior density intervals represent some form of optimal choice.

For us this could mean that we should proceed as follows: Consider a model of cosmic strings (stable, metastable, superstrings) plus SMBHBs and construct the PDF in the cosmic-string model parameter space (Gmu, sqrt(kappa), P, etc.) after marginalizing over all other SMBHBs and noise parameters. Then, select those point in the cosmic-string parameter space with the largest PDF values until the total probability in the selected region of parameter space integrates to 95%. That should yield highest posterior density regions that we are interested in.

Apart from this, it is not clear to me how we should choose our priors when we switch from a "detection analysis" (e.g., a cosmic strings only run, without any SMBHBs on top of the cosmic-strings signal) to an "upper-limits analysis". @Andrea, You were saying that we may have to change our priors from log-uniform to uniform in this case, which reminds me of this paper here https://arxiv.org/abs/2009.05143 and the discussion on why old PTA limits on A_GW were actually stronger than the best-fit values of the amplitude that we are now seeing in the data. The take-away from 2009.05143 is that uniform priors can sometimes be dangerous and lead to too aggressive upper bounds, which is also what has been discussed in the NG12.5 paper [ https://arxiv.org/abs/2009.04496 ]. @Siyuan, If I remember correctly you were also commenting on this issue in the chat yesterday. Could you say again what thoughts you had on this and how we should best choose the priors in the "upper-limits analyses"?

@all, Let me know what you think.

Best regards, Kai.

On 9/1/22 18:32, Ken Olum via IPTA-cosmic-strings wrote:
Hi, all.  There was some discussion on the call about how to set upper
limits.  Is there are some literature about this or some standard
techniques?  

I thought about these ideas once, something like this:

Suppose there is a GWB from SMBHB with unknown gamma and A, given by
some priors.  In addition there may be a cosmic string background with
some particular G mu.  The question is whether our observations are
consistent with being a realization of these backgrounds plus noise.  If
G mu is too large, we would expect a larger signal than we see.  If it
is much too large, then 95% of the time we would see a larger signalman
we do, and that is the limit we're looking for.

This depends somewhat on the priors for the SMBHB signal and the noises.
A large string signal is more consistent with observation if it is not
added to some other sources of signals and noise.  But I don't think you
need a prior for G mu here at all, because the question is about
specific values of G mu.

One might ask a different question instead: suppose we have SMBHB and
also strings and we make a PDF of the parameters.  Then we marginalize
over the SMBHB parameters, make a PDF of the G mu and find the 95%
point.  This does depend on the G mu prior.  For example, if we were
extremely confident of some large G mu beforehand, it might still have >
5% probability after unfavorable observations.

So I guess there is really a cultural question.  We would like to say
"these observations rule out G mu > ... at the 95% level".  We would
like to mean the same thing that others mean when they make similar
statements.  What is that?

Since we're not going to detect any strings in the present data, this
limit is probably the most important takeaway from our paper.  So we
should be careful about our choices of what to calculate and how to
explain what we calculated.

                                        Ken