[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [IPTA-cs] Bayesian upper limits



Hi All,
I largely agree with you all here and I wanted to emphasize a few points. 

Our paper showed that using log-uniform priors on the individual red noise models was less biased, but in fact the least biased option is to only include red noise for pulsars where there is a significant detection of that noise. I think that is a bit far from the standard PTA analysis,  so not necessary. But I think it's important to keep in mind when making these decisions. 

In some sense, the SMBBH GWB is "noise" in the string search since it's not the signal you're searching for. I'd definitely keep the priors log-uniform on the detected signal (whether you're saying it's a common process or a GWB.) This point is largely moot since the posteriors should converge to be very similar, no matter the prior, for a large detection.

The weighting trick Siyuan was referring to is really useful. I used it in the paper a lot. However,  you can only use it when you want to change the prior on only one parameter, since the covariances can change things across multiple parameters (unless they really are independent.)  Also, as Siyuan said, you need enough samples across the prior range. This effectively means it's usually possible to go from log-uniform to uniform, but not the other way around.

As for non-amplitude priors, I'm not sure there is the same physical motivation to look at uniform priors. Unless the larger parameter always make the amplitude larger. Really ULs are for setting limits on non-detections, and so innately concern the amplitude parameters.

Lastly, one uses the uniform prior for ULs not necessarily because it is more conservative, but rather because the choice of the lower limit of the prior range doesn't effect the UL as strongly. It can for the log-uniform prior.

I think Ken is definitely correct though. These choices should be made carefully because they may become the norm for all SMBBH + other signal searches. 

Sorry I can't attend these meetings, but let me know if there any specific things I can help with!

Cheers,
Jeff

On Fri, Sep 2, 2022, 9:07 AM Chen Siyuan via IPTA-cosmic-strings <ipta-cosmic-strings cosmos phy tufts edu> wrote:

Hi all,


yes, with Bayesian sampling you get credible regions that are based on your initial beliefs (priors), the 95% upper limit is more correctly described as the upper bound of the 90% credible region (from the median of the posterior distribution). All the other parameters are marginalized, in the sense that you look at the integrated posterior distribution from the chain.

In the past upper limit analyses in PTAs have used uniform priors on the GWB amplitude and importantly also the pulsar red noise amplitudes. The GWB prior was set so that your 90% credible region would be relatively independent from the choice of the lower amplitude bound as long as it is low enough to be considered close to 0. From the https://arxiv.org/abs/2009.05143 paper, the main result as Kai says is a caution on the choice of priors. The reason in the past to use uniform priors on the pulsar red noise is to assume that as much power is in the pulsar compared to the GWB. As shown in the paper above, this leads to potentially lower upper bound of the 90% credible region.

In my view, a more conservative upper limit should use log uniform prior for the pulsar red noise amplitudes. The question is what prior to use for the GWB amplitude and whether there is a way to convert between the uniform and log uniform prior choice. I think there is in principle, since you know the prior distributions, the uniform is the 1/x weighted distribution of the log uniform. If each GWB amplitude posterior sample from the log uniform analysis is weighted by 1/sample, that should take exactly into account that the uniform prior suggests a low amplitude GWB x times less likely than the log uniform prior. You can also go from the uniform to the log uniform prior choice. (I think the main requirement is that you have enough samples at all amplitudes.)

I have sent Andrea a little code snippet, so he can compute upper limits from the log uniform runs he already has. It would be good to have a confirmation by running the same analysis with a uniform prior.

For the cosmic strings, I think, for the stable strings with only Gmu as the parameters, we should be able to get a upper limit in the strings+SMBHB analysis with this method. It should be somewhat comparable to previous PTA upper limits.

For the metastable and super-strings, I am not fully sure what to do. If I am not wrong, the kappa parameter is also a log scale parameter, it can also be arbitrary extended to lower values. So, a upper limit will be depending on the prior choice of the other parameter (Gmu). I think we can show the log-log uniform prior Bayesian sampling and compute a 95% highest posterior density interval/contour. We could compute 4 'upper limits' on the 2 individual parameters. Each parameter upper limit can be computed from 1) reweighting only its distribution and 2) reweighting both distributions. This gives 2 x 2 upper limits. You can extend this to the superstrings for more different upper limits. We could then just quote the highest upper limit.


Let me know, what you think.


Cheers

Siyuan



-----Original Messages-----
From:"Kai Schmitz via IPTA-cosmic-strings" <ipta-cosmic-strings cosmos phy tufts edu>
Sent Time:2022-09-02 17:42:42 (Friday)
To: ipta-cosmic-strings cosmos phy tufts edu
Cc:
Subject: Re: [IPTA-cs] Bayesian upper limits

Hi Ken / All,

(Andrea, Siyuan, see my comments below.)

Thanks for your message. My impression is that (please correct me if I am wrong) many of the points you are raising boil down to the fact that we will eventually be constructing Bayesian credible intervals rather than frequentist confidence intervals.

Just quoting from Wikipedia [ https://en.wikipedia.org/wiki/Credible_interval ] (sorry, I don't have a better reference at hand at the moment), Bayesian credible intervals are indeed (a) prior-dependent and (b) not unique. In particular, the underlying philosophy differs from the frequentist situation, which is why we must be careful using formulations such as "this and that happens 95% of the time", which sounds like as if we were able to do repeated measurements as in a frequentist setting.

So what we can do in the Bayesian case is to state a possible (but not unique) 95% credible interval, which reflects our subjective belief that the true value of the model parameter (Gmu) is contained inside this interval with a probability of 95%. But that does not mean that values of the model parameter outside the 95% credible interval are necessarily highly disfavored. Some of these values may be contained in our 95% credible interval after all if only we decide to choose a different approach to construct the interval.

Wikipedia discusses several approaches to constructing Bayesian credible intervals (highest posterior density intervals, equal-tailed intervals, etc.) and states that, from a decision theory perspective, the highest posterior density intervals represent some form of optimal choice.

For us this could mean that we should proceed as follows: Consider a model of cosmic strings (stable, metastable, superstrings) plus SMBHBs and construct the PDF in the cosmic-string model parameter space (Gmu, sqrt(kappa), P, etc.) after marginalizing over all other SMBHBs and noise parameters. Then, select those point in the cosmic-string parameter space with the largest PDF values until the total probability in the selected region of parameter space integrates to 95%. That should yield highest posterior density regions that we are interested in.

Apart from this, it is not clear to me how we should choose our priors when we switch from a "detection analysis" (e.g., a cosmic strings only run, without any SMBHBs on top of the cosmic-strings signal) to an "upper-limits analysis". @Andrea, You were saying that we may have to change our priors from log-uniform to uniform in this case, which reminds me of this paper here https://arxiv.org/abs/2009.05143 and the discussion on why old PTA limits on A_GW were actually stronger than the best-fit values of the amplitude that we are now seeing in the data. The take-away from 2009.05143 is that uniform priors can sometimes be dangerous and lead to too aggressive upper bounds, which is also what has been discussed in the NG12.5 paper [ https://arxiv.org/abs/2009.04496 ]. @Siyuan, If I remember correctly you were also commenting on this issue in the chat yesterday. Could you say again what thoughts you had on this and how we should best choose the priors in the "upper-limits analyses"?

@all, Let me know what you think.

Best regards, Kai.

On 9/1/22 18:32, Ken Olum via IPTA-cosmic-strings wrote:
Hi, all.  There was some discussion on the call about how to set upper
limits.  Is there are some literature about this or some standard
techniques?  

I thought about these ideas once, something like this:

Suppose there is a GWB from SMBHB with unknown gamma and A, given by
some priors.  In addition there may be a cosmic string background with
some particular G mu.  The question is whether our observations are
consistent with being a realization of these backgrounds plus noise.  If
G mu is too large, we would expect a larger signal than we see.  If it
is much too large, then 95% of the time we would see a larger signalman
we do, and that is the limit we're looking for.

This depends somewhat on the priors for the SMBHB signal and the noises.
A large string signal is more consistent with observation if it is not
added to some other sources of signals and noise.  But I don't think you
need a prior for G mu here at all, because the question is about
specific values of G mu.

One might ask a different question instead: suppose we have SMBHB and
also strings and we make a PDF of the parameters.  Then we marginalize
over the SMBHB parameters, make a PDF of the G mu and find the 95%
point.  This does depend on the G mu prior.  For example, if we were
extremely confident of some large G mu beforehand, it might still have >
5% probability after unfavorable observations.

So I guess there is really a cultural question.  We would like to say
"these observations rule out G mu > ... at the 95% level".  We would
like to mean the same thing that others mean when they make similar
statements.  What is that?

Since we're not going to detect any strings in the present data, this
limit is probably the most important takeaway from our paper.  So we
should be careful about our choices of what to calculate and how to
explain what we calculated.

                                        Ken


--
IPTA-cosmic-strings mailing list
IPTA-cosmic-strings cosmos phy tufts edu
https://cosmos.phy.tufts.edu/mailman/listinfo/ipta-cosmic-strings