Hi Ken,
Is the suggestion really that upper limits should depend on a prior for
the quantity being limited?
They definitely do depend on the type of prior (like all things Bayesian) and the range of the prior. The uniform prior is usually chosen so that one is at least shielded from the range dependence. And like all things Bayesian the dependence on the prior matters less and less the larger the S/N ratio. Which is, of course, not very useful when one is setting upper limits.
The range dependence is pretty easy to understand. If one is basically sampling from the prior below some threshold amplitude and you move the lower bound of the prior lower and lower, then the 95% interval will move lower and lower.
This seems rather scary, and also not what I would want.
I totally agree with this statement. And this is what sent me down the rabbit hole of studying all of the prior dependencies of standard PTA analyses. I sort of came to the personal conclusion that Bayesian ULs are misleading, at least when one is used to frequentist ones, but basically the best we've got.
Suppose I come up with some particle physics theory in
which there should be strings with some G mu. I wonder whether this
possibility is already ruled out. I would like to look at a paper and
see whether observations are strongly at odds with this G mu. This
shouldn't depend on what the authors of that paper used as a prior for G
mu. I don't care anything about other possible G mu; I just want to
find out what observations have to say about my suggestion.
I'm not sure one can use a single upper limit from an analysis to do this. See below. In this case you could look at a log-uniform posterior and say something like the probability of the amplitude being between x & y is P, but that statement would have a LOT of caveats, and might not hold up with more informative datasets.
(That said, a properly constructed sensitivity curve, that included the GWB could be useful for this question, but comes from the noise analysis and doesn't include an actual detection analysis. )
I don't mind the upper limit on G mu depending on the prior for other
quantities, such as A_GWB. That seems to make sense: If the data is
well explained by a GWB, then introducing strings will make the fit
worse. But if the requisite GWB amplitude is disfavored by a prior, then
strings look better. On the other hand, one would hope that there is
not a very large dependence on this prior.
Yeah, as I mentioned, I wouldn't include the GWB (Common process) unless it was very significant, in which case the prior shouldn't matter very much.
I find any use of uniform priors pretty scary. Suppose I put a uniform
prior on G mu from 0 to 10^-6. That means that 99.9% of the time, G mu >
10^-9. Such signals are clearly incompatible with observations. So,
using this prior, I would conclude that cosmic strings are ruled out at
the 99.9% level.
Is this how you would read the upper limit though? Using your example, the range is just the starting point. Say you have a very uninformative data set, and the 95% upper limit comes out as 0.95e-6 (It can't be 10^-6 in your example, since the analysis returning the prior will by definition mean the 95% UL is below the upper end of the prior.) In this case I would have said that cosmic strings with G mu > 0.95e-6 have been ruled out at the 95% level, but even that statement is pretty misleading when compared to the standard frequentist statement.
Frequentist: In 95% of universes, and the model used here, the amplitude is below "this threshold".
Bayesian: Given our chosen set of priors and prior ranges, and the model used here, our single realization of the data gives a 95% chance of the amplitude being below "this threshold".
Here is a reference from our paper that I found really enlightening about these topics.
This is the abstract
Here is a more (particle) physics applicable article.
I think that one has to make some "best practice" choices and explain to readers all of the caveats.
Cheers,
Jeff