Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarification of randomness in estimation of BFs for point hypothesis tests #1750

Open
tfjaeger opened this issue Mar 6, 2025 · 0 comments

Comments

@tfjaeger
Copy link

tfjaeger commented Mar 6, 2025

The hypothesis function takes an argument "seed", and its documentation clarifies that this is "A single numeric value passed to [set.seed] to make results reproducible." Many users of brms probably do not appreciate what aspects of the hypothesis test introduce randomness, and/or will be confused if the BF (and p_posterior) vary between repeated identical calls to the hypothesis function.

As far as I can tell, only the sampling of the prior samples---which only happens for point hypotheses---introduces randomness into the hypothesis test (because priors for different parameters are assumed to be independent), and this can only affect results if the hypothesis contains at least two parameters? It might be worth clarifying this in the documentation (or to at least state that results are only expected to be non-deterministic if point hypotheses are tested/the Savage-Dickey method is used)?

As an aside, I was wondering why the density_ratio function evaluates the density at a fixed number of points along the sampled parameters (n = 4096). This would seem to cause very unstable density_ratio estimates over the point estimate when the prior and/or posterior is very flat, but perhaps that is intended?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant