There’s a long history of trying to extract experts’ beliefs about probability distributions when there is no data to estimate the distributions directly. The MATCH Uncertainty Elicitation Tool (Morris et al., 2014) offers five methods.
The roulette method is the most intuitive to me. You are provided with a blank histogram, with range and number of cells of your choosing. As you click, MATCH guesses the distribution using a least-squares procedure, choosing between normal, Student’s t, scaled beta, gamma, log normal, and log Student’s t. You can also override its guess. This then means you can look up the quantiles and use those to influence your clicks, e.g., if the median or extreme quantiles are off what you believe, you can add or remove cells to drag the quantiles where you think they should be.
Here’s an example for the range 0 to 10 and with grid height 10 and 20 bins:
There are a couple of methods that ask for quantiles (either quartile or tertile). Another that asks for three probabilities, where you can choose the parameters. The default probabilities requested are \(P(0 < X < 0.25)\), \(P(0.75 < X < 1)\) and \(P(0 < X < 0.5)\), when \(X \in [0,1]\). Finally, there’s a hybrid option which requests the median and two probabilities. This latter option also feels intuitive to work with, particularly again when looking at the fitted distribution and peeking at quantiles.
There’s also the SHELF R Package, which includes a bunch of Shiny apps that are available directly on the web too.