When you run a choice-based conjoint or MaxDiff study, the software estimates how much each respondent values each attribute level, usually with Hierarchical Bayes (HB).
These models have hundreds or thousands of unknowns, so there is no single formula to solve. They are estimated with Markov chain Monte Carlo (MCMC): an algorithm that explores the space of plausible values step by step and builds up a picture of each one.
The catch: an MCMC sampler always produces numbers when it finishes, even if it never found its way to the right answer.
Many conjoint tools don't show you whether the sampler worked. Ordeen shows three diagnostics on every result (R-hat, effective sample size and divergences) and stops any model whose R-hat is above 1.1. Here is what they mean.
1. R-hat: did independent explorers draw the same map?
Imagine sending four walkers (in MCMC, chains) to map a mountain range in fog, each starting from a different random spot.
If the model converged, all four find the same peaks and map the same ground. Their maps look almost identical.
If it did not, one walker is stuck on a hill in the west while another wanders a valley in the east. Their maps disagree.
R-hat (the potential scale reduction factor; Gelman & Rubin, 1992) compares the spread between the chains with the spread within each chain:
$$\hat{R} \approx \sqrt{\frac{\text{between-chain variance} + \text{within-chain variance}}{\text{within-chain variance}}}$$
Close to 1.00: the chains agree. Current guidance recommends below 1.01 for final results (Vehtari et al., 2021).
Above 1.1: the chains clearly disagree, and the results depend on where they happened to start.
A sampler that runs a single chain can't calculate this directly; an analyst has to rerun the estimation with different random seeds and compare.
2. Effective sample size: how many independent draws do you really have?
Each step of a Markov chain starts from the one before, so neighbouring draws are correlated. Effective sample size (ESS) says how many fully independent draws your correlated chain is worth.
In Ordeen, a CBC study runs 4 chains of 3,000 iterations: 1,000 warm-up steps that are discarded, then 2,000 kept per chain, so 8,000 kept draws in total. If the smallest ESS across parameters is 1,200, that parameter's estimate is about as precise as 1,200 independent draws. If ESS is very low (under 100, say), intervals on share and willingness to pay will be noisy.
3. Divergences: did the sampler fall off a cliff?
Ordeen uses the No-U-Turn Sampler (a form of Hamiltonian Monte Carlo), which uses gradients to move efficiently through the parameter space. Where that space has very sharp curvature, often because of sparse or contradictory data, the simulation can go off course. Each time that happens, the sampler records a divergence. Zero, or very few, divergences mean it explored the space cleanly.
Why Ordeen withholds a model when R-hat is above 1.1
In pricing research, a confident-looking chart from a model that did not converge is worse than no chart, because a team may set a real price from it. So Ordeen applies two automatic checks:
The convergence check. If the largest R-hat across parameters is above 1.1, the fit is marked as failed and its results are not shown.
The willingness-to-pay check. Willingness to pay divides by the price effect. If the price effect changes sign across posterior draws (the model can't confirm that a higher price reduces demand), the WTP figure is flagged and not shown, instead of dividing by numbers close to zero.
Related reading: Methodology · What is Hierarchical Bayes? · The Krinsky-Robb method
Sources
Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472.
Hoffman, M. D., & Gelman, A. (2014). The No-U-Turn Sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593–1623.
Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: an improved R-hat for assessing convergence of MCMC. Bayesian Analysis, 16(2), 667–718.
