I have used all three of these samplers in real research, often without much thought about why I chose one over another. emcee was my first sampler, NUTS arrived when I moved my likelihoods to JAX, and dynesty came in whenever someone asked for a Bayes factor. Eventually I realised I could not say which one I would trust on a problem I did not already understand.
Most sampler comparisons measure speed, but speed means little if the samples are wrong, and you can only check wrongness when you know the right answer. So I chose three problems small enough to solve exactly, on a dense grid or analytically, and scored each sampler against the truth rather than against the others.
NUTS was the fastest, by a factor of two to four on smooth problems, but it failed completely on the multimodal problem, and its diagnostics said so clearly. emcee was fine on the easy problems but gave confidently wrong answers on the multimodal one, with no warning in its default output. dynesty was the only sampler that found every mode and produced a correct evidence, though its evidence error bars were optimistic.
The three test problems grew progressively harder. The first was a control: fitting Ωm and H0 in flat ΛCDM to 40 simulated Type Ia supernovae, giving a roughly elliptical posterior. The second was a curved, thin ridge, fitting Ωm and the dark energy equation of state with H0 fixed. The third was a multimodal problem: finding the frequency of a sinusoid observed on 40 nights over 80 days, which produces several separated peaks.
Source: Towards Data Science · Summarized by HeadlinesBriefing