This paper presents a Bayesian model updating methodology for dynamical systems with geometric nonlinearities based on their nonlinear normal modes (NNMs) extracted from broadband vibration data. Model parameters are calibrated by minimizing selected metrics between identified and model-predicted NNMs. In the first approach, a deterministic formulation is adopted, and parameters are updated by minimizing a nonlinear least-squares objective function. A probabilistic approach based on Bayesian inference is next investigated, where a Transitional Markov Chain Monte Carlo is implemented to sample the joint posterior probability distribution of the nonlinear model parameters. Bayesian model calibration has the advantage to quantify parameter uncertainty and to provide an estimation of model evidence for model class selection. The two formulations are evaluated when applied to a numerical cantilever beam with geometrical nonlinearity. The NNMs of the beam are derived from simulated broadband data through nonlinear subspace identification and numerical continuation. Accuracy of model updating results is studied with respect to the level of measurement noise, the number of available datasets, and modeling errors.