We study the dynamics of a two-mode Bose-Einstein condensate in the vicinity of a mean-field dynamical instability. Convergence to mean-field theory (MFT), with increasing total number of particles N , is shown to be logarithmically slow. Using a density matrix formalism rather than the conventional wavefunction methods, we derive an improved set of equations of motion for the mean-field plus the fluctuations, which goes beyond MFT and provides accurate predictions for the leading quantum corrections and the quantum break time. We show that the leading quantum corrections appear as decoherence of the reduced single-particle quantum state; we also compare this phenomenon to the effects of thermal noise. Using the rapid dephasing near an instability, we propose a method for the direct measurement of scattering lengths.