Pleiotropic fitness trade-offs and their opposite, buttressing pleiotropy, underlie many important phenomena in ecology and evolution. Yet, predicting whether a population adapting to one ("home") environment will concomitantly gain or lose fitness in another ("non-home") environment remains challenging, especially when adaptive mutations have diverse pleiotropic effects. Here, we address this problem using the concept of the joint distribution of fitness effects (JDFE), a local measurable property of the fitness landscape. We derive simple statistics of the JDFE that predict the expected slope, variance and covariance of non-home fitness trajectories. We estimate these statistics from published data from the Escherichia coli knock-out collection in the presence of antibiotics. We find that, for some drug pairs, the average trend towards collateral sensitivity may be masked by large uncertainty, even in the absence of epistasis. We provide simple theoretically grounded guidelines for designing robust sequential drug protocols.