This paper introduces an improvement of the “Shake-The-Box (STB)” (Schanz, Gesemann, and Schröder, Exp. Fluids 57.5, 2016) technique using the polynomial calibration model and the line-of-sight constraints (LOSC) to overcome the refractive interface issues in Lagrangian particle tracking (LPT) measurement. The method (named LOSC-LPT) draws inspiration from the two-plane polynomial camera calibration in tomographic particle image velocimetry (Tomo-PIV) (Worth and Nickels, Thesis, 2010) and the STB-based open-source Lagrangian particle tracking (OpenLPT) framework (Tan, Salibindla, Masuk, and Ni, Exp. Fluids 61.2, 2019). The LOSC-LPT introduces polynomial mapping functions into STB calibration in conditions involving gas–solid–liquid interfaces at container walls exhibiting large refractive index variations, which facilitates the realization of particle stereo matching, three-dimensional (3D) triangulation, iterative particle reconstruction, and further refinement of 3D particle position by shaking the LOS. Performance evaluation based on synthetic noise-free images with a particle image density of 0.05 particle per pixel (ppp) in the presence of refractive interfaces demonstrates that LOSC-LPT can detect a higher number of particles and exhibits lower position uncertainty in the reconstructed particles, resulting in higher accuracy and robustness than that achieved with OpenLPT. In the application to an elliptical jet flow in an octagonal tank with refractive interfaces, the use of polynomial mapping results in smaller errors (mean calibration error < 0.1 px) and thus more long trajectories identified by LOSC-LPT (13,000) compared with OpenLPT (4,500) which uses the pinhole Tsai model (mean calibration error > 1.0 px). Moreover, 3D flow-field reconstructions demonstrate that the LOSC-LPT framework can recover a more accurate 3D Eulerian flow field and capture more complete coherent structures in the flow, and thus holds great potential for widespread application in 3D experimental fluid measurements.