We present deep James Webb Space Telescope (JWST)/Mid-Infrared Instrument (MIRI) F560W observations of a flux-limited, Atacama Large Millimeter/submillimeter Array (ALMA)-selected sample of 28 galaxies at z = 0.5–3.7 in the Hubble Ultra Deep Field (HUDF). The data from the MIRI Deep Imaging Survey (MIDIS) reveal the stellar structure of the HUDF galaxies at rest-frame wavelengths of λ > 1 μm for the first time. We revise the stellar mass estimates using new JWST photometry and find good agreement with pre-JWST analyses; the few discrepancies can be explained by blending issues in the earlier lower-resolution Spitzer data. At z ∼ 2.5, the resolved rest-frame near-infrared (1.6 μm) structure of the galaxies is significantly more smooth and centrally concentrated than seen by the Hubble Space Telescope at rest-frame 450 nm (F160W), with effective radii of R
e
(F560W) = 1–5 kpc and Sérsic indices mostly close to an exponential (disk-like) profile (n ≈ 1), up to n ≈ 5 (excluding active galactic nuclei). We find an average size ratio of R
e
(F560W)/R
e
(F160W) ≈ 0.7 that decreases with stellar mass. The stellar structure of the ALMA-selected galaxies is indistinguishable from a HUDF reference sample of all galaxies with a MIRI flux density greater than 1 μJy. We supplement our analysis with custom-made, position-dependent, empirical point-spread function models for the F560W observations. The results imply that a smoother stellar structure is in place in massive gas-rich, star-forming galaxies at “Cosmic Noon,” despite a more clumpy rest-frame optical appearance, placing additional constraints on galaxy formation simulations. As a next step, matched-resolution, resolved ALMA observations will be crucial to further link the mass- and light-weighted galaxy structures to the dusty interstellar medium.