Microstructure evolution of three Al-Ag alloys with different Ag contents (1 wt. % Ag, 5 wt. % Ag, and 15 wt. % Ag) was studied by positron annihilation spectroscopy during the aging process. In situ measurements of the positron lifetime and Doppler broadening of annihilation radiation indicate the fast formation of Ag-rich clusters during natural aging of the alloys. The formation of Ag-rich clusters was further confirmed by coincidence Doppler broadening measurements. The Ag signal reflected by the Coincidence Doppler broadening spectrum increases with increasing Ag content and is further enhanced after subsequent artificial aging at 140 °C. This might be due to the increase in the size of Ag clusters. The temperature dependence of the Doppler broadening spectra between 10 K and 290 K was measured for the Al-Ag alloys after natural and artificial aging. Detrapping of positrons from Ag clusters with increasing temperature was observed for all the three Al-Ag alloys after natural aging and for the Al-1 wt. % Ag after artificial aging. This indicates that Ag clusters act as shallow positron trapping centers. The thermal detrapping of positrons becomes ambiguous with increasing Ag content in the alloy and is nearly invisible in the artificially aged Al-5 wt. % Ag and Al-15 wt. % Ag. The positron binding energy of the Ag cluster is roughly estimated to be about 18.8 meV and 50 meV in the Al-1 wt. % Ag sample after natural aging and artificial aging at 140 °C, respectively, which suggests that the confinement of positrons in the quantum-dot like state depends on the size or chemical composition of clusters. Theoretical calculations confirm positron trapping by Ag nanoclusters, and the confinement of positrons is enhanced with increasing Ag cluster size.