We perform a multivariate analysis of Higgs-pair production via the decay channel HH → bbγγ at the future 100 TeV pp collider to determine the trilinear Higgs self-coupling (THSC) λ 3H , which takes the value of 1 in the standard model. We consider all known background processes. For the signal we adopt the most recent event generator of POWHEG-BOX-V2 to exploit the NLO distributions for toolkit for multivariate data analysis (TMVA). Through the technique of boosted decision tree (BDT) analysis trained for λ 3H ¼ 1, compared to the conventional cut-and-count approach, the signal-to-background ratio improves tremendously from about 1=10 to 1 and the significance can reach up to 20.5 with a luminosity of 3 ab −1 without including systematic uncertainties. In addition, by implementing a likelihood fitting of the signal-plus-background M γγbb distribution with optimized bin sizes, it is possible to determine the THSC with the precision of 7.5% at 68% CL even at the early stage of 100 TeV hadron collider with 3 ab −1 .