Inspired by the recent developments in the fields of quantum distributed computing, quantum systems are analyzed as networks of quantum nodes to reduce the complexity of the analysis. This gives rise to the distributed quantum consensus algorithms. Focus of this paper is on optimizing the convergence rate of the continuous time quantum consensus algorithm over a quantum network with N qudits. It is shown that the optimal convergence rate is independent of the value of d in qudits. First by classifying the induced graphs as the Schreier graphs, they are categorized in terms of the partitions of integer N . Then establishing the intertwining relation between one level dominant partitions in the Hasse Diagram of integer N , it is proved that the spectrum of the induced graph corresponding to the dominant partition is included in that of the less dominant partition. Based on this result, the proof of the Aldous' conjecture is extended to all possible induced graphs and the original optimization problem is reduced to optimizing spectral gap of the smallest induced graph. By providing the analytical solution to semidefinite programming formulation of the obtained problem, closed-form expressions for the optimal results are provided for a wide range of topologies.