In order to solve the problem of excessive communication overhead in the public-key encryption scheme based on the Learning With Error (LWE) problem, this paper proposes to construct the publickey matrices based on chaotic systems. Specifically, by taking advantages of Logistic mapping, a long pseudo-random sequence is generated from the initial state and the bifurcation parameter. The non-uniform chaotic sequence is further tweaked towards a pseudo-random sequence admitting uniform distributions. The public-key matrix of LWE is therefore constructed efficiently. Simulations confirm the correctness and efficiency of our scheme.