In-memory computing is a promising non-von Neumann approach to perform certain computational tasks efficiently within memory devices by exploiting their physical attributes. However, the computational accuracy achieved with this approach has been rather low, owing to significant inter-device variability and inhomogeneity across an array as well as intra-device variability and randomness from the analog memory devices. Bit slicing, a technique for constructing a high precision processor from several modules of lower precision, is a promising approach for overcoming this accuracy limitation. However, a systematic study to assess the precision ultimately achieved by bit slicing with analog in-memory computing has so far been lacking. In this work, we assess the computational error from bit slicing when performing in-memory matrix-vector multiplications. Using accurate models of phase-change memory crossbar arrays, we demonstrate that unlike in digital processors where bit slicing is used to extend the dynamic range of the number representation, bit slicing with in-memory computing should aim at minimizing the error from the analog matrix representation through averaging within a given dynamic range. The results are validated using a prototype phase-change memory chip and the impact on the neural network inference accuracy on CIFAR-10 and ImageNet benchmarks is evaluated.