In this paper, a non-linear precoding algorithm with low out-of-band (OOB) radiation in a massive multiple-input multiple-output (MIMO) system is proposed. Massive MIMO is a system with more than a few hundred antennas implemented in a base station and is characterized by better spectral efficiency and higher throughput. Full digital massive MIMO may require lower resolution in digital-to-analog converters (DACs) since each high-resolution DAC consumes a large amount of power. In massive MIMO systems with low resolution DACs, an output signal designed by nonlinear processing has been investigated. The conventional scheme focuses only on a sum rate or errors in received signals and causes large OOB radiation. This paper proposes an optimization criterion that takes OOB radiation power into account. Gibbs sampling is used as an algorithm to search for a sub-optimal solution for this criterion. Numerical results obtained through computer simulation show that the proposed criterion reduces the mean OOB radiation power by a factor of 10 as compared with the conventional criteria. Furthermore, the proposed criterion achieves larger sum rate over all subcarriers as compared with an error based criterion. On the other hand, as compared with a sum rate based criterion, the throughput on each subcarrier shows less variation and that demands a smaller number of link adaptation options although the average sum rate of the proposed criterion is smaller.