In this study, we propose a method to predict the visibility of images composed of different color schemes using Convolutional Neural Networks (CNN), which introduces the Multi-Stage Color Model (MSC Model), a human color vision model. Using the MSC model, a method has been proposed to predict the visibility of color schemes on a computer by reproducing the way humans see colors. In this study, the response values of multiple cell layers by the MSC model are used as input values to the CNN. In constructing the CNN, we propose a CNN with an improved structure to extract the features for the visibility between images, and attempt to improve the prediction accuracy of visibility. The CNN is trained on the color scheme visibility data collected in a pairwise comparison experiment, and the trained CNN is used to predict visibility for unknown color schemes. The results show that the construction of a CNN with the MSC model improves the accuracy of color scheme visibility.