Nonlinear blind source separation is the process of estimating either the original signals or mixture functions from the degraded signals, without any prior information about original sources. The key idea is to recover the sources by estimating an approximation function so as to approximate the inverse of mixing function. However, in practice, the approximation function is derived from some estimation algorithm with finite sample size, which leads to the performance loss. In this paper, we work on the convergence and asymptotic analysis of the separation approach, which uses the flexible approximation to extract the nonlinearity of mixture function so that to make the problem linearly separable. The analysis stems from the performance of a mismatched estimator that accesses the finite sample size. By providing a closed-form expression of normalized mean squared error (NMSE), we can present a novel algebraic formalization that leads to the upper bound on the estimation error. The simulation results show that if the flexible approximation can extract the nonlinearity of mixing functions, the minimized NMSE can be achieved as the sample size tends to be infinity. This implies that the algorithm is feasible to separate the distortion of the nonlinear mixture.