With the rapid adoption of smartphones and tablets, more and more remote medical diagnostic applications have mushroomed. Tongue Diagnosis (TD) is a kind of noninvasive diagnostic technique, which offers significant information for health conditions. However, it is rather tough to extract the tongue from a high-quality image, in which there is a definite large area of the tongue, to say nothing of extracting the tongue from a digital image captured by photographers who often lack the necessary skills using different mobile front facing cameras. Fundamentally, automatic tongue image segmentation is difficult due to two special factors: the particularity of the tongue and the diversity of the image. Our paper first addresses these problems by proposing a new end-to-end iterative network for tongue image matting, which directly learns the alpha matte from the input image by correcting misunderstanding in intermediate steps. Neither user interaction nor initialization is required. In addition, we create a large-scale tongue image matting dataset including 7,0680 training images. Compared with other high-performance algorithms, our algorithm achieves the true sense of the pixel-wise automatic tongue segmentation.