Cardiovascular diseases are the major cause of death worldwide. Magnetic resonance imaging (MRI) is often used for the diagnosis of cardiac diseases because of its good soft tissue contrast. Furthermore, the fibrosis characterization of the myocardium can be important for accurate diagnosis and treatment planning. The clinical gold standard to visualize myocardial scarring is late gadolinium enhanced (LGE) MRI. However, the challenge arises in the accurate segmentation of the endocardial and epicardial border because of the smooth transition between the blood pool and scarred myocardium, as contrast agent accumulates in the damaged tissue and leads to hyper-enhancements. An exact segmentation, is essential for the scar tissue quantification. We propose a deep learning-based method to segment the left ventricle’s endocardium and epicardium in LGE-MRI. To this end, a multi-scale fully convolutional neural network with skip-connections (U-Net) and residual units is applied to solve the multiclass segmentation problem. As a loss function, weighted cross-entropy is used. The network is trained on 70 clinical LGE MRI sequences, validated with 5, and evaluated with 26 data sets. The approach yields a mean Dice coefficient of 0.90 for the endocard and 0.87 for the epicard. The proposed method segments the endocardium and epicardium of the left ventricle fully automatically with a high accuracy.