Nowadays, earphone acts an increasingly vital role in human computer interaction. Throughout this trend, earphone-based user authentication, a new modality of biometric authentication, is winning more attention. Previous research leverage ear canal related features to authenticate users. These methods establish a binary classification model, based on the assumption that the identity information of all invalid users is given. However, they may not be effective when confronting new attackers.In this paper, we propose EarAE, a lightweight auto-encoder to address this problem. The basic idea is to model the authentication as a one-class classification problem, with features of the valid user as the single training class. Specifically, during training, the ear canal features of the valid user are fed into an autoencoder and reconstructed. EarAE learns to minimize the loss function between the input and the reconstructed counterparts. In this way, valid users will have a much lower loss function score than invalid users. Finally, a threshold is set to distinguish between valid/invalid users. Experimental results show that EarAE can achieve an averaged balanced accuracy of over 96% across 30 subjects. This demonstrates a promising prospect of EarAE to serve as a reliable means of user authentication.