To meet the practical operation requirements of manipulators in industry, service and collaboration scenarios, trajectory tracking accuracy and system stability of manipulators are usually regarded as critical control objectives. This paper proposes a novel manipulator control method with three-level constraints, called the real deep Lagrangian network (Real-DeLaN). In this method, real data (also called physical data) are collected from the physical manipulator and utilized to train the Lagrangian dynamics network model to improve the migration ability from simulation to reality. The torque output of the dynamic model is corrected by the real-time calculation of friction in the network. The contact force on the end effector of the manipulator is compensated based on the principle of virtual displacement work. The experimental results show that Real-DeLaN can better control the joints to perform trajectory tracking, reduce the friction error of the manipulator, and show better anti-interference ability.