We propose Direct Multi-scale Dual-stream network (DMDnet) for pedestrian detection. DMDnet takes a full-size image as input and detects pedestrians of various sizes directly without extracting proposals or using resampling. To improve detection accuracy we adopt dual-stream architecture that combines two types of features that are branched off from two different layers. The lower layer observes the proper sizes of receptive fields depending on the scales of pedestrians; the upper layer contains contextual information. The whole network is trained end-to-end. In experiments DMDnet yields state-of-the-art detection accuracy on the Caltech pedestrian benchmark with new annotation, and has quite fast detection speed.