Recently introduced RGB-D cameras are capable of providing high quality synchronized videos of both color and depth. With its advanced sensing capabilities, this technology represents an opportunity to significantly increase the capabilities of object recognition. It also raises the problem of developing expressive features for the color and depth channels of these sensors. In this paper we introduce hierarchical matching pursuit (HMP) for RGB-D data. As a multi-layer sparse coding network, HMP builds feature hierarchies layer by layer with an increasing receptive field size to capture abstract representations from raw RGB-D data. HMP uses sparse coding to learn codebooks at each layer in an unsupervised way and builds hierarchical feature representations from the learned codebooks in conjunction with orthogonal matching pursuit, spatial pooling and contrast normalization. Extensive experiments on various datasets indicate that the features learned with our approach enable superior object recognition results using linear support vector machines.