Emotion and theme recognition in music plays a vital role in music information retrieval and recommendation systems. Deep learning based techniques have shown great promise in this regard. Realising optimal network configurations with least number of floating point operations per second (FLOPS) and model parameters is of paramount importance to obtain efficient deployable models, especially for resource constrained hardware. We propose a novel integration of stand-alone self-attention into a Visual Geometry Group (VGG)-like network for the task of multi-label emotion and theme recognition in music. Through extensive experimental evaluation, we discover the ideal and optimal integration of stand-alone self-attention which leads to substantial reduction in number of parameters and FLOPS, yet yielding better performance. We benchmark our results on the autotagging-moodtheme subset of the MTG-Jamendo dataset. Using mel-spectrogram as the input, we demonstrate that our proposed SA-VGG network requires 55 % fewer parameters and 60 % fewer FLOPS while improving the baseline ROC-AUC and PR-AUC.