To avoid the frequent wakeups of general spoken dialogue systems, we studied continuous-speech spoken dialogue systems. The experiment results show that continuous-speech systems can efficiently enhance the success rate of interactions by 8%, in contrast with discontinuous ones. The full-duplex continuous-speech dialogue systems, which are continuously listening surrounding voices, are often interfered with by invalid sounds, such as the voice played on TV. Therefore, we have studied the audio recognition algorithm to reject those noises, and the experiments show that the recognition accuracy of this algorithm can reach about 94%. In this paper, we propose joint decision-making of audio and semantics, which can effectively reject invalid speech and further improve the success rate of interaction in continuous-speech spoken dialogue systems.