In smart buildings, physical components (e.g., controllers, sensors, and actuators) are interconnected and communicate with each other using network protocols such as BACnet. Many smart building networks are now connected to the Internet, enabling attackers to exploit vulnerabilities in critical buildings. Network monitoring is crucial to detect such attacks and allow building operators to react accordingly. In this paper, we propose an intrusion detection system for building automation networks that detects known and unknown attacks, as well as anomalous behavior. It does so by leveraging protocol knowledge and specific BACnet semantics: by using this information, the alerts raised by our system are meaningful and actionable. To validate our approach, we use a real-world dataset coming from the building network of a Dutch university, as well as a simulated dataset generated in our lab facilities.