In complex indoor environments, many localization techniques have been investigated subject to low-cost, rapid system deployment, and competitive location accuracy over the past two decades. In this paper, we propose an environment-aware tracking scheme to solve the tracking problem on smartphones in indoor environments, when Bluetooth Low Energy (BLE) beacons are deployed. In contrast to the conventional BLE proximity method, we make following three features in our tracking scheme. First, the influence of complex non-light-of-sight effects on the BLE received signal strengths in the indoor environments have been analyzed. Second, we adopt an optimal deployment strategy in indoor building, then modify BLE signals based on a probabilistic classification model of environments. Third, with the prevalence of smartphones, we extract smartphone-inertial measurement unit along with BLE readings to learn user’s behavior preferences and input to tracking algorithm. With this approach, we can increase the localization accuracy while reducing human cost when constructing indoor fingerprint database for anonymous buildings using smartphones. Experiment results shows that our proposed method is superior to the conventional works.