Abstract
Property-based testing uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from combinatorial testing, a powerful and widely studied testing methodology, to modify the distributions of our random generators so as to find bugs with fewer tests. The key concept is combinatorial coverage, which measures the degree to which a given set of tests exercises every possible choice of values for every small combination of input features.
In its “classical” form, combinatorial coverage only applies to programs whose inputs have a very particular shape—essentially, a Cartesian product of finite sets. We generalize combinatorial coverage to the richer world of algebraic data types by formalizing a class of sparse test descriptions based on regular tree expressions. This new definition of coverage inspires a novel combinatorial thinning algorithm for improving the coverage of random test generators, requiring many fewer tests to catch bugs. We evaluate this algorithm on two case studies, a typed evaluator for System F terms and a Haskell compiler, showing significant improvements in both.
For the full version, including all appendices, visit https://harrisongoldste.in/papers/quick-cover.pdf.
Chapter PDF
Similar content being viewed by others
Keywords
References
Arcuri, A., Briand, L.C.: Adaptive random testing: an illusion of effectiveness? In: Dwyer, M.B., Tip, F. (eds.) Proceedings of the 20th International Symposium on Software Testing and Analysis, ISSTA 2011,Toronto, ON, Canada, July 17-21, 2011. pp. 265–275. ACM (2011). https://doi.org/10.1145/2001420.2001452, https://doi.org/10.1145/2001420.2001452
Bell, K.Z., Vouk, M.A.: On effectiveness of pairwise methodology for testing network-centric software. In: 2005 International Conference on Information and Communication Technology. pp. 221–235. IEEE (2005)
Braquehais, R.M.: Tools for discovery, refinement and generalization of functional properties by enumerative testing (October 2017), http://etheses.whiterose.ac.uk/19178/
Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: Maher, M.J. (ed.) Advances in Computer Science - ASIAN 2004, Higher-Level Decision Making, 9th Asian Computing Science Conference, Dedicated to Jean-Louis Lassez on the Occasion of His 5th Cycle Birthday, Chiang Mai, Thailand, December 8-10, 2004, Proceedings. Lecture Notes in Computer Science, vol. 3321, pp. 320–329. Springer (2004). https://doi.org/10.1007/978-3-540-30502-6_23, https://doi.org/10.1007/978-3-540-30502-6_23
Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: adaptive random testing for object-oriented software. In: Schäfer, W., Dwyer, M.B., Gruhn, V. (eds.) 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008. pp. 71–80. ACM (2008). https://doi.org/10.1145/1368088.1368099, https://doi.org/10.1145/1368088.1368099
Claessen, K., Duregård, J., Palka, M.H.: Generating constrained random data with uniform distribution. J. Funct. Program. 25 (2015). https://doi.org/10.1017/S0956796815000143, http://dx.doi.org/10.1017/S0956796815000143
Claessen, K., Hughes, J.: Quickcheck: a lightweight tool for random testing of haskell programs. In: Odersky, M., Wadler, P. (eds.) Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP ’00), Montreal, Canada, September 18-21, 2000. pp. 268–279. ACM (2000). https://doi.org/10.1145/351240.351266, https://doi.org/10.1145/351240.351266
Colbourn, C.J., Cohen, M.B., Turban, R.: A deterministic density algorithm for pairwise interaction coverage. In: Hamza, M.H. (ed.) IASTED International Conference on Software Engineering, part of the 22nd Multi-Conference on Applied Informatics, Innsbruck, Austria, February 17-19, 2004. pp. 345–352. IASTED/ACTA Press (2004)
Comon, H., Dauchet, M., Gilleron, R., Löding, C., Jacquemard, F., Lugiez, D., Tison, S., Tommasi, M.: Tree automata techniques and applications. Available on: http://www.grappa.univ-lille3.fr/tata (2007), release October, 12th 2007
Courcelle, B.: Fundamental properties of infinite trees. Theor. Comput. Sci. 25, 95–169 (1983). https://doi.org/10.1016/0304-3975(83)90059-2, https://doi.org/10.1016/0304-3975(83)90059-2
Gill, A., Runciman, C.: Haskell program coverage. In: Keller, G. (ed.) Proceedings of the ACM SIGPLAN Workshop on Haskell, Haskell 2007, Freiburg, Germany, September 30, 2007. pp. 1–12. ACM (2007). https://doi.org/10.1145/1291201.1291203, https://doi.org/10.1145/1291201.1291203
Girard, J.Y.: Interprétation fonctionnelle et élimination des coupuresde l’arithmétique d’ordre supérieur. Ph.D. thesis, Éditeur inconnu (1972)
Havrikov, N., Zeller, A.: Systematically covering input structure. In: 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. pp. 189–199. IEEE (2019). https://doi.org/10.1109/ASE.2019.00027, https://doi.org/10.1109/ASE.2019.00027
Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering 30(6), 418–421 (2004). https://doi.org/10.1109/TSE.2004.24
Kuhn, D.R., Higdon, J.M., Lawrence, J., Kacker, R., Lei, Y.: Combinatorial methods for event sequence testing. In: Antoniol, G., Bertolino, A., Labiche, Y. (eds.) Fifth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Montreal, QC, Canada, April 17-21, 2012. pp. 601–609. IEEE Computer Society (2012). https://doi.org/10.1109/ICST.2012.147, https://doi.org/10.1109/ICST.2012.147
Kuhn, D.R., Kacker, R.N., Lei, Y.: Practical combinatorial testing. NIST special Publication 800(142), 142 (2010)
Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE transactions on software engineering 30(6), 418–421 (2004)
Kuraj, I., Kuncak, V., Jackson, D.: Programming with enumerable sets of structures. In: Aldrich, J., Eugster, P. (eds.) Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. pp. 37–56. ACM (2015). https://doi.org/10.1145/2814270.2814323, https://doi.org/10.1145/2814270.2814323
Lämmel, R., Schulte, W.: Controllable combinatorial coverage in grammar-based testing. In: Uyar, M.Ü., Duale, A.Y., Fecko, M.A. (eds.) Testing of Communicating Systems, 18th IFIP TC6/WG6.1 International Conference, TestCom 2006, New York, NY, USA, May 16-18, 2006, Proceedings. Lecture Notes in Computer Science, vol. 3964, pp. 19–38. Springer (2006). https://doi.org/10.1007/11754008_2, https://doi.org/10.1007/11754008_2
Lampropoulos, L., Gallois-Wong, D., Hritcu, C., Hughes, J., Pierce, B.C., Xia, L.: Beginner’s Luck: a language for property-based generators. In: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017. pp. 114–129 (2017), http://dl.acm.org/citation.cfm?id=3009868
Lampropoulos, L., Hicks, M., Pierce, B.C.: Coverage guided, property based testing. PACMPL 3(OOPSLA), 181:1–181:29 (2019). https://doi.org/10.1145/3360607, https://doi.org/10.1145/3360607
lcamtuf: AFL quick start guide. http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt (Apr 2018)
Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: Ipog: A general strategy for t-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07). pp. 549–556. IEEE (2007)
Livinskii, V., Babokin, D., Regehr, J.: Random testing for c and c++ compilers with yarpgen. Proceedings of the ACM on Programming Languages 4(OOPSLA), 1–25 (2020)
Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. pp. 643–653 (2014)
Mariani, L., Pezzè, M., Willmor, D.: Generation of integration tests for self-testing components. In: Núñez, M., Maamar, Z., Pelayo, F.L., Pousttchi, K., Rubio, F. (eds.) Applying Formal Methods: Testing, Performance and M/ECommerce, FORTE 2004 Workshops The FormEMC, EPEW, ITM, Toledo, Spain, October 1-2, 2004. Lecture Notes in Computer Science, vol. 3236, pp. 337–350. Springer (2004). https://doi.org/10.1007/978-3-540-30233-9_25, https://doi.org/10.1007/978-3-540-30233-9_25
Marlow, S., Peyton-Jones, S.: The Glasgow Haskell Compiler. In: Brown, A., Wilson, G. (eds.) The Architecture of Open Source Applications, vol. II. Available online under the Creative Commons Attribution 3.0 Unported license (March 2012), http://www.aosabook.org/en/ghc.html
Mathis, B., Gopinath, R., Zeller, A.: Learning input tokens for effective fuzzing. In: Khurshid, S., Pasareanu, C.S. (eds.) ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020. pp. 27–37. ACM (2020). https://doi.org/10.1145/3395363.3397348, https://doi.org/10.1145/3395363.3397348
Mista, A., Russo, A., Hughes, J.: Branching processes for quickcheck generators. In: Wu, N. (ed.) Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell, askell@ICFP 2018, St. Louis, MO, USA, September 27-17, 2018. pp. 1–13. ACM (2018). https://doi.org/10.1145/3242744.3242747, https://doi.org/10.1145/3242744.3242747
Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2) (Feb 2011). https://doi.org/10.1145/1883612.1883618, https://doi.org/10.1145/1883612.1883618
Padhye, R., Lemieux, C., Sen, K., Simon, L., Vijayakumar, H.: Fuzzfactory: domain-specific fuzzing with waypoints. Proc. ACM Program. Lang. 3(OOPSLA), 174:1–174:29 (2019). https://doi.org/10.1145/3360600, https://doi.org/10.1145/3360600
Pałka, M.H., Claessen, K., Russo, A., Hughes, J.: Testing an optimising compiler by generating random lambda terms. In: Proceedings of the 6th International Workshop on Automation of Software Test. pp. 91–97. AST ’11, ACM, New York, NY, USA (2011). https://doi.org/10.1145/1982595.1982615, http://doi.acm.org/10.1145/1982595.1982615
Reddy, S., Lemieux, C., Padhye, R., Sen, K.: Quickly generating diverse valid test inputs with reinforcement learning. In: Rothermel, G., Bae, D. (eds.) ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. pp. 1410–1421. ACM (2020). https://doi.org/10.1145/3377811.3380399, https://doi.org/10.1145/3377811.3380399
Regehr, J., Chen, Y., Cuoq, P., Eide, E., Ellison, C., Yang, X.: Test-case reduction for C compiler bugs. In: ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, Beijing, China - June 11 - 16, 2012. pp. 335–346 (2012). https://doi.org/10.1145/2254064.2254104, http://www.cs.utah.edu/~regehr/papers/pldi12-preprint.pdf
Reynolds, J.C.: Towards a theory of type structure. In: Robinet, B. (ed.) Programming Symposium, Proceedings Colloque sur la Programmation, Paris, France, April 9-11, 1974. Lecture Notes in Computer Science, vol. 19, pp. 408–423. Springer (1974). https://doi.org/10.1007/3-540-06859-7_148, https://doi.org/10.1007/3-540-06859-7_148
Runciman, C., Naylor, M., Lindblad, F.: Smallcheck and lazy smallcheck: automatic exhaustive testing for small values. In: Gill, A. (ed.) Proceedings of the 1st ACM SIGPLAN Symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008. pp. 37–48. ACM (2008). https://doi.org/10.1145/1411286.1411292, https://doi.org/10.1145/1411286.1411292
Salecker, E., Glesner, S.: Combinatorial interaction testing for test selection in grammar-based testing. In: Antoniol, G., Bertolino, A., Labiche, Y. (eds.) Fifth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Montreal, QC, Canada, April 17-21, 2012. pp. 610–619. IEEE Computer Society (2012). https://doi.org/10.1109/ICST.2012.148, https://doi.org/10.1109/ICST.2012.148
Sarkar, K., Colbourn, C.J.: Upper bounds on the size of covering arrays. SIAM J. Discrete Math. 31(2), 1277–1293 (2017). https://doi.org/10.1137/16M1067767, https://doi.org/10.1137/16M1067767
Stephens, N., Grosen, J., Salls, C., Dutcher, A., Wang, R., Corbetta, J., Shoshitaishvili, Y., Kruegel, C., Vigna, G.: Driller: Augmenting fuzzing through selective symbolic execution. In: Network and Distributed System Security Symposium (NDSS) (2016)
Usaola, M.P., Romero, F.R., Aranda, R.R., de Guzmán, I.G.R.: Test case generation with regular expressions and combinatorial techniques. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2017, Tokyo, Japan, March 13-17, 2017. pp. 189–198. IEEE Computer Society (2017). https://doi.org/10.1109/ICSTW.2017.38, https://doi.org/10.1109/ICSTW.2017.38
Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in Ccompilers. In: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, San Jose, CA, USA, June 4-8, 2011. pp. 283–294 (2011). https://doi.org/10.1145/1993498.1993532, http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf
Yun, I., Lee, S., Xu, M., Jang, Y., Kim, T.: QSYM : A practical concolic execution engine tailored for hybrid fuzzing. In: 27th USENIX Security Symposium (USENIX Security 18). pp. 745–761. USENIX Association, Baltimore, MD (2018), https://www.usenix.org/conference/usenixsecurity18/presentation/yun
Zeller, A., Gopinath, R., Böhme, M., Fraser, G., Holler, C.: The fuzzing book. In: The Fuzzing Book. Saarland University (2019), https://www.fuzzingbook.org/, retrieved 2019-09-09 16:42:54+02:00
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2021 The Author(s)
About this paper
Cite this paper
Goldstein, H., Hughes, J., Lampropoulos, L., Pierce, B.C. (2021). Do Judge a Test by its Cover. In: Yoshida, N. (eds) Programming Languages and Systems. ESOP 2021. Lecture Notes in Computer Science(), vol 12648. Springer, Cham. https://doi.org/10.1007/978-3-030-72019-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-72019-3_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72018-6
Online ISBN: 978-3-030-72019-3
eBook Packages: Computer ScienceComputer Science (R0)
