Software bugs come in two varieties - ones that fail outright and ones that don’t. Obvious failures typically have straightforward fixes and can be guarded against with tests. However, in the realm of computer modeling and simulation, we must all contend with another much more stealthy type of bug that is perhaps best summarized with the following quote from mathematician George E. Box, “All models are wrong. The practical question is how wrong do they have to be to not be useful.” Try as we might, there is no testing framework in the world that can reliably identify such a bug when it causes no failure in simulation and no obviously incorrect result. All that we are left with is a result that is too wrong to be practical and practicality is something that can only be judged by the humans who are using that result. For this reason, there is really only one way to reliably find and address this second type of bug: a software community with eyes on the code.
Spend an hour on the Ladybug Tools or Pollination forums and you will no doubt find many cases of this bug-fixing process at work. We recently had a particularly good example that involved a collaborative effort across several advanced users and software developers. At the heart of the issue, both the Ladybug Tools Grasshopper plugin and the Clima web tool produce wind rose graphics and, while each software package produced a result that appeared correct, the two were not in agreement until a week ago. Had both software packages been closed source, all of us may have had to simply resign ourselves, taking a side based on brand loyalty or some other intangible criteria. But, when software is open source (as is the case for both Ladybug Tools and Clima), we all have a much better option - look at the code and see if we can identify the issue. And that’s exactly what the Ladybug Tools community members and Clima developers did, digging deeper and deeper until finally the problematic line of code was found and fixed.
Some people think that a forum full of bug reports is a sign that a given software package is unreliable, but those of us with experience know not to judge things so shallowly. As long as that forum also has evidence that these bugs are investigated and acted upon, then more bug reports often means that the software is more trustworthy; not less. Particularly when it comes to building simulation, where there are an uncountable number of steps that could render a result too wrong to be useful, we should not underestimate the power of open source communities and their vigilant policing efforts over the code. In the words of Linus Torvalds, the grandfather of Linux, “given enough eyeballs on the code, all bugs are shallow.”.
Most software companies will cloak their simulation processes and logic with Intellectual Property (IP) protections. In light of our extensive experience with Ladybug Tools, Pollination takes a hybrid approach to IP. We want our users to see exactly what commands they are running and how their input models are set up. So, any part of Pollination that is concerned with the science of building physics and computer modeling are open source. This includes simulation engines, as well as, all of the libraries that translate models from Pollination to these engines. We believe that no user should ever face an IP barrier to making their simulations more accurate or better representations of the real world. If a user wants to model a new technology or test a new calculation methodology, they can do so by editing the source code. Similarly, if someone senses that there may be a bug in our code, they should be able to track it down and get to the truth without worrying about IP issues. The only things we keep closed source are related to convenience, ease of use, and speed of model creation, simulation, or communication.
Be rest assured, when you buy into the Pollination ecosystem, you can trust the results because you can trust the process! Read more about the Principals guiding the development of Pollination here.
Special thanks for the wind rose example above goes to Hunyum Murya (who first identified the issue), Charlie Brooker (who found the problematic line of code), Saeran Vasanthakumar (for corroborating the issue) and Giovanni Betti of the Clima development team (for ultimately fixing the bug in Clima).
