Phantom Bugs: A Lesson in Troubleshooting
As part of the Geocortex Essentials product team, we have people who fix bugs. Typically we discover a bug ourselves or hear of the issue through a customer experiencing issues in their environment which we haven't seen before. We take measures to replicate the issue, get to the bottom of it, and fix it.
But what do you do when you have an issue being reported from a few disparate customers (thus confirming its existence) which can't be replicated in-house? Recently, our support department received screenshots from several different customers looking like this:
Locally, we've never seen Essentials do this - not on 64bit operating systems, not with IE6, nor in any combination of weird server environments we could concoct. So, we spent days trying to replicate this issue with a hunch it had something to do with the new CSS Multiplexer control we'd written for 1.5.2. This requires a bit of a tangent:
Certain versions of Internet Explorer prevent any one web page from adding references to more than exactly 32 CSS (cascading style sheet) pages. Don't ask me why, but that's the limit. Because Geocortex Essentials has a large number of web controls which are independent and modular, many of them come with their own CSS registration; thus, it didn't take long before we discovered this limit (now this was a bug we encountered which warrants its own blog post altogether). The solution: the CSS Multiplexer. A web control which combines all CSS style sheets referenced on the page into one gigantic style sheet. I won't get into the details, but it worked and improved performance as a side effect.
So when we see screenshots that look like CSS isn't doing its job, the natural inclination is to blame the CSS Multiplexer. Long story short... this inclination proved wrong and it was not a bug. After a few days of wrestling with the CSS Multiplexer, and delivering patches to customers who were experiencing the issue only to discover the patches didn't resolve the problem, it became evident that resolving issues using a black box approach wasn't working. Our support department got in touch with one of the customers who was experiencing this issue and asked if we could set up an online meeting where we could see the bug and troubleshoot onsite. They agreed. Within one hour we were able to determine that a setting in IIS was preventing the CSS Multiplexer from fetching the appropriate style sheets and the customer was up and running.
We're grateful to have customers willing to help us track down an issue like this. Of course, finding a resolution is of mutual benefit, but they could've just as easily asked us to investigate this with someone else. Moving forward, I think we'll be much quicker to go down the bug observation route much sooner if we're having trouble replicating it.
Incidentally, we'll be publishing a knowledge base article on our support center for those who are experiencing this issue, and our developers have some ideas of ways we can get Essentials to "trick" IIS into behaving correctly to prevent this from occurring in the future.