Computers and Superstition

Computer systems in the best case are built in carefully defined layers. The user clicks on a button: the browser sends a request to the web server; the server program issues some commands to send this information and retrieve that; perhaps a database adds information to a table and sends information back; the web server sends a new page for the browser to display. Here the browser is built to display pages and send requests, the web server to send pages and handle requests, the database to save and retrieve information.

In theory, there is a clean separation of responsibilities. The person writing the programs the web server will run need not worry about how the browser works or the details of queries running against the database. Those who configure the database need not concern themselves with what programs will use the database--maybe it will be a script on a web server, maybe the data will be pulled into a spreadsheet. And there is a hard boundary at the database: for the most part commands and data go in, responses and data come out, but one need not worry about what happens inside the database.

In practice, things are not so cleanly separated. Often enough the person writing the scripts for the web server will also write the queries against the database. That person may have to know some of the quirks of the different browsers, and how they follow or break standards--admittedly this is a fairly shallow knowledge. The person writing queries against the database may have to look very closely to see why something runs slowly and how it can be made to run faster.

I have found little in my work that is more frustrating than coming up against one of these layers, needing just a bit of information from the other side, and being unable to get it. It is well to know that steps A, B, and C will produced result D; if D always comes about, that all one needs to know. But if every tenth application produces E, then one needs to look further. That can be very hard to do.

I have been trying to figure out why certain "integrations", methods for receiving outside data into the Great Plains accounting system, sometimes work properly and sometimes don't. I set these up as copies of another integration. After looking carefully at them all for discrepancies, I have come to a number of conclusions:

• The integration I copied was copied from some other source.
• At least one element of the source integration is otiose: it might have made sense in an integration that is a generation or three removed, but not here.
• Setting up systems in this fashion is nearer superstition than to sound technical practice.
• The explanation of the discrepancy is simple, a matter that could be explained to me by an expert in ten minutes.
• Finding that expert is not a technical but a social problem, and could be all but insoluble.

I have tried what I could think to try. The logs produced when the integration runs with logging set to the "trace" level resemble the box scores of baseball games, which is to say that they will tell you what happened but not how. The database level trace on the other hand recorded almost a thousand operations to record three invoices. I gave up on it, but may have to go back.

Trying a change with a notion of what it will effect, and how, and how one can tell that it produced that effect, is experimenting. Changing something because maybe it will help is superstition. It is too easy to fall into superstitious coding, and it always makes me uncomfortable.