CPAN is Your Friend

The past couple of weeks have offered some lessons in the use of Perl to manipulate HTML and use HTTP.

1. Don't try to tidy up weirdly tagged  HTML with regular expressions: HTML::TreeBuilder is your friend. (But remember the implications if you use $node->replace_content.)
2. Don't try to replace HTML entities (á and so on) with regular expressions: HTML::Entities is your friend.
3. Remember that Encode::encode('utf8', $string) does not turn $string into UTF-8 Unicode; it gives you an octet string, which after all is what HTML::Message wants. You do not want Encode::decode here.
4. If you have multiple parameters of the same name to send to a URL with POST,  call HTML::Request::Common::POST with "Content => [p => 1, p => 2] ". Do not call it with "Content => {p => [1,2]}, for this subroutine regards a list reference as specifying a file to load. You will scratch your head wondering why your script is trying and failing to open a file named "1", and it will take you a moment to check back with the documentation and figure it out.

I can go weeks or months without needing to look at CPAN, but what a wonderful resource it is! A couple of hours getting acquainted with HTML::TreeBuilder made it possible to write a script that saved hours of tedious work.

Unix is 40!

IEEE Spectrum this month recognizes the 40th birthday of Unix. Anyone who uses an Apple computer running OS X, anybody with an Android phone, has a Unix variant running on that device. The chances are that most of the web sites one visits run at least part of their stack on Unix variants.

Spectrum's account is worth a look for techies, and I suppose for anyone interested in the way that interesting projects sometimes get started and run beneath the notice of management in large organizations. It is well known in techie circles that UNIX got started when Ken Thompson ported a computer game from a GE-465 computer to a PDP-7.  But I had never heard that

End runs around AT&T's lawyers indeed became the norm—even at Bell Labs. For example, between the release of the sixth edition of Unix in 1975 and the seventh edition in 1979, Thompson collected dozens of important bug fixes to the system, coming both from within and outside of Bell Labs. He wanted these to filter out to the existing Unix user base, but the company's lawyers felt that this would constitute a form of support and balked at their release. Nevertheless, those bug fixes soon became widely distributed through unofficial channels. For instance, Lou Katz, the founding president of Usenix, received a phone call one day telling him that if he went down to a certain spot on Mountain Avenue (where Bell Labs was located) at 2 p.m., he would find something of interest. Sure enough, Katz found a magnetic tape with the bug fixes, which were rapidly in the hands of countless users.

Dennis M. Ritchie, RIP

Dennis M. Ritchie, creator of the C programming language, co-author of The C Programming Language, and one of the creators of the UNIX operating system, died on October 8.

UNIX is not that visible to most persons who don't work with technology, unless of course they use Apple computers--OS X is Berkeley UNIX under the hood. Yet a lot of the servers on the Internet, not to mention a lot of the machines that handle details one just doesn't see--the databases for banks, reservation systems, phone systems--use some UNIX variant.

It is most unlikely that anyone can read this post without using some software that was written in C--a large fraction of the web servers on the Internet run UNIXes or Linux, which are largely written in C; most of the rest likely use the Microsoft stack, and I dare says that a great deal that  in C, with much of the rest in C's descendants C++ and C#.

C has its flaws. It was written by people who worked at Bell Labs for their use and the use of others who worked at Bell Labs, i.e. really smart people. In the hands of the careless--occasionally including the Bell Labs staff--a language that assumed you knew what you meant could lead to trouble. C didn't care whether you checked your bounds or the return values from your system calls. Want to scribble all over your stack? No problem! Want to read 10 pounds of data into a 5 pound bag? Sure! UNIX also has its quirks, some of them C-related, some related only by the priorities of the creators.

But there are an awful lot of us who have made our livings working with the products Dennis Ritchie had a large hand in. Believe me, I'm grateful to the man.

Languages and Letters

In looking up something John Ousterhout had written, I noticed with surprise that his home page is http://www.stanford.edu/~ouster/cgi-bin/home.php.

The last three letters were the source of the surprise, for long ago Ousterhout developed the scripting language Tcl (tool-command language), which serves decently enough for web development. Or it served so anyway, for the last updates to AOLServer happened in 2010, and to Apache mod_tcl in 2003. There doesn't seem to be much talk about either of them lately, for the first page on a Google query for either retrieves postings from years ago. There may not be many patches, either, if somebody comes up with a zero-day exploit.

Tcl has its maddening aspects; with a bit of searching one can find Richard Stallman's diatribe against it. It has in its favor that it was built to be readily extensible. I learned it first because "expect" was built on top of it, a tremendously useful tool for some system administration tasks. I've used it via Tk (Toolkit, a handy tool for setting up graphical user interfaces), "oratcl" for running jobs under control of the Oracle "intelligent agent", and then with AOLserver. And I didn't know how much I liked its "upvar" until I looked without success for an equivalent in PHP.

Data

Many years and a couple of jobs ago, I twice got on a plane to a customer site, there to help them recover from a corrupted file system. In both cases, the unit that failed was a 25 megabyte disk. Now, Apple will sell you an iPod with 16 gigabytes of storage, nearly 700 times as much as fit on these units. The iPod will fit nicely in a pocket, but these units came in a cabinet about 2 feet wide by three feet deep by a foot and a half high. The prudent user backed them up, generally using 8 inch floppy disks that held 1.2 megabytes. The main drive (and I suppose the floppy--I never looked) was driven by a rubber belt like a car's fan belt--if it dropped off the system would halt.

Recovering from file system corruption required backing all recoverable files off to the floppies, a few dozen at a time. This made for a long day and a late night, and one never got all the data back. At one of the sites, I sat in on the inquest where a manager a couple of levels up, with perfect southern manners, made it clear how inappropriate such disruptions were; I met some volatile and expressive customers at that job, and at least one who was outstanding at profanity, but there was none I'd fear more to disappoint than the lady with perfect manners.

The other week I found myself the customer in need of help. An old computer crashed, and damaged a couple of file systems beyond recovery. Pretty much everything I needed was backed up, but the backup and restore software depended on a number of settings that had changed between the old operating system and the new one I had restored. After co-workers were unable to help me, I put in a call to the vendor's support line, where I was promised a call back within two hours. The first cogent response came 21 hours after my call, by which time I had figured out a way around the difficulties.

I have seen disks develop from the 25 megabyte units to 1 terabyte disks (400 thousand times larger, that is). I have seen the backup media go from 8 inch floppy and 1 inch tape to 4 mm tape, to DLT  through a couple of generations of LTO. The throughput rates and capacity have astonished me at each jump. Yet it seems that you can never get data back as quickly as you'd care to . You can get gigabytes and soon terabytes back as fast as once you could megabytes. But now it's GB and TB you need back, not MB.

Recent Lessons on Databases

To the technically indoctrinated,

1. It is an important and elegant feature of relations that the tuples within them have not top-to-bottom ordering. This is because tuples are mathematical sets, and sets have no order. It is from relations that relational databases take their name; the tables and views within a relational database are just special cases of relations.
2. Concatenating a number and a string of characters produces a string of characters. This resulting string supports the operations valid for strings, not for numbers. If one is to sort it, one will sort it in lexicographic order and "101 Dalmatians" will always precede "3 Musketeers" in a non-decreasing sort.
3. In fact, distinguishing the "101" and the "3"  within their strings violates "First Normal Form", according to which the attributes of a tuple must be atomic.

For points 1 and 3, see any good introduction to relational databases, for example C.J. Date's Introduction to Database Systems.

To those not technically indoctrinated,

1.  If I didn't want it in order, why would I have ordered it before I gave it to you?
2. Of course 3 is less than 101.
3. First Normal Form?

I must say that Microsoft has learned about point 2, I'd bet through hard experience; if you have MP3s of "12 Days of Christmas" , "100 Bottles of Beer on the Wall" , and "4 Green Fields" in a directory, Windows Explorer will display them in numeric order when it orders by name.

I must also say, as the technically indoctrinated one in this particular back-and-forth, I was grateful to the other party for her independently deciding to zero-pad the numbers of items, which resolved the problem.

A Three-Language Day

• In the morning, cleanup of some old PL/Sql code that I neglected to fix months ago.
• Ongoing, testing and modification of some Python to rewrite DXF (CAD interchange) files of floor plans to label offices with their occupants.
• In the afternoon, some work with T-SQL first to encapsulate some SQL I don't want to have to rewrite time and again, second to debug somebody else's that was giving a wrong answer.

And had I the time and energy, I guess I could have moved on to JScript for the network administrator's work.

RIP Tom West, Begetter of "a New Machine"

Tom West, who led the development of the Data General (DG) MV/8000 minicomputer, died last Thursday. He achieved fame 30 years ago when Tracy Kidder wrote about the project in The Soul of a New Machine, which won a Pulitzer Prize.

I read a bit of the book when excerpts appeared in a magazine, then a bit later read a roommate's copy. Later still, after I had some experience with the MV/Eclipse line, my wife was assigned the book for a management class, and I read it yet again. A couple of points were clearer on that reading:

• The decision to incorporate the 16-bit instruction set imposed certain constraints. The MVs had four general-purpose registers, for example; for another, one could address words but not bytes with a register-relative address.
• The lead architect of the team had wanted a VAX-like instruction set. The VAX instruction set was elaborate, just the thing if one were writing an operating system in assembly language.
• About the time that the book came out, researchers in California were discovering how hard it was to speed up the VAX to the point they wanted, and considering alternatives. The new model of "reduced-instruction-set computing" (RISC) produced some startlingly fast machines, and ate into the old minicomputer market at the high end while the 386 started to nibble at the other.

By the early 1990s, the DG users' group magazine quoted West as saying that customers "wanted to get their MIPS from gun-gray boxes running UNIX." This was pretty much true. With the Aviion, based on Motorola's 88000 RISC processor, DG made a very good effort to give the customers what they wanted. Alas, Motorola was one of many companies to discover that trying to fight the x86 machines was pointless. EMC bought up the wreckage of DG  for the Clariion storage line.

I enjoyed working with the MV/8000 and its successors, and for that matter their 16-bit predecessors. The AOS/VS operating system did many things well, the command-line interpreters were well thought out, and the assembler easily enough learned. I'm grateful for the chance to have worked with those machines.

Two for the jargon file

After certain embarrassments in the news recently, it seems we could use

Unpatchy, noun: The Apache server that came with your Linux distribution three years ago, and to which you have never applied any security patches.

And from a couple of years ago

Preloopsarian, adjective: In that state of programming innocence in which one has discovered assignment and conditions, but not iteration.

For a classic example of the latter see Chapter 3 of Jon Bentley's Programming Pearls. I regret to say that it is not one of the chapters at the on-line site.

Caught in the Details

Last week I had time to do what I had meant to do the week before, set up a Linux server for another group at work. The requirements were simple: Apache + PHP, using a SQL Server database on one of our Windows servers. This proved to be a little more complicated than I thought:

• FreeTDS (for the database layer) went in without problems, so that was easy enough. Configuring obbcinst.ini and odbc.ini was easy.
• My own preference with PHP is to use PDO for database work. But libc6 decided that PDO was taking indecent liberties with the stack. So, no PDO.
• ODBC? Well, I've never used it under Linux, certainly never with PHP.Something I Googled spoke of it faking the binding of variables on SQL Server. So, not ODBC.
• The mssql_connnect was fine.
• Then I discovered why PHP's odbc procedures fake  binding against SQL Server; SQL Server doesn't want to bind variables against anything but a stored procedure. Who knew?
• Still, mssql_query worked well enough in "SQL Injection Mode", concatenating stuff from $_POST into the query text. This was reassuring, but not what I had in mind for the long run.
• And then I found that my stored procedures, which ran nicely enough in SQL Server Management Studio, wouldn't execute with PHP's mssql_execute.
• There's always Google, and somebody helpfully explained that one must set the version to 8.0 in /etc/tds.conf. Gee, how did I miss that? I edited the file, and just like that, the stored procedures worked as intended.
• Until I rebooted after squeezing down memory, and mssql_connect quit working. Eventually I worked out that the httpd server needed to have LD_LIBRARY_PATH set.

By the time I was done, and had written up instructions for the other team, I was tired and distracted. When I spoke to one of them and she changed the topic, it took a moment to catch up.

Yet having said that, I have to acknowledge how much simpler it is to set up a Linux machine now. Fedora and VMware/VSphere take so much of the pain out of installation, and no doubt other distros are comparably good. To one who can remember the monitor shaking and chuffing as XFree86 tried to evaluate it, this seems like magic.