Date: Thu, 22 May 2003 21:25:03 -0500
From: Steve Krueger <s-krueger@sbcglobal.net>
Reply-To:  steve@grape-krueger.com
To:  lispm-hackers@lists.unlambda.com
Subject: [LMH]The Cold Loader and a Story

You may know that a load band is merely a saved virtual memory image.  Explorers 
need a LOD band and a MCR band to boot.  You can easily make a LOD band by running 
an appropriate save-band command.

If you think about that for a moment, you may wonder how the Lisp system code is 
initially placed into a band.  It is clear that once you have a band you can make 
a new band by loading some more files or making other changes in your working 
storage and then converting it to a load band by running save-band (or whatever 
the exact function name is).

But, that initial load band is a problem.  Taking files of Lisp code and turning 
it into a load band, is a much different problem than the other operations on bands.  
MIT solved this problem with the cold loader.

The cold loader takes Lisp code that is cross-compiled for a LispM environment, and 
reads it into a new virtual memory image.  TI used the cold loader for two 
purposes: to build load bands from scratch so that they would not contain anything 
extraneous; and for fundamental changes like the VM2 work that changed the tag 
encoding, the instruction set and the format of several object types.

To construct the virtual memory image, the cold loader formats objects into the 
memory image using the object format of the target.  By making changes in the 
definition and functions of the cold loader, the target band may be very different 
than the environment in which it was run.

Now if you think about a cold loader for a while, you realize that Lisp presents a 
formidable challenge in that eval-when-load forms can be arbitrarily complex.  
They can call function, macros, create any data type.  The cold loader was able 
to handle most of the simple cases, but anything very complicated is just pushed 
onto the cold-load-list variable.  The idea is that these forms will all be eval'ed 
by (mapcar #'eval cold-load-list) when the new load band is first run.

The environment of a new cold load band is very fragile.  The biggest problem is 
that only a small, essential part of the lisp environment was loaded with the cold loader.  
In particular, the flavor system and the error system weren't loaded. The window 
systme and file system were simple partial implementations of those functions.  
If anything run in this environment tries to call a function that wasn't loaded, 
it crashes.  Without the error system, the error handler didn't work.

So, the contents of the cold-load-list can prove fatal.  Files in the cold load were 
carefully written and the order in which they were loaded by the cold loader was 
carefully crafted.  A fairly innocent change to a cold loaded file, could make 
its cold-load-list items depend on other files that were not loaded into the 
cold load, or that were loaded later in the cold load and so had not had cold-load-list 
initializations performed.  The result was always a crash.

You may actually see a comment in some of the system source files that mention 
that it is a cold load file.  Now you know what that means.

Now for a story:

When we first were working to bring up the lispm software on Explorer (I), the 
microcode and lisp sources had been adapted to the TI hardware.  It took weeks 
before we got through the microcode initialization to the point of the first disk 
read from the cold-load load band.  And weeks still until it first successfully 
called the initial-lisp-function.  Every new instruction used seemed to have microcode 
bugs, so it took days before we executed much code at all.  At that point, we began 
debugging the execution of the cold load list. Some of the cold-load-list items 
referred to non-existant functions or to uninitialized modules.  Finally, we got 
out to the lisp environment.  I have a photo of the first words printed on the 
Explorer (I) screen.  I'll scan and upload it at some point.

The cold load contained a function that would read in the rest of the system files.  
Again the order was quite significant, as was the proper use of eval-when-load forms.  
Again it tooks weeks of work.  Getting the cold-load file system to work with the 
TI NuPI was a challenge as the NuPI wasn't fully functional.  Each distinct 
macroinstruction seemed to require debugging the microcode.  It probably took a 
couple of weeks to load the first file.  Each new file took days.  The name of 
each file would be printed as it loaded.  We got one, then two then three files 
listed.  Eventually six then ten files were listed.

Then one day, the file names just kept coming.  There were cheers in the system 
integration lab as it went further and further, then it reached the last line of 
the screen and said "** More **".  It took a moment for use to realize that we 
couldn't proceed.  The keyboard wasn't yet complete.  Then we burst out laughing.

After a bit of celebration we devised a work around to this problem.  We built 
the cold-load band with the keyboard buffer pre-filled with a space.

A few names for posterity: The core microcode integration team for Explorer (I) 
was Wayne Gibson, Phil Mueller and myslf.  TI's work on the cold loader was 
by Rick Blair.  Our microcode debugger was called Seymour and was written by 
Paul Fuqua.  The hardware connection to an Explorer was via a hardware box that 
tapped into the processor to give good visibility to the internal memories and 
registers, called a Spyport.  Glenn Manuel built the Spyport.


