July 2003

This document holds some of my most profound thoughts on what is required to build a reflective system like the DBOS. Each one of these points is something I missed and paid for in development time.

Every computing exvironment needs a compiler/interpretor

Richard Stalman knew this. He started his journey toward a Free software panacea by making the gcc compiler. It is this compiler that all GNU/Linux depends upon and makes the system of free software possible.

If you are making a new system, you should choose a platform that already has a compiler (or macro facility) available, or has one built into the language. Languages like Smalltalk, Lisp, Haskell and ?Python? should be considered for development of a new system.

When I started I did not know this. I chose Java to build my system, and as a result I spent too many hours building a temporary compiler.

Meta programs are really slow (and will be for a while)

Meta programming, in any or its various forms, allows the programmer to solve a classes of problems in a general way. Unfortunately these general solutions require more variables to be managed by the program and more run-time is consumed. From my experience each level of meta programming decreases your code size, but increases you run time an order of magnitude. To help reverse the efficiency loss you must cache and hash partial solutions; effectively reverising your gains in program size.

This means that the language you choose will have little effect on the development time of a multi meta program. The efficiencies gained by a particular language (either expressiveness or raw speed) will be dwarfed by the optimizations that will be required.

Compilers could solve this problem, but are not yet smart enough (or do not have enough information) to help with these cache and hash techniques. The most compilers offer is plethora of optimization variables that have to be considered by the human programmer and adjusted.

Since interpretation is a form of meta-programing we can see it is slower than direct execution. You can also see a compiler as a caching technique (machine code is cached) to increase performance. Note the size and complexity of a compiler is greater than an interpreter, but the actions are essentially the same.

Recalling my point about needing a compiler/interpretor (above), I have spent more time caching and hashing than making a compiler.

Reflective Systems are Fragile

The majority of difficulty I have had with building a reflective system is dealing with the implicit recursion that results. It would be a little help to have a tool that identifies cyclic dependencies in your code, but that will not solve the problem with identifying the bases cases that can result. This high chance of introducing a bug is one aspect to how reflective systems are fragile.

Another frailty is the fact that changing one part of the system can lead to dramatic effects elsewhere, especially if the change has a bug in it. I have almost no idea how to prevent this. One solution could be to establish versions of itself. This would be good for debugging, but would not eliminate the existence of a covert bug that will only show itself months later. If old versions were to act as backups, they would have to held forever; for contingency.

The combined effect of these two frailty aspects (the high chance of introducing a bug, and the high sensitivity to bugs) makes reflective systems difficult to build and maintain. It may be a long time before we build the tools that allow us to solve these problems. Until then, the reflective core of any system will have to be maintained by an elite few, and managed much like any software product is managed now. It will have to be released in versions, with upgrades that convert existing user code to new standards. Any reflective system will not appear dynamic and user-interactive until these frailty problems are solved.

April 2002