Subject: source access vs dynamism From: Erik Naggum <erik@naggum.no> Date: 1999/08/23 Newsgroups: comp.lang.lisp Message-ID: <3144404199547949@naggum.no> once upon a time, I was very much opposed to Free Software on a variety of political grounds. then I thought that it could do a lot of good, partially motivated by seeing a lot of problems that came with lack of access to source code. these days, I largely think both Free Software and Open Source are very serious mistakes, because they don't solve the core problems, they make them much, much worse. suppose one operating system (or language) lets you open an already open file by calling OPEN or whatever on the open stream. since this is clearly beneficial to anyone who can do it, let's assume people start to do it and demand the same functionality elsewhere. one easy way to do this is to use a function other than OPEN to do this, but this does not scale well: every request for a feature would need a new OPEN, and you would have to decide _which_ OPEN to call both in your code and in the implementation of the new functionality. clearly, this is a very messy situation and migration from one system that did support a feature to one that did not would be a function of the number of features people might have added. this problem would be solved if people had the source and could fix things themselves and then channel the fixes back to a central place that could "take over the world" with better versions. consider the Unix way, which used to have the function call DUP, but recently has /dev/fd/<N> -- you can fake /dev/fd/<N> with DUP, but to do that requires getting under the skin in OPEN and parsing a file name and all sorts of cruft that the file system code would normally do. ugly, but it would be necessary, right? the reason that source access is needed is the same as the cause of the problem: the inability to modify system behavior non-intrusively. if one could modify OPEN on any given system to acquire new behavior _without_ compiling a new OPEN or, even if you did, without re-linking the entire system (thus the need for dynamic linking and similar solutions, too), source code access would be entirely immaterial. so suppose you wanted to wanted to add DUP functionality when OPEN was called on /dev/fd/<N>. you could grab the old OPEN definition and call it anonymously, while redefining OPEN to be your particular function, which acted only on a match to such pathnames, or you could redefine OPEN once and for all and add your stuff via a well-defined hook system, or advice or whatever. whether you would try the new thing first or only after trying the argument the old way had failed would be an orthogonal design decision, subject to how easy error handling is and the expected frequency of failures. this way of doing things would in any case cause a few simple improvements over static source: (1) the core system could upgrade its OPEN without the need to migrate the changes. (2) if you did it right, you could also inspect, add, remove, and reorder your changes, and you wouldn't need the source for either of these operations, either. (3) bugs introduced by changes could be fixed the same way the changes were put in place, as opposed to rebuilding the system with the previous version of the dynamic library or whatever didn't fail. (4) you could communicate your desire for a system-wide improvement to someone who could put it in the official sources and you could add your changes conditionally. compare this to a world where no dynamism is available and people don't think in dynamic terms, anyway. you need to make OPEN change behavior so of course you need access to the source! you want this functionality everywhere, not just in particular programs, so of course you need the ability to modify it and rebuild from scratch! since this creates a maintenance nightmare, of course you want your changes to be part of the core system, too. to get that to happen, of course you need to be part of a huge movement where people compete with eachother for features they think they need and the best way to implement them. this is incredibly exhilarating, so of course this is the solution to all the world's many problems with missing features in software. suppose you drive down the highway and you suddenly want to go some nice place you saw just before you pass a forest. you veer off the road, plunge into the wilderness and promptly decide that you need four-wheel drive, a huge cutting device in front of your car, much better shock absorbers, a bigger engine that could actually run on swamp water instead of getting all drowned, and then need an amphibious vehicle to get across the river on the far side of the forest. you can't get all that right away after you made up your mind to veer off the road, and This Is Surely Somebody's Fault, so now you start this big protest movement to have auto manufacturers use Open Car Design by making your own cars and giving them away so people can modify them. the obvious morale of this story is that it would have been a little smarter to see if there were another way to get there over existing roads or ferries or whatever, even if that would cost a lot less and be faster if not Your Way Right Now, and which didn't need all this add-on stuff to your car, and perhaps you would just get there instead of getting all worked up about all the neat stuff that other people had done to their cars while claiming that if it weren't for Open Car Design, they would actually never have gotten where they are, and they sure wouldn't have built this very versatile car, which they talk about a lot more than where they have used it and for what. suppose someone adorns a commercial vehicle with a proprietary navigation system that let you discover any and all ways to get from point A to point B by talking to it, including scheduling and cost estimates so you could decide whether it was worth it, but refused to give it away to Open Car Designers. why would anyone who weren't more interested in building cars than Getting There insist on home-made Swiss Army Cars with Open Car Design when the _real_ goal had become attainable? why would Open Car Designers _not_ call the producer of the navigation system and find a way to deploy it almost everywhere? and would they insist that a home-made navigation system would be better than buying it from whoever invented it? although a ridiculous example, I don't see why anyone would insist on a way to do things that does not solves the real problem, but rather solve an ancilliary problem that is a problem only because someone is very stubborn. I keep wondering when and under which conditions the models employed by Free Software and Open Source make sense. I don't think it makes sense when the design is _already_ flexible and open and so malleable that it's used as an argument against it by the very same people who want to change everything, but only Their Way. in other words, if all you have is Unix, certain problems are so staggeringly tall that it takes an extra special genius to see what the _real_ problem is because all the people who are trying to climb it make it look as if doing something else is stupid. if you have, say, Common Lisp and good access to the exposable parts of the internals, with advice support, hooks, etc, why do you _need_ the source code? not that it wouldn't be useful, but would the demand be as great if the Smart Route to Yonder Place was at least available to intelligent people instead of requiring everybody to scale every mountain just to prove a point about inaccessibility? the problem is that scaling such mountains is much more fun than actually doing anything useful with computers. mostly, computers are used to do Really Boring Stuff, especially if you don't have any dynamism, either, and if you're young and getting into computers because it is supposedly exciting, you discover that writing quality software for real people is tremendously hard and the real rewards are pretty far apart in real life, so it's a lot more fun to spend one's time cutting another cycle off an interrupt handler or bolt another semi-dynamic feature onto a system as flexible as a block of granite. the solution is dynamic _software_, but Open Source (etc) is the best way to ensure that this never happens on a large scale, because a pale shadow of the dynamism people actually need is immediately achievable using the static model that people accept without question through fixing a source file and recompiling. ever better ways to overcome this inherent problem are created, such as Java and interpreted languages, dynamic linking and lots of nifty features. to help people _find_ the solution, however, we need to show them that the problem is aggravated by the way they think they are solving the problem today: by making the static software model even more entrenched and unmaintainable. I think Free Software and Open Source addicts will get down from their current high and become tremendously frustrated when the obvious problem of having millions of pieces of source code available and they actually can't move in any direction because the ability to make modifications that they have so lauded in the past turn into a very serious problem when much bigger problems require modifications to much larger systems than the ones people today think are usefully modified locally. I predict that "migrability" will be the one factor that causes people to choose dynamic software over static software, however modifiable, because while the software has been the most valuable for a long while (because of its cost of development), the data we stuff into the programs and ask them to safekeep for us will be so much more valuable than the software. static software is totally unable to deal with graceful migration and the need for the cost of changes to be linear in the size of the changes. however, writing "migrable" software is not a task for novices or people who are not fully aware of the impact of their decisions, and I think that too much access to source code will cause a lot of people to be exceedingly good at stuff that is antithetical to long-term solutions. it seems that programmers need to be made as aware of the problem of not killing the patient while fixing some problem as the medical profession, because as long as the static software world considers "crash and reboot" to be an option, we won't get anywhere worth going. Free Software and Open Source may well be the last gasp of the static software world and its "success" may well cause companies who believe in static software to fail while users betray their own long-term needs in exchange for the smidgeon of dynamism that source code affords. in the long run, software that adapts and can migrate from one half-expressed need to another as real users discover what they actually want, _should_ win. resisting the sometimes attractive non-solution of unrestricted source access and full modifiability may be the only way to get what we really need. #:Erik -- save the children: just say NO to sex with pro-lifers