Subject: Re: Lisp's future From: Erik Naggum <erik@naggum.no> Date: 31 Jan 2004 01:48:29 +0000 Newsgroups: comp.lang.lisp Message-ID: <2004-031-ATW-KL2065E@naggum.no> * Erann Gat | What this debate is really about, it seems to me, is no different | from the old deliberative vs. reactive debate that took place in AI | robotics during the 80s. This debate is really about whether the act of programming produces or consumes information. Of course it does both. This odd discussion is similar to the age-old discussion about how to write an essay. Some people believe that you should think carefully about everything, construct a skeleton and a plan and proceed to flesh it out, in the belief that you should have all the information you need to write the essay up front and that the writing itself should not produce more information, indeed that thinking of something you had not thought of to begin with is some sort of failure condition, like lack of planning or lack of concentration on the task at hand. Other people believe that formulating an idea in words /creates/ the idea and that you should only define a purpose, a goal condition, and then proceed to write your ideas down while monitoring that you are on the right course. (Monitoring is how consciousness works to begin with and with some training becomes automatic and emotional: you feel good when you go in the right direction and bad when you go in the wrong direction. People who plan compulsively are usually the kind who think that emotions have no business in directing attention or focusing effort.) Creative people also argue that premature planning solidifies unformed ideas that should be expressed fully if they are allowed to form at all, because almost all ideas appear good until put to the test, and the best test is not to think about what you would do, but to actually do it. (The word «essay» itself is curious -- it originally meant an attempt at expressing something, but the written attempt has become such a standard model that the word has come to mean a particular style of exposition and the history of the word is all but forgotten.) Most of the arguments in favor of one or the other of these positions (when both are clearly necessary and correct) are made for the sole purpose of pushing a pendulum that has swung too far in one direction back in the other direction, so the context of each argument is that the antithetical position has become too strong, for whatever reason and by whatever measure. We have not arrived at the stable position that is satisfactory to all parties, so we keep pushing the pendulum too much in our direction when we perceive that it has been pushed too much in the other direction by others. In all likelihood, the stable position is one where different people exhibit different capacities and tendencies and where the belief in one answer has been replaced by a more pluralistic view. In my view, the planners are not only wrong, but so averse to feeling bad that they prevent themselves from feeling good about what they do and therefore have no direct experience of the pleasure of thinking properly and so resort to their plans to avoid the pain of realizing that they do not think well. Herbert Dreyfus has argued that it is extremely important to be emotionally attached to the success of one's work if one is to progress beyond advanced beginner to competent or even proficient, and point out that students who work very hard to avoid feeling the pain of failure never become competent enough to feel the joy of competence and success. It is no accident that those who recoil from the pain of failure are those who are most likely to ask for «rules» or reach for the «answer book». Some people are very good at predicting what will happen and plan for both the likely and the unlikely, but then become nervous wrecks when their planning turns out to be faulty and they have to re-plan, which they resist until it is already too late. Others are not so good at prediction and planning, but are very good at setting goals, and they try things out, compare it with their desired results, learn from each failure, accept that neither other people nor the world in general are obliged to do their bidding, and they never become nervous wrecks in the face of hardship, even disasters, but frequently set new goals as they find that some goals were unreachable at a reasonable cost. In the real world, it takes all kinds to make a good society, and the only track record we know from people who believe that only one kind of person should exist is incredibly murderous tyrannies. A lot of (dysfunctional) people believe that all those who do not obey every rule should be punished or killed, and they spend their time blocking other views, with such phraseology as «unconditionally bad» about them while they misrepresent them in order to destroy the ability of others to present them in a rational discussion. Typical behavioral patterns with these people include hostility towards any violation of their plans and rules, and when they see some person doing something they do not like, they believe that the proper course of action is to do what they do not like on purpose and in some demented idea of retaliation and retribution, even though they are never directly affected, instead of working to rectify or improve the global situation. Breaking rules and upsetting plans cannot be mistakes to these people, only willful acts of destruction, which only severe punishment can cure, and they are never sufficiently humble to even consider the possibility that they are the first to break any rules on purpose. People who plan too much are also unable to enjoy or cope intelligently with surprises, for any surprise is evidence of their insufficient planning. The important distinction between planners and creators is whether an action taken for a given purpose does or does not provide feedback to the decision machinery that determines the next action. Listening to feedback is not an option if you want to retain your mental health, but listening to feedback will frequently mean that any plans you had must be discarded. The more energy you have poured into planning, the less likely you are to listen to feedback that it is junk. If you make up your mind that the world is different than it is, that it «ought» to be different, you will run into psychological problems, the larger the more you insist. A lot of people believe that standing fast in their beliefs is the hallmark of integrity and that it is the duty of all people of true belief to force other people to obey their beliefs. These days, we call them fundamentalists and terrorists, but in the world of software development, we have no such useful labels for those of deranged psychology and highly irrational beliefs, but they are nonetheless present in the industry. The most effective test they can be put to is whether /all/ actions taken by programmers are able to provide feedback to the planning and scheduling of the whole project, or only /some/ actions (of the selection of which they demand to be the final arbiter). If /all/ actions taken by programmers are used for feedback, the best way to view the act of programming is that of producing information about the problem to be solved. Some of it may be wrong, some of it may be of no use, some of it may be unexpectedly true and need work to be proven, but all of it is used as feedback. This is obviously not the dichotomous relationship with planning that some people think it is -- if you are both intelligent and of sound mental health, you do not plan your actions, you plan your results and the acceptable costs, and you trust both your own competence and that of others in actually accomplishing those goals without detailed instructions. Some of the time, however, an action may produce information that will affect the results and the costs, and it is at this point that a rational manager listens and an irrational manager ignores the available corrective feedback. When all actions are used for feedback, the consequence of making mistakes will be a corrective and appropriate response, because everything everybody does /matters/. If only /some/ actions taken by a programmer are used for feedback, the result will necessarily be that programmers will perform actions that are not used for feedback with reckless abandon. In this regime, the already tyrannical manager will write ever more detailed plans to enforce his model of reality, the stronger the more wrong it is, just like people who have to be taken care of by society in general because they have stopped accepting input from the real world. In business, however, those who keep pushing for something they believe in tend to be remembered when they have had a string of successes, but there is now a growing literature on the stupidity of the highly intelligent in both managing their lives and their money. (E.g., Why Smart People Can Be So Stupid, edited by Robert J. Sternberg, ISBN 0-300-09033-1.) It is quite possible for people who watch others who have succeeded by pushing for something they believed in to believe that it was pushing for something they believed in that produced the success, but in logic this is known as the post hoc ergo propter hoc fallacy. In the real world, pushing for something you believe in generally causes massive and lasting failure, but if you accidentally get it right, pushing may magnify the success. This is like performing an objectively insane action which by sheer accident happens to associate itself in the mind of the deranged with some random success condition that has nothing whatsoever to do with the action. You could have worn the same tie the last three times you won a contract or did a good presentation, and then suddenly it becomes your «lucky tie». Lots of people do this, and such things as astrology and numerology or the importance of winning New Hampshire because past presidents did, find a receptive audience in the highly unscientific population. The more /selective/ you are in the feedback you accept, the more insane your reasoning will become as you will necessarily reject corrective feedback that would have led to better reasoning. The point at which it becomes rational and productive to impose one's will on reality or other people is perhaps the most interesting aspect of the philosophy of ethics. It is not understood to any significant degree how to determine this point in advance, so ethical thinking is best thought of as a systematic approach to respond to objections, but this imposes a requirement to object, and to do so rationally and only constructively, which is not as well-defined as it ought to be. The history of modern societies provide us with the evidence that we have swerved aimlessly in this regard and are both unsystematic and highly irrational in our objections and the expression thereof, especially in cases where the severely misguided notions of «good» and «evil» are involved, and people conjure up feelings of suffering «injustice» instead of coping intelligently with the situation at hand. There are people so devoid of rational ethical thinking that when they observe something they object to, they do exactly the same thing on purpose in order to express their objections, and then fail to understand that they have done something objectionable. Human rights, for instance, is an expression of a fuzzy line of demarcation around an area where people are not allowed to impose their will on any other person for any reason whatsoever. The U.S. Constitution was also thought to have protected certain rights even under serious threats, but turned out to be completely irrelevant to the American people. Social security is an attempt to draw up an area where every citizen must accept the cost of sustaining other citizens who are unwilling or unable to sustain themselves, and those who object to this system tend to ignore the fact that the absence creates slavery and people so desperate that no law can bind them. The political debates are mostly about which areas of public and personal life must yield to the imposition of the will of others. This is far from the obvious and well-known that the mark of the highly irrational and tyrannical pretends it is: /Nothing/ is «unconditionally bad»; the will of some people who have made up their mind about both reality and people /must not/ be imposed or accepted to be imposed on others to the detriment of the ability to object both rationally and constructively. Being intelligent and reasonable human beings with the particular kind of brain that we possess, we are still not only fallible, but very far from omniscient. We are also naturally inclined to commit a fallacy known as conformance bias, which makes us want to prove that what we already believed was true and to believe faint affirmations even in the presence of rejection. Another dangerous problem with puny human brains is that counter-information tends to be rejected because opening every decision up for re-evaluation would consume infinite resources, and it also takes too much time and energy to re-evaluate enough to be able to know whether some counter-information is important or not. For these reasons, we have set up the extremely inhuman methodology known as «science»: Figuring out that a particular causal relationship is for real independent of our desire for it to be is really hard work. Most people much prefer to believe that what they would like to be true is also actually true than to do all the hard work to determine what is and what is not actually true, not the least because most of what people believe is actually dramatically false. Given what we know about human fallibility and society-wide myths and beliefs, it should be obvious that we simply cannot afford to be selective in what feedback we are willing to accept. Every single relevant, corrective feedback we refuse to consider will come back to harm us. So, too, with every rational, constructive objection to what we have done, but those who are least able to accept corrections are unsurprisingly also those who are least able to express corrections and objections in a way that other people can make use of, and they are the first to jump to conclusions about other people's motives and rush to accusations and all sorts of counterproductive idiocy. Instead of thinking when they meet corrective feedback, they defend themselves and stick to their plans precisely /because/ those plans are in danger of proving insufficient or wrong, and this is much more likely to happen if the plans have been very costly to produce. To strike a balance between planning and creativity, we need to zoom out and look at the available measurements of the success we have with our various approaches. To do this, we need to be aware of the goals we have for our work at a sufficiently abstract level. The more we plan, the more concrete our goals become, however, and the danger of planning to avoid costly mistakes in achieving our goals is that we no longer remember what we were originally heading out to accomplish. The most productive plan, then, must be the plan that is not concrete, but abstract, yet has definite test criteria. The distinction between a plan and a vision is therefore only in the availability of the tests we can employ to know if we have gotten where we wanted to get. This is the kind of plan that is incommunicable to computers at present, so we are in no danger of confusing a program with its intention, just as we are never in danger of confusing a test harness with the objective of the whole endeavor. The notion that planning at the detail level is a necessary product of good planning at the abstract goal level is itself a product of the kind of personality disorder that prevent creative people and experts from producing the desirable results. This kind of planning is only a good thing if you plan to hire monkeys who respond well to electric shocks when they deviate from their instructions. Human beings are not computers or physical devices and planning for intelligent people is different from planning for morons. Most planners also believe that they are vastly more intelligent than the people who are going to implement the plan in practice. More often than not, however, they are not noticeably more intelligent and the belief that one person can plan what another person should do is a very good way of making the person who plans unable to accept negative feedback from the stupider person who implements it and the person who performs the planned steps is unlikely to invoke their intelligence when the plans are bad, so excessive planning is a very good way of destroying productivity and /avoiding/ the abstract goals of an enterprise entirely. However, we must take the planners' word for it -- they could never employ their creativity and intelligence in the face of unexpected situations and their fear of failure is probably well-founded, so if a person demands detailed plans, we should feel a profound pity for their damaged psychology and inferior brainpower and help them write their plans and ensure that they measure only irrelevant properties of the system being developed so they are kept busy without messing up for anyone who have do to deal work. In this regard, we should go talk to the social science people, because they educate a lot of the bureaucrats who prevent politicians from doing more damage than they absolutely have to with their misguided micro-controlled plans. -- Erik Naggum | Oslo, Norway 2004-031 Act from reason, and failure makes you rethink and study harder. Act from faith, and failure makes you blame someone and push harder.