Subject: Re: Common Lisp programmer productivity From: Erik Naggum <erik@naggum.net> Date: Mon, 11 Feb 2002 09:22:19 GMT Newsgroups: comp.lang.lisp Message-ID: <3222408140336006@naggum.net> * Software Scavenger | I think most of you agree that learning Common Lisp and using it for real | work should generally make a programmer more productive at that work. It depends largely on the goal of the programming task. If the goal is to make something "work", you grab the tool that is already almost there and just whip something together to get all there, and Common Lisp is not "almost there" for a large set of common tasks today. (Those who want Common Lisp to be "almost there" with a huge lot of stupid tasks go off to whine about libraries and small syntax issues and go create Arc or whatever, instead of seeing that being "almost there" is not an asset at all, because the more "almost there" you get, the fewer things are within reach at the same distance, and those things that are much farther away than others are curiously considered "unsuitable" for the language once it has succeeded in getting close enough for comfort for other tasks.) If the goal is to make something of lasting value, which does only what it is supposed to do and does not fail randomly, but has tightly controlled and predictable failure modes with graceful recovery, then most other languages and tools are so lacking in support for those problems they were not specifically created to handle well, that _they_ are unsuited for top quality software and can only achieve this at great expense. This is where I think Common Lisp really excels. Those who argue that Common Lisp should be "almost there" for a lot of common tasks, would like people to learn to use it with simple tasks and then move on to more complex tasks. I have never seen any supporting evidence of this transition, and much evidence of the contrary: That if you use some tool for the simple tasks, it becomes so much harder to write truly robust code and solve complex problems. I think this has two important reasons that have to do with us being human beings: (1) If we have too much comfort and predictabilty, abandoning the comfort zone is very hard to do and we lose the predictability in an area where we think we have sufficient expertise to predict most developments, and that means that if you are good at slapping things together, you will not have the experience to feel comfortable in writing robust code even in you "home language". (2) Learning how to write robust code and solving complex problems is a difficult process of its own, and _habituality_ dictates that the languages and tools we use for such purposes should be very different from what we do for simple tasks, like the difference between cheap ball-point pen used to jot down a shopping list and the calligraphy equipment used on diplomas, invitations, or like the difference between a newspaper on recycled trash paper that yellows and crumbles in three months and a book on acid-free virgin paper that is expected to last. I think we do not use the same tools, languages, and equipment for that which we want to last and that which we want only in the short term for accidental reasons: I think they communicate intent simply in that choice: Using a calligraphic pen and expensive ink to sign a check in your grocery store would be way too weird. For Common Lisp, this means that writing small programs that solve common problems is mere "training" for the real thing, getting used to the language, experimenting, etc, and that is all well and good, but it must be recognized to be "training", not the end result and purpose, like it is with crud like Perl. So while it is crucially important to be fluent in Common Lisp, it must have the purpose of being used in programming tasks that are more complex and more difficult than what you can do with "almost there" tools. Looking back at my varied carreer, I see that I have always been drawn to these kinds of things: Ada, SGML, Common Lisp; I recognize that I am not a "tinkerer" who is satisfied with all the one-liners that stood between me and some "really" interesting problem, I took the problems I faced seriously enough to commit myself to solving them for good. This is why I love Common Lisp so much -- it lets me take problems seriously and solve them seriously. (As for the Open Source element here, I am quite happy about "sharing" bits and pieces, but I invest too much in what I really want to do to want to give it away, _especially_ to "tinkerers".) | But the big questions are how much and how fast. In my view, the big question is how. Common Lisp can make programmers productive in a very different way than, say, Java can, because when faced with the same superficial problem, Common Lisp programmers see a different problem than Java programmers, and therefore very different solutions. I think Common Lisp programmers think more of creating a system in which the problem is solved, while Java programmers think more of fitting the problem into the system of solutions provided by their language. This means that a Common Lisp programmer would generally spend much more time thinking about the problem and its solution than a Java programmer, who applies his restricted "solution world" to the problem instead of applying his intelligence and knowledge of the real world to the problem. In that thinking phase, a Common Lisp programmer is much more likely to make serious mistakes, go down dead ends and bakctrace, etc, than a Java programmer would. In the end, I think Java is so hard to program in that a Java programmer and a Common Lisp programmer will emerge with a solution at the same time, but the Common Lisp solution will provide the means to solve the next hundred problems in the system with very little effort, while the Java solution will require repeating the same development process for ever one of those problems, and may find that they made serious mistakes and went down dead ends _much_ later than the Common Lisp solution did, leading to full reimplementations in the face of failure to predict future demands, while the Common Lisp solution would have been much better thought out and have prepared for those future changes. In summary, the first solution in Common Lisp and Java will probably be completed with differences in implementation costs that drown in the statistical noise, but the value of Common Lisp will show over time, with te changes that are required to keep it working well, with changing ideas about what the system should "really" be solving, etc, because the design has been made with an eye to systems-building. | If you spend several years gaining CL experience and building a personal | library of CL stuff, what are the productivity results likely to be over | those several years, and how fast are those results likely to be | changing? And how big a factor is your personal library, e.g. your | functions and macros which you reuse in different applications? I think each area in which you work requires its own libraries of such stuff, and that the "almost there" property of, say, Java, is that it is such a large common library of stuff that its immediate appeal for those tasks is quite strong, but every business that bases its operation on a large system of solutions, will bring much more to the programming task that is unique to that business than it can accept from the outside, anyway, so there is a certain balance that can be struck here. I think Java has some really huge advantages over Common Lisp that it would be completely futile to fight, but Common Lisp also has some amazing differences from other languages that Java in particular has no chance ever of adopting. The question, then, is whether you model your programming tasks according to what is easy to do in Java, or whether you focus on your business and user needs and develop a system of your own. Please note that the preceding supports, e.g., Franz's pricing model. However, when it comes to building a large software system today, we have a serious problem: (1) You cannot build a large system from small pieces without getting into a huge logistical problem. (2) People only know small pieces, and have experience only with small pieces (which you can see immediately if you try to study some of the larger Open Source projects). So, (3) arguing for tackling the inherent cost of complexity up front is futile with today's managers and programmers because they are simply ignorant of complexity theory and even management of complexity. This means that a Common Lisp system in our time must start off as some silly component-based crufty glued-together crap, and while Common Lisp was exceptionally good for prototyping in days when billions of lines of code were not readily available, it is not good for prototyping today. That is, we must learn from building far smaller systems before we can embark upon the large ones, for no better reason than that we have lost the skills necessary to run large software projects. (Industry reports seem to indicate that the rate of software project failures has been increasing and that this is one of the reasons for the negative trend in new software development projects.) So how do we _start_ using Common Lisp today when the cost of smaller projects is relatively higher than smaller projects in "almost there" languages, when most projects are believed to much smaller than they actually _should_ be? | I know these questions are too vague and general for good specific | answers, but I would like to get some discussion of these issues and | whatever vague hints, anecdotes, etc., might throw some light on such | issues in different ways. I hope the above has been useful. /// -- In a fight against something, the fight has value, victory has none. In a fight for something, the fight is a loss, victory merely relief.