Why lisp is *Unpopular*

Lisp is a great programming language which has lot of power; however it is far from being most popular programming language. If you see lisp from a distance you will find:

1. Orthogonal syntax

2. Very quick to learn the basics

3. Very power full when it is used in high level project

4. You write less code as it has most powerful macro system

But still lisp is not the language of choice. It is a fallacy of programming world. Why? This question has stumped me number of times.

This post is my attempt to understand why it is so.

Prefix notation

Lisp is a programming language where prefix notations are used extensively. Prefix notations are good for computer but not for human being. From very old age human civilization and its mathematics is built on top of infix notation. We comprehend mathematical expressions in infix. You can argue with educational system but you cannot ask to change the year old mathematical notation to help a programming language to be understandable. That is simply not possible. This seems to me a perfect road block for lisp and it has to survive with that.

It will not be out of the turn to tell you a story about my friend. She was looking at Practical Common Lisp, and told, “How do you write 4*3+6 in common lisp?”

I replied (in writing), “(+ (* 4 3) 6)”.

She looked at it and closed the browser which was rendering “Practical Common Lisp”.

Mathematics behind lisp

Believe it or not all programmers do not like mathematics. They lie in the pantry, in conferences, in blogs. But the truth is a handful among us really like mathematics and care about it. Moreover it is not necessary to have deep understanding of mathematics to develop programs, or to be a programmer for that matter.

But lisp is an exception to this idea. To have a fair idea about lisp you have to understand some mathematics, like higher order functions. To understand functional programming paradigm one have to devote some weeks to lambda calculus. If one does not understand this he can write programs in lisp but it will be distant from the lisp style. The very idea of Turing machine and lambda calculus is fundamental to computer science but at the real world of software development those ideas do not cross our mind. To be honest with everyone I learnt C programming language without any knowledge of both Turing machine or lambda calculus.

There are self educated programmers who are very good in their trade and they care a damn about those theoretical computer science topics.

So lisp (and proper lisp style) is beyond many programmers and the target audience is less.

Programming language politics

Politics is pervasive; programming languages are not an exception to it. Big corporation look for development environments which are well supported, backward compatible, developers are available, etc. This is true in all large organizations or profit making corporations. When you decide about writing software you try to find out a middle ground. Lisp is powerful, 100 java average programmers = 10 or 2 great lisp hackers, you write less code to get more out programming language – all of them so very true. But when I was asked, “Show me one problem that I cannot solve with Java/Ruby/C++/Python?” I knew as all of them are Turing complete, so there is no problem that cannot be solved by Java/Ruby/C++/Python.

Although you have to learn more lexical separator in those programming languages than lisp still lisp looses. This is *PLP* - Programming Language Politics, we have to live with it. People will reject our ideas of development in lisp and the reason will never be purely technical; rather techno-political.

Code – data Equivalence – Hard to understand

I said, “In lisp code is data and data is code”.

Ron said,”What?”

People do not understand code data equivalence, because they are tuned by other programming languages. Those programming languages are their bread and butter so the immortal suggestion of “Unlearn “is “Unreal”. This is the bottom line of any lisper. This is *THE BASIS* of lisp macros. And it is so away from all other brick-mortar programming languages that no one understands its power without coding for many months and many macros. I got the feel of the matter when I read first ten chapters of Paul Graham’s “On Lisp”. This adds to another stiff cliff in any newbie’s learning curve.

Macros

Lisp macros are different beast, a beast when under you command do wonderful job for you. As any other programmer, especially *I know all* programmer like me, need some time to understand that lisp macros are really different.

It took me couple of week to realize that, when I realized it I felt like a stupid school kid with all red score card who has been slapped by his father for his exceptional result. Then I understood,”Oh! Yeah lisp macros are different.”

It opened my mind about programming as whole. Like when I learnt lisp – I learnt map, list, mapcar and all other cool lisp techniques. Before my lisp journey I was struggling with C++ STL concepts, after I learnt lisp all the things fell into place.

But the point here is that lisp macros are the last challenge that one has to understand (if not master).

Run-of-the-mill programmers

In many places it has been mentioned that lispers are not *Run-of-the-mill* programmer. They are different (in some cases arrogant), may be programmer elite. But popularity of a programming language depends on the acceptability of the language not on its power. Natural languages evolve out of existing languages to a simpler but verbose form. The same is not true for programming languages. There is no programming linguistic tree which has a common root node for syntax or semantic or paradigm. Lisp’s elite status may be the singular disadvantage to the general programming cloud.

We all play the role of *run-of-the-mill* programmer in any given project while it comes to coding and debugging. So *run-of-the-mill* programmers are necessary for any project to succeed and sustain. If your target is not *run-of-the-mill* guys you are bound to lose one day or the other.

Undermining Microsoft Windows

Not many lisp software works properly in Microsoft windows. The lion share of desktop application runs on windows box, but many lisp apps do not work.

Lisper world over has to take up the challenge to support Microsoft version of all the application that they are going to release. The word support can be even this one liner: “I do not care about windows. If it works fine else just do not disturb me.” At least some who cares about Microsoft compatibility will not use your stuff in his software.

Undermining Microsoft is bad, because Microsoft to computers is like photosynthesis to trees – it is a necessary evil.

Imperative Programming:What is *Wrong*?

What is *wrong* with the imperative programming? That’s the question that came up in a discussion where I was part; the dicussion group comprised of people from diverse software background and most of them has their lineage to C or C++. The points that came up during that discussion were well spread over the day to day activities of the programmers working in various types of projects.

Actually if you see from a distance it is absolutely ok to write a piece of code in your favorite imperative language, but if you take some time and code the software in your favorite functional programming language it feels and looks better; of course your choice of functional programming language has to be good.

Side Effects

Side effects make lot of problems while coding in imperative languages. Say you have a program that runs into several functions and as imperative language like C does not support higher order functions (in true sense) your functions are dependent on what we know as *GLOBAL VARIABLES*(to preserve the global state). Lets assume that you set an error flag if something has gone wrong in some function. As each function is interlinked in producer-consumer relationship with each other, once a function has done its job you have to see whether there is any error set or not. So your code will be sprinkled with codes like this:

i = fn1();

if( !errorset())

fn2();

This is not only bad while typing but also an example how a programmer loose his ability to think. How? As our example goes if you do it in several files and several functions; one fine morning loose the intent to think what may go wrong in simple function which is just checking whether a file exists or not! And I can tell you this happens because this happened with me.

Other than this, think about the time that I spent error handling. People might say that this is because you are using a programming language which does not support better error handling mechanisms to help the programmer. I agree, but as we know the imperative programming languages will always depend on Hoare Logic, i.e. it will have the notion of pre and post conditions. If we see both the paradigms of programming (lambda calculi and Turing machines) depends on those conditions but pure functional programming does not have side effects as it does not change (although lisp programs can do that) the values of its parameter.

But when I code in lisp I know that if I need an output from one function to another I will using the first one as a higher order function to the later. Then the second function only need to know what it expected out of the calling function if something goes wrong in the higher function it can handle it then and there and end the execution flow gracefully. This brings me to next topic execution flow.

Execution flow: Where am I?

Yup! This is a nightmare for any programmer in the imperative programming world; a bunch of programmer I knew once wrote a simulation program for Intel 8085 microprocessor but after each cycle of execution they didn’t know from where the program exited. It used to exit with unhandled error each time in Windows. If you think who they were I can tell you I was in that group. It took us some effort to fill that gap.

Now you might say that boy that doesn’t happen in real world! Oh yes that does not happen but we always have way to complete the process and most of the time the process is return-return-return till you come to main and then do simple check up and clean up of your mess with memory and then die by returning 0. No problems with that but think that if an execution starts and you know from the first command line checking itself that the user has made a mistake but you still follow your way of closing after eating some good processor time.

But in lisp you can’t loose your track! Even if you want you can’t, simple. How? If you are creating a properly designed lisp code with functional style you will be calling functions on top of one another so it will go on till the boundary of your hardware! It may fail some times but you will not be loosing your track. Even in error without any extra effort from your side you will be getting messages like: “Debugger invoked on ….”! I say, it’s just cool.

LOC – Yeah the greatest enemy of code!

I think Steve has given enough reason in his post so I need not prove this. But yes if I look at my C code some times I cry! How much I have typed for so little. Take the famous, old and boring “Hello World” program. In C you have to type at least 5 lines. In lisp REPL it’s only “Hello World”. Even a fancy function will be 2 lines. So you save 60% of typing effort while working with “Hello world” program.

And as the saying goes with lazy programmer I feel good while I use my “TAB” key for auto complete in XEmacs.

It happens especially if you need to work with raw data. By the time I finish structure filling code in C, I bet will be able to complete a subset of other tasks like some parsing also.

The crux of the matter is doing more out of less. If this is not good enough for you rethink about becoming a programmer.

Macros

My first impression about C macros was, “Here is subject which I need to learn because I really do not know what is wrong with my code when I use them.” The best macro I have used in my C code:

# define PRINTLN printf(“\n”);

And believe me it worked wonderfully for me. People in that discussion never understood what I was talking about when I said “function returning macro” or “code writing macros”. Because they do not know what a power a macro can have if you are programming lisp.

(I can go on talking about macros but I should refrain as this is not a sub-topic but requires major effort and can span several posts.)

Logical Nirvana – How Programming Languages and Our Thoughts Are Linked

A programming language can change the way you think about programming; this is true but not the whole truth. Most of us when we code we try to think about a problem and we try to figure out how to solve the problem in a programming language. I do not want reiterate the same example of how BASIC does not allow you to think about recursion blah, blah, blah; but what I want to point out here is that at the time of solving a problem we actually solve the problem in our mind first and then we start coding. At the time of coding we start thinking in a programming language not before that. If some problem can be solved by a human being it can be solved by a computer. It may be tedious to put the logic but it can be done. Like as we have programs which can play chess better than most of us. So this simply means we just map our logic, which we develop in our mind, to a programming language.

Now this mapping requires us to translate the problem in a computer language. Now for example if we need to do an operation which is very low level we can do it very easily. But if the operation is complex we need to do a lot of sub operations, so at the time of translating the logic to a computer action we need to give bridging of more fundamental statements.

Now how a programming language can help you in doing this mapping? Simple it will provide you operations which are close to your logical system. It might not match the exact logical unit that you have in your mind but it can significantly reduce the number of statements that you write in that programming language. So if a programming language allows you to do some job in 10 statements and another programming language allows you to do the same job in 3 statements then it is better to use the second programming language. Plain and simple! But this is not so simple in this world and that’s why we have programming languages which require at least 4 lines to print “Hello World!!!”, moreover those programming languages are so called *MARKET LEADERS*.

Now let us return to our discussion on programming language and thinking process. The problem of mapping get worse if you are working in a programming language which is vary far from your solution in your brain; then you have to start from your end (Top-down approach) or from the programming language end(bottom up) and bring these two things to a single point and then it should work. What we are trying here is building of higher order logic from existing logical system in the programming language. So end of the day we are doing nothing but mapping of two logical systems.

Here I feel that the greatest realization of every programmer is what we are trying, while programming is not only solving a problem but also understanding our logic; the logic which lies beneath our mind. So the question is if a programming language does not allow us to think beyond certain point should we continue with that or let our thought go to wildest of imaginations where the boundaries are endless with a programming language that has no limitations.

So let’s reach out for that logical nirvana.

Thanks for reading.

Evolution of Programming

The RISC architecture is good, as they say anything small and working is good! This is my fellow programmers is a philosophy of *nix*; how many years has gone by? Stupid thirty years and do not question it; because that’s the best idea we have got and we are following it. Doing great stuff, building systems on top of it, I agree. But has it ever came to your mind that why this is like THIS? Why we have to use the same old RISC for our computing? The answer is we have stopped innovating in the real sense. We now know that we need more data types, arithmetic operations and better instruction set to carry out an operation. But at the end of the day all we have is a *MOVE* or *STORE* or *LOAD* and we say that Lisp, Java or Ruby runs slowly. These programming languages are not close to low level and thus will take some time to actually execute it in our cute processors.

Tomorrow if we try to develop a programming language which is more higher level we will not be able to convince our bosses that this is the right way of doing things. Because they will tell us, “look at the performance man!” And an innovation will die. Once upon a time C was considered higher level language than machine language, yes true but you need to write a 20 lines of code to reverse a link list or worse link list is not a basic data types.

I do not blame any one but tell me is it fair to ask a student of computer science to write a link list reversal program in Java. I don’t think so. If you can declare link list as a data type and you have a method to reverse it please reverse it using that; and if your computer is taking more time to do that for link list of 10000 elements blame it on the computer not the programming language even if the C/C++ implementation is taking less time. This is because of the abstraction that this high level programming language is providing. C/C++ on the other hand is good because there is very low level abstraction in it and you have to build rest of the thing.

In programming world this is like saying to the programmer it’s my way or highway. So if the instructions set are evolved up to the level of say C then what happens? The magic starts then you can still program in C as now you still have a choice program in assembly language; but it will be better to choose a higher level language.

We have come from binaries to pneumonic to assembly, but we are long due for the next higher level let us just take the next step, and which programming language we should choose is up for debate…..or is it not clear?