Testing compilers and business applications – what is the difference?

The value of “programmer tests” is far from being a consensus. I am not even talking about Test-Driven Development here, just the practice of programmers building up a suite of automated tests that validate a piece of software that they are working on.

By the way, I am totally on the programmer test bandwagon, have been since I read the “Test infected” article by Beck and Gamma back in 2001. When I see luminaries in the software development field dissing programmer tests, I am totally baffled.

Let me try an analogy. I wonder if they would also question the value of a good set of test fixtures for a language implementation (be it a compiler, interpreter, virtual machine, what have you).

…when I see some household names in the software development field dissing programmer tests, I am totally baffled.


Anyone that has ever set eyes on the codebase for a compiler or interpreter for any programming language must have noticed a large set of test programs that the compiler development team maintains to ensure the compiler is correct, exploring how the compiler or interpreter handles multiple variations of features in the source language. Often the tests not only look for pass/fail outcomes, but inspect the result to ensure the expected output was produced. This is not a modern practice – I am going to guess that it has been the common practice since compilers existed (that is more than 60 years now), because it is just too obvious and technologically trivial to have been missed even in the first years of Fortran.

The way I (and many others) see it, any program for any domain is implementing a language. This language is made of concepts that exist in the domain at hand, and defines the valid ways you can deal with those concepts. For instance, for an ecommerce application, you have products, and a shopping cart. You can add products to a shopping cart, you can change the quantity for a specific item or remove it, you can empty the shopping cart, you can proceed to checkout (if your cart is not empty). That is a language, and the application implements that language, just like a compiler, or interpreter.

In other words, there is no fundamental difference in testing compilers and business applications.

Automated tests are analogous to test programs that you submit to a compiler or interpreter to test it. You play with some feature and see if it produces the expected results. You try variations that you suspect could be tricky for the language implementation to handle. You look for interactions between features that you suspect may result in some weirdness. You do that for every feature you can think of, from the most important, to the most trivial, and you can only be sure the language implementation is complete when, for every language feature, you have at least one instance of a test input that exposes that feature.

What you don’t do is tell your QA teammates to take your compiler for a spin and see if it works, or ship and hope your users will do it, and wait for bug reports for the many silly mistakes you will for sure have made.

If you are a responsible developer, you just don’t ship code to the poor users if you haven’t yourself submitted the code to a decent battery of test cases that ensure that, at least in the situations you and your team could think of, the program being tested performs as expected.

Also, writing programmer tests is a sign of empathy towards your fellow developers. When you add test coverage for a feature you implemented, you are also providing a safety net for your fellow developer teammates, which will be making changes to the codebase. By having a solid test suite, they don’t have to manually ensure not only the feature they are working on is correct, but that they haven’t broken any other feature ever implemented by the team.

…writing programmer tests is a sign of empathy towards your fellow developers.

Of course, programmer tests are not a guarantee that there will never be bugs lurking somewhere in the codebase being tested. Even the most used compilers for programming languages have bugs found by their users. You may still want QA in the team. And you will still get bugs reported by users. But QA, and of course, users, finding bugs should be the exception, and limited to non-obvious scenarios. [Of course, when you get a bug reported, you write a test case that reproduces that bug, fix the bug so that test - and all existing tests - now pass, and you just made your test suite a little better.]

Email this to someoneShare on FacebookShare on LinkedInShare on Google+Tweet about this on Twitter

Poll results: languages and frameworks for developing business applications

A couple of weeks ago I started a poll asking the following questions:

What are the best language and frameworks for building business applications? And why?

The poll was promoted on Twitter, Google+, HackerNews, Reddit, DZone, a couple of language-agnostic forums on LinkedIn, and of course this blog (which is picked up by the Planet Eclipse aggregator),

There were 46 responses, which is not huge but is a decent sample size. I am no statistician, and I am sure the research has many flaws. But I think the data is still interesting.

Languages: Java against the rest of the world

I know Java is a pretty popular language, and even more so for business applications. But I did not expect to see the following picture:

Java got almost half of the votes (20 of 46). Five languages tied in second place, with only 3 votes each: C#, Ruby, Scala, Python, and of all things, C++.

C++ would seem a tad too low level for developing business applications, so it was a bit of a surprise. However, it is one of the oldest of the languages mentioned, it has a mature library/framework ecosystem, and it was the most popular language before Java became mainstream (considerng only those mentioned here, which does not include C), so I guess that explains it.

Another notable aspect is diversity: 12 languages divided the 26 votes that didn’t go to Java, with each language getting between 1 to 3 votes (which would be enough for a second place). This level of diversity was something I expected and actually hoped for. I tried to emphasize that this survey was about understanding what makes developers personally like in languages and frameworks, as if it was only up to them to choose, no previous experience being expected, and even if they had no hope of ever being able to use those languages in their jobs. But given the number of responses for Java with Spring (see below), which have been the status quo for 5-10 years, I think I could have done a better job at that.


The table below shows all framework responses, lumped together by language:

For languages other than Java, this is not statistically relevant, but I hope it gives interesting pointers for people looking to play with the less mainstream languages. Also, I am not sure what RoR is doing there on the Xtend line, but hey.

Since Java got the majority of the votes by a landslide, an analysis of the numbers for Java frameworks exclusively is warranted.

Spring’s dominance is impressive, but not totally unexpected. Spring is more of an umbrella brand, covering frameworks for all sorts of aspects of an application, so it is very likely any Java developer building a business app to be using at least one of the Spring frameworks.

Other than Java EE (which if lumped with EJB would have 4 votes), the remaining got one mention each. Again, definitely not statistically relevant, but if you are a Java developer, maybe take a look at those you had not heard of before.

What drives developers?

The chart below shows the most common reasons respondents provided for their choice of favorite language/framework:

Responses were broken into Java (20 respondents, in blue) and non-Java (26 respondents, in green) – just in case it would give some insight into how Java and non-Java developers may value things differently (I took interest into the “Community” and “Maturity” items). The overall averages would be in between those bars. Finally, note that Clojure, Groovy and Scala were counted as part of the Non-Java camp, even though they are JVM languages.

The following options appeared only once in the responses: “Robustness”, “Scalability”, “Want an excuse to learn it”, “Cloud deployment”, “Integration”.

I expected “Want an excuse to learn it” to be more popular – given how this survey tried to get people to take chances. Most of the motivations chosen seem to belong to the “professional software engineer” mindset, as opposed to the “passionate programmer” viewpoint I was hoping to encourage.

Note that except for “Cloud deployment”, “Ease of deployment” and “Integration” (the last two originally phrased in a more language-specific way), all motivations above were suggested options, that respondents could just check. In retrospect, one major omission in the suggestions was Portability. I would guess many would have picked it, as most of the languages chosen are historically available on multiple platforms.

Motivations per language

Finally, I will leave you with one last chart.This one gives you an overview of what things are appreciated on the basis of the language (and its frameworks):

There is little significance in this data though. For languages other than Java, the sample was too small.

What are your conclusions?

You are encouraged to take your own conclusions and post them on the comments section. I had fun doing this little exercise, and am thankful to all those who took the time to respond.

For me, my personal conclusion is that there are quite a few languages people have been using or looking forward to using. But it seems that Java will continue to be the safe choice for developing business applications for a long time. It would be interesting to see what a similar survey would show 5 years from now.

Finally, if you want to try to dig something else up, the raw responses (minus identifying info), consolidations and charts are available in a Google Sheet.

Email this to someoneShare on FacebookShare on LinkedInShare on Google+Tweet about this on Twitter