cloudfier.com and abstratt.com servers moved to new hosting

The cloudfier.com and abstratt.com servers (including this blog) moved to new hosting. Please let me know if you find broken links or any issues. If you do, please try pinging the server you found it at. You should see something like this:

ping abstratt.com
PING abstratt.com (54.244.115.27) 56(84) bytes of data.
64 bytes from ec2-54-244-115-27.us-west-2.compute.amazonaws.com (54.244.115.27): icmp_req=1 ttl=51 time=0.860 ms
64 bytes from ec2-54-244-115-27.us-west-2.compute.amazonaws.com (54.244.115.27): icmp_req=2 ttl=51 time=0.992 ms
64 bytes from ec2-54-244-115-27.us-west-2.compute.amazonaws.com (54.244.115.27): icmp_req=3 ttl=51 time=0.848 ms
^C
--- abstratt.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.848/0.900/0.992/0.065 ms

If not, it is probably a temporary DNS propagation issue.

Authenticating users in Cloudfier applications

Up until very recently, Cloudfier applications had no way to authenticate users – there was a login dialog but all it was meant for was to allow users to assume an arbitrary identity by informing the name of an existing user.

That is no longer the case. The latest release build (#29) addresses that by implementing a full-blown authentication mechanism. Now, when you try to access a Cloudfier app, like this one, you will be greeted by this login dialog:

login

which allows you to sign up, request a password reset and sign in either with proper credentials or as a guest.

For more details about how authentication works in Cloudfier applications, check the new authentication documentation.

BTW, kudos to Stormpath for providing such a great API for managing and authenticating user credentials. Highly recommended.

What’s next?

Glad you asked. Next is authentication’s bigger friend: authorization. Right now any user can do anything to any data, and of course that is not reasonable even for the simplest applications. Stay tuned for more on that.

On MDA and reverse engineering

This appeared somewhere in a discussion in the MDA forum on LinkedIn:

If legacy code exists, MDA can be used to capture the models from the existing code. Extracting models from legacy code is difficult, but it is much better than having someone go through the code and create a model in their head

I would pose reverse engineering has nothing to do with MDA. MDA is really about transformations in one way only: from more abstract to more specific models (while reverse engineering is the opposite).

I am not saying though that one cannot use models obtained with reverse engineering in a MDA context, but that is out of the scope of MDA. But I’d go as far saying that in the general case reverse engineering is not a good approach for producing platform independent models. Reasons:

  1. good models require care and intent in representing the business rules and constraints in the domain being addressed, and that is really rare to see in handwritten code (Exhibit A: the incipient popularity of Domain Driven Design). If something is not there to begin with, a tool cannot extract it.
  2. manual implementation results naturally in high variability in how the same the domain and technical concerns are addressed (independently and in coordination).

Those two things make it really hard (impossible, I’d say, other than in exceptional cases) for a reverse engineering tool (that is based in some sort of pattern matching) to identify code elements and map them back to platform independent models that are not only accurate and complete, but also well designed.

Reverse engineering can be useful in getting an initial approximation of the PIMs (say, covering structure only, but not dynamics or behavior), but that will require significant manual work to become properly designed models.

New Cloudfier release supports many-to-many associations

One clear gap Cloudfier used to have was lack of support for many-to-many associations. That has now been implemented all the way from back-end to the UI.

For instance, in the ShipIt! sample issue tracking application, a user can watch multiple issues, and an issue can be watched by multiple users:

class Issue
end;

class User
end;

association WatchedIssues
    navigable role watchers : User[*];
    navigable role issuesWatched : Issue[*];
end;

UI

…which in the UI means that is now a way to link issues as watched issues for a user:

(and vice-versa from Issue, as the relationship is navigable both ways). Once the user triggers that action, they can pick multiple target objects (in this case, issues) to pair the source object (in this case, a User) up with, by clicking the “connector” button on the target entity instance’s toolbar (the second from left to right):

which once triggered shows a notice confirming the objects have now been linked together.

I will admit this UI may require some using to. It is just a first cut, and I am interested in suggestions from those of you less UX-challenged than me.

REST API

Accordingly, the application’s REST API allows querying related objects using a URI in the form:

…/services/api/<application>/instances/<entity>/<id>/relationships/<relationship-name>/

for instance:

…/services/api/demo-cloudfier-examples-shipit/instances/shipit.Issue/10/relationships/watchers/

produces a list of all users watching the base issue:

[
  {
    uri: ".../instances/shipit.User/2",
    shorthand: "rperez",
    type: ".../entities/shipit.User",
    typeName: "User",
    values: {
      ...
    },
    links: {
      ...
    },
    actions: {
      ...
    },
    relatedUri: ".../instances/shipit.Issue/10/relationships/watchers/2"
  },
  {
    uri: ".../instances/shipit.User/8",
    shorthand: "gtorres",
    type: ".../entities/shipit.User",
    typeName: "User",
    values: {
      ...
    },
    links: {
      ...
    },
    actions: {
      ...
    },
    relatedUri: ".../instances/shipit.Issue/10/relationships/watchers/8"
  }
]

and to establish links, you can POST a similar representation to the same URI, but you only really need to include the ‘uri’ attribute, everything else is ignored.

New tour video

There is also now a new tour video, this time with audio and much better image quality, if you gave up on watching the original one, please give this one a try!

How can modeling be harder than programming?

One argument often posed against model-driven development is that not all developers have the skills required for modeling. This recent thread in the UML Forum discussion group includes a very interesting debate on that and started with this statement by Dan George:

Don’t take his comments about the orders of magnitude reduction in code size to mean orders of magnitude reduction in required skill. I think this is the reason model-driven development is not mainstream. The stream is full of programmers that could never even develop the skills necessary to use MDD. Humbly, I know that I’m still a wannabe.

which was contested by H.S. Lahman

I have to disagree (somewhat) with this assessment. Yes, there is a substantial learning curve and OOA models need to be developed much more rigorously than they are in most shops today. Also, one could argue that the learning curve is essentially the same learning curve needed to learn to do OOA/D properly.

and later by Thomas Mercer-Hursh:

There is something confusing about the idea of good modeling being hard. After all, all one is doing is describing how the system is supposed to work without having to worry about the implementation details. If one can’t do that, then how is one supposed to manually create a correct, working system?

I sympathize with Lahman’s and Thomas’ points (and share some of their puzzlement), but I do agree with Dan’s initial point: modeling can be harder than programming.

Separation of concerns? Not in the job description

The fact is that one can deliver software that was apparently appropriately built (from a QA/product owner/user point-of-view) and yet fail to fully understand the constraints and rules of the business domain the software is meant to serve.

Also, even if a developer does understand the business requirements at the time the solution is originally implemented, it is unfortunately very common that they will fail to encode the solution in a way that clearly expresses the intent so that it would be easy for other developers (or the author at a later time) to correlate the code to business requirements (as proposed by Domain Driven Design). That leads to software that is very hard to maintain, because it is hard to understand, or hard to change without breaking things. Model-driven development is a great approach for proper separation of concerns when building software (the greatest, if you ask me). However, as sad as that is, proper separation of concerns is not a must-have trait for delivering “appropriate” software (from a narrow, external, immediate standpoint). Ergo, one can build software without modeling, even implicitly.

I don’t think those things happen because developers are sociopaths. I think properly understanding and representing the concerns of a business domain when building software is a very desirable skill (I would say critical), but realistically not all that common in software developers. But how can hordes of arguably proficient programmers get away without such skill?

Delivering software the traditional (programming-centric) way often involves carefully patching together a mess of code, configuration and some voodoo to address a complex set of functional and non-functional requirements that works at the time of observation (a castle of cards is an obvious image here). Building software that way makes it too easy for one to be overwhelmed by all the minutia imposed by each technology and the complexity of making them work together and lose track of the high level goals one is trying to achieve – let alone consciously represent and communicate.

Conclusion

So even though I fully agree with the sentiment that proper programming requires a good deal of modeling skills, I do think it is indeed possible to deliver apparently working software (from an external point of view) without consciously doing any proper modeling. If you stick to the externally-facing aspects of software development, all that is valued is time to deliver, correctness, performance, and use of some set of technologies. Unfortunately that is all that is required for most development positions. Easy of maintenance via proper separation of concerns is nowhere in that list. And model-driven development is essentially an approach for separation of concerns on steroids.

What do you think?

Checking the current state of a UML state machine

In Cloudfier, we use UML as the core language for building business applications. UML is usually well-equipped for general purpose business domain-centric application modeling, but that doesn’t mean it always does everything needed out of the box.

Case at hand: assuming one is developing an expense reporting application and modeled an expense’s status as a state machine (in TextUML):

class Expense
    /* ... */
    attribute status : Status;
    operation review();
    operation approve();
    operation reject();
    operation submit();

    statemachine Status
        initial state Draft
            transition on call(submit) to Submitted;
        state Submitted
            transition on call(approve) to Approved
            transition on call(reject) to Rejected
            transition on call(review) to Draft;
        terminate state Approved;
        terminate state Rejected;        
    end;
end;

How do you model the following in UML?

Show me all expenses that are waiting for approval.

Turns out there is no support in UML for reasoning based on the current state of a state machine.

Creative modeling

So, what do you do when UML does not have a language element that you need? You extend it, in our case, using a stereotype applicable to the LiteralNull metaclass (in TextUML):

stereotype VertexLiteral extends LiteralNull
    property vertex : Vertex;
end;

So, a vertex literal is a value specification, more specifically, a variant of LiteralNull, that can refer to a Vertex, which is a metaclass that represents the states (including pseudo-states) in a state machine.

Notation, notation

In terms of notation, I chose to make State/Vertex literals look like enumeration literals: Status#Approved or Status#Draft. So, back to the original question, this is how you could model a query that returns all expenses that are in the Submitted state:

    static operation findAllSubmitted() : Expense[*];
    begin 
        return Expense extent.select ((e : Expense) : Boolean {
            return e.status == Status#Submitted
        });
    end;

If you are thinking to yourself: I didn’t know UML had queries or closures!?, well, it usually doesn’t. See the posts on SQL queries in UML and Closures in UML for some background on this.

Note also that if you wanted to refer to the symbol Status from a class different than the one enclosing it you will need to qualify it (i.e. Expense::Status#Submitted).

Show me more!

You can run the Expenses application showing state machines and state-based queries in Cloudfier right now (login is “guest” or any of the employee names you will see later).

The entire Expenses sample application (currently 150 lines of generously spaced TextUML) is available on BitBucket. You can also easily check it out into Cloudfier so you can run your own copy of the application on the web (there is nothing to install). Give it a try!

What do you think?

Your feedback (questions, support or criticism) to any of the ideas presented in this post is very welcome.

UPDATE: I started a thread on the subject on the UML Forum group, and turns out you can do this kind of reasoning in OCL, but indeed, not in UML itself. Well, now you can.

Yet another Orion-based site: cloudfier.com

Okay, we are live.

I just put the last finishing touches on the developer site at cloudfier.com.

The developer site, develop.cloudfier.com, is powered by Orion. Cloudfier’s instance of Orion has several features to support modeling with TextUML, such as:

  • Syntax highlighting
  • Outline
  • Validation
  • Auto-formatting
  • Templates

and I have a picture to prove it:

but wouldn’t you rather see for yourself? If you are shy because you don’t know how to model in TextUML, just make sure you create a file with a “.tuml” extension and use the content assist templates to get a model going. Or if you are feeling lazy, just clone this Git repository: https://bitbucket.org/abstratt/cloudfier-examples.git

But what and who is Cloudfier for you may ask. I won’t tell you here though. Please go to cloudfier.com, give it a quick read. If you don’t get it, please let me know in the comments – a main goal now is to ensure the main page can get the message across.

TextUML Toolkit finally gets continuous integration thanks to Tycho and CloudBees

TextUML Toolkit 1.8 is now available! You can install it as usual using http://abstratt.com/update as the update site. There is also a snapshot update site, which will work from within Eclipse only:

jar:https://repository-textuml.forge.cloudbees.com/snapshot/com/abstratt/mdd/com.abstratt.mdd.oss.repository/1.0/com.abstratt.mdd.oss.repository-1.0.zip!/

This is a transition release where the TextUML Toolkit now uses continuous integration for producing builds via Eclipse Tycho, as opposed to developer initiated builds from the IDE. This benefits contributors (the development setup is much simpler), but primarily users – since it is now so much easier to obtain the source code and generate a release users can expect much more frequent releases, and hopefuly more goodies from occasional contributors.

Talking about frequent releases, if you don’t mind living on the bleeding edge, I invite you to install the TextUML Toolkit from the snapshot update site (that is what you get if you install the Toolkit using the Eclipse Marketplace Client). That way, features or fixes will become available to you a day after they have been committed.

This release contains a number of new features and bug fixes added since 1.7 was released a year ago, but we are not documenting those yet. You will see those properly promoted in a future release. Our focus now was to get our release engineering act straight, and I think we succeeded, thanks to Tycho.

Finally, we would like to thank CloudBees for their generous free plan that allows us to set up Jenkins continuous builds for the TextUML Toolkit at no cost. On that note, we are applying for a FOSS plan so we can have our build results available for everyone to see, and as a bonus, enjoy a slightly higher monthly build quota. As you can see, we are already living up to our side of the deal by spreading the word about their cool DEV@cloud product. :)

UPDATE: CloudBees is now providing the TextUML Toolkit project with a free DEV@cloud instance.

Adding State Machines to TextUML and AlphaSimple [take 1]

I decided to go ahead and finally implement support for state machines in TextUML and AlphaSimple.

This is an example of what a state machine will look like (take 1), based on fig. 15.33 in the UML specification 2.4:


(...)
statemachine Phone

  initial state
    entry { self.startDialTone() }
    exit { self.stopDialTone() }
    transition on digit to PartialDial;

  state PartialDial
    transition on digit to PartialDial
    transition when { self.numberIsValid() } to Completed;

  final state Completed;

end;
(...)

A state machine may declare multiple states. Each state declares a number of transitions to other states. Each transition may be triggered by many events (or none), each denoted by the keyword ‘on’, and may optionally present a guard constraint (using the keyword ‘when’). The initial state is the only one that may remain unnamed. The final state cannot have outgoing transitions, but just like any other state, it may declare entry/exit behaviors.

What do you think? I did try to find existing textual notations for UML, like this and this, but none of those seem to be documented or look like covering all the UML features I want to support. Any other pointers?

Feedback wanted: invariant constraints in AlphaSimple/TextUML

I am working on support for invariant constraints in AlphaSimple/TextUML.

Some of the basic support has already made into the live site. For instance, the AlphaSimple project has a rule that says:

A user may not have more than 3 private projects.”

This in TextUML looks like this:


class User 

    attribute projects : Project[*] 
        invariant Maximum 3 private projects { 
            return self.privateProjects.size() <= 3
        };
        
    derived attribute privateProjects : Project[*] := () : Project[*] {
        return self.projects.select((p : Project) : Boolean {
            return not p.shared
        });
    };

end;

(Note the constraint relies on a derived property for more easily expressing the concept of private projects, and that backslashes are used to escape characters that otherwise would not be allowed in identifiers, such as whitespaces.)

What do you think? Does it make sense? I know the syntax for higher order functions could benefit from some sugar, but that can be easily fixed later. I am much more interested in feedback on the idea of modeling with executable constraints than in syntax.

Wading in unknown waters

I am in the process of modeling a real world application in AlphaSimple and for most cases, the level of support for constraints that we are building seems to be sufficient and straightforward to apply.

I have though found one kind of constraint that is hard to model (remember, AlphaSimple is a tool for modeling business domains, not a programming language): in general terms, you cannot modify or delete an object if the object (or a related object) is in some state. For example:

"One cannot delete a project's files if the project is currently shared".

Can you think of a feature in UML that could be used to address a rule like that? I can't think of anything obvious (ChangeEvent looks relevant at a first glance, but there is no support for events in TextUML yet).

Any ideas are really appreciated.

MDD meets TDD (part II): Code Generation

Here at Abstratt we are big believers of model-driven development and automated testing. I wrote here a couple of months ago about how one could represent requirements as test cases for executable models, or test-driven modeling. But another very interesting interaction between the model-driven and test-driven approaches is test-driven code generation.

You may have seen our plan for testing code generation before. We are glad to report that that plan has materialized and code generation tests are now supported in AlphaSimple. Follow the steps below for a quick tour over this cool new feature!

Create a project in AlphaSimple

First, you will need a model so you can generate code from. Create a project in AlphaSimple and a simple model.


package person;

enumeration Gender 
  Male, Female
end; 

class Person
    attribute name : String; 
    attribute gender : Gender; 
end;

end.

Enable code generation and automated testing

Create a mdd.properties file in your project to set it up for code generation and automated testing:


# declares the code generation engine
mdd.target.engine=stringtemplate

# imports existing POJO generation template projects
mdd.importedProjects=http://cloudfier.com/alphasimple/mdd/publisher/rafael-800/,http://cloudfier.com/alphasimple/mdd/publisher/rafael-548/

# declares a code generation test suite in the project
mdd.target.my_tests.template=my_tests.stg
mdd.target.my_tests.testing=true

# enables automated tests (model and templates)
mdd.enableTests=true

Write a code generation test suite

A code generation test suite has the form of a template group file (extension .stg) configured as a test template (already done in the mdd.properties above).

Create a template group file named my_tests.stg (because that is the name we declared in mdd.properties), with the following contents:


group my_tests : pojo_struct;

actual_pojo_enumeration(element, elementName = "person::Gender") ::= "<element:pojoEnumeration()>"

expected_pojo_enumeration() ::= <<
enum Gender {
    Male, Female
}
>>

A code generation test case is defined as a pair of templates: one that produces the expected contents, and another that produces the actual contents. Their names must be expected_<name> and actual_<name>. That pair of templates in the test suite above form a test case named “pojo_enumeration”, which unsurprisingly exercises generation of enumerations in Java. pojo_enumeration is a pre-existing template defined in the “Codegen – POJO templates” project, and that is why we have a couple of projects imported in the mdd.properties file, and that is why we declare our template suite as an extension of the pojo_struct template group. In the typical scenario, though, you may would have the templates being tested and the template tests in the same project.

Fix the test failures

If you followed the instructions up to here, you should be seeing a build error like this:



Line	File		Description
3	my_tests.stg	Test "pojo_enumeration" failed: [-public -]enum Gender {n Male, Femalen}

which is reporting the code generated is not exactly what was expected – the template generated the enumeration with an explicit public modifier, and your test case did not expect that. Turns out that in this case, the generated code is correct, and the test case is actually incorrect. Fix that by ensuring the expected contents also have the public modifier (note that spaces, newlines and tabs are significant and can cause a test to fail). Save and notice how the build failure goes away.

That is it!

That simple. We built this feature because otherwise crafting templates that can generate code from executable models is really hard to get right. We live by it, and hope you like it too. That is how we got the spanking new version of the POJO target platform to work (see post describing it and the actual project) – we actually wrote the test cases first before writing the templates, and wrote new test cases whenever we found a bug – in the true spirit of test-driven code generation.

Can you tell this is 100% generated code?

Can you tell this code was fully generated from a UML model?

This is all live in AlphaSimple – every time you hit those URLs the code is being regenerated on the fly. If you are curious, the UML model is available in full in the TextUML’s textual notation, as well as in the conventional graphical notation. For looking at the entire project, including the code generation templates, check out the corresponding AlphaSimple project.

Preconditions

Operation preconditions impose rules on the target object state or the invocation parameters. For instance, for making a deposit, the amount must be a positive value:


operation deposit(amount : Double);
precondition (amount) { return amount > 0 }
begin
    ...
end;

which in Java could materialize like this:


public void deposit(Double amount) {
    assert amount > 0;
    ...
}

Not related to preconditions, another case assertions can be automatically generated is if a property is required (lowerBound > 0):


public void setNumber(String number) {
    assert number != null;
    ...
}

Imperative behavior

In order to achieve 100% code generation, models must specify not only structural aspects, but also behavior (i.e. they must be executable). For example, the massAdjust class operation in the model is defined like this:


static operation massAdjust(rate : Double);
begin
    Account extent.forEach((a : Account) { 
        a.deposit(a.balance*rate) 
    });
end;

which in Java results in code like this:


public static void massAdjust(Double rate) {
    for (Account a : Account.allInstances()) {
        a.deposit(a.getBalance() * rate);
    };
}

Derived properties

Another important need for full code generation is proper support for derived properties (a.k.a. calculated fields). For example, see the Account.inGoodStanding derived attribute below:


derived attribute inGoodStanding : Boolean := () : Boolean { 
    return self.balance >= 0 
};

which results in the following Java code:


public Boolean isInGoodStanding() {
    return this.getBalance() >= 0;
}

Set processing with higher-order functions

Any information management application will require a lot of manipulation of sets of objects. Such sets originate from class extents (akin to “#allInstances” for you Smalltalk heads) or association traversals. For that, TextUML supports the higher-order functions select (filter), collect (map) and reduce (fold), in addition to forEach already shown earlier. For example, the following method returns the best customers, or customers with account balances above a threshold:


static operation bestCustomers(threshold : Double) : Person[*];
begin
    return
        (Account extent
            .select((a:Account) : Boolean { return a.balance >= threshold })
            .collect((a:Account) : Person { return a->owner }) as Person);
end;        

which even though Java does not yet support higher-order functions, results in the following code:


public static Set<Person> bestCustomers(Double threshold) {
    Set<Person> result = new HashSet<Person>();
    for (Account a : Account.allInstances()) {
        if (a.getBalance() >= threshold) {
            Person mapped = a.getOwner();
            result.add(mapped);
        }
    }
    return result;
}

which demonstrates the power of select and collect. For an example of reduce, look no further than the Person.totalWorth attribute:


derived attribute totalWorth : Double := () : Double {
    return (self<-PersonAccounts->accounts.reduce(
        (a : Account, partial : Double) : Double { return partial + a.balance }, 0
    ) as Double);
};  

which (hopefully unsurprisingly) maps to the following Java code:


public Double getTotalWorth() {
    Double partial;
    partial = 0;
    for (Account a : this.getAccounts()) {
        partial = partial + a.getBalance();
    }
    return partial;
}

Would you hire AlphaSimple?

Would you hire a developer if they wrote Java code like AlphaSimple produces? For one thing, you can’t complain about the guy not being consistent. :) Do you think the code AlphaSimple produces needs improvement? Where?

Want to try by yourself?

There are still some bugs in the code generation that we need to fix, but overall the “POJO” target platform is working quite well. If you would like to try by yourself, create an account in AlphaSimple and to make things easier, clone a public project that has code generation enabled (like the “AlphaSimple” project).

11 Dogmas of Model-Driven Development

I prepared the following slides for my Eclipse DemoCamp presentation on AlphaSimple but ended up not having time to cover them. The goal was not to try to convert the audience, but to make them understand where we are coming from, and why AlphaSimple is the way it is.

And here they are again for the sake of searchability:

I – Enterprise Software is much harder than it should be, lack of separation of concerns is to blame.

II – Domain and architectural/implementation concerns are completely different beasts and should be addressed separately and differently.

III – What makes a language good for implementation makes it suboptimal for modeling, and vice-versa.

IV – Domain concerns can and should be fully addressed during modeling, implementation should be a trivial mapping.

V – A model that fully addresses domain concerns will expose gaps in requirements much earlier.

VI – A model that fully addresses domain concerns allows the solution to be validated much earlier.

VII – No modeling language is more understandable to end-users than a running application (or prototype).

VIII – A single architecture can potentially serve applications of completely unrelated domains.

IX – A same application can potentially be implemented according to many different architectures.

X – Implementation decisions are based on known guidelines applied consistently throughout the application, and beg for automation.

XI – The target platform should not dictate the development tools, and vice-versa.

I truly believe in those principles, and feel frustrated when I realize how far the software industry is from abiding by them.

So, what do you think? Do you agree these are important principles and values? Would you call B.S. on any of them? What are your principles and values that drive your vision of what software development should look like?

Interview on perspectives on MDD and UML

I had the honor of being interviewed by Todd Humphries, Software Engineer at Objektum Solutions, on my views on UML and model-driven development. Here is an excerpt of the interview:

Todd Humphries: Did you have a ‘Eureka!’ moment when modelling made sense for the first time and just became obvious or was there one particular time you can think of where your opinion changed?

Rafael Chaves: When I was first exposed to UML back in school it did feel cool to be able to think about systems at a higher level of abstraction, and be able to communicate your ideas before getting down to the code (we often would just model systems but never actually build them). The value of UML modeling for the purpose of communication was evident, but that was about it. I remember feeling a bit like I was cheating, as drawing diagrams gave me no confidence the plans I was making actually made a lot of sense.

After that, still early in my career, I had the opportunity of working in a team where we were using an in-house code generation tool (first, as many have done, using XML and XSLT, and later, using UML XMI and Velocity templates, also common choices). We would get reams of Java code, EJB configuration files and SQL DDL generated from the designer models, and it did feel a very productive strategy for writing all that code. But the interesting bits (business logic) were still left to be written in Java (using the generation gap pattern). It was much better than writing all that EJB boilerplate code by hand, but it was still cumbersome and there was no true gain in the level of abstraction, as we would model thinking of the code that would be generated – no surprise, as there was no escaping the facts that we would rely on the Java compiler and JUnit tests to figure out whether the model had problems, and in order to write the actual business logic in Java, we had to be very familiar with the code that was generated. So even though I could see the practical utility of modeling by witnessing the productivity gains we obtained, there was a hackish undertone to it, and while it worked, it didn’t feel like solid engineering.

It was only later, when…

Visit The Technical Diaries, Objektum team’s blog, for the full interview. Do you agree with my views? Have your say (here or there).

MDD meets TDD: mapping requirements as model test cases

Executable models, as the name implies, are models that are complete and precise enough to be executed. One of the key benefits is that you can evaluate your model very early in the development life cycle. That allows you to ensure the model is generally correct and satisfies the requirements even before you have committed to a particular implementation platform.

One way to perform early validation is to automatically generate a prototype that non-technical stakeholders can play with and (manually) confirm the proposed model does indeed satisfy their needs (like this).

Another less obvious way to benefit from executable models since day one is automated testing.

The requirements

For instance, let’s consider an application that needs to deal with money sums:

  • REQ1: a money sum is associated with a currency
  • REQ2: you can add or subtract two money sums
  • REQ3: you can convert a money sum to another currency given an exchange rate
  • REQ4: you cannot combine money sums with different currencies

The solution

A possible solution for the requirements above could look like this (in TextUML):

package money;

class MixedCurrency
end;

class Money
  attribute amount : Double;
  attribute currency : String;
  
  static operation make(amount : Double, currency : String) : Money;
  begin 
      var m : Money;
      m := new Money;
      m.amount := amount;
      m.currency := currency;
      return m;
  end;
  
  operation add(another : Money) : Money;
  precondition (another) raises MixedCurrency { return self.currency = another.currency }
  begin
      return Money#make(self.amount + another.amount, self.currency);
  end;
  
  operation subtract(another : Money) : Money;
  precondition (another) raises MixedCurrency { return self.currency = another.currency }  
  begin
      return Money#make(self.amount - another.amount, self.currency);      
  end;
  
  operation convert(anotherCurrency : String, exchangeRate : Double) : Money;
  begin
      return Money#make(self.amount * exchangeRate, anotherCurrency);
  end;  
end;
        
end.

Now, did we get it right? I think so, but don’t take my word for it.

The proof

Let’s start from the beginning, and ensure we satisfy REQ1 (a money sum is a pair <amount, currency>:

[Test]
operation testBasic();
begin
    var m1 : Money;
    m1 := Money#make(12, "CHF");
    Assert#assertEquals(12, m1.amount);
    Assert#assertEquals("CHF", m1.currency);
end;

It can’t get any simpler. This test shows that you create a money object providing an amount and a currency.

Now let’s get to REQ2, which is more elaborate – you can add and subtract two money sums:

[Test]
operation testSimpleAddAndSubtract();
begin
    var m1 : Money, m2 : Money, m3 : Money, m4 : Money;
    m1 := Money#make(12, "CHF");
    m2 := Money#make(14, "CHF");

    m3 := m1.add(m2);    
    Assert#assertEquals(26, m3.amount);
    Assert#assertEquals("CHF", m3.currency);
    
    /* if m1 + m2 = m3, then m3 - m2 = m1 */
    m4 := m3.subtract(m2);
    Assert#assertEquals(m1.amount, m4.amount);
    Assert#assertEquals(m1.currency, m4.currency);
end;

We add two values, check the result, them subtract one of them from the result and expect the get the other.

REQ3 is simple as well, and specifies how amounts can be converted across currencies:

[Test]
operation testConversion();
begin
    var m1 : Money, result : Money;
    m1 := Money#make(3, "CHF");
    result := m1.convert("USD", 2.5);
    Assert#assertEquals(7.5, result.amount);
    Assert#assertEquals("USD", result.currency);
end;

We ensure conversion generates a Money object with the right amount and the expected currency.

Finally, REQ4 is not a feature, but a constraint (currencies cannot be mixed), so we need to test for rule violations:

[Test]
operation testMixedCurrency();
begin
    try
        Money#make(12, "CHF").add(Money#make(14, "USD")); 
        /* fail, should never get here */
        Assert#fail("should have failed");
    catch (expected : MixedCurrency)
        /* success */
    end;
end;

We expect the operation to fail due to a violation of a business rule. The business rule is identified by an object of a proper exception type.

There you go. Because we are using executable models, even before we decided what implementation platform we want to target, we already have a solution in which we have a high level of confidence that it addresses the domain-centric functional requirements for the application to be developed.

Can you say “Test-driven modeling”?

Imagine you could encode all non-technical functional requirements for the system in the form of acceptance tests. The tests will run against your models whenever a change (to model or test) occurs. Following the Test-Driven Development approach, you alternate between encoding the next requirement as a test case and enhancing the model to address the latest test added.

Whenever requirements change, you change the corresponding test and you can easily tell how the model must be modified to satisfy the new requirements. If you want to know why some aspect of the solution is the way it is, you change the model and see the affected tests fail. There is your requirement traceability right there.

See it by yourself

Would you like to give the mix of executable modeling and test-driven development a try? Sign up to AlphaSimple now, then open the public project repository and clone the “Test Infected” project (or just view it here).

P.S.: does this example model look familiar? It should – it was borrowed from “Test Infected: Programmers Love Writing Tests“, the classical introduction to unit testing, courtesy of Beck, Gamma et al.

Testing code generation templates – brainstorming

We would like to support automated testing of templates in AlphaSimple projects. I have been “test-infected” for most of my career, and the idea of writing code generation templates that are verified manually screams “unsustainable” to me. We need a cheap and easily repeatable way of ensuring code generation templates produce what they intend to produce.

Back-of-a-napkin design for code generation testing:

  1. by convention, for each test case, declare two transformations: one will hardcode the expected results, and another will trigger the transformation to test with some set of parameters (typically, an element of a model). We can pair transformations based on their names: “expected_foo” and “actual_foo” for a test case named “foo”
  2. if the results are identical, the test passes; otherwise, the test fails (optionally, use a warning for the cases where the only differences are around layout, i.e., non significant chars like spaces/newlines – optionally, because people generating Python code will care about layout)
  3. just as we do for model test failures, report template test failures as build errors
  4. run template tests after model tests, and only if those pass
  5. (cherry on top) report text differences in a sane way (some libraries out there can do text diff’ng)

Does that make sense? Any suggestions/comments (simpler is better)? Have you done or seen anything similar?

What is the focus of analysis: problem or solution?

Is the purpose of an analysis model understanding the problem or proposing a solution? I have discussed this a few times with different people. This is how I used to see it:

  • Analysis deals with understanding the problem domain and requirements in detail
  • Design deals with actually addressing those (functional and non-functional) requirements
  • A detailed design model can be automatically transformed into a working implementation
  • An analysis model can’t, as in the general case, it is not possible to automatically derive a solution based on the statement of a problem.

Rumbaugh, Blaha et al in “Object-oriented modeling and design” (one of the first OO modeling books) state the purpose of analysis in OO is to model the real-world system so it can be understood; and the outcome of analysis is understanding the problem in preparation for design.

Jacobson, Booch and Rumbaugh (again, now with the other “two amigos”) in “The unified software development process” state that “an analysis model yields a more precise specification of the requirements than we have in the results from requirements capture” and “before one starts to design and implement, one should have a precise and detailed understanding of the requirements”.

Ok, so I thought I was in good company there. However, while reading the excellent “Model-based development: applications“, to my great surprise, H. S. Lahman clearly states that contrary to structured development, where the focus of analysis is problem analysis, in the object-oriented paradigm, problem analysis is done during requirements elicitation, and the goal of object-oriented analysis is to specify the solution in terms of the problem space, addressing functional requirements only, in a way that is independent of the actual computing environment. Also, Lahman states that the OOA model is the same as the platform-independent model (PIM) in MDA lingo, so it can actually be automatically translated into running code.

That is the first time I have seen this position defended by an expert. I am not familiar with the Shlaer-Mellor method, but I won’t be surprised if it has a similar view of analysis, given that Lahman’s method is derived from Shlaer-Mellor. Incidentally, Mellor/Balcer’s “Executable UML: a foundation for model-driven architecture” is not the least concerned with the software lifecycle, briefly mentions use cases as a way of textually gathering requirements, and focuses heavily on solution modeling.

My suspicion is that for the Shlaer-Mellor/Executable UML camp, since models are fully executable, one can start solving the problem (in a way that is removed from the actual concrete implementation) since the very beginning, so there is nothing to be gained by strictly separating problem from a high-level, problem-space focused solution. Of course, other aspects of the solution, concerned with non-functional requirements or somehow tied with the target computing environment, are still left to be addressed during design.

And now I see how that all makes sense – I struggled myself with how to name what you are doing when you model a solution in AlphaSimple. We have been calling it design based on the more traditional view of analysis vs. design – since AlphaSimple models specify a (highly abstract) solution, it couldn’t be analysis. But now I think I understand: for approaches based on executable modeling, the divide between understanding the problem and specifying a high-level solution is so narrow and so cheap to cross, that both activities can and should be brought closer together, and the result of analysis in approaches based on executable modeling is indeed a model that is ready to be translated automatically into a running application (and can be quickly validated by the customer).

But for everybody else (which is the vast majority of software development practitioners – executable modeling is still not well known and seldom practiced) that is just not true, and the classical interpretation still applies: there is value in thoroughly understanding the requirements before building a solution, given that the turnaround between problem comprehension, solution building and validation is so damn expensive.

For those of you thinking that this smells of BigDesignUpFront, and that is not an issue with agile or iterative approaches in general – I disagree. At least as far as typical iterative approaches go, where iterations need to comprise all/several phases of the software development life cycle so they can finally deliver results that can be validated by non-technical stakeholders. As such they are still very wasteful (the use of the word agile feels like a bad joke to me).

Approaches based on executable modeling, on the other hand, greatly shrink the chasm between problem analysis and conceptual solution modeling and user acceptance, allowing for much more efficient and seamless collaboration between the problem domain expert and the solution expert. Iterations become so action packed that they are hardly discernible. Instead of iterations taking weeks to allow for customer feedback, and a project taking months to fully cover all functional requirements, you may get a fully specified solution after locking a customer and a modeler in a boardroom for just a day, or maybe a week for bigger projects.

So, long story short, to answer the question posed at the beginning of this post, the answer is both, but only if you are following an approach based on executable modeling.

What is your view? Do you agree with that? Are you an executable modeling believer or skeptic?

UPDATE: make sure to check the thread on Google+ as well.

Upcoming presentation on code generation and executable models @ VIJUG

This coming Thursday I will be doing a presentation entitled “Code generation – going all the way” to the Vancouver Island Java User Group.

The plan is to take the audience from the most basic ideas around generating code from models, visiting approaches increasingly more sophisticated, analyzing their pros and cons, all the way to full code generation based on executable models.

In the process, we will be taking a cursory look at some code generation tools in the market, culminating with a preview of the upcoming release of AlphaSimple, our online modeling tool, which will support executable modeling and full code generation.

What:

May 2011 VIJUG Meeting – Code generation: going all the way (official announcement)

Where:

Vancouver Island Technology Park, Conference Center Room – 4464 Markham Street, Victoria, BC

When:

Thursday, 26th May 2011, 18:00-20:00

If you are in Victoria and think developing business applications got just way too complicated and labor-intensive, and that there must be a saner way to build and evolve them (no matter what platforms you use), come to this presentation and learn how executable models and full code generation can fix that.

Update: these were the slides presented:

Help wanted: Converting closure-based iterators into plain loops

Smalltalk, Ruby, Groovy and other languages allow one to implement loops using closures. But so does TextUML/UML. Given the primary use case of TextUML/UML is to generate code, one thorny question is how to generate code from a UML model using closures for implementing loops through collections into a language, like Java or C, just as one would normally write loops over collections in those closure-free languages.

Here are some examples of how to translate from closure-based loops (in TextUML, but the specific syntax shouldn’t matter) to ordinary loops (in Java, but again, syntax specifics shouldn’t matter):

forEach

In TextUML


self->units.forEach((u : Unit) { 
    link ProjectUnits(project := clone, units := u.clone()) }
);


In Java


for (Unit u : this.getUnits()) { 
    clone.addUnits(u.clone());
}

select

In TextUML


return Project extent.select((p : Project) : Boolean { return p.shared });

In Java


Set<Project> result = new HashSet<Project>();
for (Project p : Project.allInstances()) { 
    if (p.isShared()) {
        result.add(p);
    }
} 
return result;

collect

In TextUML


return Project extent.collect((p : Project) : User { return p->owner });

In Java


Set<User> result = new HashSet<User>();
for (Project p : Project.allInstances()) { 
    User owner = p.getOwner();
    result.add(owner);
} 
return result;

count

In TextUML


return Project extent.count((p : Project) : Boolean { return p.shared });

In Java


int count = 0;
for (Project p : Project.allInstances()) { 
    if (p.isShared()) {
        count++;
    }
} 
return count;

In AlphaSimple, we got much of what is needed above in place. There are though some additional challenges posed by the need of chaining those collection primitives, and the need for mapping the data flow that chains them together to an unchained form, using local variables in the target language. These last two aspects have been keeping me awake at night. If you feel like throwing a light (with strategies, references) on how to address that, by all means go for it, it is pretty dark in here right now… :)

Pitching AlphaSimple

Not clear on what AlphaSimple is about? Isn’t the value we aim to provide clear?

Does this 7-slide pitch deck (including title) help you understand it?

If not, please let us know. Or if it is clear, but you don’t think it would work, please help us understand why. Or even if you think it is right on the money, some reassurance is always welcome.

Note that the web site hasn’t yet changed to reflect the more recent focus on code generation that the slide deck hints at (in addition to the existing focus on requirements and solution design validation). That is on our to-do list. Expect significant changes (messaging and service features) this Spring.


This is the inaugural post on the “business of software” category. Expect many more in the future as we work on turning AlphaSimple into a product that fulfills our mission: to bring model-driven development to the masses, and stopping hordes of developers from writing so much code. Stay tuned.