QCon highlights

A QCon presentation style review (mostly)

At QCon this year the influence of Presentation Zen was to be seen everywhere. But I must admit: it didn’t help too much. Sure, the quality of the talks, its content was as great as ever, but I am not sure Garr Reynolds’s influence helped the presentations that much. It mostly felt artificial. The slides may have improved, but they looked so similar, unoriginal. And the presenters were still always craning their necks, standing behind the lectern, or if they dared to stray away, they had to return immediately to push the forward button (please buy a decent presenter, like: this for OS X or this for Windows) One of the exceptions: Sam Newman’s talk on From Dev to Production: Better Living through Build Pipelines and Teamwork, who seemed so at ease with his material, so natural that it was pure delight to watch. I’ve been told that it might be difficult, to get him to stop talking, but in this presentation it came together nicely. Thank you, Sam.

Simon Wardley did a slick presentation presentation as well. His explanation of the cloud “movement” from a historical and economic viewpoint, was spot on. I will especially take with me his view, that it’s not a question ‘if’ you are going to move to the cloud, but ‘when’, because computing power is becoming a commodity, and if you treat it otherwise you will fall back, because you will not be able to innovate on top of it as fast as others are doing.

His presentation on the other hand was completely over the top, with 400+ slides, well rehearsed, but after some time it became tiresome. He simply overdid it, seemed to enjoyed himself so much, that he might as will did it in front of a mirror. Please, cut it down next time.

The slides I probably liked most, were Nat Pryce’s. Simply because they were original: He completely did them by hand on a tablet PC, brushing them up with InkScape. It surely helps that he has a very legible handwriting

He ran a little bit through his different case studies, I think it would have been better to have at most two examples and going a bit deeper into his material on testing asynchronous systems. Still his advice was helpful.

The funniest presentation came from Dan North, I hope they publish the video asap. Without Dan’s presentation, the slides are only half as fun. This in no means says that the slides are only half good, it just means that Dan is a great presenter. You can follow him on Twitter, I think it’s worth it.

And I have to follow his advice to buy me a rubber duck! Yes, a rubber duck: whenever I am furiously hacking on something, losing myself; I then look at my rubber duck and asking it: is this worth it? And the rubber duck just looks back: duh? Ok, it’s not worth it, thank you. Very helpful.

Ralph Johnson on the other hand was rather boring, talking about refactoring; nothing new.

Optimizing CPU cache access in Java

The presentation that absolutely made my head explodes, was “LMAX – How to do over 100K contended complex business transactions per second at less than 1ms latency”. They really pushed performance optimization in Java beyond the limit I had ever considered. They even padded a data structure with additional bytes to ensure that two essential parts (the head and the tail of their own list implementation) were to be placed in two different cache rows of the CPU, to reduce contention. Most of their other optimizations were not so far of, but – for me – unheard of in the Java world.

Facebook create their own PHP compiler & runtime

Where LMAX pushed the Java limits, Facebook created their own PHP compiler to improve the performance of their web site. And they open sourced it (see: HipHopBlog and HipHop), because they had profited in so many ways from open source software that they want to give something back. Great move. But what really resonated with me was their “culture”. They …

1. … create everything in very small teams
2. … still try to rather push some new features out there and fix any (scaleability) problems when they occur

Sure the last principle does not work for everyone, and you have to weigh in the image problems when something really goes wrong, but sometimes we simply are way beyond sanity. Sometimes it feels that an IT department is mostly trying to avoid getting something through the door. Yes, too often those who have to stand up in the middle of the night aren’t those that caused the problems in the first place. But what about taking Sam’s advice and wear the pager yourself for a week after the release. Okay then you have to have access to production systems yourself, and so on yadda, yadda, yadda. Try it. At least ask (“we can’t possibly do that”), and then ask again (“we have never done it”)– and again (“well …”).

Let it REST

Stefan Tilkov is still trying to convince everyone to use REST. Luckily he has lost some of his zeal and even acknowledges that there are some difficulties in adopting the REST mindset. But most of his claims were less provocative then he might have thought. So I could agree with him most of the time. (And I must still thank him, because his zeal provoked me into blogging in the first place, see my very first blog entry.

But I must admit that the best part was “stolen” from Jim Webber (which he explicitly and happily said so himself. Please check out the keynote by Jim and Martin Fowler from QCon 2008): Should an ESB be the base of your SOA? Or which problem does an ESB solve?

If you happen to stumble upon the usual enterprise systems spaghetti landscape:

The idea of an ESB that cleans up the mess might be tempting. Everything now finally looks orderly, everything has its place. Something we like: Order. Cleanliness.

But what happens if you open up the lid?

Oops.

His best argument for REST (from my point of view) is this: When you look at most APIs, they are 90% CRUD, so why use something that might be better for the last 10%? This undermines my strongest argument, that some important services really do not fit the REST style. Yes, they do not fit well, but is that a good argument for the WS-?

Books

As usual I had to buy some books, which I will probably never read (have a look at my library at Library Thing). But, oh, I already finished “Confessions of a Public Speaker” (Scott Berkun, 2010), nice little book, you will probably have finished it, in one or two evenings. Nothing really new, some good stories, even if a little bit rambling along, but good entertainment after a full day of conference sessions.

The other one is “Web Design for Developers” (Brian Hogan, 2010), not yet sure if it was a good choice, but still learned quite a bit on color schemes. Would have preferred less micro-recipes for Photoshop and more about the thought processes, but still a good read up to now.

The idea that the business should learn IT is typical of technical people and completely misguided

Some time ago I wrote the following:

“The idea that the business should learn IT is typical of technical people and completely misguided” (Ron Palmer)

Yes, but then please don’t complain. Go away. You are talking about stuff you do not understand and I can’t take you seriously.
There is a Dilbert cartoon where the Pointy Haired Boss is doing a project estimation; claims that everything he doesn’t understand is simple and goes on: ‘Creating a multi-tier architecture for multiple channels … that will be three minutes’.

Do you listen to your IT department when they tell you, that doing this project in the given time frame will make maintenance difficult? No you won’t, you say that “they stifle the creative entrepreneurialism that is critical to advancing the state of the business”.
But would you skip maintenance on your car knowing that the risk of losing a tire will increase every month?

You do not listen … and then you complain. You create christmas wishlists and then you complain about not getting everything and especially not getting the thing that wasn’t on your list. But that was obviously needed by you.

So don’t learn about IT but make it your responsibility that they understand your needs. Would you have an architect build your house without checking the plans making sure that he understood your needs, without regularly checking the progress?

It was written, when I was involved in a project where the business side (again) complained that our software did not do what they expected. And it was (again) a misunderstanding about a sentence in their specification, that to us had a complete different meaning than what they expected (and vice versa). So perhaps you can forgive me the angry style. Instead of whining, let me try to improve on the situation.

Let’s draft a contract: It is our responsibility to understand your needs and translate them into software. We do that in the best way we know of. And “best” here means, as fast and cheap as possible (“cheap” in the long or short run: you choose). Because if you need something now, and know the trade off of getting it ASAP, we will do it. If we (think we) see a better way to support your business, it is our obligation to propose such a solution. But to be able to do that you must help us in any way you can. So that we are enabled to learn, what is important to you and what is not. And this does not end with you handing over a “specification”. It involves you, to write the specification (with us) in the best way you know of. It involves you, when we have questions and need clarifications. It involves you listening to us, when we explain trade offs for various solutions. It involves you, when we have a prototype or even a first version ready, to thoroughly try and test it. Anything that has been in that first version, that you would like to change later, will become more expensive to change over time. Adding a basement after the roof has been set is kind of difficult.

Does anyone read the Ivy documentation?

I know there are write-only-programming-languages, but isn’t there write-only documentation as well? Case at hand: The Apache Ivy documentation.

Some years ago I stumbled upon Ivy and we tried to use it to replace our home grown Ant solution. But after some weeks we simply gave up, nobody but our in-house build guru could understand it and he refrained from supporting the build, if it used Ivy. Some month later, we/he gave in. I am not sure this was a wise move.

Now, searching again for something less convoluted than Maven, I am looking into Gradle and Buildr (interestingly all but Gradle are part of Apache).

Since Gradle uses Ivy, I took another look at it, i.e at its documentation:

<project xmlns:ivy="antlib:org.apache.ivy.ant" name="hello-ivy" default="run">
     ...
     <target name="resolve" description="--> retrieve dependencies with ivy">
        <ivy:retrieve />
     </target>
  </project>

“[...] Note that in this case we define a “resolve” target and call the retrieve task. This may sound confusing, actually the retrieve task performs a resolve (which resolves dependencies and downloads them to a cache) followed by a retrieve (a copy of those file in a local project directory)”

Yes, it sounds confusing. I had to read that twice, just to be sure I haven’t simply garbled the sentence in my head. We have a resolve target and a retrieve task that performs a resolve. Why isn’t the “resolve” target simply called “retrieve”? But directly after that sentence, there is the rescue: “Check the How does it work ? page for details about that.” Let’s move on.

First, we see a great looking image:

and then (explaining “resolve”):

“The resolve time is the moment when ivy actually resolve the dependencies of one module. It first needs to access the ivy file of the module for which it resolves the dependencies.



Then, for each dependency declared in this file, it asks the appropriate resolver (according to configuration) to find the module (i.e. either an ivy file for it, or its artifacts if no ivy file can be found). It also uses a filesystem based cache to avoid asking for a dependency if it is already in cache (at least if possible, which is not the case with latest revisions).”

I am not a native speaker, so I won’t try to fix this gobbledegook, but please could someone review this text. I think it is nearly unchanged (or even more complicated) than some years ago. But if i understand it correctly, “resolve” does retrieve the files from the enterprise repository to the cache. “retrieve” then copies them to the project workspace:

“What is called retrieve in ivy is the fact to copy artifacts from the cache to another directory structure. This is done using a pattern, which indicates to ivy where the files should be copied.

For this, ivy uses the xml report in cache corresponding to the module it should retrieve to know which artifacts should be copied.”

Ok, I give up.

Objective-C? plain ugly

Now using a Mac for over a year I would really like to do some programming for OS X. Just to learn something new (and do some “real” work instead of creating slides).

But Objective-C?

  int main ( int argc, const char * argv[ ])
  {
     NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
     NSLog ( @”Programming is fun! ”) ;
     [pool drain];
     return 0;
  }

plain ugly. Mixing Smalltalk style message passing with C, throwing in some brackets like parentheses in Lisp? It gets worse: we have again to maintain an interface and an implementation file for each and every class. And even if there is now a garbage collector in Objective-C 2.0, you cannot use it when programming for the iPhone. Duh!

MacRuby

So I looked elsewhere, what about MacRuby? From Objective-C (Smalltalk-style):

  [person setFirstName:first lastName:last]

to MacRuby’s

  person.setFirstName(first, lastName:last)

or

  person.setFirstName first, :lastName => last

To me that looks rather, hm, unbalanced? The first argument has other syntactic sugar than the second. Ok, I understand that there are limitations to Ruby’s malleability, and it sure is better than RubyCocoas:

  person.setFirstName_lastName(first,last)

which even has other problems. I am not sure that I would like

  person <= (firstName: first, lastName: last)

better, if that would be possible in Ruby.

Conclusion: undecided.

F-Script

Some time ago, when I was looking for a Smalltalk implementation for OS X (and don’t start with that ugly Squeak) I stumbled upon a Script language called F-Script. Its main goals are to provide

• an embeddable scripting language for OS X applications
• to have an interactive shell (REPL) to experiment with the APIs and to prototype solutions

It is not intended to write complete applications, but John Sterling has a nice blog entry, that shows how it could be done.

Perhaps this is the way to go. Remember: I do not want to create a real application, just to play around and have fun.

Cards on the Wall

“If you want to make God laugh, tell him about your plans”
(Woody Allen, probably)

When it comes to creating project plans, I still use the “cards-on-the-wall” technique I first read about in 2001. This technique is rather old fashioned (compared to the current agile mainstream), but I still like it.

The main dimensions I am using are people (aka “staff”, aka “resources”) and time:

In reality it looks much less clean:

Each tasks should be exactly 5 days, but that is rarely the case, so you have to mark the estimated effort on the index cards (either with a simple number, or a visual cue: just some boxes (at most five, and some empty place holders.) You can add an ID, some description (preferably on the back) and what ever you like. Just keep it simple.

The main advantages for me are:

• Everyone (up to 6-7 people) can participate, can understand the reasoning, discuss and finally the whole group decides on this plan.
• It is much easier and faster to move tasks around, group them, put them aside for a moment etc. On a computer screen I can never do that

Sure it’s still a plan (someone is laughing). And therefore will be changed. The first planning round is just to see, if the project is feasible, to discover blunders, previously overlooked dependencies and especially missing tasks. And at the end of each iteration, we will adapt the plan — sometimes even more often.

Much too often, there are no user stories, there is just a plain old “functional requirements” document. So this is the best technique I could find to create such a plan. The alternative is much too grim: sitting on my computer, at best with one other person, and creating some monster of a wallpaper that no one ever wants to change and no one understands — me included.

But here’s the catch: in most cases someone wants this plan in electronic form and wants some progress report based on that plan. Let’s not start a discussion, if that is the most effective way to track a project. Let’s say it is just part of some contract or something like that (in my experience you can convince people higher up, that this plan is sufficient; as long as you take some pretty pictures as the one above for record keeping, but it takes time, it takes at least one successful project).

So how do I create that plan? Back to some project planning monster? I haven’t yet found a program that has a view like the one above, where I can …

1. … assign tasks by drag and drop to some “resources”
2. … still move taks around

At least for the first requirement I finally have found a tool, at least if you are using a Mac: OmniPlan. You can see it in action for yourself here. If only the dependencies would be visible and if I could still directly move a task around (change dependencies, prioritize, etc.) in that view.

Groovy makes Java Strings mutable

Just stumbled about an entry in the Groovy mailing list

def input = '''<tag id="12" />'''
def tag = new XmlParser().parseText(input)
tag.attribute("id").value = "ABCD"
println tag.attribute("id")
==>
AB

Oops. The object returned by tag.attribute(“id”) is of type String (i.e. java.lang.String) and not of some type XmlAttribute. And the ‘value’ is the private final (sic!) variable containing the String’s content (a char array). And Groovy allows us to change that value … making String mutable.

Even funnier:

...
println tag.attribute("id").value
==>
[A, B, C, D]

I always had the feeling, that Groovy was programmed first and then designed afterwards. In the last years they progressed nicely, but this makes me doubt again. This issue has been known since May 2007. But I think they just realized what it means: “Yikes! String is mutable!“

Update (April, 24th): Created a new issue, because the existing issue is only concerned with violating the “private” contract and not with violating the “final” contract.

Please vote for the issue to be fixed.

REST vs. WS-* … Does it help me to achieve my goal?

The discussion about REST vs. SOAP seems to be endless, so — why bother?

Because there seems to be a thorough misunderstanding between the two camps that needs to be resolved. A misunderstanding that pertains to different values hold by each camp. Perhaps “values” is to big a word and viewpoint might be enough, but the discussion evokes so many emotions so much “you just don’t get it” so much ferociousness, “values” seems to be the right notion. (This is my second try to put these thoughts into the right words (I even created my own blog for this), you might want to read that discussion first.)

There are two central issues at play:

1. “The Web” as an entity that can be abused
2. API design

I think the latter is more important and more fundamental, but let me try to get the first out of the way.

The “abused” Web

… or better: the abused HTTP protocol.

Proponents of the REST approach consider using HTTP as a mere transport as abusive (e.g. Tilkov, Vinovsky). For me, using a word like “abuse” sounds like putting HTTP on a throne. You have to worship it. Using it for another purpose as was intended is against the … what? As far as I remember, SMS were initially not intended for use by mere mortals, so sending a message via SMS is an abuse? James Snell puts it right:

“there will always be multiple technologies. We use the technologies that best suit the problem we’re trying to solve. It just so happens that HTTP is well suited to address a hell of a lot of problems.”

If a tool helps me to achieve a goal, I will use it. Perhaps there are better tools or more efficient ways to do things. We can talk about that, but “abuse”?. Do I abuse a hammer when I use its handle to get something out from under the bed?

I hope you can agree with me, but Stefan Tilkov throws in the following analogy:

“would you consider using a RDBMS and then defining a single table only, with two columns, ‘id’ and ‘content’ (the latter a BLOB) an abuse of the RDB”

Yes, it sounds like using a RDB in the wrong way, but if I happen to have a RDB at hand, and the data is unstructured, or simply easier store as a BLOB, so why not? Does it help me to achieve my goal? Sorry, I do not “believe” in “the Web”, the “HTTP protocol” as entities that should be worshiped and defended against the unbelievers.

So if you think that WS-* is an abuse, please let us simply state that we disagree on this point and move on. Can we?

API design

WS is centered around verbs (what you can do with things) and REST is centered around resources (the things itselves). I could stop here, because I think, that’s all there is. But then why is the debate still going on and on and on? Because this is a cultural issue. And as long as you are part of one culture it is very difficult to see value in another culture. Let me try to explain that.

When you have to integrate a bunch of legacy systems that all have some kind of (R)PC-API, it just feels natural to wrap them into WebServices. You do not have to translate all the existing verbs into their RESTful equivalent. In my comment to Stefan, I offered the following existing API as an example:

 (d) scoreCustomerMortgageRisk(customerId)
     // i.e. the customer has send in a mortgage application,
     // that should now be "scored" to decide if we can offer
     // him a credit contract

 (e) promoteToCustomer(personId)
     // a person that is in our system, will now become a
     // customer (an account will be opend, etc.)

 (f) calculateNetInterestRate(credit)

and Daniel Yokomizo translated them into one possible RESTful API:

 (d) PUT /mortage-application/{id}/score

 (e) POST /people/{id}/promotions
     =returning> 303 /customer/{id}

and with (f) he cannot see the “resource”, perhaps because there was no explanation in my comment, but he correctly assumes that this really is nothing but a calculation and he even thinks that RPC might be the right thing (so, not everyone is a zealot in the REST camp).

I think this is the best, most concrete illustration of the different mind sets. For me this looks artificial. Is “mortgage-application” a resource? Is “score” a verb or a resource? I have to translate something that looks natural to me into something unusual. If I had to document the REST services, I still would describe (e) as “Promote to Customer”, because this is the culture I grew up with. But if I had to design an API for resources that are accessible via the Web from scratch, I might try REST, especially if I have (more) people in my team that are experienced in a RESTful style. It still all comes down to my central mantra “Does it help me to achieve my goal?”

Some final remarks and disclaimers

I think XML Schema is a mess and WSDL even more so, but I would like to have a rather formal description of the XML messages I have to send and can receive. Not to generate code, but to improve my code as a consumer of a RESTful web service.

There is a difference, whether you are creating an API that is really used between two parties over the Internet, or if you are (just) connecting systems inside your company using a “platform independent” mechanism. In the latter case I would mostly prefer an RPC style in the previous case I would very thoroughly think about my prospective users (if I want as many users of my services as soon as possible, REST is much simpler to use than WS-*).

I do not question that I can translate each and every verb of my RPC-style API into the four HTTP verbs and some deliberate URL. I know about Turing machines, but I think we all are happy that we do not have to program Turing machines directly. Restricting myself to a set of very strongly defined verbs, might even help, but I fear that eventually the URIs will be very arbitrary. If that is true, nothing will be gained.