This post is about software change management, the problem and procedures involved with maintaining progress and quality of a team-developed software solution over time.

As Alex puts it in his unusual DailyWTF post this is a discipline that is neither part of software development nor software engineering. And more so, the inherent problems are hardly recognized by most developers nor architects. For most IT people this is located somewhere in the Q-Space, something somebody has to take care of and usually only gets on your nerves. I am no exception to that. It took me years to realize that this is the spine of the software development process. Initially almost any approach will do, but when team and solutions grow, a weak spine will bend and finally break.

When asked about software change management most developers will think of SCM systems like Suversion, Git, or Perfore – the tools that we are faced with when generating change in the first place.

The frustrating part of change management starts when selectively moving changes from one “solution version”, say a branch, to another: Most everybody will have had these super frustrating moments when merging changes has turned into an endless, painful double maintenance experience.

Here is a simplified view of places that receive changes (typically a branch in some SCM, at the far right end in many cases a runtime):

The problem does not lie in re-merging of feature branches – illustrated on the left side of the flow diagram. Those flows are rather well addressed by standard SCM workflows (more true for Git than Subversion though).

Neither does the final push of changes to the execution environment pose a problem – typically a straightforward forward-merge.

The merge approach breaks down when managing the conflicting interest of feature integration and selective feature release into pre-production stabilization. I.e. when crossing the “divide” in the diagram:

• Without seeing features work in integration, all talk about feature readiness is moot. This is particularly true for solution/application development. There is a need to have changes enter into integration as soon as possible. On the other hand:
• You do not want to stall development. There is no pause. In order to qualify for production a subset of all change history must be isolated from receiving further changes to be tested (and usually fixed) to qualify for production.

The reason the second issue is hard lies in the nature of solution development. Technologists suggest to approach integration problems by reducing the size of components and attempts to define slowly changing component interfaces, so that you can version and release on disjoint parts of the code base. In application development however complexity has a tendency to spread out: End-user requirements typically do not map well to module boundaries so that there is constant overlap between completed developments and half-done features.

The selective push of features from integration to stabilization has a number of technical problems and some more interesting process problems. The decision to release a set of changes that make a feature should be with the ones being ultimately responsible to also fix problems later on: The development team, or preferably the individual developer who introduced the changes. Technically changes made by different developers may interfere with each other so that management of a change release queue is necessary.

There is other approaches out there. For example the Facebook style “release everything from trunk every week” (and hence avoiding selective integration at all, see here). But that applies “only” to the Facebook frontend where features can even be turned on and off selectively, and where the backend takes the burden of maintaining consistency (plus it’s functionally a rather simple application anyway). I doubt that that would work well for Facebook backend development.

So much for the problem description.

Coincidentally I know from my past experience that this is one of the problems that traditional SAP software logistics solves quite nicely. Unfortunately I was not able to identify a publicly available tool that addresses these problems.

How are we doing this today? We muddle through, try to identify complete snapshots, stall development…. it sucks.

It would be extremely cool to have a tool that

1. allows developers to group changes into “feature change sets” and release them for integration into another code line
2. manages a change set queue and implements workflows to resolve dependencies on the queue of change sets

Know something? Please let me know as well.

References

• http://thedailywtf.com/Articles/Release-Management-Done-Right.aspx
• http://www.facebook.com/video/video.php?v=10100259101684977

This post is about asynchronous work in back end software. It is part one of a series of two posts.

This part looks at use cases and motivations for server-side background work – some of which most developers will have run into – and we prepare the stage for the implementation described in the second post: Getting Work Done with Quartz and Z2.

With background work or asynchronous work we denote activities that happen outside the scope of user interactions – but within the user’s business scope of course.

In essentially any not completely trivial business software, there is some work that is best offloaded to a later point in time or to different computing resources. Some typical use cases are

• Updating a search index or some analytical data (can be expensive and time consuming)
• Following up on some state change of a business entity, e.g. by sending emails
• Producing a state change of a business entity due to an expiration condition

These examples already give a good clue about the main motivation for background work:

1. It has to be done once in while – but cannot be done on-demand as it is too expensive (either in time or space)
2. It has to be done, but it is not necessary to have the user wait for it (and hence impact the user experience)
3. It has to be done at some defined later point in time (e.g. order to be revisited two weeks after creation).

Another, great motivation to put work into the background is …

4. … that the operation is not reliable and may need to be tried again at some later point in time (e.g. mail server problems, overload, etc.) and finally that
5. … that resources may be used more efficiently, if work can be queued up and processed when resources are available rather when there is peaks of user load
6. … to move the work elsewhere so that e.g. front end resource are not impacted and so that user experience is not harmed

All of these cases fall into one of the following two categories:

Asynchronous GET (AGET): The operation is idempotent. The operation generates some information, that – if nothing else is changed – stays unchanged. Computing analytical insight is an example for this case. This kind of work can be postponed, or repeated as convenient. The reason for an “Async GET” is an optimization.

Asynchronous POST (APOST): The operation implies a state change. It should be performed either as soon as possible or at some given point in time. An APOST corresponds to a user interaction. An APOST typically needs to some input data for processing. Once done, the operation may not be repeated. The reason for an APOST is an optimization of user experience or some intended deferment.

The analogy with the HTTP method GET and POST is intended and complete. In fact, all the sample use cases above, any use case I have encountered, can be modeled as scheduled GET or a deferred POST request. The analogy carries further: A scheduling system should indeed allow to trigger an AGET or retrigger an APOST at any time in a way that an administrator can completely infer the meaning and target of the “request”.

The analogy reduces the scheduling system to where it belongs: Managing activity triggers in time. This includes:

• Some concurrency control: In many cases it is more efficient that the scheduling system doesn’t even try is some other activity is still ongoing rather than blocking on the application layer.
• Ability to re-attempt some activity for some number of times and with some delay
• Some visual management of pending or failed activities

More on that in “Getting Work Done with Quartz and Z2.”. Stay tuned!

This post is about Apache Maven and why I think it does not deliver what many developers hope to get from it.

Maven is a build and dependency management tool. Maven claims to be something more general, but I think that is misleading:

<quote from=”Maven Getting Started Guide“>
At first glance Maven can appear to be many things, but in a nutshell Maven is an attempt to apply patterns to a project’s build infrastructure in order to promote comprehension and productivity by providing a clear path in the use of best practices. Maven is essentially a project management and comprehension tool… .
</quote>

Whatever… let’s call it a build tool for the moment. When people start looking into Maven, that’s is the problem they are trying to solve. All the other aspects of Maven, i.e. all the other stuff it does, I’d call a consequence of the environment it was initially developed for and where it grew up.

Beyond general curiosity that is also why I looked into Maven. I even recommended Maven to address software production issues for a non-trivial software system. I would not do that anymore. Here is why.

When I started looking into Maven I wanted to solve the problem of “maintaining a solution consisting of several interrelated modules without caring much about build problems”. As it turned out, that’s not what Maven is built for.

How Does It Work?

Let’s not look into how Maven invokes a compiler or build plugins and such. The more significant aspect is that Maven relies on repositories that hold artifacts, by name and version, typically produced by other Maven builds. In fact, you have one locally. When Maven cannot find what it needs, it will go to remote repositories and fill your local repository that then serves as a cache. Also Maven plugins, the code that actually implements the very thing you wanted it to do in the first place comes from there (which is consistent). That’s why Maven, on first call for anything, seems to virtually “download half the internet” (which, no, I am not complaining about. That is just bootstrapping).

Maven projects are described in a Project Object Model (POM). Ignoring the details, the POM describes project type, some version numbering and dependencies (by name and version). Dependencies are resolved using the repositories.

You put stuff into a local repository by installing it. You put it into remote repositories by deploying it (see references below).

So in short, apart from actually invoking plugins that transform some source code artifact into something better process- or  machine-processable, Maven is all about managing artifacts between your local development environment and some remote storage that provides missing dependencies to you and others.

What Problem Does It Solve?

Maven was built for Apache. And that’s where the approach makes most sense. Many  projects are single-project really with dependencies to stuff that is out of individual reach, so that Maven’s repository approach provides an automated, defined interface to getting a hands on some other project, in the form of its (binary) output, it’s released artefacts.

So, Maven is great, if you want to integrate into the Apache Eco-System as you are right in it – from the start – and you have well-defined interface to express your dependency onto other people’s projects.

Maven for Solution Development?

When developing and maintaining a software system that comprises of any non-trivial number of modules however, you will want:

• Integrity: You do not want to fiddle around and dive deeply into your dependency version vector. There should be one place to look for what is being used in your system, and their should be no doubt whatsoever where that comes from.
• Modifiability: Sooner or later you will need to fix something that initially was taken from elsewhere and is now part of your system.
• Repeatability: Whether you look at your system code now or in two years. If and when you need to build it, the result should be guaranteed to be identical.

So… thinking about it, what this means is that you – at least – need to take ownership of the surrounding infrastructure. I. e. you need implement/set up your own shared repositories to guarantee availability and repeatability. You may need to take over local versions of other projects to make sure you can patch. So that’s a principle problem that may not be obvious in the beginning: You are now in charge of operating a non-trivial distributed infrastructure.

Secondly, you buy into the complexity of managing non-trivial version vectors although you really didn’t mean to build a network of independent components that somehow get intelligently integrated into a solution. You just wanted to work on your solution. Maybe even in a Product Line Engineering style. But you definitely didn’t want fine-grained module level versioning.

So in short, something that is useful for a large number of relatively small and independent groups of interest does not apply well to highly interdependent groups that work on a large, modular software system.

Conclusion

Well… you can guess where this is going. I am convinced that solution development and maintenance demands a system-centric approach – and I have seen that successfully implemented. Solving it by adding secondary tools and configuration from the “outside” that eventually has a similar complexity as the actual solution simply doesn’t cut it.

References

• History of Maven: http://maven.apache.org/background/history-of-maven.html
• Funny rant: http://tech.puredanger.com/2009/01/28/maven-adoption-curve/
• Maven vs. Ant: http://www.iternum.com/knowhow/guidelines/maven-vs-ant/document.pdf
• Plugin reference: http://maven.apache.org/plugins/index.html
• System Centric Development

Lately we have been visiting a potential customer of ours that is interested in learning more about z2 and eventually using it to increase development productivity, maintainability as well as customizability of customer installations.

The approach behind z2, i.e. what we like to call System Centric, to a large extent and regardless of the implementation implies or at least fosters these attributes. As there is little to no development setup, no build fixing, and integration happens essentially as soon as meaningfully possible, productivity naturally goes up. As sources and customization is always at hand, maintaining and customizing further works just as well as the solution at hand intrinsically allows.

What is strikes me is that developers and operations professionals with an ERP background (disclaimer: not even SAP in this case) typically get this right away.

That is strangely contrasted by the observation that Java developers have a much harder time to notice the waste of time and the added complexity that  gets dragged along. That may be due to being so deeply used to the working mode  that it has become close to a natural law – nothing to question anymore.

Another reason could be that every now and then some toolset manages to creates a level of hype and a so deeply convinced following that tools expand beyond their space of usefulness. In other words, there is a lot of “golden hammers” out there. One now historical example is the Enterprise Java Beans (EJB) technology, that actually started out as Remote Objects – which makes sense – and then got perverted into some sort of locked away, XML polluted, and finally slowly starved “Java Objects Deluxe”.

One more recent example is the Maven toolset. Not saying that it doesn’t have it’s place and problem to solve. It does. Development and maintenance for business solutions is however not part of the Maven problem space. More on that in another post.

In the specific case, the development team had lost a lot of time fiddling with OSGi dependencies, building and deploying bundles, keeping installations in sync, keeping version vectors straight and more technical aspects of solution assembly – none of which added value to the solution.

When storing massive amounts of data in a distributed storage the Map/Reduce (M/R) algorithm provides a generic and effective means to compute data transformations spanning large datasets. The underlying assumption is that each storage node (think: machine) provides computing power that can be used for custom, parallelized data processing and benefits strongly from data locality.

At Original1 we use Apache Hadoop, a popular open source implemention of a distributed file system (HDFS) and of an M/R runtime in conjunction with Apache HBase, a NoSQL database, to store large numbers of rather simply structured, versioned data records.

In order to provide analytical insight across all data records, we use Map/Reduce jobs that compute aggregations, look at anormalities in record history and more.

One of the attractive aspects of Map/Reduce jobs is that it is your code that is executed directly where the data is stored. Not some restricted query language. That is, on the other hand, also one of the biggest weaknesses: The M/R implementation is a very weak and limiting application container.

This post is about how the z2-Environment can be used to overcome this limitation and how to integrate M/R seamlessly into your solution.

What’s the Problem?

In order to run your M/R implementation you need to package the job’s code into a JAR archive and schedule the job with Hadoop’s Job Tracker. Your JAR gets eventually downloaded to all data nodes and executed locally. This is no unreasonable limitation, if that’s all you want to do: Run M/R jobs. Typically however, data analysis is just one part of the overall solution and in order to make sense out of the data, you will want to reuse a lot of the code that you used elsewhere, e.g. domain types, service implementations, utilities.

Also, whatever is the result your job delivers, frequently it will need RDBMS stored data to be completed and/or stored. And no, contrary to some enthusiasts, HBase/Hadoop or other NoSQL storage engines will put no end to relational databases.

So, in short, while the technology is an interesting addition, it is poorly integrated.

How Z2 Fixes This

The integration of z2 with Hadoop is an example of using the embedded mode of z2 (see embedded). When scheduling an M/R job, below is an example of how that happens, z2 will actually upload a JAR with Hadoop for execution. That JAR however is completely generic and all it does is to start the z2 runtime, retrieve the actual Job component and delegate the actual Hadoop interface calls (Mapper, Combiner, Reducer) to the z2 hosted implementation.

The embedded z2 works as always: It checks whether local updates from the connected SCM are required and updates (and compiles if applicable) on demand. Your Job component can do what any other component of your solution can do, e.g. re-use code, connect with a database etc. It is defined where all the rest is defined, in the very same module/component structure.

Anatomy of an M/R Job

As part of the integration, the Hadoop module provides a component type com.zfabrik.hadoop.job. A component of this type is supposed to provide an implementation of the interface IMapReduceJob that describes everything a Job implementation has to say about its execution:

public interface IMapReduceJob<MKEYIN,MVALUEIN,MKEYOUT,MVALUEOUT,RKEYOUT,RVALUEOUT> {
    String TYPE = "com.zfabrik.hadoop.job";
    void init(String name);

    boolean hasMapper();
    boolean hasCombiner();
    boolean hasReducer();

    Mapper<MKEYIN,MVALUEIN,MKEYOUT,MVALUEOUT> getMapper(Configuration configuration);
    Reducer<MKEYOUT,MVALUEOUT,MKEYOUT,MVALUEOUT> getCombiner(Configuration configuration);
    Reducer<MKEYOUT,MVALUEOUT,RKEYOUT,RVALUEOUT> getReducer(Configuration configuration);

    Job configure(Configuration configuration) throws Exception;
}

The methods are relevant at different parts in the life cycle of an M/R job. In order to schedule, the configure method will be called. Also, at the time of configuration, the job will be asked about what M/R phases it supports (hasMapper, hasCombiner, hasReducer).

At execution time, on the data nodes, the same implementation will be asked to provide concrete implementations (getMapper, getCombiner, getReducer). Hence, the interface is really just a simple abstraction of a job definition.

For example, the configure method for the Word Count example (Hadoop’s Hello World program, see http://wiki.apache.org/hadoop/WordCount), would look like this:

 public Job configure(Configuration conf) throws Exception {
    Job job = new Job(conf);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);
    FileInputFormat.addInputPath(job, new Path(this.inPath));
    FileOutputFormat.setOutputPath(job, new Path(this.outPath));
    return job;
}

Very much like the original – if not simpler. The difference is that the configuration is preset by z2, in order to abstract the actual connectivity information (which should of course not be part of the component), and that the actual job submission, for similar reasons, is not in the job component implementation.

Jobs can be scheduled programmatically or from a simple web interface for execution on any configured hadoop cluster or locally (for debugging).

Programmatic scheduling looks like this:

IJobConfigurator jc = IComponentsLookup.INSTANCE.lookup(jobComponentName,IJobConfigurator.class);
Configuration c = IComponentsLookup.INSTANCE.lookup(hadoopConfigurationComponentName,Configuration.class);
jc.configure(c);
jc.submit();

The first lookup is for the Job Component. E.g. the Word Count job implementation. The second lookup is for the Hadoop client configuration component. Another component type that simply holds a core-site.xml file (Hadoop’s properties in XML file format) that defines how to connect to Hadoop (and possibly HBase) and more.

Be aware that programmatic scheduling is a common, although not completely trivial use-case: In most cases M/R jobs will be scheduled either on a by-time basis (i.e. via some cron-type scheduling) or as the result of some data change that invalidates some previous computation. Smart scheduling of jobs is a worthy subject on its own (i.e. a related post just got scheduled).

Where Can I Get That?

This technology is used by Original1. We are planning to release it in a new distribution (working title z2@hadoop) anytime soon (i.e. as soon as we get to it). That’s why this post is called “Part 1″ – Part 2 will point to the actual distribution and documentation. If you want it earlier – talk to us.

Summary

The integration of z2 with Hadoop is another strong case of the System Centric approach. Via a seamless integration, Hadoop can be added to your solution space without the need to change the structure of other parts and with full re-use capabilities.

Software projects, eventually, go through one or more “Cycles of Agony”.

At the start of a software project, opportunities look promising, productivity and motivation is high. It takes experience and sweat to come up with a somewhat stable initial structure that is instructive, scales with the problem space, is sufficiently modular (i.e. keeps complexity manageable).

Regardless how well you manage all necessary compromises (e.g. productivity vs. process), how well you were able to foresee required tools and technologies, how well the business case was understood: Projects (if not abandoned) eventually run into a state of unproductivity that shows one or more of the following symptoms:

• Developers are unsure on how things are supposed to be done,
• The architecture tends to show unneeded abstractions of previous abstractions,
• Conceptual integrity breaks down,
• a general fear of touching things that “worked before” (no daring)

It’s unavoidable. Inherently, the business scenario at hand is only partially understood and will most likely change over time, inherently new use-cases, new technologies to integrate with, new customer requirements will show up (why spend money on developing a new software otherwise at all?). Hence the probability to make only the right far-reaching decisions is essentially zero.

The Agony Cycle looks somewhat like this:

What to do, when you are in the “agony” part of the chart? Well, it depends!

If the solution at hand is a “cash cow” (rather than a star), it may well make sense to not dare big change and consciously  move on into indefinite maintenance (and for developers to leave). Otherwise though, tough decisions are required to get you back on track. Previous decisions have to be reverted, possibly requiring non-trivial refactorings that do not result in immediate benefits. Although you (hopefully) have a product already, you need to pay a price that doesn’t linearly add obvious value (a concept that looked much more acceptable in the beginning). Whereas in the beginning of the cycle craftmanship was essential, now the willingness for change is key to further success.

In larger organizations, the risk of change may be considered too high and the need of change is politically undesired. That’s one reason why we see seemingly inexplicable failure to innovate on existing products by big players in the market.

This post is inspired by a discussion with a development lead of a company nearby and by the blog post “What about complexity in software“.

The specific discussion was about the idea of using a scripting language (think PHP, Python, Ruby, Groovy et al) for developing business applications and integration features.

A perfectly reasonable approach. Everything inside me rejected it.

But:

There is more context to the discussion of course – which is why the formulation above is slightly unfair to the reader. But before going into any detail, let’s disect the situation a little bit.

For simplicity let’s assume you already have some software asset that you need to develop further in some specified but rather broad direction (i.e. expectations unclear as so often – but there is a distinct feeling of necessity).

Now, every time you make a decision about adding some tool (or technology, or team split, or…) into the big picture  you do not only make a step in some direction but you do also add complexity and limit further possibilities. While you can achieve more short term, it will typically be hard to revert that other decision or to make a competing, challenging, similar decision later on.

How hard changing a decision is, depends on how easily the change can be implemented. When changes can be implemented more or less atomically and others involved are not confronted with the unexpected, we tend to call that a refactoring**. All other cases are revolutions, causing significant pain before showing tangible improvements. If the pain and risk factor is large and your organization is sufficiently political, decisions will probably not be taken at all (or late, or for political reasons) and you should generally consider yourself screwed (bye bye progress!).

That said you should naturally strive to delay the unavoidable and:

1. Make sure you really need to pick a tool at all.
2. Make sure to pick a tool that carries as far as possible – albeit possibly not the most sexy one short term (so that you can live with it longer – even if you do not love it).
3. Make sure to pick a tool that can scale with your problem space.

Back to the Point

Enough of that and back to the beginning of the post and the specific question of whether to use a scripting language for application development is a smart thing to do.

Scripting languages (at least the ones named above) are tuned for productivity. The general idea is provide for productivity by requiring little declarative “overhead”, provide simple and versatile data structures, high-level rich operations, and be relaxed about type safety and sometimes syntax.

Scripting languages are great for “wide but thin”: Shallow layering of code, mostly local data structure, many variations of the same pattern. Heavy re-use of non-trivial data structures and interfaces, deeply designed call hierarchies, performance-critical code are not what you should bring to the party.

Considering bullet point two above, if you know you are in for “wide but thin” and you can handle other known implications (e.g. deployment and distribution), go for the lightness and productivity of your favorite scripting tool. If you don’t know what you are in for, pick a tool that carries further – i.e. Java (it was about time I’d say that!), or C#, or C/C++ even (depending on your problem domain again – of course) – although that may be perceived as harder.

—

**) The ability to refactor instead of “throw away and redo” or “live with the pain eternally” cannot be praised enough.

In my previous life at SAP, I was first part of the team designing and developing the runtime of the Enterprise Portal product and later I was part of the Java EE Engine (a.k.a. SAP Netweaver/Java) team.

Both systems were initially really simple. Simple in the sense that a runtime developer could typically work on the whole system at once – as far as he or she was concerned at least. It would all fit into one development workspace. If you wanted to test or debug something: No problem. Just check out the whole thing, build it on your machine, and run it.

Life was good.

That changed. Bug reports were coming in for some already (internally) released version. How to find out what was really running on the other side? How to reproduce the bug in your environment? Still you could stretch things somehow. People might allow you to deploy patches and try again. Still somehow manageable. We got used to it.

But then things got more complex. And all the sudden you found yourself not anymore at the more comfortable lower layers but rather sandwiched between some framework you used (that library over here) and users of your code (this complex HR application over there). Add the versioning problem and the mess is complete.

At some point in time setting up a complete environment required some non-trivial installation and tweaking that could easily stop you for half a day. Potentially leading to the devastating outcome that the weekly build of service pack x actually didn’t really have the fix you expected because change list 52304 did not make it into that week’s build. Needless to say that developers rather stayed longer with what they had. Longer than healthy: Integration problems eradicated all deadline fantasies out there.

Was that just lack of discipline? Not really… it was a mix of non-human-manageable complexity and the superhuman forensic instinct required to identify exactly that source code revision made your environment and whose fault it was (in a system of hundreds of components that were last updated at various points in time in various source code repositories with a deep dependency graph). I guess there is a way to set things up so that everything is easily trackable in such complex environments. I doubt that anybody has managed. Not over any significant time. Not with more than a handful of interdependent components that are developed at the same time.

Life was not good anymore.

One day I was working on a Web Service interop and performance test. One side implemented in an ABAP system, the other side implemented on Netweaver/Java.

Something was broken on the ABAP side. Couldn’t get it to work. Called a developer: „Hey, ABC is not working in BYX, any idea why?“, „Hmm.. let’s see… (click, click,…) ah yes, there is that bug XYZ. It’s a trivial fix. Ok. fixed. Can you try again?“. Wow… works!

Something was broken on the Java side. Couldn’t get it to work. Called a developer. „Hey, DEF is not working on my installation. Any idea why?“. „What version do you have?“. „Should be DWI CW34“, „Let’s see… (wait)… I think that bug TFB was fixed in CW42“. „Can you send me a patch?“. „Hmmm…(wait)… you should deploy the latest SDAs (deployables) from here,… but let’s see… they require this and that.. you could try?!“. Well… ok,.. maybe… maybe better not.

That is what System Centric is all about:

Know what is in the system. Be able to modify anything in the system down to the single line level. Allow system-local modification. Be self-contained: Always keep the overall consistency by definition and without dependency on external infrastructure.

In a system centric approach the difference between the source and configuration store vs. the executing runtime is the same as the difference between a blueprint of a house and an actual house. The runtime is nothing more than an actual instance of the system exactly as described and configured in the system’s repository. When looking at the repository you would say “this is the system”, just as when looking at the blueprint of a house you would say “this is the house”. We like to say “The source is the system“.

Contrast that with a server that is build of a mix of binary components pulled from many different places cobbled together with a number of deployed applications from even more indefinite sources. Phew…

Want more? Go here: http://www.z2-environment.eu/v20doc.

As you have undoubtedly learned by now, z2 compiles on its own, on-demand. It uses the Eclipse Compiler for Java (ECJ) for that. It used to use the build it Javac from the JDK more on that in some other post.

Lombok is a rather cool (although maybe slightly disturbing) approach to add getters and setters and more to pretty plain Java objects w/o typing them. A minor feature in a way – and maybe not really important – but definitely a noise remover. And a cool example to show off how well z2 holds things together even when enhancing the compilation step.

Using Lombok, you can use POJOs w/o the need to declare setters and getters before you actually have a good reason to do so:

package test;

import lombok.Getter;
import lombok.Setter;

public class MyBean {

    @Getter
    @Setter
    private String text;
}

Lombok, under the hood makes sure that methods setText, getText are available.

Compiling Java source code to Java byte code is  just one of the things that need to be happening before code can be executed – in some cases. To give extensions a chance to be able to enhance what happens during compilation, z2 offers a compiler API.

Both use-cases apply:

Modifying source code before compilation: In some cases it is most handy to prepare source code before compilation already – like a preprocessor. This is what we use for Lombok below. If you know Lombok, that may come as a surprise. Read on.

Modifying byte code after compilation: Other tools understand byte code already and better than source code. So you can do that as well. This method was used in the z2@Spring distribution supporting weaving of the AspectJ aspect of the spring framework. No idea what that means? Read on here: Spring features in Z2, Spring AspectJ.

But back to Lombok: Here is a short descriptions how it’s done. More links below.

The compiler API is very simple. There is only one interface: ICompiler, that has one method:

boolean compile(String compName, File[] src, File dest, ClassLoader cl);

Whenever called, the job of a compiler is to transform source files in the folders passed in src or compilation results of other compiler invocations in the folder passed as dest as desired.

The set of compilers to be used is a Java component specific configuration. I.e. on a Java component descriptor (the z.properties file) you specify the set of compilers to apply as well as their invocation order. E.g. like this:

java.compile.order=lombok,java

The default setting is “java.compile.order=java”.

Now to the construction of the actual extension:

A compiler extensions is declared as a z2 component (of course). In its component descriptor it declares its type, its unique id as a compiler, as well as its implementation class:

com.zfabrik.component.type=com.zfabrik.compiler
compiler.id=lombok
component.className=com.zfabrik.impl.LombokCompiler

The implementation class uses the Delombok source converter tool as follows:

1. Create a temp folder at <component root>/gen/lombok
2. Compute a class path based on the class path class loader (see above). Delombok requires a class path.
3. Process all source files in the src folders using Delombok
4. Copy the processed source files back to where they were found originally

View the source code at: LombokCompiler.java (use z2_base/z2_base)

If you have a running z2 setup (see Check it out), you can play around with this by simply checking out the projects at

http://www.z2-environment.net/svn/z2_base/trunk/offline/lombok

(again: use z2_base/z2_base when asked)

into your eclipse workspace and sync’ing your runtime. The simple web app at /lombok demos a simple Lombok processed Bean in a JSP.

There is one pre-requisite however: You need to have the tools.jar from your JDK on your classpath. The simplest way to do so is to copy tools.jar from JAVA_HOME/lib to JAVA_HOME/jre/lib/ext.

Why is this cool?

1. We reduced the application of lombok to your Java code to a mere one-liner
2. From the outside, e.g. when using the Eclipsoid plugin to fullfill project dependencies, it’s as if the Java Beans were indeed normal Java Beans in the first Place.
3. On the downside you need a JDK rather just a JRE and need to provide the tools.jar (that contains the Javac that Delombok requires).

Links:

• Lombok
• Check out z2_base

My name is Henning Blohm. I am co-found and co-owner of a small software startup, ZFabrik Software, located near Heidelberg in Germany.

Previously I spent a significant part of my life at SAP AG, mostly working on Java topics. SAP does not do as much Java as some people may believe – I learned a lot by working in an environment where Java was more in a defending position rather than the champion. By far, most of the code SAP customers get to enjoy is written in ABAP, SAP’s proprietary programming language. Arguably an old-fashioned, hardly beautiful language. But don’t be fooled: It’s not about the language. It’s all about the software engineering approaches behind it.

A common tune of this blog will be just that: Things that suck in “Java-style” software engineering vs. other approaches.

One tangible result of my learnings at SAP is the z2-Environment: A “System Centric” approach to Java development (w/o breaking Java’s neck). I am sure that will play a big role in this blog.

Some other technologies that we use heavily in projects will certainly play a role. Most notably Hadoop/HBase and the Vaadin UI toolkit.

It was about time. Wanted to get started on this a long time back: A blog about all things “z”.