Lately my colleague Udo has been busy adding Groovy support to Z2. For those who are not aware, Groovy is a scripting language that is deeply rooted in the Java platform. While Groovy differs in Syntax, Groovy classes interoperate seamlessly with Java classes.

The cool aspect of integrating Groovy, apart from Groovy itself, is that the integration is extremely smooth and required no adaptation of Z2 as such. It’s all just yet another compiler. Using the ability to use Groovy code or any mix of Groovy code with Java code just means add the Groovy Add-On  to your system and to add the declaration

java.compile.order = groovy

to a Java component declaration. That’s it.

We prepared a little sample that highlights just a few capabilities such as

• Mixing Java and Groovy
• Using Groovlets and GSPs
• Using the Groovy-based Spock test specification framework

Check it out at Groovy in Z2 sample.

When I talk about Z2 I usually show a picture like this one:

I do that for a purpose of course. It is meant to provide the impression that everything is driven from the repositories. I.e. it’s system centric. To propagate changes, you only need to propagate changes. Nothing else requires thinking. Does that come across?

That picture captures a real development and production landscape of a previous project (except that I added the hubcr lately). So this is not an idea, it is meant to explain something existing. Hence it lacks all that was deemed obvious to the reader (developers familiar with the technology) at the time. The obvious at that time is of course all that matters when talking to people not familiar with Z2.

It does show repositories and change flow along some dev → staging → production chain. It shows developer environments on the bottom, some automated testing environment on the top left, some staging installation at the top middle and a production landscape on the right.

Coming back to the motivation for this blog’s title: As runtimes will pick up changes by themselves, because there is no deployments and the likes, there is no such thing as a roll out in the sense of a complicated activity. Changes can be applied and made effective at minimal impact on the operational complexity. If you need fix a single line of code, than that’s the most complex thing you need to do.

A few examples:

• A pre-production environment, called “staging” here, provides a de-coupling from daily development progress for production qualification. From a developer perspective, fixing the staging system means to check out a Z2 core, fire it up, test local changes, commit/push.
  Due to its pre-configured environment, the developer’s local staging installation will pull anything needed from the right repositories and would even use a shared staging application database.
• Running automated integration tests means exactly that: Running them. There is virtually no environment to write setup scripts for. Changes will be picked up automatically.
• Applying a critical hot-fix in production requires to apply the minimal change possibly (after testing in pre-prod) in the production code base and to trigger a synchronization on the hub, restarting the servers as appropriate.

What makes repairs minimally invasive is not the amount of changed code, it’s the little amount of work to be done and the little risk to be taken when applying changes.

After some discussion following the HubCR description in the previous post I thought it would be nice to describe more useful ways of deploying Z2 with some pros and cons. After all, that’s good input to next version’s documentation anyway (note to myself: Add as Wiki content).

Let’s get started with THE standard out-of-the-box of operating Z2: All Z2 homes connect directly to an SCM (Subversion or Git) and nothing else is ever needed:

For development setups there is no better choice: Everything is intrinsically as up-to-date as possible and necessarily self-contained. Also for production setups (in that case the SCMs represent a production branch) this is great, if you do not worry about having source code being downloaded to production machines (a consequence of the local compilation step). It’s great for essentially the same reasons the development setup is cool: Minimal infrastructure required, no inconsistencies, simple scale-out, complete change history.

When deployments get bigger or you are worried about source code on production machines, the Hub Component Repository (HubCR) provides another level of scale and isolation without violating the pull-deployment approach. The HubCR looks like the regular component repository abstraction to reading nodes:

It prepares a pre-compiled view that does not contain source code anymore. Also it removes the need of compilation on production machines (which is actually not a big deal, as that would happen in parallel anyway). More significantly it takes away load from the SCMs. Should the HubCR nodes eventually become a bottle neck, these could be clustered as well. (Clustering of HubCR is not available today!).

The price to pay in comparison with the development setup is that a change will not be rolled out to the production environment before the HubCR node has been synchronized. And of course, you need to maintain a HubCR node in the first place.

A future application of the HubCR is to support automated updates for remotely hosted on-premise solutions. The hub side would provide the updates to remote systems and enforce licenses.

But now, let’s leave the pull-deployment approaches aside and see, if we can find some good reasons for push-deployments.

If your execution environment is far away and connectivity is poor or cannot be guaranteed at all and you still want to make sure there is no source code on production machines, you may create a binary export using the Distribution Exporter tool.

That tool creates a source code free repository structure that is semantically equivalent to the original source repositories (as seen from Z2) – much like the HubCR does it – but copies the results into a folder structure that can be used to create another repository.

In the case below, this is yet another SCM, that has production system configuration (which typically differs from development configuration):

Turning the arrow has non-trivial implications (i.e. you need to push and delta computation is not trivial). However as there is a version-controlled store that holds the production config, you still have good control over who changed what and when. In contrast to the HubCR, the repository can be hosted elsewhere – so that network access is not required or does not need to perform particularly well.

The last case to look at is – once more – very different from the others. Suppose you want to ship your solution in a simple bundle that can be installed into a remote file system.
In that case, you would use the Distribution Exporter tool and bundle the result as a file system component repository with a Z2 core that references it.

This is the closest to a traditional deployment and can easily be implemented via a build script.

It’s been a while already. Forgot to put it up in the blog: Version 2.1 of the z2-Environment is ready for use and learn. All the new stuff has been described in this previous post.

Among the features we plan for 2.2 the following stand out:

• Eclipsoid-style Z2 support for IntelliJ IDEA
• The Hub component repository

The goal of the first item is clear: Resolve a classpath for modules in the IDE corresponding to your runtime state (read on here for example) and offer the same developer convenience that can be achieved using the Eclipse plugin for users of the IntelliJ IDE.

The Hub component repository (or HubCR) requires some more background. By default, a Z2 installation (a z2 Home) accesses the version controlled storage that holds the system definition directly to download and prepare (and compile) modules as needed. This is the underpinnings of the system centric, pull-deployment based approach. In some situations however, it is not desirable to have source code on production servers at any time. This may be for compliance reasons or for fear of risking theft of intellectual property.

This is where the HubCR comes in. As a principle, all modules, all component resources, essentially anything the Z2 runtime knows about is served by component repository implementations.  There are implementations for Subversion, Git, file system folders, development workspaces and now, the latest addition, for another Z2 server that provides a consolidated, source-code free and pre-compiled view onto production resources (the HubCR provider).

So, instead of having production systems read and process source code directly, an intermediate node provides a semantically equivalent but pre-processed view onto the system definition:

The way the HubCR does that is by maintaining a pre-compiled and source-code stripped snapshot of the original production configuration. At the same time, the HubCR is just a regular z2 Home that runs the production config.

To the real production nodes (on the right in the diagram) however, the HubCR presents everything but the HubCR and other remote component repositories.

As a result, production systems can be completely separated from the source level details of the system definition. They don’t even see authentication details to the configuration store – only those necessary to access the HubCR service. At the same time, the pull semantics are preserved and updates can flow in and will be distributed consistently as for any other Z2-based system.

Lately I have been working on (re-) designing the operational mode of a Java-based solution that includes Hadoop and HBase and is to be scaled out in a “cloud-style” manner. In other words: changing the set of nodes that define the operational environment may change dramatically over the course of time in number and in other parameters (like the exact OS used) – and hence that change must be implementable as cheap as possible. And, if you do not want to get overwhelmed by scale complexity later on, this is something to definitely keep in mind right from the beginning.

Complexity again

Complexity is a killer for clustered solutions. For systems that may need to scale into the hundreds of nodes, any complexity avoided up-front is not just a risk-reduction. It’s a definite must-have.

In order to reduce complexity, you need to narrow dependencies between your stuff – that you can fully control – and their stuff – that you have reduced control of. In an ideal world, you would not need anything like a node local OS in the first place. Of course (and unfortunately) that is not realistic. The next best solution would obviously (!) be that all nodes get a copy (and updates) of everything they need by something as simple as running an rsync, an Subversion checkout, or a “Git pull”.

How to Install

The other important aspect, next to narrowing OS dependencies, is locality of what you install and users.

Linux package managers (rpm, dpkg) are a great way of configuring your OS as far as it concerns basic, stable components where incompatibilities are highly unlikely. If you can, getting along without them is better though. Anything you can configure to run from where you copied it to is naturally superior in simplicity, consistency, clean removal, clean update, ability to install parallel version – i. e. in all practical matters.

Also, and I personally think this is a quality that hardly gets the recognition it deserves, software that can be installed and run from a folder of your choice can be used by developers just the way it is used in production.

Package managers solve a problem in providing to a Linux distribution with dependencies. They do not necessarily solve your problems: On a later solution packages may have become incompatible with previous versions (as in the case of Ganglia for example). The JDK you used may not be available anymore, etc. You do not want the Linux distribution providers or other third parties to determine when you need to change your solution! You need to be on top of that!

Here’s my recipe:

• Everything possible is installed in and executed by one and the same user account
• All software as far as possible is installed in one simple folder layout that has one folder per software component
• Moving software into location is generally good enough. No post-copy transformation is acceptable. At most environment variables may be used to convey topological information.

Specifically, the JDK, hadoop, hbase are installed by folder checkouts.

Fortunately, Java provides a rather solid isolation against OS matters to the extent that you can install a Java Runtime Environment essentially by copying it and you generally do not need native libraries that would have to be installed “ouf of band” (Admittedly as “malloc uses excessive memory for multi-threaded applications” shows the JDK insulation is sometimes not strong enough).

Automation

Just like people may look alike but do different things, so do nodes in your cluster. Doing different tasks translates to running different processes with potentially different configurations and may also mean to install different software components.

Fortunately there are tools that automate preparation of a node configuration as well as making sure that exactly the right stuff runs.

I chose CFEngine. While its configuration language sucks, the underlying model (promise convergence) is cool and it has hardly any dependency on anything else (which is arguably the coolest feature of CFEngine). So in short the process of adding a node to the cluster means:

1. Add the node to the policy server stored configuration (i. e. designate a purpose – which can be based on IP ranges for example – and hence be “automatic”).
2. Throw CFEngine at the node (install one .deb or ,rpm).
3. Bootstrap the local CFEngine for the policy server.

Can be done even by an SSH-remoted script.

Once up, CFEngine will check every few minutes whether promises still hold. This includes configuration updates but also making sure your server is still running the right processes and services (assuming your promises are designed accordingly).

The other thing that is cool about CFEngine is that it can run in user-space. That is you could split your configuration into promises that require root-permissions (which better be few and robust) and leave the ugly stuff to user space configuration. If there is anything you want to hide from user space processes (e.g. private keys), you need to block root access as much as possible.

Finally some words on

Monitoring

Along with the ability to bring your solution to the nodes and to adapt it as needed and to make sure all necessary things are up, you can make sure monitoring agents are their and running. Agent based monitoring tools such as Ganglia and Zabbix provide a powerful information interface to the piece you are looking at. The “push from agent” model has other advantages:

• As agents call the monitoring hub, they may self-register
• If the agent doesn’t talk anymore, it is clear the node is in trouble
• Less contention: A node gone bad will not hamper information retrieval (i.e. no blocked and waiting connections).

Ps.: Z2

In the setup above, Z2 is used as execution environment for application code, either to run some actual service endpoint, to run background jobs, or to run MapReduce jobs as in How to Hadoop. Z2 is installed by a core check out (as usual) and pulls its updates on its own.

My first encounter with Vaadin was during the “Orignal1 project”. At the time we were looking for a user interface framework that was suitable for an externally available administration interface that was expected to

• support few but demanding users that should get a rich user experience;
• have a very streamlined, repetitive look and feel – no surprises when moving between views;
• not be layout driven (no pixel perfect visuals);
• become rather data centric and will require a lot of desktop-style interaction for data maintenance;
• become subject to a lot of extensions over time.

Nobody in the team knew about Vaadin at that time (as far as I can remember – although the larger part of the team was located in Helsinki!). There was a lot of experience with design-driven mass-user Web sites on the team, also some for snappy, highly interactive rich client sites. Plain HTML or Sencha-style rich clients were quickly dismissed though for the problem of balancing productivity, extensibility, and user experience in the niche we were moving into.

When Vaadin was brought up, it only took a few samples to be convinced – despite its name that doesn’t sound too impressive for anybody not from Finland.

The downside of Vaadin is that one frequently has the feeling that the APIs could be somehow more elegant. On the plus side, considering our constraints, you get:

• It’s all Java code. No other markup required (except for theming of course)
• As its all Java, you can build composite controls, your own utility controls, anything unless when you want different HTML (in which case you need to compile GWT widgets).
• As it is all Java, you can put your composite controls and utilities in re-use modules – including their resources (e.g. images).
• As it is all Java, you can refactor directly from your IDE
• Productivity is only limited by the quality of your application structure and re-use assets (and your dev environment of course)
• It comes with a rich control set and there is an impressive add-on community

So, currently at least, if somebody asks for a toolkit to build intranet and “complex+few users+very much under maintenance” user interface, Vaadin is a safe choice.

I promised a sample. It’s here:

Sample-vaadin-spring-hibernate

It is very much like the other samples in our Wiki. And as before we use Spring to inject dependencies into the UI layer whereever needed. Which is kind of the point here. Vaadin was used just as a drop-in replacement for other UI implementations and it simply fit in -  without any bending and plumbing.

This post is about isolating intrusive frameworks by classloaders so that they stop getting in the way of what you are actually supposed to do: Build a maintainable, consistent, well-defined solution to a business problem.

This post is a logical continuation of the previous post Protecting Java in a Modular World – Context Classloaders.

So you have been building your applications carefully watching out to choose a consistent set of tools and 3^rd party libs that allow impressive features with little implementation effort and now you are going to integrate Eclipse BIRT (the reporting tool) and boom!

• There is a seemingly inifinite number of dependencies brought in that may or may not match the versions you already have (and that may or may not match the versions the other framework you integrated requires) – so your code stops working when satisfying BIRTs demands.
• Even, if you would get that sorted out – you will need to sort it out once more after the next upgrade.
• All the problems in Protecting Java in a Modular World – Context Classloaders apply once more.

You are right in the middle of Java’s DLL Hell.

And Eclipse BIRT is just one example. Apache Hadoop is another and there is countless others out there.

Naively adding all the required libs into one class loading scope typically creates a huge mess that a mere human can hardly get under control at one point in time – much less so over time.

How to fix that?

Isolation!

Make sure that BIRT lives in its own, dead end class loading scope. In other words, make sure that BIRT operates as if it was the end of the food chain – or more specifically that its classloader is not delegated to by other modules of the solution – or only for very dedicated purposes.

In plain Java the way to achieve this is to create a child classloader of main application class loading scope. The isolated code can be accessed via reflection or by “accessor” style implementation classes that implement some parent loader provided interface and that are added to the BIRT scope.

Here is some almost not-so-pseudo code that shows the initialization:

public class BIRTService {
  private BIRTService impl;

  private void init(File birtJarsFolder) {
    try {
      // collect jars
      List<URL> urls = new LinkedList<URL>();
      for (File j : birtJarsFolder.listFiles(
        new FileFilter() {
          public boolean accept(File pathname) {
            return pathname.getName().endsWith("*.jar");}
          }
        )
      ) {
        urls.add(j.toURI().toURL());
      }

      // construct class loader
      URLClassLoader cl = new URLClassLoader(
        urls.toArray(new URL[urls.size()]),
        this.getClass().getClassLoader()
      );

      // load impl
      this.impl = (BIRTService) Class.forName(
        "sample.BIRTServiceImpl",
        false,
        cl)
      .newInstance();

    } catch (Exception e) {
      throw new IllegalStateException("BIRT Service init failed",e);
    }
  }

  /*
   * delegate
   */
  public void methodA() {
    this.impl.methodA();
  }
}

Of course, in addition make sure that context classloaders  are set (as described in in the other blog) to the BIRT scope – on every invocation in this case – in the implementation of that service. Otherwise BIRT will still fish in the applications other scopes – which is asking for trouble.

So, in terms of class loaders we would get the following scheme:

Now apart from the complexity of packaging stuff the right way, what if you use stuff that BIRT cannot work with – or may not work with in the future? Following the (otherwise robust) class loading delegation pattern, types that you provide with your application will be provided to BIRT with preference over its own types. In other words, if you use a commonly used library in a version you prefer but BIRT cannot live with, you will still need to follow BIRT’s regime to make things work.

In other words: To do it right, you need more, better, stronger modularization!

Specifically, you need more class loading separating modularization

Think about it: If the modules of your application would hide and manage their visibilities as we just did to isolate BIRT, you could actually keep clean scopes.

In Z2 you would set up modules to achieve the following class loading scheme (slightly abstracted):

Instead of loading an implementation class from a class loader you instantiated, you would look up a component and leave the module mechanics to Z2.

(In OSGi you could use the BIRT bundles – and spread out the mess – or hide BIRT into one bundle achieve a similar scheme as above).

Now, when done, what do we get:

• BIRT is confined to a leaf on the class loading hierarchy – minimizing the impact BIRT has on other modules.
• BIRT only sees what is needed – minimizing the impact other modules have on BIRT.

Other frameworks pose similar or harder challenges and what we did not even get into is how to provide extensions to something like BIRT but that’s another topic.

This post is about seemingly tricky class loading stuff in Java. It is actually not really tricky, there is just a lot of missing-the-point advice out there and it is one of those problems that you typically start worrying about when you already had the problem for a while and have built up a lot of wrong ideas about it.

In particular this post we look at the widely used but hardly noticed Thread Context Classloader mechanism in Java.

Java Classloading recap (skip if you think you’re a pro)

Java class loading is separating name spaces in the type system. That is actually not as complicated as it sounds. It is just saying that there are different groups of types at runtime and types in different groups (class loading scopes) may not be able to access each other or only one may access the other but not vice versa. What goes into a class loader scope, i.e. what is provided by one class loader, is defined by the environment – typically all types of a module or an application (in Java EE).

(Disclaimer: OSGi makes it all a little more complicated in its own way – but the effects described below still show.)

This has various important implications: On the one hand it allows you to isolate stuff from each other. E.g. you can run several web apps that have the same JAX-RS servlet (as an example) by name but in different implementation versions. This type of isolation is the underpinning of all hot deployment within one JVM process.

In fact, types in the JVM are considered identical if and only if they match in name and defining loader (i.e. the class loader that actually provided the class).

The standard class loading setup is that a class loader has one parent and when asked to load a class it will ask its parent before trying to check its resources (typically .jar, .class files)

That is very good. It assures consistency, which filtering or child-first policies (that many app servers offer as problem solvers ) do not. And losing consistency in class loading may give you some short term gain for a usually super awful midterm pain.

Modular systems like OSGi or Z2 for that matter (still need to check on Jigsaw) modify this pattern slightly by having a view from higher or by delegating to more than one loader. But the basic premise holds true. Simplified you end up in tree-style visibility relationships where B and C share the types of A, but A has no access to the types provided by B nor C.

Here, A may be the System class loader, the one that provides java.lang.String and B and C are some module loaders in some modular runtime. Now, the fact that B and C share the types of A is the reason they can instantiate, pass and use implementation of types of A to each other without trickery . On the other hand, B can be removed without requiring to unload C (if done badly however, C may still hold Objects from B without knowing even knowing of their type!).

OK. Let’s stop the general stuff here and let’s talk about context class loaders.

A Thread’s context classloader. What is that good for?

Context classloaders actually solve an important problem. Imagine you have a functionality F in A that is supposed to be used in B. As an example for F think about a Web framework like Stripes, Vaadin, or essentially any such toolkit. In non-trivial cases, you may want to extend F to use extension types and or configuration files that are supplied in B. Wouldn’t it be good if A could actually look into B for that? That is exactly what context class loaders are for. They transport a caller scope into the callee. This mechanism is used all over the place. It is used in JNDI to find initial context factories (and more), in Jetty (or any other Web container), BIRT, Hadoop, javax.mail… just to name a few.

You may never have noticed, because application containers set them for you whenever the control flow passes their access point. For example the Web container sets the context class loader to the one of the Web app for every request.

The general coding scheme used is

ClassLoader thatLoader = Thread.currentThread.getContextClassLoader();
Thread.currentThread.setContextClassLoader(thisLoader);
try {
  // do something
} finally {
  Thread.currentThread.setContextClassLoader(thatLoader);
}

And that is the main advantage of the context class loader approach: It’s really simple.

Unfortunately, that is also its biggest problem: Most coders are hardly aware of the mechanism and accidentially either do not set it correctly when it is advised to do so, or have a class loader passed on when it would be better not to do so.

In modular environments (e.g. OSGi as well as Z2) when the control flow passes through different class loading scopes, there may be no one automatically switching contexts for you as that is prohibitive due to performance and complexity considerations.

Furthermore, many frameworks like to hold on to context class loader they pick from some initial call and use these later on. This holds for example for Eclipse BIRT and for Apache Hadoop and essentially any other framework that manages its own threads.

In this case, when eventually passing on control, the context class loader may be set to some or some other Web app, for example, and the framework at hand may start looking into the Web app’s class loading scope which can not only lead to using implementations you didn’t expect, it may also mean that re-deployment leaves old implementations in use (because the old loader is still held on to).

That leads us to

Rule #1: Make sure you set the context class loader to a suitable scope when using some non-trivial framework in a modular environment.

Note that any new thread (or java.util.Timer) will take over the context class loader from the instantiating thread. I.e. there is not much required to make rule #1 a necessity.

A more constructive situation comes up when you intentionally integrate a framework that manages threads to call your code back at some later point in time. For example the Quartz Scheduler does so (see also Getting Work Done Asynchronously). When used in a Web app, Quartz’s threads will pick up the class loader from the web app and as long as the job’s implementation is in the Web app, things will work as “naively” expected.

In modular environments however, the job’s code may reside in some other scope than Quartz itself. Quartz is effectively used as an application container.

In that case you need some generic job wrapper that not only locates the actually job implementation and its module, but also sets the context class loader to the one of the module. Of course, Quartz falls into the category for which rule #1 applies. I.e. when starting the scheduler, make sure the context class loader has Quartz and your job wrapper. Not more.

So we have:

Rule #2: When integrating asynchonous callbacks in a modular environment, abstract callback retrieval and set the context class loader appropriately.

This concludes the context class loader piece. In a next installment (now available: Protecting Java in a Modular World – Isolating the Bad Guy), we will look at how to tame beasts like Eclipse BIRT by confining it into an isolated class loading scope.

In the meantime try searching for class loading problems around BIRT and enjoy (or feel sorry).