Refactoring-safe referencing of bean properties

Currently starting to overhaul an old idea for a rather handy (sort of DSL’ish) query API in Java that exposes some properties I very much miss in APIs like Query DSL.

A typical problem when designing data access APIs or any other API that binds some data structure to Java Beans is that you cannot directly refer to bean properties in a refactoring-safe way when constructing expressions. To do so you make use of string constants, thereby denoting property names redundantly. The advantage of bean properties over string constants however is that refactoring tools recognize usage throughout a complete codebase, so that changing internal data naming is a straightforward and low-risk task.

The approach taken by tools such as the (dreadful) JPA criteria API or Query DSL is to offer generation of Companion Types for bean types. The companion types expose access to property names and more. As code generation – in particular code generation  involving the IDE – that generates code that is referenced by name from hand-typed code, extends the compiler food chain to an even more intrusive beast – even introducing IDE dependencies – this approach is not only ugly, it asks to trouble mid-way and cannot be a sane choice long term.

Here is another approach:

Based on the Java Bean Specification we have a one-to-one relationship between bean properties and its read methods (the getters). In Java, method meta-data is not as directly accessible as class meta-data via the reflection API. That is, unlike

MyBean.class.getName()

to access a class name in code, there is nothing like

MyBean.getOrder.method.getName()

for methods. In order to retrieve the property association via a getter method, we can however make some careful use of byte-code trickery. Using the Javassist library, we can generate a support extension – a meta bean – of the original bean type, that, when invoking its getters, provides us with the associated property name.

In essence this works as follows (see the code below): After retrieving a meta bean (that may be held on to), invoking a getter leaves the corresponding property name in a thread local. A helper method reads the thread local and resets it. So, continuing the example below,

MyBean mb = MetaBeans.make(MyBean.class);
System.err.println(MetaBeans.p(mb.getOrder()));

would output

order

As a neat extension, the artificially created getters return (whenever sensibly possible) meta beans itself, and when finding a non-empty property name held by the thread local storage, instead of setting it to the property name, the property name will be appended, so that

//...
import static MetaBeans.*;
//...

MyBean mb = make(MyBean.class);
System.err.println(p(mb.getOrder().getId));

would output

order.id

Using this approach requires no further tooling whatsoever, can easily be extended to other use-cases, is completely refactor-safe, and comes at diminishing costs.

Note that the implementation below is not made to run with module-system-type class loader setups, is somewhat crude, and is really just meant to illustrate the idea. Consult the Java Assist API for more information on managing class pools.

Here is the MetaBeans class:

 public class MetaBeans {
    private static ThreadLocal<String> properties = new ThreadLocal<String>();

    /**
     * Create a meta bean instance. If not eligible, this method throws an IllegalArgumentException.
     * 
     * @param beanClass the bean class to create a meta bean instance for
     * @return instance of meta bean
     */
    public static <T> T make(Class<T> beanClass) {
        return make(beanClass,true);
    }

    /**
     * Create a meta bean instance. If not eligible, return null.
     * 
     * @param beanClass the bean class to create a meta bean instance for or null, if the class is found to be not eligible.
     * @return instance of meta bean
     */
    public static <T> T makeOrNull(Class<T> beanClass) {
        return make(beanClass,false);
    }

    /**
     * Track meta bean invocations and return property path.
     */
    public static String p(Object any) {
        try {
            return properties.get();
        } finally {
            properties.set(null);
        }
    }

    /**
     * Internal.
     */
    public static void note(String name) {
        String n = properties.get();
        if (n==null) {
            n = name;
        } else {
            n += "."+name;
        }
        properties.set(n);
    }

    //
    // private 
    //

    // actually provide an instance
    private static <T> T make(Class<T> beanClass, boolean nullIfNotEligible) {
        try {
            Class<?> c = provideMetaBeanClass(beanClass, nullIfNotEligible);
            if (c==null) {
                return null;
            }
            return beanClass.cast(c.newInstance());
        } catch (Exception e) {
            throw new RuntimeException("Failed create meta bean for type "+beanClass,e);
        }
    }

    // try to provide a meta bean class or return null if note eligible
    private static Class<?> provideMetaBeanClass(Class<?> beanClass, boolean nullIfNotEligible) throws Exception {
        // check eligibility
        StringBuilder b = checkEligible(beanClass);
        if (b.length()>0) {
            if (nullIfNotEligible) {
                throw new IllegalArgumentException("Cannot construct meta bean for "+beanClass+" because: \n"+b.toString());
            }
            return null;
        }
        String newName = metaBeanName(beanClass);
        // check if the class can be found normally or has been defined previously
        ClassPool pool = ClassPool.getDefault();
        CtClass cc = pool.getOrNull(newName);
        if (cc==null) {
            // ok, need to construct it.
            // start constructing
            cc = pool.makeClass(newName);
            // as derivation of the bean class
            CtClass sc = pool.get(beanClass.getName());
            cc.setSuperclass(sc);

            // override getters
            for (PropertyDescriptor pd : Introspector.getBeanInfo(beanClass).getPropertyDescriptors()) {
                String pn = pd.getName();
                Method g = pd.getReadMethod();
                if ( (g.getModifiers() & (Modifier.FINAL | Modifier.NATIVE | Modifier.PRIVATE)) ==0) {

                    // fetch return type (pool will retrieve or throw exception, if it cannot be found)
                    CtClass rc = pool.get(g.getReturnType().getName());
                    // create the new getter
                    String body = "{"+
                        MetaBeans.class.getName()+".note(\""+pn+"\");"+
                        // add a cast as Java Assist is not great with generics it seems
                        "return ("+g.getReturnType().getCanonicalName()+") "+MetaBeans.class.getName()+".makeOrNull("+g.getReturnType().getCanonicalName()+".class);"+
                    "}";

                    CtMethod m = CtNewMethod.make(
                        rc,
                        g.getName(),
                        new CtClass[0],
                        new CtClass[0],
                        body,
                        cc
                    );
                    cc.addMethod(m);
                }
            }
            return cc.toClass();
        } else {
            return Class.forName(newName);
        }
    }

    private static String metaBeanName(Class<?> beanClass) {
        String newName = beanClass.getCanonicalName()+"__meta";
        return newName;
    }

    private static StringBuilder checkEligible(Class<?> beanClass) {
        StringBuilder b = new StringBuilder();
        if (beanClass.getPackage().getName().startsWith("java.lang")) {
            b.append("No meta beans for standard types\n");
        }
        try{
            beanClass.getConstructor();
        } catch (NoSuchMethodException nsme) {
            b.append(beanClass.toString()).append(" has no default constructor\n");
        }
        return b;
    }
}

 

 

On integratedness or the math of updates

Last year, in a talk at Codemotion Berlin (see here) I described as one of the hurdles in keeping development productivity up when systems grow the poor model match between runtime and design time. Turns out that was an awfully abstract way of saying “well something like that”.

At last I am a little smarter know and I’d rather say it’s about the integratedness.

This post is about:

What makes us slow down when systems grow large, and what to do about it?

 productivity_goes_down

 

A lot of things happen when systems grow. And there is more work on this topic around than I could possibly know about. In fact, what I will concentrate on is some accidental complexity that is bought into at some early stage, then neglected, and typically accepted as a fact of life that would be to expensive to fix eventually: The complexity of updates as part of (generic) development turnarounds.

While all projects start small and so any potential handling problem is small as well, all but the most ignorable projects eventually grow into large systems, if they survive long enough.

In most cases, for a variety of reasons, this means that systems grow into many modules, often a distributed setup, and most-definitely into a multi team setups with split responsibilities and not so rarely with different approaches for deployment, operations, testing, etc.

That means: To make sure a change is successfully implemented across systems and organizational boundaries a lot of obstacles – requiring a diverse set of skills – have to be overcome:

Locally, it has to be made sure that all deployables that might have been affected are updated and installed. If there is a change in environment configuration this has to documented so it can be communicated. Does the change imply a change in operational procedures? Are testing procedures affected? Was there a change in the build configuration? And so on.

Now suppose for an arbitrary change (assuming complete open-mindedness and only the desire for better structure) there is n such steps that may potentially require human intervention or else an update will fail. Further more assume that we have some minimal probability p that we run into failure than the probability that an update succeeds immediately is at most:

(1-p)^n

What we get here is a geometric distribution on the number of attempts required for a successful update. That means, the expected number of attempts for any such open-minded update is:

1/(1-p)^n

which says nothing else but that

Update efforts grow exponentially with the number of obstacles.

While the model may be over-simplified, it illustrates an important point: Adding complexity to the process will kill you. In order to beat an increasing n, you would have to exponentially improve in (1-p) which is … well … unlikely.

There is however another factor that adds on top of this:

In reality it all works out differently and typically into a sort of death spiral: When stuff gets harder because procedures get more cumbersome (i.e. n grows), rather than trying to fix the procedure (which may not even be within your reach) the natural tendency is be less open-minded about changes and rather avoid the risk of failing update steps altogether by constricting one’s work to some well-understood area that has little interference with others. First symptoms are:

  • Re-implementation (copy & paste) across modules to avoid interference
  • De-coupled development installations that stop getting updates for fear of interruption

Both of these happen inevitably sooner or later. The trick is to go for later and to make sure boundaries can be removed again later (which is why in particular de-coupling of development systems can make sense, if cheap). Advanced symptoms are

  • Remote-isolation of subsystems for fear of interference

That is hard to revert, increases n, and while it may yield some short term relieve, it almost certainly establishes an architecture that is hard to maintain, makes cross-cutting concerns harder to monitor.

With integratedness of the system development environment, I am referring to small n‘s and small p‘s. I don’t have a better definition yet, but its role can be nicely illustrated in relation to to other forces that come into play with system growth: The systems complexity and its modularity. While the system grows so does (normally) its overall complexity grow. To keep the complexity at hand under control we apply modularization. To keep the cost of handling under control, we need integratedness:

forces

One classic example of an integrated (in the sense above) development and execution environment is SAP’s ABAP for its traditional ERP use-case. While ABAP systems are huge to start with (check out the “Beispiele” section in here), customers are able to add impressively large extensions (see here).

The key here for ABAP is: Stuff you don’t touch doesn’t hurt you. Implementing a change makes it available right away (n=1 for dev).

References

  1. Lines_of_Code (Beispiele), German Wikipedia
  2. how many lines of custom ABAP code are inside your system?, SAP SCN
  3. System-Centric Development

 

The Linda Problem of Distributed Computing

Suppose an important function of your solution is pricing calculation for a trading good.

What is the more appropriate solution approach:

  1. You develop a software module that implements pricing computation
  2. You develop a REST server that returns pricing computation results

I am convinced that more than a few developers would intuitively chose b).

Taking a step back and thinking about it some more (waking your lazy “System 2”) it should become clear that choice a) is much stronger. If you need to integrate pricing computation in a user interface, need a single process deployment solution, AND a REST interface – it’s all simple adaptations of a). While having b) gives little hope for a). So why chose b)?

This, I believe to be an instance of a “conjunction fallacy”. The fact that b) is more specific, more tangible, more representative as a complete solution to the problem makes it more probable to your intuition.

Back to the observation at hand: Similar to the teaser example above, I have seen more than one case where business functions got added to an integration tier (e.g. an ESB) without any technological need (like truly unmodifiable legacy systems and the like). An extremely poor choice considering that remote coupling is harder to maintain, has tremendously more complex security and consistency requirements. Still it happens and it looks good and substantial on diagrams and fools observers into seeing more meaning than justified.

Truth is:

-

Distribution is a function of load characteristics not of functional separation

-

(or more generally speaking: Non-functional requirements govern distribution).

The prototypical reason to designate boxes for different purposes is that load characteristics differ significantly and some SLA has to be met (formally or informally). For many applications this does not apply at all. For most of the rest a difference between “processing a user interaction synchronously” and “performing expensive, long-running background work asynchronously” is all that matters. All the rest is load-balancing.

Before concluding this slightly unstructured post, here’s a similar case:

People love deployment pipelines and configuration management tools that push configuration to servers or run scripts. It definitely gives rise to impressive power-plant-mission-control-style charts. In reality however: Any logical hop (human or machine) between the actual system description (code and config) and the execution environment adds to the problem and should be avoided (as probability of success decreases exponentially with the number of hops).

In fact:

-

The cost of system update is a function of the number of configurable intermediate operations from source to execution

-

and as an important corallary:

-

The cost of debugging an execution environment is a function of the number of configurable intermediate operation from source to execution

 -

More on that another time though.

This post was inspired by “Thinking, Fast and Slow” by Daniel Kahneman that has a lot of eye-opening insights on how our intuitive vs. non-intuitive cognitive processes work. As the back cover says: “Buy it fast. Read it slowly”

 

Dependency Management for Modular Applications

While working on a rather cool support for Maven Repositories in Z2 (all based on Aether – the original Maven dependency and repository implementation), I converted several scenarios to be using artifacts stored in Maven central. This little piece is on what I ran into.

It all starts with the most obvious of all questions:

How to best map Java components (Z2 term) from Maven artifacts?

As Java components (see here for more details) are the obvious way of breaking up stuff and sharing it at runtime, it is natural to turn every jar artifact into a Java component – exposing the actual library as its API. But then, given that we have so much dependency information, can we map that as well?

Can we derive class loader references from Maven dependencies?

Maven has different dependency scopes (see here). You have to make some choices when mapping to class loader references. As compile dependencies are just as directed as class loader references, mapping non-optional compile dependencies to class loader references between Java components seems to be a rather conservative and still useful approach.

At first glance this looks indeed useful. I was tempted to believe that a lot of real-world cases would fit. Of course there are those adapter libraries (like spring-context-support), that serve to integrate with a large number of other frameworks and therefore have tons of optional compile dependencies. Using them only with non-optional compile dependencies on the class path completely defeats their purpose. (For example spring-context-support is made to hook up your Spring application context with any subset of their target technologies – and only that needs to be on the effective class path.)

But there are other “bad” cases. Here are two important examples:

  • Consistency: The Hibernate O/R Mapper library integrates with the Java Transaction API (JTA) – and it definitely better should. So it does have a corresponding non-optional compile dependency on JTA, but not on javax.transaction:jta but rather JBOSS’s packaging of it. Spring-tx (the transaction utilities of the Spring framework) however has a compile dependency on javax.transaction:javax.transaction-api. Using both with the trivial mapping would hence lead to some rather unresolvable “type misunderstanding” a.k.a. class cast exception. So there is consistency problems around artifacts that have no clear ownership.
  • Runtime constraints: The spring-aspect library, that enables transparent, automatic Spring configuration for objects that are not instantiated by Spring, associates a Spring application context with a class loading namespace by holding on to an application context by a static class member. That is: Every “instance” of the spring-aspect library can be used with at most one application context. Hence, it makes no sense what so ever to share it.

So there are problems. Then again: Why split and share at runtime with class loader isolation anyway? Check the other article for some reasoning. Long story short: There is good cases for it (but it’s no free lunch).

Assuming we want to and considering the complexities above, we can conclude that we will see some non-trivial assembly.

Here are some cases I came up with:

mod-dep-cases

  1. Direct mapping: Some cases allow to retain the compile dependency graph one to one as class loading references. This does however require to not rely on anything that is expected to be present on the platform. E. g. “provided” dependencies will not be fulfilled and most likely we end up in case 3.
  2. Isolated Inclusion: For the case of of spring-aspects, it assumes to be a class loading singleton. It is one for each application context. So, if you want to have multiple application contexts, you will need multiple copies of spring-aspects. This is true for all cases where some scope that can be instantiated multiple times is bound to some static class data.
  3. Aggregation: the not so rare case that a dependency graph has a single significant root and possibly some dependencies that should be excluded as they will be satisfied by the environment, it is most practical to form a new module by flattening the compile dependency graph (omitting all exclusions) and satisfy other runtime requirements via platform class loading means. The role model for this is Hibernate.
  4. Adapter Inclusion: Finally, we consider the case of a library that is made to be used in a variety of possible scopes. A great example is spring-context-support that provides adapters for using Spring with more other tools than you probably care in any single setup. If spring-context-support would be used via a classloading reference, it could not possibly be useful as it would not “see” the referencing module’s choice of other tools. By including (copying) it into the module, it can.

Conclusion

If you follow the “normal” procedure and simply assemble everything into one single web application class loader scope, you can spare some thinking effort and get around most of the problems outlined above. If you however have the need to modularize with isolated class loading hierarchies, you need to spend some time understanding toolsets and choosing an assembly approach. This does pay off however, as something that happened accidentally has now become a matter of reasoning.

References

  1. Z2 Maven Repo Support
  2. Z2 Java Components
  3. Maven Dependency Scopes

 

User Friendly Production Updates

February is almost over – so little time to spend on this blog. Here is a short post on something cool we did a while back in Z2.

The original request was along the lines of

 

Can we do an software upgrade without interrupting a user’s work?

 

This is in the context of the not so insignificant class of applications that require users to operate on some non-trivial but temporary and yet rich state before a persistent state change of the system can be performed. That is, applications that keep non-trivial session state where kicking out users means a real loss of time and nerves and is more than unfriendly.

Still – what if there is an important update to an intranet application that should be applied and, say, it should be tried by some group of users after lunch?

Applying a software upgrade to a running application without interfering with the work progress of currently logged-in users has some natural limitations. For example, smart data migrations will be extremely hard to get right (in a stateful scenario). But anything above the domain level might actually work.

Theoretically the most natural approach would be to temporarily store user session data somewhere, “replace” the application and load the user session data into memory again. Practically speaking however, it will be hard to find complex Java applications that use session serialization and would be assumed to reliably save and restore a user session. Furthermore, during the time of the application restart there would still be some downtime that may be taken as a system failure by users.

So, instead of doing something smart why not do something really obvious:

 

Leave the application running until the last user has logged off (or was logged off due to session expiration). Present the new application version to all user that log in after the update.

 

In fact, the approach we took leaves the whole (frontend) application stack running until the last session “running” has become invalid. It is implemented by the Gateway module and described in detail in the wiki. Here is a short summary:

Normally, a Z2 in “server mode” has at least two processes running: A home process that serves as a watch dog and synchronization service plus a webWorker node that runs the (Jetty) Web server and whatever Web applications are supposed to be up.

With the Gateway module this setup is altered (by configuration) in that the actual Web server entry point is now running in the home process and forwarding requests on a by-session scheme to the actual webWorker process. Worker processes, such as the webWorker, can now be detached from the synchronisation procedure. That is, instead of being stopped because of changes, detached worker processes are simply left unaffected from any updates until nobody needs them anymore:

steady_workers

Using that mechanism, users can decide to complete their current work and decide to upgrade at their convenience by logging off and on again.

Finally, this is another cool show case of how beneficial worker process control within the execution environment is.

References

Secondary Indexes in HBase

Fundamental problems in solution development can be deceiving – in particular when considered from a conceptual distance (i.e. being somewhat clueless). At one time they look too hard to address as one more of the many side topics one has to keep under control, another time they turn into something completely manageable – only just to hit you hard once you turn around again.

One such topic that kept me busy lately is secondary indexing for HBase stored data.

As such, HBase does not have any built-in support for secondary indexes. All data is ordered by row keys and that’s it (in fact, HBase does not store rows but rather versioned columns).

Why is it that HBase does not have secondary indexes while products in the same space, such as Cassandra and Hypertable do? I would put it this way:

Due to the distributed nature of Bigtable style database systems, there is no good single solution to secondary indexing – but one rather has to chose between trade-offs to find a suitable approach for a given use-case

(Start you exploration here).

That would not necessarily need to mean that there could not be a variety of implementations available out of the box. Unless there is some stronger type system support and more agreement and experience with the various approaches, I guess, this will however still take some time.

This post is on three different approaches I learned about lately and what I think are their main qualities (with some interesting links).

Global Index Table

This is the most obvious (and still useful) approach. In its simplest version: Given some row r with row key r.key to index by some field value r.val, store a mapping

${r.val} / ${r.key}

as row keys in another table. Finding all rows with r.val==x then simply means to scan for the row keys following “x/” . As invalid index entries can be readily identified by checking the row for the expected field value keeping a consistent index is really easy by making sure that index entries are written before row updates are written and by cleaning out stale entries once in a while. So, the implementation is really not that hard and and it is effective (see also Global Index Table references below). Due to its simplicity and robustness, it would be highly desirable to have some utility implementation generally available.

Furthermore it is space-efficient, indexes can easily be regenerated if things go bad, you need no further software but HBase, and you can be really smart about the value to index as its all application based.

There are significant downsides as well: Getting a single row via the index is two roundtrips and, worse, scanning rows via the index implies to – in the worst case – to have randomly distributed “point gets”.

scnd_index_gt

 

The latter is obviously the most significant downside and if you need to efficiently scan rows via an index this approach is not for you.

Region-level Index

An approach that is more focussed on adding efficiently to write secondary indexes to the database implementation is to add region server or region level indexes. That is, integrated with HBase’s Write-Ahead-Log (WAL), indexes are written directly and atomically for data within the region server local regions to the region server. So, every region server holds indexes for its data. This approach is implemented by HIndex and IHBase.

This is good for the write performance. Reading data via the index does however require to contact each region in question, as there is no way of knowing where not to look for index content (e.g. unlike the global index table that uses regular HBase row key ordering).

scnd_index_rli

Effectively that means that read-throughput does not scale up with the number of region servers, but due to parallelization it does not degrade with data size either. So, in a way, like a classic RDBMS it does not scale vertically in throughput but unlike a classic RDBMS it does scale vertically in space.

Covering Index

Covering indexes are essentially materialized views without a name. That is, instead of fishing for the data via some key indirection, the actual data is copied into the index table which employs row keys corresponding to the indexed data rather than the original row key.

So, index-based queries don’t need to go anywhere but just into the right table where all data is already present. Consequently read performance is great.

scnd_index_cover

Updating the index however is much more complicated now. All copies need to be updated when changing data – unlike the global table approach there is no simple verification possible while still preserving the performance attributes.

While this approach does perfectly preserve the scalability attributes of HBase, indexes may still come at a steep price: Every new index potentially duplicates a significant portion of the original data.

Secondary indexing in Phoenix, a SQL-implementation for HBase, is based on this approach. And indeed query-performance is one of the strong selling points while there is still great effort put in making sure index updates are essentially atomic (and WAL-integrated) and performing well.

Summary

Not being really deep into the implementation details of the more complicated approaches, I must admit that I find Region-Level-Indexing the least attractive option. It looks like something that is asking for trouble in large setups – which is after all exactly what you expect to happen, once you come looking for HBase.

Covering indexes are really interesting – for analytical applications: If you need fast query performance over a large data this looks good. In analytical applications you typically know about the dimensions that need to be present in result sets.

For OLTP-style applications scanning via indexes is often a less prominent access method compared to single but complete row retrieval for display and manipulation by a variety of conditions. That is, global index tables may still be the more space-efficient and flexible means to that end, while still being sufficiently read efficient.

Ideally it should be (and quite possibly is already) possible to combine both approaches.

————-

References

Indexing on Hbase

  1. Secondary Indexes and Alternate Query Paths, HBase documentation,
  2. Musings on Secondary Indexing in HBase, Jesse Yates
  3. HBASE-2038, HBase issue tracker discussion

Global index tables

  1. Consistent Enough Secondary Indexes, Jesse Yates
  2. Musings on Secondary Indexes, Lars Hofhansl

Region-Level indexing

  1. HIndex at Github
  2. HIndex at Intel
  3. IHBase

Covering Indexes

  1. Phoenix
  2. Phoenix Secondary Indexing

Other indexing tools for HBase

  1. Culvert
  2. Lily

Other Big Table style database products with secondary indexing

  1. Hypertable
  2. Cassandra (and here)

Extend me maybe…

Initially I wanted to write a clever piece on extension patterns for modular software systems, but then it got all too long (well… it is tto long), too boring (of course), and too cumbersome (certainly).

But at least I have some reasonably good looking diagrams that I can share with you. So here it goes.

What I wanted to get at with the original concept is that there is a common pattern of software extensibility – the addition of new implementation into a software system at some point in time different than its original definition – that starts in a very simple way (very early on in most systems) but then starts attracting more complex requirements that introduce non-trivial transition costs (and experienced decision making). Just like the modularization topic, this is one that is easily ignored in the beginning, can be worked around for a while but will eventually bite you (at which point the typical decision is to ignore harder as everyone knows it’s going to become expensive).

At least in the early 2000s, before the IoC hype, the prototypical way to implement an extension point, the conceptual or programmatic entity that extensions are associated with in the system, was some class to call to register an instance of an extension (that would typically be required to implement some particular interface):

emm_register

There is only one advantage to this approach: All you need to know is the place to call. The downsides are significant and all due to its weak definition: All extensions somehow need to get run time, regardless of whether they are actually ever going to be used or not, and there is no way to find extensions much less to control their life cycle in a defined way.

The natural inversion and improvement of this pattern is to let the extension point actively find and load extensions.

emm_find

 

This does not only allow to defer loading to the point of time of use, it also allows to be clever about what to load and how to find it. In the wild, a simple, convenient and yet powerful implementation is established by using an IoC container like the Spring framework with lazy-loading and interface-based bean binding. Some more intelligence can be added by requiring custom annotations and configurations.

The limiting factor of this approach is its lack of modularization. All problems discussed in  Modularization is more than cutting it into pieces apply. Beyond that, the fact that extension enter the system at some unknown time (from a platform’s perspective) and the expectation of future compatibility make it ever more desirable to avoid conflicts due to sharing one large class loading name space.

So, given a module system (you might guess which one I have in mind here – but I am willing to grant you to think of other ones as well – and after all, the subject as discussed here is sufficiently generic), you need to be able to find and load some implementation from another module. In addition the implementation loaded should be expected to adhere to some API as otherwise the consuming extension point will hardly be able to make sense of it.

emm_module_system

 

From this high, no further qualification of the specifics is possible. However, the extent to which it is possible to intelligently query for extension point implementations and the associated costs are an important consideration. For example, in OSGi, an extender (think the Web application feature) can look into any bundle installed. In turn however, all bundles need to be present and introspected with all their resources on all nodes. In Z2, querying by type and configuration properties can be done without downloading any resources locally. But to benefit from this, all relevant data must be defined statically (which is typically the case). And of course there is no distribution effort, as information is centrally provided by the repository.

Now that we can separate type name spaces into modules and that we can load late, there is two more missing pieces to complete the picture. For once, it should be possible to replace or remove a module and the extension point should respond to that in a meaningful way. This is particularly true at development time. Secondly, whatever is expected to be implemented by the extension may not necessarily be what the consumer can make sense of.

An excellent example for the latter is Web applications. Web applications are a natural extension point. That is, the actual extension is a web application, the whole Servlet API  (in the case of Java) the corresponding API, and the consumer is whatever subsystem requires the Web application to be running.

In order to handle that, we should be able to isolate the actual extension point type and its implementation from its consumers and its suppliers – the ones that look for implementations and those that provide them.

emm_module_factory

 

OSGi’s Web Application extender is one such case. Z2′s Web App component type works rather similarly. The mechanism of component types is very light-weight and addresses both aspects: Management of runtime dependencies and transformation between component resources and consumer interfaces.

One final practical example can be found here with Z2′s Vaadin addon: Extensions implement Vaadin UI components, e.g. a tab sheet in a multi-tab Web application. The provided components can be queried and consumed by the main Web application. The Vaadin component component type implementation makes sure that an invalidation of an extension, unloads the consuming session so the system stays in a consistent runtime state.

References

In-system testing re-invented

This post is about z2Unit an in-system, unit/integration testing enhancement to the z2-Environment.

What is it all about?

In software testing, as far tools are considered it all starts with Unit Testing. Unit testing translate into writing and running code that checks the behaviour and results of other code. Typically tests are executed out of context. That is, during testing, code that normally runs in the context of, say, a Web application is now executed on its own. Unit testing is extremely useful for code that is very algorithmic, needs to process possibly complex input in possibly complex ways.

The process of providing a functional environment to the tested code is called mocking. Unit testing stops being useful, when the effort to mock a required environment starts to outweigh the actual testing.

On the other end of the scale we find automated black-box integration testing where application behaviour is verified completely from the outside. In case of Web applications, tools like Selenium (just to name one of many) can be used. Black-box testing is extremely useful to make sure typical user flows are indeed still working. A fast-paced roll-out pipeline is practically unthinkable without this.

Now, there is a place in between those two that is highly interesting: In-system integration tests. These shine where unit tests stop being useful and where black box tests cannot reach into: Testing code in its real target environment without any limitation on verifying data changes or use of data types. In cases where code is highly integrated with the environment, this kind of test is less effort to write (and to follow up on refactorings) due to less mocking, and, as for black-box tests, it tells you more about the actual system behaviour in the real.

This is unlike so-called in-container testing, that essentially aims at simplifying the mocking effort by providing a complete (and yet specialized) application container as part of the mock.

Now what about z2Unit?

The idea behind z2Unit is to provide in-system tests based on the JUnit framework as conveniently as possible. In short: Just have a unit test class with all the usual features and all the expected life cycle instantiated within its Java component context. Fortunately this turns out to be relatively straightforward.

In principle JUnit works like this:

junit_plain

A client, say your IDE or some Build Tool like ANT or Maven, runs tests by invoking the JUnit Core. The JUnit Core in turn identifies which Runner to use. The Runner provides test meta data like what tests to run and in what order or filtering and eventually orchestrates the test execution during which test events are reported back to the core.

So there is actually a protocol between JUnit Core and the runner. Z2Unit uses this feature to remote the actual test execution using a custom runner on the client side:

z2unit

(click on the picture to enlarge). On the server-side any custom runner is supported. It is really all just about remoting the protocol. In fact, this is no new invention: When your IDE executes local tests, it does essentially the same.

This means:

  • For the client is all looks the same: IDE, ANT, Maven…
  • Running a server-side test is just a click in the IDE – you can actually use this to script the server-side

How to use it

If you want to try, the best way is to follow the How to Z2Unit.

Practically a test class looks like this:


@RunWith(Z2UnitTestRunner.class)
@Z2UnitTest(componentName="my_tests/java")
public class AZ2UnitTest {

    @Test
    public void someTestMethod() {
        Assert.assertTrue("This is just a test method",true);
    }
}

For an example of a custom runner, see Running Spock Tests with z2Unit and Groovy.

There is virtually no restriction on the server-side. An advanced test case may fully integrate with Spring injection and have an application context initialized before execution to be a first class application citizen (without any other effort). Here’s a slightly anonymized excerpt from real world test case:

@RunWith(Z2UnitTestRunner.class)
@Z2UnitTest(
    componentName="com.xyz.abc.vault",
    dependencies={"com.xyz.abc.vault/applicationContext"}
)
@Configurable
public class BitBillServiceIntegrationTest {

    @Autowired
    private ThisService bbs;
    @Autowired
    private ThatRepository repo;

    @Test
    @Transactional
    public void testStatesUpdateAndVerification() throws Exception {
...

Ok, so where are all the other in-system test frameworks?

Z2Unit is a piece of z2. If you have a look at the implementations (see here) you will notice that it is almost trivial. So how come there is no other in-system test frameworks? After all Cactus seems to have died a while back.

Why do people pull out big and complicated monster test-deploy-into-some-fake-test-container-with-more-mocking-than-sense guns?

In case you did actually try you will have noticed that z2Unit benefits enormously from system-centric approach of z2: What you see in your IDE is what is running in your local runtime. There is no other web app to deploy (z2Unit is part of the base distribution). There is nothing to configure and Eclipsoid will make sure your IDE has all the right types without investing any intelligence.

Hence I suppose it is the lack of lack of integratedness, the de-coupling of development approach from the execution environment that is sometimes a little blinding and makes people accept crazy detours.

References

  1. JUnit Framework
  2. How to z2Unit
  3. Running Spock Tests with z2Unit and Groovy

Modularization is more than cutting it into pieces

Any half-decent developer knows that a monolithic code base does not scale. Not even for a single developer. There are many reasons for that, the most important being that humans are simply not made to concentrate on more than a handful of concepts at a time (I have a hard time memorizing more than four items on my shopping list).

We need abstractions that help us isolate conceptual complexity so we can focus our concentration on few isolated aspects at a time. One such helping abstraction is that of modularization of a code base into logical units to keep scope and reach of changes bound to a limited set of responsibilities. Followed by clean separation of module interfaces/APIs from their implementation details (i.e. implement information hiding, see as far back as Parnas).

This is the common practice in server-side Java (EE) development: Break the code base into modules, manage cross-module dependencies, have a build tool assemble everything into a deployable WAR or EAR archive. This approach has many striking positive aspects:

  • Expectation on the runtime environment in terms of module management is zero. It’s ok that it is all treated as a Web application, although the Web presentation aspect may be the least of all.
  • Any build tool can be made to copy JARs around and ZIP them up to yet another archive.

Let’s call this a soft modularization. Even without going for extreme cases like BIRT, this approach breaks down at scale of code base size and complexity and third party reuse. This is because every module adds a debt in terms of dependencies and because of simply being around. The former is due to conflicting or potentially conflicting use of some other libraries. Even if there is no type / version conflicts at some point in time (or none you noticed yet), any third party component upgrade might change that at any time.

The latter, i. e. what I called the effect of being around, refers to runtime penalties due to longer code introspection, adding to messy Spring configuration, spreading out its own configurations… adding noise and complexity to manage.

Get runtime isolation or get stuck

So we do really need (and should want) more runtime isolation that helps us keeping those “having too much in one basket” problems under control.

One true success story of code isolation is what all modern operating systems do: Process isolation. Practically speaking however, (real or pseudo) cross-process integration of software modules is simply not cutting it in terms of invocation performance (anybody remember EJB’s with remote interfaces as application components?).

So we want sharing of objects and memory while still being able to protect modules running into type conflicts.

Ah… so classloaders again?

For Java this means class loaders. Class loaders help achieve the following:

  • Controlled visibility of types – meaning conscious isolation as well as conscious sharing of types
  • Sharing of memory between modules and
  • direct, in-process invocation with the ability to pass data by reference

In essence a modularization approach based on the class loading mechanism loads types of a module with a module-specific class loader and has some more or less clever schemes to make types loaded from other class loaders visible to modules based on some configuration. In fact, this is the underpinning of any modular Java runtime (be it open source or proprietary) – that I am aware of at least.

Note: This is no way to build an OS. But for building medium sized solutions or the service endpoints of large distributed applications it is a useful compromise between assembly flexibility and reliability and runtime performance.

Plus, with a little care, and at least during development, the class loader isolation allows for module reload at runtime and hence shorter round trip cycles compared to restarting (or even redeploying) a complete application (and much cleaner and better defined than any hot swap technology or the likes).

It does however imply that you loose one big pro from the list above: You need runtime support, one way or the other, that manages your modules and a corresponding class loader topology. The good old tomcat days are over. Time to grow up!

Transitions

Now it would seem natural that you could go easily from one level of complexity control measure to the next. But that is not so. The following, admittedly somewhat complicated picture tries to illustrate different levels of achievements over progress (or requirements) in time and scale vs. complexity required to understand with some transitions:

modularization

The sequence of transitions while progressing from the left to the right is not as linear as it seems. While the first transition is the most natural, it gives a model that in reality does not lend itself to a smooth next transition in most cases. As soft modularization is also rather soft in its requirements, stricter isolation – required to make the isolated modules model scale – may be harder to achieve when starting so later (at which point in time it typically shows that good separation of concerns and APIs existed more on paper than in code).

Given a messy, badly-modularized codebase, it is tempting to try to simplify matters by doing some big cuts and split things up into two or more applications that are used in a distributed or co-located way.

That is, ignoring the third model in the picture and jumping directly for the distributed model. Doing that in order to solve a problem of a bad code base is almost always a mistake due to the operational overhead implied when going distributed. That is, unless you have a convincing scale, security, or technology-driven reason, you should never go functionally distributed. And if, try to make this a problem of “we are all the same – but we do different things” (i.e. still one code base, service endpoints differing in what parts are actively used).

But those modular runtimes suck!

Yes,…, yes,.. OSGi burnt so much ground. But before getting to that, let’s see what you really want. First of all, it should be possible to keep things the way they are as far as possible. That is, you do not need any dogmatic teaching about “how to do things the right way” (by people that most likely never implemented anything non-technological) that throws you into months long, certain to fail refactorings, while having no experience with the new environment anyway.

So, if everything is softly modularized today, you will want a smooth path into strongly modularized – just as much as you see benefits coming in.

Secondly, third-party libs should be naturally usable as is of course. I.e. without looking to deep, collecting package names or anything like that.

Then there is the much undervalued and underrated awareness required for the context class loader (again, see the other blog) but otherwise that’s that. And most Java libs have learned their lesson a while back so that they typically do work nicely in a multi-classloader , modular environment.

As now types may live in isolation, how do you actually create instances? There obviously needs to be a way to create services and the like from code that is not visible from other modules. So you need some runtime support for that. That sucks, because you need to adapt. In turn, you have a right to expect something that doesn’t try to teach you but rather tries to serve you.

For that purpose OSGi has deep at its heart the service registry. Z2 uses a resource lookup approach (that btw. seamlessly integrates with JNDI and Spring – which is important).

OSGi sucks and fails

Ah now I said it. OSGi, being the most prominent approach to modular Java, deserves some words. In fact, I wanted to write a blog article about it for a long time, but never had the energy. And it’s failing anyway as far as I can tell. It’s maybe a little unfortunate to have no standard modularization approach. On the other hand: Quite possibly that would just mean to make everybody equally unhappy.

Anyway, there are a lot of reasons for OSGi’s failure – in the Enterprise application space at least. Only about half of it due to technological aspects. Let’s say that OSGi is far too complex to be useful and far too code-centric, failed to position itself in any useful category, and tries to convince you about things that are not helpful to you (i.e. how to modularize THE RIGHT WAY) without helping you much to actual match that with your code base at hand.

Some concluding words

This article got a little out of control. The subject is probably to wide to make for a good, short blog. One little final, very personal conclusion: I doubt that there is a single runtime approach to modularization in Java. It is one of those attributes that need further specifics on how it’s going to be used and for what kind of applications or solutions, and most importantly, what is the application life cycle and development approach associated with it. Most likely it’s all of these aspects that drive the way to implement module support (if any). That’s definitely true for Z2. Here development productivity, application life-cycle requirements go first, it’s modularization approach is just a necessary conclusion.

References

  1. Modularization with Z2 
  2. JAVA components in Z2
  3. OSGi
  4. On the Criteria to Be Used in Decomposing Systems Into Modules, David Parnas

Scoping JPA

This post is about the following question:

If you bought into using the Java Persistence API (JPA) and your solution gets bigger and bigger, should you a) limit the scope of where managed objects are visible and, b) if so, how?

There is a lot of bad things to say about JPA and a few good and rather strong points to it. On the bad side you might bring forward, that the abstraction is incomplete (need to understand SQL anyway), JPQL is not algebraically closed, large persistence units (PU) imply a loading time and memory penalty without even touching a single persistent bit, getting started is ugly, a given persistence unit cannot be extended from other PUs. In short, it violates so many significant properties of the relational model, it can become a little embarassing at times (see also [1] and [2]).

On the plus side however, simple things are simple, the technology is mature, and – most importantly – any Java developer knows how to use it (or can be expected to). It is the standard API to access relational databases in Java. We do use JPA and it is for the latter reasons.

One key problem – in particular when leaving the realm of trivial applications, is that JPA-managed objects have a rather non-intuitive runtime behavior that can be the source of a lot of problems when not knowing the details of the application assembly extremely well: Detaching of objects with or without pre-fetch, lazy vs. eager fetching, accidental persistence updates without business-level validation due to simply calling a setter from a previously unforeseen control flow.

So in fact, there is a development scalability problem. A lack of clean separation of concerns, as in modularization and, as we will see below, to be mitigated via a modularization approach.

That is, to answer the first question, the visibility of domain types as JPA managed objects should be limited, exactly when encapsulation and information hiding becomes an issue due to the size or complexity of the solution at hand.

When projects start off small and don’t care about de-composing early, layering typically looks like this:

getting_started

The bold, dotted line in the picture above emphasizes that in JPA, the entity types of the persistence unit and their metadata do indeed represent the database stored model completely. And hence all Java binding of the domain model is in one place. Unlike JDBC-POJO -mapping tools like myBatis, this is inherently not use-case driven, potentially overlapping views onto the data model.

When the code base grows and gets modular (one way or the other – be it with or without runtime support of modules and separation of API and implementation), the natural tendency, out of convenience and simplicity, is to spread the use of managed entities across otherwise separated concerns:

too_big_to_maintain

(The light arrows imply visibility in the picture above.)

Now all the issues mentioned in the beginning start to show. Concerns that should feel separated stay tightly coupled due to implicit requirements when operating with entities directly.

We are finally about to tackle the second question.

As functions get separated into modules to achieve encapsulation, so should the overall data model be de-composed into separated concerns. That is, different persistence units should be used to represent different parts of the data model. In terms of Domain-Driven-Design (see [3]) this corresponds to the concept of bounded contexts. So, you might say PU==Bounded Context. But of course, it means nothing else but breaking up the overall model. Note that the resulting splits may indeed overlap in the database schema, at least for reading, but not in the code architecture.

Once the split of persistence units, or bounded contexts, is in place, the visibility of JPA types should be bounded to modules that require deep integration with persistent structures. Or put differently, the functional abstractions that structure all the code providing behaviour associated with the persistence unit may require a modular approach due to its complexity and so the bounded context spans several modules. One of these then naturally holds all type definitions of the JPA persistence unit. All other access to the respective bounded context should be hidden behind higher level API abstractions. In fact, the classical term subsystem seems quite adequate to capture this scope. Ideally a subsystem comprises of only one module allowing to completely hide domain types from visibility by other parts of the solution:

two_bcs

As an example consider some large business context as something that is naturally subject to functional extensions and that grows to many modules over time, while some conceptually more strictly constraint user management subsystem can be completely encapsulated into one module.

That brings another rather lengthy post to an end.

As a final remark note that all that has been discussed above may well apply to JPA persistence in particular, due to its specifics as laid out in the beginning. At the end of the day however, the choice of persistence access technology is rather irrelevant with respect to our conclusions. In order to make sure data is consistently validated and consistently exposed and interpreted, limiting all direct access and all application level representation of model data to one coherent implementation is a smart move in any case.

References

[1] “An Introduction to Database Systems”, Chris Data

[2] “The Vietnam of Computer Science”, Ted Neward,

[3] Domain-Driven-Design, Eric Evans