Blog of Dmitry Buzdin

Validating JSON with Schemas

2011-07-22T15:46:00.005+03:00

Introducing JSON Schemas

The world is in transition from XML to alternative document description languages such as JSON. Some of the reasons of moving from XML to JSON are the following:

JSON is a native Web format supported by most of the browser JavaScript engines. No conversions or usages of proprietary XML parsing extensions required.
Easier to read and to write, since it is more about data than markup. This format can be successfully edited in plain text editors. No more duplication of tag names at the beginning and at the end of each line.
Smaller document size, which saves some bandwidth.
Simplicity - no CDATA, no comments allowed, only few data types.
Flexibility - no schemas by default.

Lack of schema validation support is considered as a feature by some of JavaScript and other dynamic languages protagonists. However there are different scenarios and in some of them you would wish for such thing as JSON schema. To mention some of the example usages of JSON validation:

You are building a REST interface for external team or application or even public service with certain restrictions on data formats and sets of processed properties.
You are having application configuration stored in JSON format and it should be backward compatible at all times.
You are storing JSON to document-oriented database such as MongoDB and want to have certain indexed fields present in each document.
You want to write automated test suite, which validates JSON content.

In mentioned cases you would like to ensure successful evolution of your document format. You have certain mandatory properties and limitations like max, min or format checks. And it is for sure required to validate documents, which come from external sources and can not be trusted before storing them in the database.

In XML world these types of problems are solved long ago. In JSON the solution is work in progress. This leads us to JSON Schema specification published at Internet Engineering Taskforce page [link]. The standard is in its third version and authors claim that version four will be very close to the final one and will not introduce any serious changes [reference].

Schema Sample

Assume you have the following JSON object structure.

{
  "id": 1,
  "name": "some name",
  "letters": ["a", "b", "c"],
  "details": {
    "additional": "some more"
  }
}

JSON Document Example

It demonstrates most of the JSON common features - arrays, nested objects and properties of number and string types. According to the referenced specification the JSON schema would look like the following.

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
     "id": {"type": "number"},
     "name": {"type": "string"},
     "items": {
       "type": "array",
       "items": {"type": "string",  "enum": ["a", "b", "c", "d"]}
     }
     "details": {
       "type": "object",
       "optional": true,
       "properties" {
          "additional": {"type": "string"}
       }
     }
  }
}

JSON Schema Example

The format is quite straightforward and readable. The schema describes the data format and could be used for validation purposes. Some of the JSON schema features are used:

additionalProperties - additional properties are allowed by default, but if you are sure nothing else should be present put this setting to false;
enum - it is possible to apply content checks such as enumerations;
optional - all properties are mandatory by default, but you could set them to be optional.

The schema language is quite powerful and allows advanced validation features such as Regular Expressions, numeric checks etc. Refer for the documentation for the full list of features.

Java Implementations

Schema language itself would be not so useful without successful implementations. The reference implementation in JavaScript could is ready to be used and is available here http://code.google.com/p/jsonschema/.

Implementations in other languages and platforms are underway. Situation in JVM world is not so good.
There is a problem finding a proper validation implementation working in JVM at the moment.

First attempt to come up with the implementation was JSONTools project [link]. Looks pretty much dead now. Last site update was in 2008.

Another options is to get a working validator draft from Git repository [link]. Not really production ready software, not finished yet, has relatively poor performance, but could be easily used for offline testing, since it covers most of the important features. This validator is based on Jackson JSON processing library (http://jackson.codehaus.org/). It works and does what is promised.

One implementation, which looks more promising and finished has recently been announced
https://github.com/fge/json-schema-validator. This implementation was inspired by the Git code from he previous link and is also Jackson based. It lacks Maven releases and has weird java package name "kitchen.eel", but let's hope that it will change over time. Not many people would like to have kitchen eels in their imports I guess, especially during code and design reviews.

Scala may be a good implementation language for writing such validator for JVM. It has some native JSON parsing features and nice constructs like pattern matching, which makes writing recursive parsers a simpler task.

So there is still some space for good JSON schema validator frameworks in JVM and let's wait for the new spec release. I think JSON validation will be more important topic next years as adoption is continuing. Many frameworks try to come up with custom formats and approaches, but if there is a standard, why not use that?

Oracle EMEA Users Group Leaders’ meeting

2011-05-27T15:07:00.000+03:00

I have had an opportunity to represent Java User Group Latvia on annual Oracle User Groups meeting. This year it was held in Prague on May 10th and 11th.

Andrei Vorobyov from Oracle User Group Latvia was there as well. No one else from Baltics was noticed. This event have gathered 52 attendees from EMEA region. The attendees were both Oracle and Java User Group Leaders from all over the Europe.

In two days Oracle did several marketing and strategy-revealing presentations and user group leaders have had an opportunity to participate in open discussion events. The event was quite interesting. It is nice to feel that Latvian JUG is a part of a bigger family. Open discussion were related to collaboration of User Groups and universities, using social media tools for running UGs (e.g. Facebook, Twitter, Eventbrite), attracting sponsors and members and exchanging with speakers between UGs.

Important news and rumours from Oracle, which are relevant to Java professionals. Some of them are well know for quite some time, some of them were new to me:

Oracle is not bad at all for Java (with the exception of Google law suite case).
Java licensing model stays the same for the end users. No paid Java is planned.
Java 7 release activities will start at 07.07.2011. This will be global J7 launch event with speakers travelling across the world. Let's cross fingers so Latvia is in the official list of the show.
Java 8 is almost ready and will be released in 2012.
Swag and free stuff will be available for ordering for UGs starting from June (T-Shirts and Posters).
There are chances bringing Oracle Tech Days to Riga late 2011 or 2012.
JUG London is now a member of JCP comittee and JCP will be hopefully more open from now on. This is the first time JUG is represented in JCP.
Oracle donated Hudson back to Open Source community where it belongs :)
JRockit and HotSpot will merge to Converged JVM. Mission Control and other monitoring tools from JRockit will be offered for a price as before.
IBM and Apple have joined the OpenJDK project for quite time already and Oracle is pleased with this collaboration.
Oracle Java Academy programme is being restored this year.
There is a site http://jugevents.org/jugevents/ where references to JUG events are hosted.
Oracle conference call line is available upon request for JUGs.

This was a valuable experience and the most important results and outcomes are the following:

Established contacts with European JUG leaders.
Established contacts with Oracle representatives responsible for Technical Communities.
Exchanged information on how other User Groups are operating and what are the typical challenges.
Learned more about Oracle strategy towards Java technology.

Our Java User Group actions that will follow after this event are the following:

Collaborate with European User Groups regarding exchange of speakers.
Sign-up for the Java 7 release activities in July.
Order free swag in June for giveaways on JUG events.
Start discussions about possibilities of hosting Oracle Tech Days in Riga.
Add several new "features" to JUG - feedback forms, task lists to facilitate the community.
Work harder to find interesting speakers for Latvian events.

Next year we hope to send someone to visit this meeting again. Chezch beer and the old city were also great...

Technical Conference Checklist

2011-05-10T12:39:00.001+03:00

After organizing several larger events in 2010/2011 related to Java and Agile User Group activities in Latvia we came up with the basic check list for such events in order not to forget anything for the next time. I believe that this checklist could be valuable for people starting to think organizing various User Group events. So here it is.

Before (long term)

Decide how many people we expect
Pick event date
Check for overlapping events in the region
Book premises
Prepare conference site
Find good speakers
Estimate budget
Find sponsors
Sign contracts with sponsors
Advertise the event on partner sites, blogs and specialized portals
Create early bird discounts in case of paid events

Before (short term)

Print event programme for attendees
Create T-Shirts or other conference related stuff
Plan for food and drinks
Prepare badges for attendees
Create online registration form (e.g. Eventbrite)
Prepare online-feedback form
Review and listen to presentations
Find the place for afterparty
Prepare opening speech
Prepare closing speech
Prepare for filming
Check with sponsors about banners and advertising materials
Receive confirmation from all speakers
Receive the final presentations from speakers before the event
Check the venue for microphones, beamers and other infrastructure

During

Place sponsors banners
Mention sponsors in opening and closing speeches
Announce event Twitter channel
Place Computer or Beamer with live twits
Tell about afterparty, lunch and other related activities
Elaborate on event content in introductory speech
Control speakers time (5 minutes warning)
Have coffee breaks
Have few longer breaks for networking
Help to hold Q/A session after each presentation
Announce each speaker
Tell about next planned events/activities
Film presentations

After

Publish blog post about the event
Publish presentations and videos online
Send link to online-feedback form to attendees
Summarize feedback
Send feedback to speakers
Send letter of gratitude to speakers
Hold retrospectives meeting

Playing with Play

2011-04-18T13:38:00.002+03:00

I have given the presentation on http://www.playframework.org/ at the joint Oracle and Java User Groups event in Riga last Thursday (http://www.jug.lv/events/april).

I have ignored Play framework previously, thinking that it is yet another MVC web framework with no real use-cases. However, it turned out to be very concise and well-positioned tool for building page-oriented applications. The core selling points of Play are:

Productivity - comparable to Rails-like frameworks;
Simplicity - I would say Play is simpler than Rails;
Performance - absolutely stateless, it does not encourage to use sessions at all;
REST support - seamless HTTP to Java mapping for HTML/JSON/XML representations.

Presentation slides are published on Slideshare. The demo was demonstrating most impressive Play features:

Instant Java code refresh
Java compiler errors in the browser!
Groovy templating language
Automatic HTTP request mapping
JPA support
Testing mode

Java and the Web

View more presentations from Dmitry Buzdin.

I would suggest every Java Web developer to take a look at Play, since my gut feeling is that it will take its part of the market pretty soon. Play is not too mature framework. The oldest mailing posts date back to 2009. However, it is designed and implemented taking into consideration modern trends and requirements for real web projects. Technologies like NoSQL, EhCache, Memcached, Scala are supported with help of additional modules. The framework is very easy to learn and you could start writing real code in no time.

Since Play does not bring any advanced RIA stuff you could easily put GWT on top and let Play to handle REST backend and provide page navigation model.

Advanced Web Application Development with GWT

2011-03-24T14:57:00.000+02:00

Finally, the podcast from my 10th of March session in SkillsMatter London premises is available online (http://goo.gl/2IKz1).

This was an introductory lecture on what GWT really is and is not. It has some interesting parallels of how GWT fits with current state of IT technology. One of the key points is that JavaScript demand increases with each day. More and more RIA application and mobile-HTML5 enabled devices are available. And as we all know JavaScript simply does not scale. I will not go in details here and I am pretty sure that there are JavaScript professionals who think differently, but it is rather a collective industry opinion, not just mine. JavaScript is a low level language with lots of portability and other non-functional problems. So either you are expert in that or you use GWT. These are the general options today (or use Flash?).

GWT Compiler Optimizations

2011-02-16T13:21:00.000+02:00

GWT compiler is slow, everyone knows that. Every time someone compares GWT with other "super Web 2.0 technology" the main drawback of GWT is stated quite simply - it is slow.

But what does make the compiler slow. Is it just a poor quality of the framework? Or is it just a lot of things going on undercover and we would really like to run them even knowing the performance implications. This post highlights performance optimizations done in GWT Compiler. I have failed, personally, to find any decent documentation on this subject other than reading the source code, which resulted in a given description. The observations are done using SVN trunk, so it may be possible that in older versions of GWT you are using some optimizations are yet missing or work a bit differently.

The entry point for the source code is JavaToJavaScriptCompiler in gwt-dev.jar package. Most of the optimizations are done in compilePermutation() and optimizeLoop() methods. Optimizations are utilizing Visitor pattern extensively and are work on GWT implementation of Java Abstract Syntax Tree (AST) (JClass, JField etc.). Visitor pattern is a common choice for tool doing source code traversal (such as FindBugs). Note that some of the optimizations are executed several times depending on optimization settings. That is done because first round of optimizations can unlock potential improvements for the second round. It could take several loops for the complex code base to "diverge" into the stable/optimal state.

On a high level the compiler does the following chain of transformations starting with the Java source code and ending with JavaScript source:

Java Source Code -> Java AST -> Optimized Java AST -> JavaScript AST -> Optimized JavaScript AST -> Obfuscated JavaScript AST -> JavaScript

Note that GWT works with the sources, not with the bytecode as this is sometimes a misconception. GWT does not invoke Java Compiler and does not depend on that. This post describes Java AST to Optimized Java AST transformation only. Other transformation steps are worth separate discussion.

And now lets get to the list of optimizations - in order of their appearance in the source code:

Pruning - removing unreferenced classes and interfaces as well as methods and fields. This is the magic optimization, which allows you to include megabytes of GWT dependencies and still produce downloadable JavaScript piece if you do not use all library features.
Finalizing - make methods, classes and variables final if there is such a possibility.
Make Static - if instance method can be done static it is rewritten to static, which means it is not overridden anywhere and does not depend on instance variables.
Type Tightening - optimization of field and parameter types to be more concrete. This is done in order to reduce type casts and instanceof checks.
Method Call Tightening - updates polymorphic method calls in order to invoke the exact version of method right away. Walks up class hierarchy and finds most suitable method.
Dead Code Elimination - eliminates irrelevant parts in logical conditions and expressions (such as a || true; a + 0), which do not make any effect on program execution. This is mostly done in areas of Java primitives and logical operators.
Method Inlining - algorithm inlines (embeds) method bodies with no more than 2 lines and no references to other external methods. The idea is that it is more optimal to embed those one-two lines in multiple places rather than leave a separate method for that.
Same Parameter Value Optimizer - detects when the literal passed through parameter is used in method calls. Uses the literal directly, which makes the parameter obsolete.
Enum Ordinalizer - replaces references to Enum variables with their ordinal values. This is usually simply int key representation.
Remove Empty Super Calls - what could be simpler to understand. Does exactly what is says - removes super() calls to empty methods.
Set of Normalization Rules - make similar operations look unified. Merging catch blocks, working with arrays etc. The goal is to transform common parts of code to be optimal for JavaScript generation purposes without changing the behavior. Replaces calls to JDK classes with calls to com.google.gwt.lang classes.
Pruning - is run for the second time as the last optimization as previous AST modifications could unlock new potential constructs, which are not referenced anymore.

This is a full list of Java AST optimizations I could spot in the source code and which are implemented in explicit manner. The source code is easy to read and well documented therefore I advice everyone interested to take a closer look.

To summarize - GWT can not be compared to other Web 2.0 technologies as those are not doing even a fraction of provided optimizations on JavaScript level. Of course server centric Ajax frameworks fully benefit from Java Compiler and JVM JIT optimizations, which are not available for GWT running in the browser unfortunately. So it is up to you what path to choose. Despite the number and complexity of optimizations GWT compiler is very stable and bugs in its core are rarely encountered. This message come both from personal experience and talks with fellow GWT users.

It is possible to disable all optimizations by passing -draftCompile flag to GWT compiler. In this case most of the optimizations will be omitted. This is the answer for impatient who can not wait for their application to recompile. And don't forget to use Development mode (which does not invoke Java to JavaScript compilation), SSD disks and Dual Core Machines in your development environment.

There is a page describing future planned optimizations. Sharing it just for curiosity purposes.

Is Java Reflection Really Slow?

2011-01-20T14:41:00.002+02:00

One of the sayings you still hear nowadays from fellow developers is that not only Java is slow, but also that Reflection is an exceptionally slow part of the JDK. This post is intended for those who believes that or want to have a constructive conversation with the believers.

Technical topics presented in this post are years old, but somehow some of the things are surprising when you discover them. So even if you are a seasoned Java developer you may find some interesting points here.

How can anyone measure that reflection is slow? The first thought is by running a benchmarking app. This could be done as simple as following.

long start = System.nanoTime();
method.invoke("reflection");
long stop = System.nanoTime();
...
long start = System.nanoTime();
"reflection".length();
long stop = System.nanoTime();

The results are quite obvious: reflection version takes around 15000 nano seconds, which is 15 micro seconds; method invocation takes 2.2 micro seconds, which is around 7 times faster.

But was it really a clean test, which shows us real results? It turns out that not exactly. The test was far from being clean. The most obvious thing is that for the first time the results will be slower, because of the possible Class Loading, initialization and other activities, which may result in excessive objects being created and Garbage Collection invoked.

The first thing to fix is to run the test for several times in order to warm-up the code. How many times you should do that to get the plausible results? One, two or ten? It turns out that you should do even more than that. Aside of usual Class Loading there are different things happening under the hood. JIT compilation may occur at some point and also internal JVM implementation optimization. It turns out that you could monitor whether any classloading and/or JIT compilation is happenning programmatically in order to know whether there were disturbances in test. This could be done by two provided JMX beans. Note that some of the JIT compilation processes are triggered only on if JVM started with -server switch. By default JVM works in -client mode assuming more random class invocation patterns than on the server and does not apply heavy optimization of the code. So if you have ever ran benchmarks in -client mode, please redo them, as they differ greatly as we will learn further.

ClassLoadingMXBean classLoadingBean = ManagementFactory.getClassLoadingMXBean();
CompilationMXBean compilationBean = ManagementFactory.getCompilationMXBean();

You could poll these beans after each test iteration and compare the timings to know whether class loading or compilation was active during last iteration. Ok, this is solved. Now more interesting topics. In Sun JVM there is a mechanism called Reflection Inflation. In two words it "inflates" reflective method invokations with bytecode implementations meaning there will be less overhead to call them. Take a look at the implementation. This process is triggered by default only after 15 iterations. It means that only 16th test iteration will be closer to what we will have on real application environment.

Now it becomes clear that the first attempt to perform performance test of reflection was oversimplified and there are plenty of details to consider. One approach to still measure the performance is to use a microbenchmarking tool. Microbenchmarkings tools are intended to bring the JVM in "stable" state prior to executing tests and run your test suite for sufficient amount of time to get statistically proven evidence of the results.

One of the frameworks I tried and quite liked was Microbenchmarking tool from Brent Boyer. It has very simple API and the only thing you should do is to put your specimen code inside Callable.

public static void main(String[] args) throws Exception {
    Callable task = new Callable() {
        final Method method;
        final FastMethod fastMethod;
        final Object[] EMPTY_ARRAY = {};
        {
            method = String.class.getMethod("length");
            FastClass fastClass = FastClass.create(String.class);
            fastMethod = fastClass.getMethod(method);
        }

        public Object call() throws Exception {
            //return method.invoke("hello");
            return fastMethod.invoke("hello", EMPTY_ARRAY);
            //return "hello".length();
        }
    };
    Benchmark benchmark = new Benchmark(task);
    System.out.println(benchmark);
}

I have included the third method call approach in the tests, which is CGLib
FastMethod. By using it you force to generate bytecode method accessor from the first execution time and exactly where you want it. This approach was used in frameworks for some time (for example in Hibernate), but nowadays people are going away from that to direct reflection invocation as it does not have a radical difference on modern JVMs.

The results for running direct method invocation, reflective one and CGLib FastMethod in both server and client modes are given below.

-client getter() :         first = 36.743 us, mean = 10.317 ns (CI deltas: -46.363 ps, +178.344 ps), sd = 3.697 us (CI deltas: -2.959 us, +4.774 us)
-client Method.invoke() :  first = 52.688 us, mean = 128.262 ns (CI deltas: -1.377 ns, +1.635 ns), sd = 17.014 us (CI deltas: -2.323 us, +3.549 us)
-client CgLib FastMethod : first = 42.709 us, mean = 27.511 ns (CI deltas: -119.732 ps, +186.596 ps), sd = 4.722 us (CI deltas: -1.398 us, +2.178 us)

-server getter() :         first = 38.303 us, mean = 3.915 ns (CI deltas: -14.731 ps, +19.287 ps), sd = 1.088 us (CI deltas: -204.310 ns, +360.180 ns)
-server Method.invoke() :  first = 44.177 us, mean = 24.041 ns (CI deltas: -77.722 ps, +124.821 ps), sd = 3.127 us (CI deltas: -925.638 ns, +1.617 us)
-server CgLib FastMethod : first = 44.941 us, mean = 14.756 ns (CI deltas: -97.488 ps, +165.397 ps), sd = 5.685 us (CI deltas: -1.988 us, +2.727 us)

On my workstation calling a method by reflection 100000 (hundred thousand) of times should cost us only ~3 milliseconds! The conclusion is that on modern server infrastructure with proper JVM settings we should not be worried about Refelction performance too much. Another conclusion is that going in the direction of direct bytecode manipulation such as CGLib FastMethod will not bring a tremendous difference in performance, but will add dependencies and complexity to your project. If someone is blaming to detect performance degradation using reflection this could be the case of -client measurements with only a few iterations. Of course this is only valid if we simply compare direct invocations with calls by reflection. If you heavily use Class introspection API and do a lot of other fancy Reflection stuff things can become really slow.

Here are the same comparison results done by Google Charts API.

So if anyone will try to blame you for writing slow reflection code you could share this results and re-run microbenchmarking tests by yourself.

Similar performance benchmark results, which triggered this post could be found here.

Simplest Code Generator with GWT

2010-10-14T14:19:00.002+03:00

Many people tend to think of GWT as a specialized Web framework. However its main essence is Java to JavaScript compiler. One of the great things you could do with that is to write your own code generators, which gets invoked during compilation process. There is limited documentation on this topic on official GWT site. Reference to RPC implementation generator can be found at Deferred Binding Page. The example given is a bit complex and require GWT source code digging to understand what is happening there.
Therefore this post represents the simplest working code generation example, which shows the general approach and its capabilities. What we need to achieve in this example is to provide a generated implementation for Function interface shown below.

public interface Function {
    public Object execute();
}

To test the results you should invoke GWT compilation and its code generation engine. It is possible to do so out of GWTTestCase. The following test checks that implemented funcionality works as expected. Now we have to make our test runnable and passing. The most important part here is invocation of GWT.create(), which is the entry point of the Deferred Binding.

public class FunctionGeneratorTest extends GWTTestCase {

    @Override
    public String getModuleName() {
        return "com.acme.gwt.Generator";
    }

    public void testGenerator() throws Exception {
        Function function = GWT.create(Function.class);
        assertNotNull(function);
        assertEquals("Hello", function.execute());
    }
}

First what you need to do is tell the compiler that the interface will be replaced by generated implementation. To do so you need to put the following instructions to your gwt.xml descriptor.

<generate-with class="com.acme.gwt.FunctionGenerator">
    <when-type-assignable class="com.acme.gwt.Function" />
</generate-with>

Now let's code the generator itself. You should extend Generator class to make it work.

public class FunctionGenerator extends Generator {
    private static final String IMPL_TYPE_NAME = 
Function.class.getSimpleName() + "Impl";
    private static final String IMPL_PACKAGE_NAME = 
Function.class.getPackage().getName();

    @Override
    public String generate(TreeLogger logger, GeneratorContext context, 
String requestedClass) throws UnableToCompleteException {
        TypeOracle typeOracle = context.getTypeOracle();
        JClassType functionType = typeOracle.findType(requestedClass);
        assert Function.class.equals(functionType.getClass());

        ClassSourceFileComposerFactory composerFactory =
            new ClassSourceFileComposerFactory(
                IMPL_PACKAGE_NAME, IMPL_TYPE_NAME);

        composerFactory.addImport(Function.class.getCanonicalName());
        composerFactory.addImplementedInterface(Function.class.getName());

        PrintWriter printWriter = context
            .tryCreate(logger, IMPL_PACKAGE_NAME, IMPL_TYPE_NAME);
        SourceWriter sourceWriter = composerFactory
            .createSourceWriter(context, printWriter);

        sourceWriter.print("public Object execute() {");
        sourceWriter.print("    return "Hello";");
        sourceWriter.print("}");

        sourceWriter.commit(logger);
        return IMPL_PACKAGE_NAME + "." + IMPL_TYPE_NAME;
    }
}

The Generator loads requested Function interface, determines its package and writes down FunctionImpl type with method execute implemented. You are completely in control of what Java code you are outputting. It is possible to add implementing interfaces, parent class, imports and annotations. Then the body of the class is written using simple String concatenation. Notice TypeOracle class, which is GWT internal Type introspection engine.
This is the simplest possible generator, but here you have limitless capabilities of introspecting existing classes, looking for annotations and then generating the boilerplate code. In fact Deferred Binding and Generators are the basis for most of the advanced GWT frameworks like gwt-log and gin. Good knowledge of Generator concept is absolute must to understand how GWT technologies work internally. The example is prepared using GWT 2.0.

ThoughtWorks Radar Demystified (GWT)

2010-09-09T11:04:00.003+03:00

I have recently discovered ThoughtWorks technology radar, which is published since January this year.

First of all I find it a very useful tool to follow the general trends in realistic software development. Before that, all radars, seen by myself, were either internal customer ones (means that they can not be shared) or radars from companies like Forrester and Gartner, which are an interesting reading, but more concentrated on comparing commercial products, like ESB and BPM providers. Main target of such reports are business people or IT management, so it does not spend much attention to exact low-level techniques and tools used for productivity gains. As my personal opinion is most of the heavyweight tools should stay in the past (exceptions exist of course), therefore I am not interested in this kind of reading.

ThoughtWorks radar is split in four parts: techniques, tools, platforms and languages. Each part of the radar is divided in four areas: hold, assess, trial, adopt. The most interesting ones are "hold" and "adopt". On "hold" means that the technology/technique is in a risky state of being discarded or have undergoing major changes, which will make its future more clear. "Adopt" means the technology/technique is usable for productive projects and efficiency gains are worth investing.

The main reasons to post this, is single particular point on the radar, which is GWT (found in Platforms section). Right now it is put "on hold", which is surprising to me, as the technology keeps gaining the momentum to become one of the major future Web technologies. Therefore I would like to understand why ThoughtWorks is thinking in this way and provide my comments on that.

Platforms - Google Web Toolkit

The current decision is to have GWT on hold. To quote ThoughtWorks from radar's April edition:

"Google Web Toolkit (GWT) offers an interesting premise: write Swing-like Java code and generate unit testable JavaScript widgets and user interfaces. From a practical standpoint this doesn’t work well. First, using code-gen to produce the artifacts is time consuming, artificially extending build times and requiring manual changes to obtain optimal package layout. Second, if the JavaScript doesn’t behave exactly as you want you will have to hack the generated code. Third, using Java to generate JavaScript means that you can’t take direct advantage of the powerful features of JavaScript or numerous libraries such as JQuery. Finally, the JUnit support is quite limited, for example code using reflection cannot be tested."

Unfortunately I have an impression that this has been written by a person who did not use GWT at all, but rather studied several articles. GWT does not make you to use code generation during development. During coding in the development mode you are simply refreshing browser, which brings client code changes automatically without recompilation or lengthy code generation. This is backed up by the way GWT compiler works. It actually parses and compiles Java sources, bypassing javac invocation. That is how dynamic recompilation and reloading is implemented. This gives you the same productivity benefits as you would get by using dynamic languages on JVM such as JRuby.

The section about "manual changes" can be generally applied to code generation tools, but is not met in GWT context. GWT can have a full build automation starting from compilation and to opening a browser on the required page.

Statement about possible JavaScript hacks is theoretically right, however from my personal experience we have not met a single bug in GWT compiler, which would make us to go into native JavaScript coding. And I am talking about nearly a dozen of production running projects using GWT features extensively. We have been using framework versions 1.3 to 2.0 and none of them have had serious compiler bugs, which altered application behavior unexpectedly.

One more thing is that you actually *can* use native JavaScript functions, for example from JQuery. It is not a recommended approach though, but with enough technical confidence this approach is easily implemented and is called JSNI (JavaScript Native Interface). Calls to JavaScript libraries are injected inside resulting generated code and the library itself can be bundled together with GWT application distribution.

JUnit support is a bit limited, that one is finally true, but the statement about reflection is totally wrong. There is no reflection possible in GWT and that is why it is not possible to test that. GWT uses Java in its own specific manner. It applies more restrictions that Java static typing model itself. To be able to properly reference all execution paths in order to apply compile-time optimizations GWT should have an understanding of a full call graph. It is impossible to do with reflective calls and that is the reasons those are forbidden in Java code compiled to JavaScript. The real limitation of JUnit is that you can't unit test widgets interactions without running test in special test harness called GWTTestCase. This is due to the fact that underlying Browser DOM implementation is required in order to draw most of the widgets. One of the most popular design patterns in GWT world right now is MVP, which main goal is to address these kind of issues and separate widgets (View) from logic (Presenter).

To Summarize

Most of the content in the radar is done in a professional manner and is worthwhile looking at, but the GWT example is either totally wrong, or ThoughtWorkers have not looked at the framework close enough yet, but still wanted to include GWT in the list, because of its current importance. My opinion is that GWT is one of the unlocking technologies for "Browser as a Platform" vision mentioned in the radar as well. Why is it so, deserves a separate post. I wish ThoughtWorkers to review their view on the GWT potential and put that at least in Trial state. Nevertheless I would recommend the radar to my colleagues to get an overall picture of what is going in the world right now and to look for technologies/practices to invest your time with best ROI.

Agile Latvia First Event

2010-08-30T14:54:00.000+03:00

On the last week we have had an opportunity to organize our first Agile event in Riga. Initially planned as small and short we ended up in having a full-house of 62 attendees (not including speakers) and a track of six sessions. The topics were varying between organizational practices to technical ones like TDD.

The general feedback on the event was positive. We received quite a number of good comments. One of the problems stated was the fact that people beginning their Agile journey were a bit frustrated, because most of the talks were done expecting listeners to have some background in Agile already. However, more experiences people were satisfied with quality and content, which is a good sign. We will have a Retrospectives session this week to identify gaps and plan better for the next time.

With first Agile Latvia being a solid success we are aiming at end of November to create a larger (full-day) international event. Stay tuned...

GWT 2.0 Development Mode on IE7

2010-06-09T17:03:00.001+03:00

We are in process of migrating our custom Widget library to GWT 2.0.3.
When opening Development Mode in IE7 the following JavaScript exception was thrown.

number: -2147024891
description: Access is denied.


at com.google.gwt.user.client.impl.HistoryImplIE6.getTokenElement(Native Method)
at com.google.gwt.user.client.impl.HistoryImplIE6.init(HistoryImplIE6.java:78)
at com.google.gwt.user.client.History.(History.java:63)
...

It turned out to be a security violation when IFRAME located code tries to access parent document.
The solution is pretty simple - adding localhost or 127.0.0.1 to list of trusted sites in Internet Explorer Security settings.

GWT Dynamic Model

2010-05-17T23:58:00.010+03:00

In the current GWT world the most popular (at least in theory) architectural style to build serious enterprise applications is MVP. Quite a lot of articles and open source frameworks are dedicated to M and V of the MVP, but to my knowledge not so many information is there how to make the M (Model part) right.

This post is giving an insight on Dynamic Model approach successfully used in several serious enterprise applications based on GWT framework.

Dynamic Model is nothing new, but a generic model, backed up with a map implementation. It means that there is no traditional getters and setters at all, but instead generic get(String id) and set(String id. Object value) methods.

Moreover nested, indexed and key based attributes are supported. All following expressions are valid and return the same value.

model.get("customers.[21].address.city");
model.get("customers.[21]").get("address.city")
((List) model.get("customers")).get(21).get("address").get("city");

Model holds an internal relations between its nodes. Possible intermediate nodes are submodels and lists. Leaf nodes could be of any object type.

It turns out that this simple data structure has many benefits especially in GWT applications.

There are two major restrictions relevant to this solution:

the more serializable Java classes you have the bigger your final JavaScript is;
there is no reflective method access available by default in GWT code (and we definetly do not want all class metadata being stored in resulting JavaScript);

By introducing Dynamic Model concept you significantly reduce the number of classes, which are being send over the wire between client and server. There is no need to support two mirrored set of objects, one for persistence, second for GWT serialization. The performance part could be solved by splitting your application via RunAsyncCallbacks since 2.0, but you still need to create dozens of dumb getters and setters.

By using Dynamic Model you are able to reference attributes by String-based mappings, which is a replacement for reflection mechanism. Just to give a hint of why we need reflection analogue on the client is this piece of code.

rule("myRule")
.when(eq(LARGE_RISK, Boolean.TRUE))
.then(make(APPROVER, MANDATORY))
.otherwise(make(APPROVER, HIDDEN));

The example demonstrates an internal DSL, which gets executed in uniform manner both on client and server. The trick is not to evaluate rules themselves, but rather pass a serialized AST to the client. It is possible to validate data on the client to show visual effects and bypass server calls, then revalidate on server to fight against HTTP requests forgery. It can not be achieved without map based model as there is no way to reference a field both on client and server (unless you use reflection emulation library on GWT).

The same Dynamic Model serves as a basis for declarative data binding.

bind(APPROVER, comboBox);

There is a certain performance overhead this implementation because GWT does too many class type checks behind the scenes. This can be disabled since GWT 2.0 using a compiler feature, but is not recommended thing to do. However, applications based on described approach are able to display hundreds of widgets bound to Dynamic Model in Internet Explorer with a reasonable performance.

The resulting solution is almost one model fits all approach and provides significant benefits on large applications, where serious effort should be spent to create and maintain models for all the screens.

To achieve proper transformation to application-side model Dozer Bean Mapper is used in our reference architecture. Next example gives a hint how declarative mappings look like in described case. It is possible to define set and get method names for custom data structure like Dynamic Model and then map nested attribute to GWT model with transformation to String.

<mapping>
  <class-a>com.sample.server.ServerModel</class-a>
  <class-b map-get-method="get" map-set-method="set">com.sample.shared.ContractModel</class-b>
  <field>
    <a>contract.startDate</a>
    <b map-get-method="get" map-set-method="set" key="startDate">this</b>
    <b-hint>java.lang.String</b-hint>
  </field>
</mapping>

The approach described here actually works in large scale applications and is running on several productive systems. This is not a feature, which can be easily recreated during weekend, so I am thinking of extracting this as a separate Open Source project. Besides the model, in the original implementation, we have simple attribute query language, data binding and GWT portable rules concept.

At the moment I am aware of the following client-server communication approaches in GWT:

Name convention based Bean -> BeanDTO parallel object hierarchies (automatically mapped by Dozer)
Sending persistent model stripped down by Gilead
Custom mapping from Domain Model to Screen Specific Model (some mapping tool or hand written)
Using Enhanced Classes Serializing GWT feature
Dynamic Model based approach described here
Anything else?