Sunday, July 29, 2012

From Hibernate to MongoDB - Part II

For those who are interested, I still don't have any large dataset results.
Last weekend I spent with datanucleus/mongo but I had some issues that noone did seriously take care of, so I decided better not to go that way (Sorry datanucleus folks, but I'm used to get support from the community). This weekend of course I started another approach with eclipselink which has jpa/mongo support, too.

Let me say, it was a whole lot more work than I expected it to be. EclipseLink has a different understanding of the JPA annotations, which caused some boring refactorings in the domain model. Mostly Fetch.Eager doesn't work with mongodb, and some other minor stuff, I found that ManyToOne(optional=false) doesn't work either, have a bugzilla for this one.

But the hardest work was making eclipselink play with spring dm, and gemini jpa. I had some issues with the latest spring release aop stuff, which didn't play with the osgi classloading anymore, so I went back to 3.0.5 which I knew was working. Then I created a FactoryBean based on the gemini jpa EntityManagerFactoryBuilder service, so that I can still use my spring managed transactions.

One word to the gemini folks: Bundle start order is not what osgi suggests. But restarting a blueprint/spring powered bundle (thats what gemini jpa does for weaving the entity classes) when their appContext isn't finished yet gives you an explosion of exceptions. By the way I am still running on dm 1.2 because of the namespace handler resolving still is a mess in blueprint. Just my two cents.

Inbetween I did some clean up work (pay your debts) that still needs some polisihing, as there was not enough time to pull through all the changes, so there are some compile errors now. But nothing evil.

One thing to say is that I have some logic that needs a complete restructruring, as there were many queries based on the relational thinking, that now don't work anymore, as eclipselink/mongo doesn't support joining. Even if it did, mongodb gives you a different persistence model and it makes sense to work with larger objects than I did with the relational model.

I have created plenty of tasks that still need to be done, I'm far away from starting to implement mutli-tenancy (which of course is the reason for all of that).

Up to now I spent round about 40hours on this (remember domain model is about 70 classes, that are fine now). Of my 20 DAO classes there are three left to restructure that depended too heavily on the relational stuff. All other stuff is working.

Load time is blazing fast now, most of the time goes into RAP UI code creating the editors. Guess this will be my new bottleneck :)

My overall impression is, that this is still young stuff. Speaking about eclipselink/mongo, most important things work, but things that don't, always throw completely insane errors, forcing you to debug into the code to understand what has gone wrong. For e.g. when there's a join in the query, a ClassCastException is thrown, that DatabasePlatform cannot be cast to MongoPlatform. eclipselink folks, there is plenty of work in there, too. But overall I am confident today that I can build my (non-profit:) business on it.

Saturday, July 28, 2012

RAP UI Testing

These days I'm evaluating UI testing methods for RAP Applications. I made some nice progress with a commercial tool called QF-Test. It has a click recording functionality, and works like a charm with the WidgetUtil.CUSTOM_WIDGET_ID way that rap offers to generate custom ids for the controls. It's really easy to setup, and seems to have no problems with the rap ui so far. I allows groovy/jython scripting and is able to evaluate JavaScript expressions on the dom under test. I am really happy that I finally can get up some ui tests quite easily. Plus I have to say that the folks at QF-Test took great care of getting me started with my evaluation license, +1 for their support.

This screenshot of a recorded Sequence is german, but it should be understandable (Klick means Click :)

Sunday, July 22, 2012

From Hibernate to MongoDB - The promise of changing the persistence layer.

This is not basically eclipse related, but it comes from my eclipse project, that I recently ported to RAP and now I plan to go multi-tenant with it.

This is not my first attempt in this direction, but this weekend I have a working proof of concept for changing my persistence layer from hibernate/hibernate search to mongodb.

I have about 80 domain classes and a bunch of handmade custom DAOs that are mostly based on the JPA Api. After playing with orientDB (I stopped that because of no "delete-orphan" there) this weekend I made an attempt with the datanucleus mongodb storage engine. To replace hibernate search, I decided to go for elasticsearch, which seems to be straightforward to build a multi-tenant index.

At first I struggled with some classloading issues, the datanucleus enhanced classes need a dependency on the jdo api at runtime. Then one major all-solution refactoring to use the mongodb string-based id on all my entities. I guess that took most of the time. I also changed  many dependend relations to embedded types, so that I can get the performance benefits mongo db promises.

Before I had load times for a single object graph (hibernate/sql server) from 3 to 4 seconds (yes one entity, you now know why I have to change that) and with mongo db and all the embedded collections you can hardly notice load time.

It's just that the relational model is not suited for complex object graphs. Even with massive parallel processing on an sql server cluster I had processing times around 10 minutes for 50k objects. I do not have values on that for mongo (need an importer first), but I guess it will work out.

I still have some exceptions and need to refine the elasticsearch integration, but over all I worked some 15 hours on it and at least I have a proof that it's doable.