Migrating to Github powered website

This blog has migrated to a new home at ethomasjoseph.com - a domain which owned for long. For a long time this blog was hosted at the domain tech.ethomasjoseph.com but was still powered by Blogger. As of today, the migration to ethomasjoseph.com is complete, after the completing the 301 redirect setup for  tech.ethomasjoseph.com.

Why?

I started using Blogger since 2006 for blogging. Even though I was not a posting frequently, I always felt restricted in the flexibility that Blogger provided me.

• The web based editor is very restricted in what editing capabilities - especially when posting code snippets.
• It is simply too complicated to customize the look and feel of the website. Blogger has its own syntax and style of how the Blogger template should be customized. I simply cannot use my tooling - SASS, LESS, NodeJS based workflows etc for any of these customization.
• I want to get away from Google. Google is becoming creepy each day. I have already changed by default search engine, and now the blogging platform. The G+ comments are also creepy and crappy, which you can't avoid nor migrate. I am getting rid of all my comments accumulated over the past years in G+, and moving to Disqus for comments.

The new website apart from hosting this blog, will be home to pages, and articles and lot of good features.

I am dicussing more on this migration on my new website - ethomasjoseph.com. See you there!

Don't be a dick technologist

Being a technologist doesn't mean that everything we create is created from scratch. We Google around, find lots of write ups, blogs, articles, and code and we take them all for writing our own solution.  However the effort we put to create our solution would vary a lot. Our solution may depend on a lot of libraries, or at times we have to copy some code and get started. All this is so usual to our normal routine work, but one thing many of us forget about is the license that needs to be respected - especially when dealing with Open Source licenses.

A decade back, people used to question the quality of the Open Source, but today with professional open sourcing in place this mindset has changed and people now have thinking that using Open source is good. However many people still believe that "Open source is free". People also wrongly believe that "Every visible source code is open source". Another wrong belief about Open Source is that for source code - "can-see is can-copy". This mindset needs to change. People must know the license that applies to any piece work (including a write-up), and must faithfully abide by the license.

But most of the time, reading the license text is very verbose and complex with legal terms that many of us would fail to understand it. But irrespective, we all must give due respect to the hard work of the people(s) who originally created the work, and which you are now using. This is rightly summarized by the DBAD license (Don't be a dick license). This license rightly brings out in simple and humorous way the courtesies that every user of the piece of work must follow. This license focuses on the moral responsibility of a person copying the work, rather than legally binding the person to do something or prevent it. It highlights the fact that many people copy other's work, and remain ungrateful, and do not attribute their work to the original work, or even worse - complain about it.

I personally may not apply a DBAD license due to its loose legal terms. However,  I would love to put its text along with my work, to let other know how important it is to show some courtesy to the creator of work. I would love to reproduce some of the context text of the license here.

As developers we all want to protect our code from dicks that try to steal, sell, infringe or just generally rip us off. For this we have licenses like GPL, MIT, etc.
Sometimes however, you release a project where you just don't care what happens to the code. For this Sam Hocevar created the WTFPL which is a brilliant license.
For those of you who want something in between, try the DBAD license.

One thing which every person and company using other's work in their own work (especially when using Open sourced work) must understand is that although the work is free to copy, the license accompanying it is still a legal license and should be abided. The copyright owner (or someone on their behalf) of the work can always sue the infringing dick. When you make loads of money by selling someone else's work, do care about buying them a pint or a coffee please, or at-least acknowledge them. Don't be a dick!

Universal Applications with Universal JavaScript

Universal Applications with Universal JavaScript from ethomasjoseph

JavaScript is going Universal - Why should this matter you?

If you program for the web, and think that you are a "backend developer", and JavaScript isn't my cup of tea, that is not true any more. You can close your eyes and wish away that JavaScript on Server side is a hype, but the fact is that JavaScript is invading the Server side space each passing day, and you "must" know JavaScript and JavaScript based rendering technologies, if you intend to create next generation web and mobile applications, or may be - just to be employed!

In the recently shared roadmap of Angular 2 in Angular U, the Angular team presented the idea of splitting Angular into two pieces: core functionality and the renderer. The concept of renderer in Angular could be used for server side rendering, or native rendering on iOS or Android.

This roadmap clearly brings Angular into the family of Universal JavaScript (also known as Isomorphic JavaScript) rendering technologies. The family of Universal JavaScript rendering technologies already include Rendr, Flux, React, Dust and LazoJS - to name a few popular ones. Coursera, Airbnb, LinkedIn, Facebook, NetFlix, Walmart are some of the high volume web applications that have already moved to Universal JavaScript. This move has already helped these websites to write applications that run universally - on server, browser and mobile apps - significantly reducing the development and maintenance effort across various channels, and more importantly - you just maintain one code and be flexible about intelligently choosing a logic execution and rendering to be on server or client side while pushing up the web applications to be much faster!

JavaScript going Universal

Despite the creation of JavaScript as a lightweight look-alike brother of Java, JavaScript eventually became a key component in enabling the Web 2.0 evolution - especially through AJAX. JavaScript has moved much beyond its initial design goals, with more standardization with ECMA, optimizations done to every release of the JavaScript engines, JavaScript is all ready to rule the web (and more platforms).

Universal JavaScript
Image Credit - Netflix

Today JavaScript is a ubiquitous runtime.  JavaScript is everywhere - from Browser,  server side, embedded device, mobile and gaming.. The language's influence continues to grow on the server landscape via Node.js, and runtimes such as Rhino and Nashorn for Java; Jurassic and V8.NET of .NET and projects such a J2V8  and React aiming to go even higher by targeting at Server side and mobile at the same time.

Digital Marketing, CMS, Ecommerce and Omnichannel

If you are into Digital Marketing and Ecommerce, you cannot discount the fact that these worlds are increasingly growing omni channel. Engaging consumers across various channels require consistent experience. I am yet to gain some more insights into usage of JavaScript for native apps and work it out, but thinking your application architecture with universal JavaScript rendering itself lends the architecture to support mobile apps as well. Which means building omni-channel experiences would be much easier, with cleaner separation of concerns. More importantly many of the frameworks advocate - "learn once and write anywhere".

My Experiences with Universal JavaScript

Working at SapientNitro, last year I had initiated a humble beginning to use Universal JavaScript rendering with Handlebars being introduced for Adobe AEM on the server side, which later paved its way into the IEA - Integrated Experience Architecture. With almost one year down the line one extremely positive outlook common with people who have worked on this concept on different projects  was the "separation of concerns" and the inherent advantages of it.

However such a model of development could be a cause of concern at times when there is no mediator who would understand the "backend" and the "frontend" frameworks patterns well, and allow the two teams to gel together. Even without such a person, projects would still move on, but what is produced at the end may not be elegant or easy to understand.

When I started out with this effort for a project for a leading financial consulting client, I learnt Handlebars, Grunt, and NodeJS to an extent that I wrote some Grunt tasks and Handlebar helpers on JavaScript. While this may look trivial to a seasoned JavaScript developer, for me it really helped a lot to understand both sides of the world and help the Interactive team and the AEM team with teething problems and integration glitches. This also helped us to create a better application "together". Looking back at the application, it is so seamless that ramping up a new member on the technology stack was really fast.

JavaScript - much beyond just rendering

JavaScript from a pure programming perspective itself is interesting - even when we leave out the goodies gained with universal rendering. JavaScript does not follow a purist programming paradigm, but it lends itself to almost any programming paradigm - Object Oriented (with prototypal inheritance), Functional or or just even no paradigm! Lot of libraries exist to help you get your flavour of JavaScript. The evolution of JavaScript specification itself is going through interesting phase - especially with ES6 advances on making the language better. The TypeScript from Microsoft makes JavaScript "strongly typed" and object oriented, while libraries like Underscore, Lo-Dash, Ramda make it a functional language. This further enables you to take the best of the different programming paradigms and write better programs. Add to that, the programs you write can run on Server or on the client. Isn't that interesting?

JavaScript based server side technologies - especially with NodeJS have advanced so well that you can use them to create full blown high-volume (I/O) web applications. Rendering page, connecting to database, crunching number or throwing out graphical charts, you name it - NodeJS applications do almost everything or even more than what you would expect from a traditional language or framework.

I would recommend that more and more people on the "backend" come forward and learn the front-end frameworks, more importantly the JavaScript. This would help us be better programmers and architects and create better applications. For when you plan to give training to the developers, do also consider that the training can also include the interested backed developer. A "frontend" aware backend architect and developer can make such a difference that you will end up more time productively creating application than arguing that "it works for me", or "please do whatever to make this work on your end". And remember the Atwoods Law - "Any application that can be written in JavaScript, will eventually be written in JavaScript." as a corollary to "The Rule of Least Power".

The improvements on the framework is ongoing. You can't wait for the perfect framework to come to learn it. Improvements in JavaScript frameworks seems to be there each passing day. If you are not already into the groove of learning JavaScript, do so. Understand the ecosystem of JavaScript - especially with things like NodeJS, Grunt, Gulp and JavaScrpt templating. Try joining your local JavaScript meetup group, or give yourself some hands on training.Try out the functional and object oriented features of JavaScript. It will make you a better and efficient programmers and architects.

Sling - Who is closing my JCR Session?

Has this happened to you? - You got the JCR session, but at times, you find that "someone" closed the JCR session! And that too - its very inconsistent. Sometimes it works and other times it does not! Well then its that you have hit the Sling / CQ development anti-pattern. And possibly this is what you see in your log:

javax.jcr.RepositoryException: This session has been closed.

As suggested by Jorg Hoh on his blog on CQ development patterns, you should rely on Sling objects as much as possible, rather than relying directly on the JCR objects. But at those cases (e.g. writing to a repository) when you will have to directly rely on the JCR repository you will have to still use the JCR Session, its recommended that you directly get the JCR session from the Sling repository, and not adapt it from the ResourceResolver, IF you do not intend to use the ResourceResolver directly.

@Reference
private ResourceResolverFactory resolverFactory;

// Wrong way of obtaining JCR Session
private Session getSessionAntipattern() throws LoginException {
 ResourceResolver resolver = resolverFactory.getServiceResourceResolver(null);
 return resolver.adaptTo(Session.class);
}

Code Listing 1: JCR Session Creation Antipattern

@Reference
private SlingRepository slingRepository;

// Correct way of obtaining JCR session
private Session getJCRSession() throws RepositoryException {
 return slingRepository.loginService(null, null);
}

Code Listing 2: JCR Session Creation Pattern

You may wonder, what's wrong. Logically thinking, why would you retrieve the ResourceResolver (and  adapt it to JCR Session), if you do not intend to use the ResourceResolver?  This logic has made the Sling developer's to close the underlying JCR Session of the ResourceResolver when the ResourceResolver object is Garbage Collected. Although, how this is done is being changed across the releases, the basic idea of releasing the resources held up (JCR Session) is still valid.

If you observe in the code snippet 1 (antipattern), the scope of the ResourceResolver object ends as soon as the method returns, which makes it eligible for garbage collection. This means that whenever the garbage collector runs, the JCR Session got through the antipattern would be closed - oops! This explains your JCR Session getting closed inconsistently.

I would like to emphasize that adapting to JCR Session from ResourceResolver itself is not wrong, if you intend to use the ResourceResolver along, other than just adapting it to JCR Session.

Hope this post helps someone save some time debugging on the JCR Session getting closed inconsistently.

Insane sanity (testing)!

You are busy writing your code on a complex piece and suddenly a team wide announcement comes - "The new code has gone in for the build, please do 'sanity' for your components".

"My unit tests would save me!" is not something which will save you here if this happens to be something which has a lot of visual and "content driven" elements. So, you break out of writing code half-way and start doing the "sanity" because it is crucial to test your components before lunch, as this code will be going to production soon after. You do this half-heartedly because this is so common for you whenever a major release happens or a content restructuring happens - and anyway "I am not a tester!". So you lose at-least one hour of your otherwise productive time to doing this insanely mundane task. On top of this, you also have your "professional testers" doing the same testing -just few extra eyeballs! This is the typical scenario if you are developing content driven (CMS) applications. Bless you if you are not so!

Insane testing
Image Credit - Wikimedia Commons

At this age in Digital Marketing, where content is the king, faster development and delivery overpowers the more structured delivery methods often followed in typical enterprise applications, where you invest for a lot of integrated and automated testing. Reason - "If we invest time to build integration testing and automation, we lose time to market". This is a broken argument often upheld by people who most often do not even care about integration testing in its real sense, and automated testing which can test the UI elements and the content of the page.

If we can invest some time in bringing about continuous integration and UI automation, a lot - really lot of time could be saved on developers doing "sanity testing". This may be difficult to achieve for shorter duration projects of the span less than 6 months - all out of their own budget, but may be easily feasible for bigger projects. Better - if this could be planned by "practice leads" who could invest on this once, and reap the benefits multiple times across projects. I do not have metrics to prove this by maths, but just a little retrospection into this would be useful.

Personally, this topic is lower on my radar. First, there are higher priority tasks for learning and experimentation on my TODO list. Second, I work more in a consulting mode than delivery, so budgeting it on  my project is not possible. Additionally, I do not have metrics to convince my client to budget this either! However, I will try to squeeze in some time in the coming days and weeks to do some research into this topics and publish my findings.

Three great advice

Act on it. Don't think too hard. Be Agile. These are the three great advice that have been my favorite picks from the book - "The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009)".

"Act on it" means that a piece of good advice needs to have an action plan on it - however small it may be. Most of the chapters of the book had the "Act on it" section which forced us to review ourselves and try to find an action plan. Without an act, the advice just rusts in our minds.

Thinking hard - "to jump or not?"
Image Credits - stefaanroelofs

"Don't think too hard" means that it is not necessary to perfect the things in the first go. We should think on it, but not "too hard" to make it the best at the first attempt. This has been especially important to me because, often trying to do the "best" often resulted to the act not being done at the first place. The best example would be writing something for the blog. I would often have such a lot of topics but often either I choose a topic which requires itself to be expressed as an essay, or just make the writeup to be the best and cover all the aspects of the topic, leaving no open loops. Both of this essentially means a lot of time, which eventually just rusts in the backlog and never gets a chance to get published.

"Be Agile" when applied to career and life means that career and life are never done. Re-iterate over, and try to make them better every time, and correct the course of action if it needs so.

Putting this together would also mean that I should be blogging more often, even with not so well rounded topics still appearing in there. And of-course, this would also apply to other aspects of life as well.

Nice is just ordinary! You need to be Extra-ordinary - Amazing - Remarkable!

Today I happened to see a TV advertisement which had a beautiful story - "Nice is just ordinary!". This happened to  rephrase what Chad Fowler had to say in his amazing book - The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009), which I had finished reading one round just today.

Winning is not about not-losing. Winning is about being awesome. Winning is about being Amazing - Remarkable!

My career, as I look back, seemed to have stagnated. I never had a roller-coaster ride in my career. I had pigeon-holed myself into the mediocrity - especially for the last 4-5 years. No one else to be blamed - just me. This was why I forced myself to read this excellent book to be inspired, and bring out the best in me. After-all I got into this profession for pure passion.

This book was in my reading list ever since 2010, but it was two weeks back that I finally ordered a paperback edition from Amazon, which I would religiously read during my travel to and from office.

The world around us always demands extra-ordinary and amazing things. Everything else is just OK, and CAN just live without. But I would like my life to be something which would create a dent in the world! Something awesome; something remarkable!

Yes, I do need a lot of overhauling. I have to invest in myself - to be a remarkable product - to create remarkable products. And this should mark the beginning of the journey to a remarkable career.

Over the next course of time, I would post critique to the book, apart from my brain dumps of some interesting ideas brewing.

The secret sauce of wrapping a jar as OSGi bundle

OSGi has been on the mainstream technologies for some time now. Many of the Java application servers now are either natively OSGi based, or support OSGi. This has also led many to package their libraries as valid OSGi bundles. But still there are a lot of libraries, that have absolutely no knowledge of OSGi - and when we have to use one of such libraries - we have to depend on one of those many open source projects which gives an OSGi bundle for the library, or prepare our own recipe to wrap them into OSGi bundles.

Before you begin

Before you pack your toolbox to wrap vanilla jars into OSGi bundles, do search public OSGi repositories and projects which can save a lot of your effort. Here are some of the popular ones:

1. SpringSource Enterprise Bundle Repository 
2. ServiceMix Bundle Project  Github Repository - https://github.com/apache/servicemix-bundles
3. Pax Tipi Github Repository - https://github.com/ops4j/org.ops4j.pax.tipi

If you still can't find what you are looking for, you may have to consider building your own! You can even consider contributing your bundles to the ServiceMix Bundle project or the Pax Tipi project.

Wrapping of vanilla jars can sometimes become complicated, especially when it is related to dynamic class generation or that of XML parsing. And that is why I am writing this - to share my experience of wrapping OSGi bundles when the library is complicated.

Choosing your tools

No surprise, I am also going to suggest the BND tool here. However, the BND tool itself is available in various flavours depending on your development workflow and tool as described in the Download page of the BND tool:

1. Bndtools - Bundling tool for the Eclipse Plugin
2. Apache Maven Bundle Plugin
3. Bnd4sbt - Bundling tool for Scala application
4. Gradle OSGi Plugin - A Gradle based OSGi plugin. Gradle is a Groovy based builder. 
5. Ant - Available as an ant task
6. Osmorc - Plugin for IntelliJ IDEA
7. Command Line Utility 

All of them however use the bnd library at its heart. However here, the BND tool command line utility can be handy to quickly wrap vanilla jars into OSGi bundles and do some basic analysis of the vanilla jars.

The choice of tool is mostly a personal preference, and also driven on your development environment and workflow. I personally prefer the Maven Bundle Plugin. However, irrespective of the tool used, there are a few fundamental principles which will help us wrap the bundles successfully. This need not be the best way, but has always helped me to build out OSGi bundles.

For the Maven users

To begin with, you can clone one of the projects - ServiceMix Bundle Project or Pax Tipi project, and add your own module to start out wrapping your bundle. If you need to manipulate the vanilla jar other than just adding the Manifest, you could consider using the Maven Shade Plugin - as it is being used in the ServiceMix bundle project.

Setting up the lab

First, I would recommend setting up an environment to test your OSGified library. You should have the following in your testing environment:

1. An OSGi container - Felix or Equinox or whatever you are using. Ensure that the version matches your target environment.
2. The target OSGified jar - yeah we are not done yet, but this is the test target. We will perfect it over some iterations.
3. The client bundle - the bundle where your code uses the API which exists in the target library (#2 above). You can use an OSGi activator (or the SCR activate) to run your code which will test if you have your library OSGified correctly.
4. The supporting bundles in your target environment. This may range from logging bundles to that of XML parsers and so on. Just be as close as possible.

All this environment should be something that is easy to provision and setup. I personally use a vanilla Felix instance with necessary environment bundles provisioned using Pax Runner. However you are free to use the tool of your choice to do this provisioning. The objective is to be able to start, test and stop the OSGi container, make changes to your target OSGified bundle and client bundle and repeat. And you do need an environment which should be easy to provision. You can also consider using Pax Exam to setup this environment.

Bundle it up!

If the above setup gave you a success right away at your first try, then you need not read the rest of this writeup. But most often, we do not get it right at the first attempt. So then you will have to do some experimentation with the packaging of your bundle which will ultimately give you the success.

Realize what you need - single or family pack?

It is important to understand and realize that what you ultimately need, and how the final target OSGified bundle will look depends on your use case. If operating in a pure and strict OSGi world, every library should be in its own bundle. In other cases, where there is no strict requirements around your dependencies being strictly separate bundles, you can combine a few libraries together to get a single bundle (or may be still a few more). However, both these ways of bundling has their own pros and cons.

Single bundles are the citizens of perfect OSGi world. This bundle would only represent one vanilla jar. The advantage of this approach is that you fully leverage the OSGi modularity. If there are multiple bundles which wants to use this library, one instance of this library would do (as long as the version requirements also match).  Your artifacts would be light. However, you may have to deal with and manage a lot of these bundles if the transitive dependencies of your target library is just too big.

Family pack - as I love to call it - is usually made by combining a related set of libraries together to form a single bundle. In Maven world, this often can be related to as the transitive dependencies. This kind of packaging may sometimes be necessitated by the library itself, which often has code which has hard, non-OSGi compliant bad OSGi practices code for loading classes and resources, which requires some other dependent libraries to be on the same classpath (OK, you need to understand that in OSGi, every bundle has its own classpath). At other times this kind of packaging avoids you to manage a lot of bundles on your target environment. However, this would mean that tomorrow if another library needs to be added, which has one of the transitive dependencies that already exist somewhere in your existing family pack bundles, either you would duplicate the library over in your second family pack, or just expand your existing family pack. Family packs at times can also lead to cyclic dependencies being created between bundles, which really is just evil.

Things to Remember

Once you have decided the kind of packaging you require, you can move ahead and start creating and perfecting your bundle. The following are some of the things to be taken care of when you start your OSGification process:

1. Know what you are doing

Unless the vanilla jar you are trying to wrap is really simple, you would have to understand OSGi a little bit deeper - especially what each of the manifest headers mean, and how it affects the classloading and resource resolution. What each of the BND instructions mean and how it affects the bundle being generated. Read the BND documentation, and also the tool specific documentation - in my case the Maven Bundle Plugin's documentation.

2. Verify the vanilla jars versus your OSGified bundle

Verify that the contents of the vanilla jar and your OSGified bundle to ensure that you preserve the packages and the resources contained therein, unless you intend it otherwise. The resources iteself may contain some configuratoin files that are required by the library to function, or it may have some licenses. You should be particularly be aware that almost all the licenses and notices require that you keep them in the derived products. If you are merging more than one library, you may have to consider renaming the resources - Maven Shade Plugin can be useful here.

3. Perfect on the contract - Import and Export packages

The Export-Package and the Import-Package are the primary contract of your OSGi bundle. What goes into these should be absolutely clear. Try to understand the library clearly, and export those packages which are API, and hide the internal implementations (some libraries hardly distinguish this fact, so that forces you to have not-so-perfect OSGi practices). Also ensure that your exported packages has a logically valid version. For instance, if you are OSGifiying itext-2.1.7, you could start with an Export-Package of com.lowagie*;version="2.1.7".

What to keep for Import-Package is trickier sometimes. This is where you would have to experiment a bit. Start with all the import packages being optional (*;resolution:=optional), and then once your client bundle is all working perfect, come back here and understand what is absolutely necessary for the library to function, and which are optional. As an example, some libraries may give out dependencies to JUnit classes. Now, why would someone depend on JUnit classes, so quietly remove them from the import packages. On the other hand, dependencies to JAXB libraries could possibly be made as optional one. All this depends on the library under consideration.

4. Ensure no BND warnings

When you are perfecting the contract of Import and Export packages, ensure that you have no BND warnings. Especially around split packages. When you get the split packages, carefully study your vanilla jars (often split package warnings will come when you are making a family pack), and understand of how would you like to prioritize the packages across the libraries to apply the split-package instructions for BND.

5. Dealing with cyclic dependencies

As mentioned before, multiple family packs can lead to cyclic dependencies. When this happens, carefully study the bundles involved in cyclic dependencies and the packages on which these links are formed, and carefully remove the cyclic dependency by either separating out one of the transitive jar as a separate OSGi bundle, or simply keep all the packages of such a library as private. Take the decision based on your situation.

6. Adjusting your client bundle

At times, your bundle may be importing certain packages from the wrong bundle (instead of what you are intending). Example of such packages may be what is exported by the system bundle which is outdated, and you instead want your client bundle to import from a later version exported by a separate OSGi bundle. In these cases, use the version as your savior and explicitly restrict the package versions so that it would import from the bundle of your choice. Often the client bundle may have a relaxed import versioning, which makes it to import from the system bundle. When the version is stricter, you would often achieve what you intend to.

7. Classloader Hell

At times you may get into situations that even after following all the instructions above, your library is unable to load some classes or resources. Carefully try to debug through the library's source code and see where actually the code fails. Also compare this execution to one in a non-OSGi environment, and you would be able to identify that in OSGi, the library is unable to instantiate a class or load a library, because of the way that the code is written to do that. In these cases, the last option may be to patch the library. Download the source code of the original library and make modifications to the offending classes and put it back to the OSGi bundle. Follow the OSGi best practices when you do dynamic class loading or resource loading.

8. Test thoroughly

Test you client bundle thoroughly with all the possible use cases of using the library. If possible use the Pax Exam to write a test suite to be sure that you are covering all the use cases. Everytime you run the above environment, you will realize - most often with OSGi resolution error messages or error stack traces that something went wrong. This information is key to identifying what has to be tweaked to get it working. Repeat the all the steps above through this step, until you are done.

9. Be license compliant

Nothing technical, but respect the licenses of the libraries you are trying to OSGify.

For AEM / CQ developers

For sometime I have been working on the Adobe's marketing platform AEM /CQ which utilizes OSGi as the application container and so requires developers to package their code as OSGi bundles and many of them get panic when they have to OSGify an external dependency - especially when it is a complicated one. I have personally found that most of these AEM developers do not understand the OSGi beyond what is there in the product documentation. And this was my primary motivation for writing this article.

I have made some shortcuts for kick-starting your OSGification process for AEM environment, by creating Pax Runner profiles for few AEM versions at my Github repository https://github.com/ethomasjoseph/aem-pax-runner-profile. Feel free to tweak and use this and contribute to this if you find this useful.

Before you say - I am done (or not)!

Bundling a vanilla jar - especially the complicated ones require a lot of patience and experience. A lot of the above instructions are dependent on the context of which library are you trying to work with and what exactly is the need of the hour. Weigh the choices to choose your track. Feel free to reach out to me (blog comments are OK), in case you want some help in OSGification of your vanilla jar.

Happy Bundling!!!

References

1. BND's official site
2. Maven Bundle Plugin
3. Apache Commons OSGi tracker
4. Instruction of creating bundle using Maven and BND
• http://felix.apache.org/documentation/subprojects/apache-felix-commons/creating-bundles-using-bnd.html
• http://spring.io/blog/2008/02/18/creating-osgi-bundles/
• http://gnodet.blogspot.in/2008/09/id-like-to-talk-bit-about-third-party.html
• https://karaf.apache.org/manual/latest/developers-guide/creating-bundles.html

Tuning your JCR Queries for the AEM & Jackrabbit OAK

Application tuning is an activity which which every developer and architect will encounter sooner or later for any application. AEM based applications are no different. One of the activities that is particularly important with AEM 6 is query tuning. AEM 6 uses Jackrabbit OAK as the repository and hence we would have to know more about OAK to tune the queries. Apart from tuning the queries itself a majority of the time you would be tuning your indexes based on the query. AEM itself uses a lot of queries within for the application to function. It is important to understand how the queries and the indexes work in AEM.

Update: This post was not written as a generic performance tuning guideline for AEM, but just on tuning the queries on OAK repository. Although a lot of the performance depends on how well tuned is your repository. Adobe's recent service packs on AEM 6, and AEM 6.1 in general are much stable and performant. Please refer to the Adobe's documentation for general guidelines on performance tuning.

While the instructions herein are primarily written for AEM applications, non-AEM applications which uses Jackrabbit OAK can also benefit.

There are no hard and fast rules that a particular index will work. This is very similar to database indexes. You would have to look at the current queries being fired and the current repository state and arrive at the best indexes. Often you arrive at the best indexes over time. You would have to apply an index and the re-evaluate your queries, then further tune your queries or try out a different index. Repeat these steps over and over again to find the best and optimized index.

However, there are some basic guidelines which should be followed, which can help you define the best indexes to you application specific needs:

Ground Rules (AEM Specific)

1. Apply the latest Service Pack available and Oak specific hotfixes (if applicable). As of date SP-2 is publicly available.
2. AEM has some built in indexes as well. Do not arbitrarily re-index these OOTB indexes. Indexing is a costly and resource-intensive operation. OOTB indexes should be re-indexed only when specifically advised by Adobe (this is a catch - sometimes you know that re-indexing a particular OOTB index will solve your problem at hand, but get a buy-in from Adobe for this). At other times you would have to create a new index (type) since the OOTB / existing index is in-efficient.
3. Be aware that there has been some important changes since OAK 1.0.9 - especially related to Lucene indexes, hence be mindful about reference resources that advice about using OAK indexes. Check the OAK's latest documenting on query and indexing.

Tools

1. If you are using AEM, the best tool to help you debug and explain your queries, find out slow and popular queries is to install Adobe AEM ACS Tools package (0.0.20 and above). Once installed, you can have access it using the URL: http://yourinstance/etc/acs-tools/explain-query.html 
2. If the above is not possible, you may have to depend on JMX to find slow and popular queries, you can use the Felix console to identify slow and popular queries on any instance 
• http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D9%2Cname%3D"Oak+Query+Statistics"%2Ctype%3D"QueryStat". 
3. You can also use the OSGi Service of org.apache.jackrabbit.api.jmx.QueryStatManagerMBean to programmatically find the slow and popular queries. 

queryStatManagerMBean.getSlowQueries().values()
queryStatManagerMBean.getPopularQueries().values()

4. Use the following debug logs for details on various aspects of query and indexing:
1. Enable DEBUG logging on  org.apache.jackrabbit.oak.query for logs to find the details on query cost calculation (explain)
2. Enable DEBUG logging on  org.apache.jackrabbit.oak.plugins.index for logs to find the details on indexing
5. Use JMX to have high level overview about lucene indexes (OOTB or custom):
• http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D4%2Cname%3D"Lucene+Index+statistics"%2Ctype%3D"LuceneIndex"
6. To look at the actual indexes, you can stop the AEM / Oak instance and use the oak-run "console" to explore the indexes among various other options:

• java -jar oak-1.1.6.jar explorer /path/to/crx-quickstart/repository/segmentstore

  (oak repository)
• On the resulting console, run the following command, to dump the index into file system:

• lc dump path/to/dump /path/to/luceneIndex/inRepository

• eg.

  lc dump path/to/dump /oak:index/myCustomLuceneIndex

• Now analyse the resulting dump using Lucene Index analyser tool Luke:

java -XX:MaxPermSize=512m -classpath luke-with-deps.jar;oak-lucene-1.1.6.jar org.getopt.luke.Luke

• The following jars needs to be used:
• The luke-with-deps.jar can be downloaded from https://github.com/DmitryKey/luke/releases
• The oak-lucene-1.1.6.jar can be downloaded from Maven repository as well.

Guidelines

The basic steps for tuning your queries can be summarized as follows:

1. Identify the slow queries in your system. This could be the one that takes a lot of time to execute primarily because there are not proper indexes and there are node traversals happening. The system itself can be queries through JMX to identify the slow and popular queries.
2.  Tune your queries to ensure that you can use indexes whenever possible.
3. Identify properties and nodes that needs to be indexed, based on the slow queries identified, and create Oak indexes for them.
4. Verify your queries and indexes after creating indexes. The indexes are optimum if the cost of the queries identified are lowest - 0 being the best.
5. Repeat steps 3 and 4 above (by even creating different indexes for the same query) till the most cost effective index is created.

Identification of Slow Queries

Identify the slow queries by monitoring the error.log. Note the queries which traverses many nodes, where the log may look something like:

org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed 1000 nodes
... consider creating an index or changing the query 

You can also identify the slowest and popular queries in the system (which the Jackrabbit Oak keeps track of) using JMX or Adobe AEM Commons as described in the "Tools" section above.

Indexing

For the same query you can create multiple indexes. However choose the one to be promoted to your production instances which prove to be the the best in terms of achieving lower cost to the query at hand.

Consider the following when deciding the type of index to be used:

1. Full Text Index Types
• Lucene & Solr
• Aggregate Index - query time aggregation
2. Lucene Property Index
• Since OAK-1.0.9 Lucene Property indexes are preferred indexes over other type of indexes
• Async
• Combines properties & full-text.
• Supports property condition, range conditions, ordering, full-text
• Index-time aggregation - performance improvements
• Using Lucene & Solr.
• Follow instructions for creating the index nodes as in the OAK's Lucene documentation.
3. Property Index
• Default Indexes
• Synchronous
• Unique or Non-Unique
• eg. UUID
• property equals query works best
4. NodeType Index
• Internally Uses Property Index
• Uses primaryType and mixins to identify nodes
5. Traversing Index
• Traverses Repo
• Does not store any data
• Used to retrieve the given path or its child nodes (sometimes the parent nodes as well)
6. Ordered Index
• Async
• Not recommended of nodes > 1000 nodes

Index Management

1. Currently, the indexes can be found under /oak:index/ on AEM.
2. Once you have identified the best indexes, plan to install them as a package on AEM.
3. Re-indexing is usually not needed. However when you see degrade in system performance, do double check your existing indexes if it would need a re-index.
4. Re-index is a costly and resource intensive operation. Use it only when any oak upgrade or  mentioned explicitly in release doc for any patch or hotfix
5. Avoid having multiple properties in index , it may slow your query , its recommended to have one index for each property.
6. Set the system property  oak.queryLimitInMemory to 200000 (or something similar). This will limit the traversal of the nodes to that number

Finally...

Jackrabbit Oak is a project under active development, especially around the area of Indexing. Hence please check official documentation for details. Indexing and tuning requires a lot of observation and practice to get perfect.

I would recommend watching a webinar on this topic by Thomas Muller and Marcel Reutegger on AEM Gems, which is a bit old (before OAK-1.0.9 was released), but immensely useful to understand indexing on Jackrabbit Oak.

Happy Indexing!!!

Writing method with dynamic return type without type casting

I have often come across this requirement across multiple projects, where we need to write a method, which can potentially return object of any class, and still you would not like to do a typecasting. Something like this:

MyClass myClass = getDynamicClass(MyClass.class);

Instead of :

MyClass myClass = (MyClass) getDynamicClass(MyClass.class);

Common example of such use case is where you would do some lookup of a dynamic service (as in OSGi) and return the object found. However, with Java 5 and above this is easy. So, if your earlier code was:

protected Object getDynamicClass(Class interfaceClass) {
    Object obj = null;
    // code to create / lookup the dynamic object
    return obj;
}

This code can be converted as follows:

protected <ReturnType> ReturnType getDynamicClass(Class<ReturnType> interfaceClass) {
    Object obj = null;
    // code to create / lookup the dynamic object
    if (interfaceClass.isInstance(obj)) {
        return interfaceClass.cast(obj);
    }
    return null;
}

This is very obvious code, but thought that sharing this would be useful. Enjoy!

Installing Java Plugin for Ubuntu / Linux for multiple users

If you are a Linux user like me and have encountered websites that require applet, you may be ok installing the default Java that comes with your distro, and most of the time you may be good to go with it.

But at times you may specifically want to install Sun Java on your system. Since Oracle has licensing restrictions, distros like Ubuntu do not have Sun Java in their repositories, and you will have to resort to some "longer" ways of installing Java as explained in here for Ubuntu.

But even by following the "longer" ways, you may be still annoyed when your browser says that Java plugin is missing (it happened for me for my Ubuntu 12.04). Or else, even when it may be working for you, it may not be working for other users on your computer, and you may have to create soft links to Java plugin executable in .mozilla/plugins directory of every user's home.

Assuming that you have already installed Java flavor of your choice, and want to enable Java plugin for your browser for every user of your computer, just create a soft link to the Java plugin executable under /usr/lib/mozilla/plugins/ directory.

cd /usr/lib/mozilla/plugins

ln -s /usr/lib/jvm/jdk1.7.0_17/jre/lib/amd64/libnpjp2.so

...and you are good to go. Now you can refresh your browser containing the applet to see that it is working, or you can test it at the Java's official test page here. For me here is how my browser looked after I configured Java as above:

Know your Geolocation right away with HTML5

Wondered how Google Maps tracks you current location? HTML5! Thats what enables most of the modern-day applications to track down your current location.

You can do it too right away!

Open up your JavaScript/Firebug console. You can hit F12 to work on most of the browsers. And paste the following JavaScript code snippet.

navigator.geolocation.getCurrentPosition(function(position){console.log(position);});

or

navigator.geolocation.getCurrentPosition(function(position){alert(position.coords.longitude + " / " + position.coords.latitude);});

Did you see a confirmation asked on your browser? Allow it! Thats it! It was so simple. You can now see your location in terms of longitude and latitude on the console. You can use this data along with Google Maps API to spot your location on a map.

Developing Java / OSGi applications on Day CQ

Day CQ is a very popular enterprise CMS (Content Management System) suite from Adobe. Although most of the time the development on Day CQ is related to CMS and most of the development can be mapped to a proper CMS system, there are times when you would need to develop some logic in your application, or need to develop some feature that may not be available readily within Day CQ. In these cases, you may need to write some application code.

Day CQ is a Java based software, and Java is the most popular way to develop custom features and application logic. To be specific, Day CQ is largely built using OSGi - a technology that enables you to develop truly modular applications. Developing any custom Java application modules also needs to be built on OSGi.

I have noticed that developing OSGi modules is often a pain for a CMS developer, largely because of the lack of full development support  (or call it a buggy development support) from the CRXDE (The bundled Eclipse based IDE), which makes OSGi module development to take long time and painful. This post intends to give more insights to a CQ developer on OSGi, and also to do faster development of OSGi modules with OSGi.

Solving eclipse error with SVN 1.6 (loading JavaHL library)

If you are using Eclipse and SVN (Subversion), there are high chances that you are using the Eclipse plugin - Subclipse. And if you recently upgraded to subclipse 1.6, there is a good chance that you have encountered the error or loading JavaHL Library, that says something like this:

Problem:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

If this is the case, then few tips to get this working. As always, I will suggest solutions with the least number of steps, and all simple.

1. When installing/updating subclipse 1.6, use the update site - http://subclipse.tigris.org/update_1.6.x/ from within Eclipse.
2. The installation screen shows the features. Make sure that all features marked "required" are selected for installation. (You will anyway be not allowed to proceed otherwise).

If it still does not work, it is highly possible that the Java native libraries required for the SVN to work together with Java (thats Java HL) is not available to the Java. Either not installed or just that Java is not able to find it.

1. Install Java HL

You can refer to here for a generic discussion on this topic, especially for Windows and Mac OS. For Linux, I have simpler solutions :).
Install through your package manager, a package named libsvn-java, that contains the JavaHL. You must make sure that it is for SVN 1.6.
For Ubuntu or other Debian based Linux, the command should be handy:

sudo  apt-get install libsvn-java

However, it is possible that the installation is requiring you to install the whole Java binaries itself, although you already have one (an old version, or a manually configured Java for example), and you are not willing to install the Java packages. In that case, just go to your package maintainer's site, and manually download the package, and unzip it to your system (yes, you will need root permissions), so that the dependencies are not installed.

If you had installations fine, without any hacks, restarting Eclipse should get your Subclipse working. If not, follow what is below.

2. Make Eclipse runtime aware of the new JavaHL binaries.

. There are two options for doing this:

1. Tell JVM/Eclipse about the JavaHL libraries when starting JVM with the system property, that points to the java library path where the JavaHL binaries are installed. eg:

      -Djava.library.path=/usr/lib/jni
   

   This option can be specified in config.ini as well.
2. Put the JavaHL binaries in paths that JVM normally searches for native libraries. Normally these paths are listed, in the error above. Above the paths are listed as:

   java.library.path = /opt/jdk/64/jdk1.6.0_20/jre/lib/amd64/server:/opt/jdk/64/jdk1.6.0_20/jre/lib/amd64:/opt/jdk/64/jdk1.6.0_20/jre/../lib/amd64:/usr/lib64/xulrunner-addons:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
   

I went for the second option, since that one seemed most non-intrusive and easy to me. Let me show it how I did that:

1. Choose any of the listed Java library paths. In a terminal, move into that directory:

cd /usr/lib/jvm/java-6-sun/jre/lib/i386

2. Make softlinks to the JavaHL libraries.

   sudo ln -s /usr/lib/jni/libsvnjavahl-1.so libsvnjavahl-1.so
   sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0 libsvnjavahl-1.so.0
   sudo ln -s /usr/lib/jni/libsvnjavahl-1.so.0.0.0 libsvnjavahl-1.so.0.0.0
   

Now restarting Eclipse should get everything working fine. If it does not, review your steps, and also read in full length about JavaHL from http://subclipse.tigris.org/wiki/JavaHL, and try the steps again.

Running tomcat/webserver on port 80/443 without root permissions on Linux

The Problem

One of the concerns for web application developers is to run their web servers on the default port (which is 80 for HTTP, and 443 for HTTPS). However, since these ports are privileged ports (ports less that 1024 are privileged ports in Linux/UNIX), the developers are either required to run the server either as a root, or run on alternative non privileged ports such as 8080 or 8443.

The Solution

While to the same problem, there may be different solutions, the most simplest of all that I found was to do local port forwarding. Again here, many do not like to fiddle with commands every time. An easier, and one time solution to this is to use a program called "guidedog".

Guidedog is a GUI based advanced and powerful networking configuration tool for Linux, that can be used for packet routing, NAT and port-forwarding. Although it is for KDE, your Linux package manager should be intelligent enough to make that work for you even if you do not use KDE. We will here use its port-forwarding feature.

Steps

1. Launch your favorite package manager tool (Synaptic/Aptitude/YUM etc) and install "guidedog". For Ubuntu I use Synaptics Package manager, or even a command would do.
   

   sudo apt-get install guidedog
   

2. Once installed, launch the application with root permissions (don't worry, you require it to do just once).
   

   sudo guidedog

   
   On Ubuntu, you can also find it under the menu Applications->Other. You will have to even edit the menu command to gksu guidedog.
3. Move to the "Forward" tab. Add a new rule, such that TCP traffic on port 80 is forwarded to port 8080 (or whatever port your server runs on). Similarly, add another rule, such that TCP traffic on port 443 is forwarded to port 8443 (or whatever port your server runs on for HTTPS/SSL connections). After you are done, your screen should look like this:
   
   
   
4. Apply the changes.

You are done. Now just hit on localhost instead of localhost:8080, next time you start your applications.

Tool power

The tool creates a system rule to forward the traffic to a different port(8080/8443) from its original destination (port 80/443) on the same machine. You can see in the tool that it is much more powerful, and can allow even port forwarding for other machines too, or even do IP Masquadering/NAT (Network Address Translation). However, explaining on these topics would outgrow the topic of this blog and would take more of your and my time too.

VNC via SSH tunneling

5 simple steps to connect through VNC to office/work PC via SSH tunneling

Situation:

You have to connect to your office PC from home via VNC, but your office PC does not have a public IP. The only way to access your PC is through one of the access point machines provided by your office that can be reached through Internet.

Concept:

VNC port is 5900. You can ssh tunnel to your office PC through your office access point, and connect to this tunnel to have VNC connection.

Assumption:

Office access point machine: accesspts1.company.com
Internal IP of your office machine: 192.168.1.31

Also assume that your Office PC is switched on, and access point machine can access your office PC via the internal IP, and can access its ports as well.

Steps:

1. SSH to access point machine. Provide password when prompted
   

   ssh username@accesspts1.company.com

2. Now that you are on access point machine, ssh from there to your office PC. Again provide password when prompted.
   

   ssh username@192.168.1.31

3. On your office pc, start VNC server. Keep this terminal open.
   

   x11vnc

4. On another terminal, make open a tunnel to your office pc via the access point. Provide the password for your access point machine when prompted.
   

   ssh -L 0.0.0.0:xxxx:192.168.1.31:5900 username@accesspts1.company.com

5. Use an VNC viewer to connect to your Office PC via the tunnel. Use the following details:
   

   Host: localhost
   Port: xxxx

Agility and more - better & faster business solutions with software - part-3

In my previous posts, I discussed about practicing agile in real life, and why adapting the agile prescriptions is important. I discussed how the basic values of Agile manifesto narrates it all. This is my last post in this series, where I will explain about practices even beyond the direct agile prescriptions, that can help build faster and quality software with smiling teams.

Some Agility and beyond

While the software creation itself is creative, the process employed to create this software is also creative. Its not just practicing agile, that helps create good software, there are more things to it. The craftsman (developer) who creates software has also to be given importance (Did I mention that before?). Let me discuss few of these things that include things from the Agile practices such as XP and Scrum, as well as those that go beyond them.

Agility and more: better & faster business solutions with software - part-2

In my previous blog post, I talked about why software development needs processes, and the need for agile in the processes. Here, I continue with the discussions, focusing on how the basic principles of agile itself has a lot to say, that can help us build better software.

Practicing agile - adapting is more important

As mentioned before, the popular practiced tools for agile process include Scrum, Kanban and XP. But often, the process models/tools are applied as a religion, being strict on what the text prescribes. This has resulted in some rumblings that has questioned the effectiveness of agile processes to deliver quality and creative solutions to business. The InfoQ book on Scrum and Kanban has highlighted the need for the process models to be adapted to their needs.

One size doesn't fit all - adapting is important.

(Image Courtesy: Robin Skyler Tell's photostream)

So what needs to be practiced is taking a lean model, and adapting it to your needs, probably by adding "prescriptions" from the stouter process models, and even removing "prescriptions" that hinder your progress, but with absolute knowledge of what is being removed. This is what Henrik has put in his InfoQ book - "Do not limit yourself to one tool!".

Agility and more: better & faster business solutions with software - part-1

Software development is a creative process, where we create software to deliver solutions to business problems (most of the time), and/or to add value to it. Although the holy duty of a software developer is to deliver solutions to business problems, and not always necessarily developing software. But when we have decided to build a software, we work as teams (most of the times), and the business problem is the central focus of the process. But, the overall quality of the developed software also depends on how we are developing the software - from the requirement capturing to coding, testing, and implementation. This highlights the need for a process at all stages of software development.

Note: Software is also developed for fun, some write it to learn programming (we all have did that), some times it is written for personal use (eg,. automating some mundane task). But most of the time, its written for business needs - let it be a game software, and ERP, the OS itself, or the search engine software - most of them eventually turn as a business or a business solution. In this post I am talking mostly about the software that deals with business and hence the software shops find time and money to be spent to develop these software. So whenever I talk of software, I mean it to be the software developed in these software shops, unless otherwise indicated.

These series of posts highlights my understanding and experience on how the Agile processes can be effectively used to improve the overall quality of the software solutions with reasonable time leads and better estimates. Why and how agile processes needs to be adapted? For Software Engineering teams, there is even more than just being lean/agile, that needs to be into the process.

(Image courtesy: Coffish's web album)

This is of course not a comprehensive discussion on any of the process tools/frameworks discussed here. Although I discuss about Scrum, XP and Kanban to certain extent, what I tell here is the basics, a skeleton that every team/organization can give thoughts to, and then adapt them to their needs. You may add whatever more is needed to meet some standards, but just don't overdo it that you have problem remembering your own processes - keep it as lean as possible. There are several literature that talks about these specific techniques. I will not talk about all the aspects of software development too! Here I will talk in general, about faster development cycle and developer productivity while achieving quality and timely software releases with less team burnouts and overtimes.