JavaScript is going Universal - Why should this matter you?

If you program for the web, and think that you are a "backend developer", and JavaScript isn't my cup of tea, that is not true any more. You can close your eyes and wish away that JavaScript on Server side is a hype, but the fact is that JavaScript is invading the Server side space each passing day, and you "must" know JavaScript and JavaScript based rendering technologies, if you intend to create next generation web and mobile applications, or may be - just to be employed!

In the recently shared roadmap of Angular 2 in Angular U, the Angular team presented the idea of splitting Angular into two pieces: core functionality and the renderer. The concept of renderer in Angular could be used for server side rendering, or native rendering on iOS or Android.

This roadmap clearly brings Angular into the family of Universal JavaScript (also known as Isomorphic JavaScript) rendering technologies. The family of Universal JavaScript rendering technologies already include Rendr, Flux, React, Dust and LazoJS - to name a few popular ones. Coursera, Airbnb, LinkedIn, Facebook, NetFlix, Walmart are some of the high volume web applications that have already moved to Universal JavaScript. This move has already helped these websites to write applications that run universally - on server, browser and mobile apps - significantly reducing the development and maintenance effort across various channels, and more importantly - you just maintain one code and be flexible about intelligently choosing a logic execution and rendering to be on server or client side while pushing up the web applications to be much faster!

JavaScript going Universal

Despite the creation of JavaScript as a lightweight look-alike brother of Java, JavaScript eventually became a key component in enabling the Web 2.0 evolution - especially through AJAX. JavaScript has moved much beyond its initial design goals, with more standardization with ECMA, optimizations done to every release of the JavaScript engines, JavaScript is all ready to rule the web (and more platforms).

Universal JavaScript
Image Credit - Netflix

Today JavaScript is a ubiquitous runtime.  JavaScript is everywhere - from Browser,  server side, embedded device, mobile and gaming.. The language's influence continues to grow on the server landscape via Node.js, and runtimes such as Rhino and Nashorn for Java; Jurassic and V8.NET of .NET and projects such a J2V8  and React aiming to go even higher by targeting at Server side and mobile at the same time.

Digital Marketing, CMS, Ecommerce and Omnichannel

If you are into Digital Marketing and Ecommerce, you cannot discount the fact that these worlds are increasingly growing omni channel. Engaging consumers across various channels require consistent experience. I am yet to gain some more insights into usage of JavaScript for native apps and work it out, but thinking your application architecture with universal JavaScript rendering itself lends the architecture to support mobile apps as well. Which means building omni-channel experiences would be much easier, with cleaner separation of concerns. More importantly many of the frameworks advocate - "learn once and write anywhere".

My Experiences with Universal JavaScript

Working at SapientNitro, last year I had initiated a humble beginning to use Universal JavaScript rendering with Handlebars being introduced for Adobe AEM on the server side, which later paved its way into the IEA - Integrated Experience Architecture. With almost one year down the line one extremely positive outlook common with people who have worked on this concept on different projects  was the "separation of concerns" and the inherent advantages of it.

However such a model of development could be a cause of concern at times when there is no mediator who would understand the "backend" and the "frontend" frameworks patterns well, and allow the two teams to gel together. Even without such a person, projects would still move on, but what is produced at the end may not be elegant or easy to understand.

When I started out with this effort for a project for a leading financial consulting client, I learnt Handlebars, Grunt, and NodeJS to an extent that I wrote some Grunt tasks and Handlebar helpers on JavaScript. While this may look trivial to a seasoned JavaScript developer, for me it really helped a lot to understand both sides of the world and help the Interactive team and the AEM team with teething problems and integration glitches. This also helped us to create a better application "together". Looking back at the application, it is so seamless that ramping up a new member on the technology stack was really fast.

JavaScript - much beyond just rendering

JavaScript from a pure programming perspective itself is interesting - even when we leave out the goodies gained with universal rendering. JavaScript does not follow a purist programming paradigm, but it lends itself to almost any programming paradigm - Object Oriented (with prototypal inheritance), Functional or or just even no paradigm! Lot of libraries exist to help you get your flavour of JavaScript. The evolution of JavaScript specification itself is going through interesting phase - especially with ES6 advances on making the language better. The TypeScript from Microsoft makes JavaScript "strongly typed" and object oriented, while libraries like Underscore, Lo-Dash, Ramda make it a functional language. This further enables you to take the best of the different programming paradigms and write better programs. Add to that, the programs you write can run on Server or on the client. Isn't that interesting?

JavaScript based server side technologies - especially with NodeJS have advanced so well that you can use them to create full blown high-volume (I/O) web applications. Rendering page, connecting to database, crunching number or throwing out graphical charts, you name it - NodeJS applications do almost everything or even more than what you would expect from a traditional language or framework.

I would recommend that more and more people on the "backend" come forward and learn the front-end frameworks, more importantly the JavaScript. This would help us be better programmers and architects and create better applications. For when you plan to give training to the developers, do also consider that the training can also include the interested backed developer. A "frontend" aware backend architect and developer can make such a difference that you will end up more time productively creating application than arguing that "it works for me", or "please do whatever to make this work on your end". And remember the Atwoods Law - "Any application that can be written in JavaScript, will eventually be written in JavaScript." as a corollary to "The Rule of Least Power".

The improvements on the framework is ongoing. You can't wait for the perfect framework to come to learn it. Improvements in JavaScript frameworks seems to be there each passing day. If you are not already into the groove of learning JavaScript, do so. Understand the ecosystem of JavaScript - especially with things like NodeJS, Grunt, Gulp and JavaScrpt templating. Try joining your local JavaScript meetup group, or give yourself some hands on training.Try out the functional and object oriented features of JavaScript. It will make you a better and efficient programmers and architects.

Sling - Who is closing my JCR Session?

Has this happened to you? - You got the JCR session, but at times, you find that "someone" closed the JCR session! And that too - its very inconsistent. Sometimes it works and other times it does not! Well then its that you have hit the Sling / CQ development anti-pattern. And possibly this is what you see in your log:

javax.jcr.RepositoryException: This session has been closed.

As suggested by Jorg Hoh on his blog on CQ development patterns, you should rely on Sling objects as much as possible, rather than relying directly on the JCR objects. But at those cases (e.g. writing to a repository) when you will have to directly rely on the JCR repository you will have to still use the JCR Session, its recommended that you directly get the JCR session from the Sling repository, and not adapt it from the ResourceResolver, IF you do not intend to use the ResourceResolver directly.

@Reference
private ResourceResolverFactory resolverFactory;

// Wrong way of obtaining JCR Session
private Session getSessionAntipattern() throws LoginException {
 ResourceResolver resolver = resolverFactory.getServiceResourceResolver(null);
 return resolver.adaptTo(Session.class);
}

Code Listing 1: JCR Session Creation Antipattern

@Reference
private SlingRepository slingRepository;

// Correct way of obtaining JCR session
private Session getJCRSession() throws RepositoryException {
 return slingRepository.loginService(null, null);
}

Code Listing 2: JCR Session Creation Pattern

You may wonder, what's wrong. Logically thinking, why would you retrieve the ResourceResolver (and  adapt it to JCR Session), if you do not intend to use the ResourceResolver?  This logic has made the Sling developer's to close the underlying JCR Session of the ResourceResolver when the ResourceResolver object is Garbage Collected. Although, how this is done is being changed across the releases, the basic idea of releasing the resources held up (JCR Session) is still valid.

If you observe in the code snippet 1 (antipattern), the scope of the ResourceResolver object ends as soon as the method returns, which makes it eligible for garbage collection. This means that whenever the garbage collector runs, the JCR Session got through the antipattern would be closed - oops! This explains your JCR Session getting closed inconsistently.

I would like to emphasize that adapting to JCR Session from ResourceResolver itself is not wrong, if you intend to use the ResourceResolver along, other than just adapting it to JCR Session.

Hope this post helps someone save some time debugging on the JCR Session getting closed inconsistently.

Insane sanity (testing)!

You are busy writing your code on a complex piece and suddenly a team wide announcement comes - "The new code has gone in for the build, please do 'sanity' for your components".

"My unit tests would save me!" is not something which will save you here if this happens to be something which has a lot of visual and "content driven" elements. So, you break out of writing code half-way and start doing the "sanity" because it is crucial to test your components before lunch, as this code will be going to production soon after. You do this half-heartedly because this is so common for you whenever a major release happens or a content restructuring happens - and anyway "I am not a tester!". So you lose at-least one hour of your otherwise productive time to doing this insanely mundane task. On top of this, you also have your "professional testers" doing the same testing -just few extra eyeballs! This is the typical scenario if you are developing content driven (CMS) applications. Bless you if you are not so!

Insane testing
Image Credit - Wikimedia Commons

At this age in Digital Marketing, where content is the king, faster development and delivery overpowers the more structured delivery methods often followed in typical enterprise applications, where you invest for a lot of integrated and automated testing. Reason - "If we invest time to build integration testing and automation, we lose time to market". This is a broken argument often upheld by people who most often do not even care about integration testing in its real sense, and automated testing which can test the UI elements and the content of the page.

If we can invest some time in bringing about continuous integration and UI automation, a lot - really lot of time could be saved on developers doing "sanity testing". This may be difficult to achieve for shorter duration projects of the span less than 6 months - all out of their own budget, but may be easily feasible for bigger projects. Better - if this could be planned by "practice leads" who could invest on this once, and reap the benefits multiple times across projects. I do not have metrics to prove this by maths, but just a little retrospection into this would be useful.

Personally, this topic is lower on my radar. First, there are higher priority tasks for learning and experimentation on my TODO list. Second, I work more in a consulting mode than delivery, so budgeting it on  my project is not possible. Additionally, I do not have metrics to convince my client to budget this either! However, I will try to squeeze in some time in the coming days and weeks to do some research into this topics and publish my findings.

Three great advice

Act on it. Don't think too hard. Be Agile. These are the three great advice that have been my favorite picks from the book - "The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009)".

"Act on it" means that a piece of good advice needs to have an action plan on it - however small it may be. Most of the chapters of the book had the "Act on it" section which forced us to review ourselves and try to find an action plan. Without an act, the advice just rusts in our minds.

Thinking hard - "to jump or not?"
Image Credits - stefaanroelofs

"Don't think too hard" means that it is not necessary to perfect the things in the first go. We should think on it, but not "too hard" to make it the best at the first attempt. This has been especially important to me because, often trying to do the "best" often resulted to the act not being done at the first place. The best example would be writing something for the blog. I would often have such a lot of topics but often either I choose a topic which requires itself to be expressed as an essay, or just make the writeup to be the best and cover all the aspects of the topic, leaving no open loops. Both of this essentially means a lot of time, which eventually just rusts in the backlog and never gets a chance to get published.

"Be Agile" when applied to career and life means that career and life are never done. Re-iterate over, and try to make them better every time, and correct the course of action if it needs so.

Putting this together would also mean that I should be blogging more often, even with not so well rounded topics still appearing in there. And of-course, this would also apply to other aspects of life as well.

Nice is just ordinary! You need to be Extra-ordinary - Amazing - Remarkable!

Today I happened to see a TV advertisement which had a beautiful story - "Nice is just ordinary!". This happened to  rephrase what Chad Fowler had to say in his amazing book - The Passionate Programmer - Creating A REMARKABLE CAREER in software development (2009), which I had finished reading one round just today.

Winning is not about not-losing. Winning is about being awesome. Winning is about being Amazing - Remarkable!

My career, as I look back, seemed to have stagnated. I never had a roller-coaster ride in my career. I had pigeon-holed myself into the mediocrity - especially for the last 4-5 years. No one else to be blamed - just me. This was why I forced myself to read this excellent book to be inspired, and bring out the best in me. After-all I got into this profession for pure passion.

This book was in my reading list ever since 2010, but it was two weeks back that I finally ordered a paperback edition from Amazon, which I would religiously read during my travel to and from office.

The world around us always demands extra-ordinary and amazing things. Everything else is just OK, and CAN just live without. But I would like my life to be something which would create a dent in the world! Something awesome; something remarkable!

Yes, I do need a lot of overhauling. I have to invest in myself - to be a remarkable product - to create remarkable products. And this should mark the beginning of the journey to a remarkable career.

Over the next course of time, I would post critique to the book, apart from my brain dumps of some interesting ideas brewing.

The secret sauce of wrapping a jar as OSGi bundle

OSGi has been on the mainstream technologies for some time now. Many of the Java application servers now are either natively OSGi based, or support OSGi. This has also led many to package their libraries as valid OSGi bundles. But still there are a lot of libraries, that have absolutely no knowledge of OSGi - and when we have to use one of such libraries - we have to depend on one of those many open source projects which gives an OSGi bundle for the library, or prepare our own recipe to wrap them into OSGi bundles.

Before you begin

Before you pack your toolbox to wrap vanilla jars into OSGi bundles, do search public OSGi repositories and projects which can save a lot of your effort. Here are some of the popular ones:

1. SpringSource Enterprise Bundle Repository 
2. ServiceMix Bundle Project  Github Repository - https://github.com/apache/servicemix-bundles
3. Pax Tipi Github Repository - https://github.com/ops4j/org.ops4j.pax.tipi

If you still can't find what you are looking for, you may have to consider building your own! You can even consider contributing your bundles to the ServiceMix Bundle project or the Pax Tipi project.

Wrapping of vanilla jars can sometimes become complicated, especially when it is related to dynamic class generation or that of XML parsing. And that is why I am writing this - to share my experience of wrapping OSGi bundles when the library is complicated.

Choosing your tools

No surprise, I am also going to suggest the BND tool here. However, the BND tool itself is available in various flavours depending on your development workflow and tool as described in the Download page of the BND tool:

1. Bndtools - Bundling tool for the Eclipse Plugin
2. Apache Maven Bundle Plugin
3. Bnd4sbt - Bundling tool for Scala application
4. Gradle OSGi Plugin - A Gradle based OSGi plugin. Gradle is a Groovy based builder. 
5. Ant - Available as an ant task
6. Osmorc - Plugin for IntelliJ IDEA
7. Command Line Utility 

All of them however use the bnd library at its heart. However here, the BND tool command line utility can be handy to quickly wrap vanilla jars into OSGi bundles and do some basic analysis of the vanilla jars.

The choice of tool is mostly a personal preference, and also driven on your development environment and workflow. I personally prefer the Maven Bundle Plugin. However, irrespective of the tool used, there are a few fundamental principles which will help us wrap the bundles successfully. This need not be the best way, but has always helped me to build out OSGi bundles.

For the Maven users

To begin with, you can clone one of the projects - ServiceMix Bundle Project or Pax Tipi project, and add your own module to start out wrapping your bundle. If you need to manipulate the vanilla jar other than just adding the Manifest, you could consider using the Maven Shade Plugin - as it is being used in the ServiceMix bundle project.

Setting up the lab

First, I would recommend setting up an environment to test your OSGified library. You should have the following in your testing environment:

1. An OSGi container - Felix or Equinox or whatever you are using. Ensure that the version matches your target environment.
2. The target OSGified jar - yeah we are not done yet, but this is the test target. We will perfect it over some iterations.
3. The client bundle - the bundle where your code uses the API which exists in the target library (#2 above). You can use an OSGi activator (or the SCR activate) to run your code which will test if you have your library OSGified correctly.
4. The supporting bundles in your target environment. This may range from logging bundles to that of XML parsers and so on. Just be as close as possible.

All this environment should be something that is easy to provision and setup. I personally use a vanilla Felix instance with necessary environment bundles provisioned using Pax Runner. However you are free to use the tool of your choice to do this provisioning. The objective is to be able to start, test and stop the OSGi container, make changes to your target OSGified bundle and client bundle and repeat. And you do need an environment which should be easy to provision. You can also consider using Pax Exam to setup this environment.

Bundle it up!

If the above setup gave you a success right away at your first try, then you need not read the rest of this writeup. But most often, we do not get it right at the first attempt. So then you will have to do some experimentation with the packaging of your bundle which will ultimately give you the success.

Realize what you need - single or family pack?

It is important to understand and realize that what you ultimately need, and how the final target OSGified bundle will look depends on your use case. If operating in a pure and strict OSGi world, every library should be in its own bundle. In other cases, where there is no strict requirements around your dependencies being strictly separate bundles, you can combine a few libraries together to get a single bundle (or may be still a few more). However, both these ways of bundling has their own pros and cons.

Single bundles are the citizens of perfect OSGi world. This bundle would only represent one vanilla jar. The advantage of this approach is that you fully leverage the OSGi modularity. If there are multiple bundles which wants to use this library, one instance of this library would do (as long as the version requirements also match).  Your artifacts would be light. However, you may have to deal with and manage a lot of these bundles if the transitive dependencies of your target library is just too big.

Family pack - as I love to call it - is usually made by combining a related set of libraries together to form a single bundle. In Maven world, this often can be related to as the transitive dependencies. This kind of packaging may sometimes be necessitated by the library itself, which often has code which has hard, non-OSGi compliant bad OSGi practices code for loading classes and resources, which requires some other dependent libraries to be on the same classpath (OK, you need to understand that in OSGi, every bundle has its own classpath). At other times this kind of packaging avoids you to manage a lot of bundles on your target environment. However, this would mean that tomorrow if another library needs to be added, which has one of the transitive dependencies that already exist somewhere in your existing family pack bundles, either you would duplicate the library over in your second family pack, or just expand your existing family pack. Family packs at times can also lead to cyclic dependencies being created between bundles, which really is just evil.

Things to Remember

Once you have decided the kind of packaging you require, you can move ahead and start creating and perfecting your bundle. The following are some of the things to be taken care of when you start your OSGification process:

1. Know what you are doing

Unless the vanilla jar you are trying to wrap is really simple, you would have to understand OSGi a little bit deeper - especially what each of the manifest headers mean, and how it affects the classloading and resource resolution. What each of the BND instructions mean and how it affects the bundle being generated. Read the BND documentation, and also the tool specific documentation - in my case the Maven Bundle Plugin's documentation.

2. Verify the vanilla jars versus your OSGified bundle

Verify that the contents of the vanilla jar and your OSGified bundle to ensure that you preserve the packages and the resources contained therein, unless you intend it otherwise. The resources iteself may contain some configuratoin files that are required by the library to function, or it may have some licenses. You should be particularly be aware that almost all the licenses and notices require that you keep them in the derived products. If you are merging more than one library, you may have to consider renaming the resources - Maven Shade Plugin can be useful here.

3. Perfect on the contract - Import and Export packages

The Export-Package and the Import-Package are the primary contract of your OSGi bundle. What goes into these should be absolutely clear. Try to understand the library clearly, and export those packages which are API, and hide the internal implementations (some libraries hardly distinguish this fact, so that forces you to have not-so-perfect OSGi practices). Also ensure that your exported packages has a logically valid version. For instance, if you are OSGifiying itext-2.1.7, you could start with an Export-Package of com.lowagie*;version="2.1.7".

What to keep for Import-Package is trickier sometimes. This is where you would have to experiment a bit. Start with all the import packages being optional (*;resolution:=optional), and then once your client bundle is all working perfect, come back here and understand what is absolutely necessary for the library to function, and which are optional. As an example, some libraries may give out dependencies to JUnit classes. Now, why would someone depend on JUnit classes, so quietly remove them from the import packages. On the other hand, dependencies to JAXB libraries could possibly be made as optional one. All this depends on the library under consideration.

4. Ensure no BND warnings

When you are perfecting the contract of Import and Export packages, ensure that you have no BND warnings. Especially around split packages. When you get the split packages, carefully study your vanilla jars (often split package warnings will come when you are making a family pack), and understand of how would you like to prioritize the packages across the libraries to apply the split-package instructions for BND.

5. Dealing with cyclic dependencies

As mentioned before, multiple family packs can lead to cyclic dependencies. When this happens, carefully study the bundles involved in cyclic dependencies and the packages on which these links are formed, and carefully remove the cyclic dependency by either separating out one of the transitive jar as a separate OSGi bundle, or simply keep all the packages of such a library as private. Take the decision based on your situation.

6. Adjusting your client bundle

At times, your bundle may be importing certain packages from the wrong bundle (instead of what you are intending). Example of such packages may be what is exported by the system bundle which is outdated, and you instead want your client bundle to import from a later version exported by a separate OSGi bundle. In these cases, use the version as your savior and explicitly restrict the package versions so that it would import from the bundle of your choice. Often the client bundle may have a relaxed import versioning, which makes it to import from the system bundle. When the version is stricter, you would often achieve what you intend to.

7. Classloader Hell

At times you may get into situations that even after following all the instructions above, your library is unable to load some classes or resources. Carefully try to debug through the library's source code and see where actually the code fails. Also compare this execution to one in a non-OSGi environment, and you would be able to identify that in OSGi, the library is unable to instantiate a class or load a library, because of the way that the code is written to do that. In these cases, the last option may be to patch the library. Download the source code of the original library and make modifications to the offending classes and put it back to the OSGi bundle. Follow the OSGi best practices when you do dynamic class loading or resource loading.

8. Test thoroughly

Test you client bundle thoroughly with all the possible use cases of using the library. If possible use the Pax Exam to write a test suite to be sure that you are covering all the use cases. Everytime you run the above environment, you will realize - most often with OSGi resolution error messages or error stack traces that something went wrong. This information is key to identifying what has to be tweaked to get it working. Repeat the all the steps above through this step, until you are done.

9. Be license compliant

Nothing technical, but respect the licenses of the libraries you are trying to OSGify.

For AEM / CQ developers

For sometime I have been working on the Adobe's marketing platform AEM /CQ which utilizes OSGi as the application container and so requires developers to package their code as OSGi bundles and many of them get panic when they have to OSGify an external dependency - especially when it is a complicated one. I have personally found that most of these AEM developers do not understand the OSGi beyond what is there in the product documentation. And this was my primary motivation for writing this article.

I have made some shortcuts for kick-starting your OSGification process for AEM environment, by creating Pax Runner profiles for few AEM versions at my Github repository https://github.com/ethomasjoseph/aem-pax-runner-profile. Feel free to tweak and use this and contribute to this if you find this useful.

Before you say - I am done (or not)!

Bundling a vanilla jar - especially the complicated ones require a lot of patience and experience. A lot of the above instructions are dependent on the context of which library are you trying to work with and what exactly is the need of the hour. Weigh the choices to choose your track. Feel free to reach out to me (blog comments are OK), in case you want some help in OSGification of your vanilla jar.

Happy Bundling!!!

References

1. BND's official site
2. Maven Bundle Plugin
3. Apache Commons OSGi tracker
4. Instruction of creating bundle using Maven and BND
• http://felix.apache.org/documentation/subprojects/apache-felix-commons/creating-bundles-using-bnd.html
• http://spring.io/blog/2008/02/18/creating-osgi-bundles/
• http://gnodet.blogspot.in/2008/09/id-like-to-talk-bit-about-third-party.html
• https://karaf.apache.org/manual/latest/developers-guide/creating-bundles.html

Tuning your JCR Queries for the AEM & Jackrabbit OAK

Application tuning is an activity which which every developer and architect will encounter sooner or later for any application. AEM based applications are no different. One of the activities that is particularly important with AEM 6 is query tuning. AEM 6 uses Jackrabbit OAK as the repository and hence we would have to know more about OAK to tune the queries. Apart from tuning the queries itself a majority of the time you would be tuning your indexes based on the query. AEM itself uses a lot of queries within for the application to function. It is important to understand how the queries and the indexes work in AEM.

Update: This post was not written as a generic performance tuning guideline for AEM, but just on tuning the queries on OAK repository. Although a lot of the performance depends on how well tuned is your repository. Adobe's recent service packs on AEM 6, and AEM 6.1 in general are much stable and performant. Please refer to the Adobe's documentation for general guidelines on performance tuning.

While the instructions herein are primarily written for AEM applications, non-AEM applications which uses Jackrabbit OAK can also benefit.

There are no hard and fast rules that a particular index will work. This is very similar to database indexes. You would have to look at the current queries being fired and the current repository state and arrive at the best indexes. Often you arrive at the best indexes over time. You would have to apply an index and the re-evaluate your queries, then further tune your queries or try out a different index. Repeat these steps over and over again to find the best and optimized index.

However, there are some basic guidelines which should be followed, which can help you define the best indexes to you application specific needs:

Ground Rules (AEM Specific)

1. Apply the latest Service Pack available and Oak specific hotfixes (if applicable). As of date SP-2 is publicly available.
2. AEM has some built in indexes as well. Do not arbitrarily re-index these OOTB indexes. Indexing is a costly and resource-intensive operation. OOTB indexes should be re-indexed only when specifically advised by Adobe (this is a catch - sometimes you know that re-indexing a particular OOTB index will solve your problem at hand, but get a buy-in from Adobe for this). At other times you would have to create a new index (type) since the OOTB / existing index is in-efficient.
3. Be aware that there has been some important changes since OAK 1.0.9 - especially related to Lucene indexes, hence be mindful about reference resources that advice about using OAK indexes. Check the OAK's latest documenting on query and indexing.

Tools

1. If you are using AEM, the best tool to help you debug and explain your queries, find out slow and popular queries is to install Adobe AEM ACS Tools package (0.0.20 and above). Once installed, you can have access it using the URL: http://yourinstance/etc/acs-tools/explain-query.html 
2. If the above is not possible, you may have to depend on JMX to find slow and popular queries, you can use the Felix console to identify slow and popular queries on any instance 
• http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D9%2Cname%3D"Oak+Query+Statistics"%2Ctype%3D"QueryStat". 
3. You can also use the OSGi Service of org.apache.jackrabbit.api.jmx.QueryStatManagerMBean to programmatically find the slow and popular queries. 

queryStatManagerMBean.getSlowQueries().values()
queryStatManagerMBean.getPopularQueries().values()

4. Use the following debug logs for details on various aspects of query and indexing:
1. Enable DEBUG logging on  org.apache.jackrabbit.oak.query for logs to find the details on query cost calculation (explain)
2. Enable DEBUG logging on  org.apache.jackrabbit.oak.plugins.index for logs to find the details on indexing
5. Use JMX to have high level overview about lucene indexes (OOTB or custom):
• http://yourinstance/system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D4%2Cname%3D"Lucene+Index+statistics"%2Ctype%3D"LuceneIndex"
6. To look at the actual indexes, you can stop the AEM / Oak instance and use the oak-run "console" to explore the indexes among various other options:

• java -jar oak-1.1.6.jar explorer /path/to/crx-quickstart/repository/segmentstore

  (oak repository)
• On the resulting console, run the following command, to dump the index into file system:

• lc dump path/to/dump /path/to/luceneIndex/inRepository

• eg.

  lc dump path/to/dump /oak:index/myCustomLuceneIndex

• Now analyse the resulting dump using Lucene Index analyser tool Luke:

java -XX:MaxPermSize=512m -classpath luke-with-deps.jar;oak-lucene-1.1.6.jar org.getopt.luke.Luke

• The following jars needs to be used:
• The luke-with-deps.jar can be downloaded from https://github.com/DmitryKey/luke/releases
• The oak-lucene-1.1.6.jar can be downloaded from Maven repository as well.

Guidelines

The basic steps for tuning your queries can be summarized as follows:

1. Identify the slow queries in your system. This could be the one that takes a lot of time to execute primarily because there are not proper indexes and there are node traversals happening. The system itself can be queries through JMX to identify the slow and popular queries.
2.  Tune your queries to ensure that you can use indexes whenever possible.
3. Identify properties and nodes that needs to be indexed, based on the slow queries identified, and create Oak indexes for them.
4. Verify your queries and indexes after creating indexes. The indexes are optimum if the cost of the queries identified are lowest - 0 being the best.
5. Repeat steps 3 and 4 above (by even creating different indexes for the same query) till the most cost effective index is created.

Identification of Slow Queries

Identify the slow queries by monitoring the error.log. Note the queries which traverses many nodes, where the log may look something like:

org.apache.jackrabbit.oak.spi.query.Cursors$TraversingCursor Traversed 1000 nodes
... consider creating an index or changing the query 

You can also identify the slowest and popular queries in the system (which the Jackrabbit Oak keeps track of) using JMX or Adobe AEM Commons as described in the "Tools" section above.

Indexing

For the same query you can create multiple indexes. However choose the one to be promoted to your production instances which prove to be the the best in terms of achieving lower cost to the query at hand.

Consider the following when deciding the type of index to be used:

1. Full Text Index Types
• Lucene & Solr
• Aggregate Index - query time aggregation
2. Lucene Property Index
• Since OAK-1.0.9 Lucene Property indexes are preferred indexes over other type of indexes
• Async
• Combines properties & full-text.
• Supports property condition, range conditions, ordering, full-text
• Index-time aggregation - performance improvements
• Using Lucene & Solr.
• Follow instructions for creating the index nodes as in the OAK's Lucene documentation.
3. Property Index
• Default Indexes
• Synchronous
• Unique or Non-Unique
• eg. UUID
• property equals query works best
4. NodeType Index
• Internally Uses Property Index
• Uses primaryType and mixins to identify nodes
5. Traversing Index
• Traverses Repo
• Does not store any data
• Used to retrieve the given path or its child nodes (sometimes the parent nodes as well)
6. Ordered Index
• Async
• Not recommended of nodes > 1000 nodes

Index Management

1. Currently, the indexes can be found under /oak:index/ on AEM.
2. Once you have identified the best indexes, plan to install them as a package on AEM.
3. Re-indexing is usually not needed. However when you see degrade in system performance, do double check your existing indexes if it would need a re-index.
4. Re-index is a costly and resource intensive operation. Use it only when any oak upgrade or  mentioned explicitly in release doc for any patch or hotfix
5. Avoid having multiple properties in index , it may slow your query , its recommended to have one index for each property.
6. Set the system property  oak.queryLimitInMemory to 200000 (or something similar). This will limit the traversal of the nodes to that number

Finally...

Jackrabbit Oak is a project under active development, especially around the area of Indexing. Hence please check official documentation for details. Indexing and tuning requires a lot of observation and practice to get perfect.

I would recommend watching a webinar on this topic by Thomas Muller and Marcel Reutegger on AEM Gems, which is a bit old (before OAK-1.0.9 was released), but immensely useful to understand indexing on Jackrabbit Oak.

Happy Indexing!!!