Fork me on GitHub

Monday, December 21, 2009

Saving a streaming video / presentation

Ever wondered to to save those streaming videos from Youtube, Metacafe, DailyMotion and so many others? I always wanted to save those streams, so that I could watch them again and again even without an Internet connection.

I am a faithful Firefox user, and one of its wonderful plugin - DownloadHelper has helped me for a long time to capture and save Youtube videos as flash video files on my local hard disk. That was quite helpful. But I always wanted to save those InfoQ videos as well, but unfortunately, DownloadHelper could not provide support for InfoQ (among others).

It was only recently, that I used the conversion feature of DownloadHelper, in which I was converting one of those Youtube videos to be converted to wmv format. I noticed, that the conversion started only after the whole of the video stream was played on my browser. And I could readily guess, that the conversion must be using one of the libraries on my machine. So I looked at the process to find that it was using ffmpeg command to do this job. I also noticed, that actually every video was buffered to the /tmp on my Ubuntu. The filename I could see was something like FlashVOZFnS.

Naturally, if a movie plays on your machine, it must be stored somewhere before being played on your browser. So now if you want to save your next streaming video, just identify where in your system does the flash player saves its movie temporarily before being played on your browser, and save that file to your videos folder, and possibly give it a .flv extension too.

Note
  1. As soon as you close the tab containing the video, the temporary file is discarded.
  2. Not all videos may be fully buffered before playing. Very long videos and webcasts may not be fully buffered.

Converting flash movies to other format (Linux)

And now if you also want to convert it into other formats, its very easy in Linux with the following commands:
FLV to WMV: 
ffmpeg -i /home/thomas/Desktop/video.flv -y -v 0 -acodec wmav1 -b 1000kbps -f asf -vcodec wmv1 /home/thomas/Desktop/video.wmv

FLV to MPG:
ffmpeg -i /home/thomas/dwhelper/Desktop/video.flv -y -v 0 -f asf -vcodec mpeg4 /home/thomas/Desktop/video.mpg

There are so many other options with this command, even up to video capturing your screen, but I am not very expert at those. So I leave it to you to experiment with those options.

Update

For any streaming video site, if you cannot see DownloadHelper to your help, make sure that you should have only this one tab containing any video to download. In other words, if you have multiple tabs with video contents, DownloadHelper seems to get confused, so you may not actually get a download link help.

---------------
Edited and removed some content for legal reasons.
---------------

Tuesday, December 15, 2009

Configuring log4j with XML in OSGi

Sometime back, I had to make perf4j work on OSGi. It required log4j as the logging library and purely demanded log4j.xml (it would not take the .properties file since the configuration must attach downstream appenders). I assumed that it should be simple, but unfortunately it was not. In the rest of this post, I describe how I could get around this, and used log4j.xml to configure log4j logging in OSGi.

Wednesday, November 18, 2009

Microsoft ASP.NET MVC - Why Java web developers should reconsider action frameworks?

Microsoft's ASP.NET MVC is a paradigm shift in web development from its pioneering component based development offered with WebForms to action based web framework. This was a natural evolution due to the pains in the component based web development. While the Java community started out with the action based frameworks, the shift of the community seems to be on a higher side for the component based web development led by JavaServer Faces (JSF) in the recent years. The shift of Microsoft towards action frameworks should make the Java web developer's heads again turn towards action frameworks and reconsider their choice towards the paradigm for web development. Perhaps a thinking about polyglot web development is needed, rather than abandoning action frameworks.


Thursday, October 29, 2009

Evolving persistence, SQL and ORMs to the next generation

Recently there has been a lot of discussion on the web describing the ORMs as a thing of the past, and its associated discussions that agreed and disagreed with their arguments presented to various degrees. The discussions even took a long conversations across the twitter space too.

Persistence models and application evolution

Persistence is important for any application. Over the years this has been done in various ways. While RDBMS and SQL were the most obvious choice till few years from now, recent times have also seen successful practical usage of other persistence models which include Amazon's SimpleDB, and Google's BigTable among others. These non-relational alternatives fall under the NoSQL (or “post-relational”) persistence models.

Naturally it becomes quite obvious that over the time, relational databases have proven inadequate to satisfy the growing needs of the Web2.0 applications. The advent of the new-generation non-relational databases are a proof to this. Every persistence model has a niche area to which they cater. The newer databases have has also made people to think about "mix-n-match" use of the various persistence models as people talk more and more about polyglot persistence.

The relation-object impedance mismatch

Since RDBMS has a solid theoretical foundation, and has been used across various applications, its existence for another century cannot be ruled out. Yet new advances in persistence like CouchDB, BigTable, SimpleDB, Graph databases, etc are most welcome steps in advancing the persistence mechanism.

Even with newer persistence models, a lot of the enterprise applications still use, and will continue to use relational databases. Applications that use up this data are largely represented as objects. While object model seem to be the most obvious choice for application development, the mismatch between the relational model and the object model comes as the reason for a lot of hues and cries that have given rise to ORMs. The learning curve and the pain in using these ORMs have now given rise to debates questioning the existence of ORMs in the future.

More than debating over the existence of ORMs itself, rather a deep look is needed, to the problem which the ORM is trying to solve. Why does the ORM exist? The most popular answer would be impedance mismatch. But the mismatch is pointed out between relational data representation versus the object model.

Overhauling the persistence mechanisms

Often the data storage mechanism and its interfacing API are seen as an integral part, including the RDBMS data storage and SQL. They are often seen as one, but are actually two separate components of the modern day relational databases. They were created at a time when the existence and usage of OO was questionable. Even now when they are used for a long time, it can be argued that SQL and RDBMS themselves have a mismatch in the paradigms. Now when OO has gained so much importance, it is time that the RDBMS and SQL should be re-looked from the OO perspective.

We can consider two aspects in any persistence mechanism:
  1. The way data is stored (relational, key-value, hierarchical , column, JSON etc) using various data structures and storage mechanisms. Closely associated to this data storage will be the related methods of storing and fetching the stored data (algorithms used to fetch data based on persistent store itself - eg relational algebra) that could be probably exposed through function calls. The function calls will be closely associated to the data store mechanism itself.
  2. Interfacing provided to interact with persistent store (SQL, NoSQL, ObjectQL, programming language API etc).
Often these two aspects are baked together (in our minds most of the time). This thinking should evolve and these two should essentially be seen as separate, and ideally pluggable. We talk about adapter pattern in programming languages, but the same concept could probably be applied to persistence mechanisms as well. The persistent storage and its storing and fetching mechanism should allow widely different interfacing provided to interact with it. DB vendors can provide few interfacing mechanism, and also allow others to write other interfacing mechanism that should be pluggable.
Persistence Interfacing
Interfacing will expose the persistent store in various forms - relational model, or object model or even other models used now and may come in future. Mapping of the datastore persistence model to the exposed model should reside with the interfacing mechanism used.

Concluding thoughts

Achieving this practically may not be very easy, but not impossible. Once this is achieved, ORMs will rather shed off the mapping weight and evolve as frameworks that enhance developer productivity on issues like caching, application scaling etc. Rather, ORMs will no longer be ORMs and probably a new breed of softwares will evolve that will take care of the cross-cutting concerns for an application when dealing with persistent stores.

This may look crap to some, but then that is what I wish. I am an application developer presenting my ideas on what a database itself can provide, and how applications can leverage the new capabilities. The issue needs to be looked at by db vendors and research scholars. DB vendors and db developers need to give deeper thoughts to the problems faced by application developers in their need to persist application data. How many of us use the bare databases (using only SQLs probably) without an application? The current forms of persistence may be right in their own theoretical framework, but it should evolve to fit in seamlessly with the application development space, where it is used.

And till that happens, I will have to carefully weigh my choices among NoSQL, SQL and ORMs, and muddle with the problems that have already heated up a lot of discussions in the application development space.

Thursday, June 11, 2009

JavaOne 2009 slides are now out!

Now the slides of the sessions at JavaOne 2009 are out and available at Sun Developer Network (SDN).

You need to be a registered user there! (For some, this sucks!)

Modularity and OSGi was one of the discussed topics in the sessions, but worried to find that Sun has not mentioned anything on it (OSGi) on the site, though it has put up the slides :).

More Goodies

Sun has put up some good articles and other stuffs on JavaOne as well on its site.

OSGi DevCon 09 - Component Oriented Development in OSGi with DS, Spring and iPOJO

This slide discusses some comparisons on the various component oriented development models with OSGi. Probably there a mention of Peaberry, would have been good. Sure, I will post some in the coming posts.

OSGi DevCon 2009 Review

Last few weeks I was more into the OSGi updates going around with the EclipseCon and JavaOne. In this and few more posts I will try to put something to capture on the recent developments on OSGi and related fields and technologies.

So here is an interesting slide I found.

Thursday, May 14, 2009

Book Review: Maven: The Definitive Guide

Sonatype recently released its 0.5 version of its Maven Book - Maven: The Definitive Guide.The print edition published by O'Reilly, the printable PDF published by Sonatype is available as a free download! I went thorough the book quickly and here is my first hand impression. The books is also available for online reading.
Maven: The Definitive Guide

Cool!

I have been using Maven now for quite sometime, and I was very happy to read the book. The book is a fit-for-all, who wants to use maven in someway or other.

It starts with as simple as downloading and installing maven, to explaining the core basics of maven, its dependency management, plugins etc, to optimizing plugins and POMS. It serves as a tutorial as well as a reference guide for Maven.
The book explains concept with ample examples. It also covers to explain some of the most useful plugins in depth. It explains the maven build cycle in depth, build profiles, maven assemblies, site generation and maven repositories, m2eclipse and nexus at depth. It also explains customizing plugins and also on writing our own plugins.

How to use the book

An extract:
This book is divided into three parts: Introductory Material, Part I, “Maven by Example”, and Part II, “Maven Reference”. The introductory material consists of two chapters: Chapter 1, Introducing Apache Maven and Chapter 2, Installing and Running Maven. Part I, “Maven by Example” introduces Maven by developing some real examples and walking you through the structure of those examples providing motivation and explanation along the way. If you are new to Maven, start with Part I, “Maven by Example”. Part II, “Maven Reference” is less introduction than reference, each chapter in Part II, “Maven Reference” deals with a focused topic and dives into as much detail as possible about each topic. For example, the Chapter 17, Writing Plugins chapter in Part II, “Maven Reference” deals with writing plugins by providing a few examples and a series of lists and tables.

While both Part I, “Maven by Example” and Part II, “Maven Reference” provide explanation, each part takes a different strategy. Where Part I, “Maven by Example” focuses on the context of a Maven project, Part II, “Maven Reference” focuses on a single topic. You can skip around in the book, Part I, “Maven by Example” is by no means a prerequisite for Part II, “Maven Reference”, but you'll have a better appreciation for Part II, “Maven Reference” if you read through Part I, “Maven by Example”. Maven is best learned by example, but once you've gone through the examples, you are going to need a good reference to start customizing Maven for your own environment.

Kudos

The work is excellent and what makes it even better is the fact that the company and the authors do not consider it complete yet! Its a beta 0.5. This is what they have to say on it:
If this book moves out of Beta, we're essentially saying that there is no more content to add and nothing is going to change. I've always thought that good books evolve over time and that they live beyond the confines of the chapters and sections that define them. A good book is an ongoing conversation and a series of interactions not just between authors and readers, but of lateral interactions between readers. That was a fancy way of saying that the book is a community.

Sonatype and the book authors have done an excellent job. I would recommend the book to anyone who uses Maven.

Tuesday, May 12, 2009

Flex Clients for OSGi apps

As I continued to work on OSGi based web apps, there was rising demands from our clients and partners about supporting Flex clients in future. I took a lead to evaluate our existing architecture consisting mainly of POJOs, supported with Spring and OSGi.

Thanks to the documentation at BlazeDS site and the refcard at DZone for helping me out. I am almost ignorant of Flex programming, but still I could make out few things. Here is my first-cut evaluation.

Flex and Java with POJO

I was happy to learn that with Spring's hailed paradigm of programming with interfaces and POJOs would make integration with Flex very simple, and would hardly require any change in how we program and architect our application.

Flex has strong ties to Java. Its server BlazeDS is based on Java remoting. Good news that with Spring BlazeDS, we can reuse our interfaces and its POJO implementations.

Spring BlazeDS as described by SpringSource:
Spring BlazeDS Integration is a new addition to the Spring portfolio, and a component of the complete Spring Web stack. This project's purpose is to make it easier to build Spring-powered Rich Internet Applications using Adobe Flex as the front-end client. It aims to achieve this purpose by providing first-class support for using the open source Adobe BlazeDS project and its powerful remoting and messaging facilities in combination with the familiar Spring programming model.


Blaze DS is based on Java remoting and web messaging from Adobe. Its primary purpose of existence is easier integration of Adobe Flex/Adobe AIR with existing Java code to create Rich Internet Applications (RIA). Spring BlazeDS leverages more on this end by its dependency injection, and simpler integration platform.

Blaze DS can be set to run on any web container such as Tomcat. It listens to a particular URL for client requests and processes them. Flex clients connect to the Blaze DS server and executes Java methods. The base transport protocol will be HTTP.

The regular client calls with Blaze DS is usually handled by a Servlet from Blaze DS. However, when Spring Blaze DS comes into picture, this is replaced by the Spring's Spring MVC Servlet, so that Spring can intercept those client calls.

Flex works on the concept of message broker. A class that will can handle client calls. The clients calls will be routed to the message broker through additional configuration specific to Blaze DS. We should be happy to expose our interfaces and its implementation classes to work as message brokers.

A sample Spring's application-context.xml file
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:flex="http://www.springframework.org/schema/flex"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/flex
http://www.springframework.org/schema/flex/spring-flex-1.0.xsd">
<!-- Spring Beans's -->
<bean id="myService" class="MyServiceImpl" />
<!-- Simplest possible message broker -->
<flex:message-broker />
<!-- exposes myService as BlazeDS destination -->
<flex:remote-service ref="myService" />
</beans>

The point to be noted above is that myService can be given any reference, including a reference to any OSGi service.

Another notable point about Spring BlazeDS Integration is that Flex can support authentication, that can easily plug in with Spring Security.

Kudos

Spring BlazeDS has not yet had its final release, but indeed a great work from Spring guys to help lift existing POJO based applications to work with Flex clients in most non-intrusive manner.

Friday, May 8, 2009

Guice getting into Java SE?

One day I may use @javax.inject.Inject as my regular Java annotation! Yes, dependency injection with annotations might become a standard Java API. Google Guice and SpringSource have partnered to standardize a proven, non-controversial set of annotations that make injectable classes portable across frameworks. The JSR has been proposed.

Read more about this at the official Google Code Blog.

Monday, April 20, 2009

Submitting form to new window

Today one of my co-workers was in an effort to submit a form that would bring up a new popup window. After some Googling, they found a solution.
<form name="datacollector" method="POST" action="new-url.do" target="_blank">
<!--
     ...
     other form elements
     ...
-->
</form>
Though this was getting them a new window, on Firefox it was appearing as a new tab. They soon found a setting in FF, that would bring it to a new window itself. However, they wanted a popup without any toolbars, menus etc.

They approached me. Thanks to one of my student days HTML/JavaScript tutorial. I was able to recollect it and give them a solution.
<script type="text/javascript">
   function submitToNewWindow(frm) {
       window.open('','popupWin','scrollbars=no,menubar=no,height=600,width=800,resizable=yes,toolbar=no,status=no');
       frm.action= "new-url.do";
       frm.target = "popupWin";
       frm.submit();
   }
</script>
...
...
<form name="datacollector" method="POST">
<!--
     ...
     other form elements
     ...
-->
      <input type="button" value"Generate" onclick="javascript:submitToNewWindow(this.form)"/>
</form>
The idea is simple. Open a window with a name (here popupWin) using JavaScript that will have no scrollbar, menu, etc. Now in the form's target, supply this window name. This would bring up the form submission's result on the specified popup window. As far as the "_blank" is concerned, it is a HTML's default name for a new blank window.

This is a basic feature of HTML, that in the target, it accepts the name of a window. Note that if a window with that name does not exist, probably a new one will be created.

This solution was tested to be working in FF as well as IE.

Though this is a simple solution, I thought to put up here on my blog, as my co-workers were not able to get anything like this on Googling.

Thursday, April 16, 2009

Cross Platform Java Daemon

Running a Java application (non graphical) would normally need a command line for it. This seems OK if we are developing small applications, but today we see enterprise applications that often needs to go and work as a server, that need to start up and go down with the OS. For such applications, the requirement of avoiding a command line gains importance. Such applications are often referred to as Daemons in the *nix space, and services in the Windows space.

Solutions?

Wondering that you have seen so many Java applications that can work without command line? Its not that Java applications without a command line is not possible. Though Java is popular for being platform independent, doing this "daemonising" task has not been of that nature. Many of the Java based server products have been doing that in various different ways, and in most cases using native code. And if your look around for Java Daemon, or Java as Windows Service, you will find separate and widely different solutions.

And there are solutions specific to platforms, for instance - the nohup and screen command in linux can get you an application without terminal. For the nohup, though your solution to stop the application, is probably to kill it.

Recently I had a similar need, I looked around and could find a bunch of solutions:
  1. Commons Daemon (C, Java)
  2. Java Service Wrapper (C and Java)
  3. Yet Another Java Service Wrapper (pure Java)
  4. Akuma


Do it yourself!

Going through even the surface of these will give you an idea of how widely the implementations vary. These are solutions that help you lift your existing application, if you yourself do not want to provide any implementation of getting daemonised. But if you are happy to learn on this concept, or want to get your code provide the "daemon" power natively with simplicity and lightweight effort, read on...

(Note: I would recommend a reading of the Commons Daemon. Also note that the code below is just ideas and concepts, you may need to evolve it further to make it production ready.)

The concepts

Any Java application needs an entry point for it to start working. This is the public static void main(String[]) method. And the common way to do this invocation is to instruct the JVM and show it the class that contains this method block, with the java command. This command will start the JVM which starts the Java program starting with the main method. The command line is blocked until the Java program completes. This is a valid scenario for small and interactive applications, but for multi-threaded server programs, we do not want the command line.

The daemon solutions have varied way to enter a Java program (other than the main method as well). But here, we will use what we already know: enter a Java program with main method and still remain detached from the command line.

There we need to understand two concepts:
1. Starting a program. Using Java, it is possible to start "any program" using Runtime.exec() and ProcessBuilder.exec(). Wondering? Yes, any program that you can run in a command line can also be launched within Java programs as well. Another notable aspect of this is that the launched program does not have a command line associated with it, and it is optional for the launching Java program to keep track of this process. Well, we will use this concept to launch a Java process inside a Java program!
2. Stopping the program For a Java program, there is no way to get the control of a process running in the system. So if we launched a process from a Java program, and lost control over it, how can we tame it to bring it down? Establish protocol! We will make the launched program to open a port and start listening to it, and when it receives a valid shutdown command, start its shutdown procedures.

Taming the terms...

Below, I will show how to do this. But before, I will introduce some terms, so that further discussions are easy.
  1. TargetApp - the Java application (possibly a server program) that we want to get detached from the command line and go "daemonised".
  2. Daemon - The class that actually goes into daemon mode and listens to shutdown commands. If you are coding the daemon inside your application, this functionality will be inside one of your classes in the TargetApp.
  3. Launcher - This program will kick off the Daemon as a process detached from the command line. Also provides interfacing to provide shutdown command to the Daemon.


Reaching out with Code

1. TargetApp

Since this can be any Java application that usually works in a loop, I will create a simple application that will keep printing a funny text to a file created in your current working directory.
package org.otw.daemon;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Date;

public class FunnyTargetApp {

   private boolean shouldRun = false;
   /**
    * @param args
    */
   public static void main(String[] args) {
       FunnyTargetApp app = new FunnyTargetApp();
       app.start();
   }

   /**
    * Starts the application.
    */
   public void start() {
       shouldRun = true;
       Thread workerThread = createWorkerThread();
       Thread commandTracker = createCommandTrackerThread();
       commandTracker.start();
       workerThread.start();
       try {
           workerThread.join();
       } catch (InterruptedException e) {
           e.printStackTrace();
       }
   }

   /**
    * Stops the application, in a clean way.
    */
   public void stop() {
       System.out.println("Stopping...");
       shouldRun = false;
   }

   /**
    * Creates a thread that mimics a continuous job as that of a server.
    * @return
    */
   private Thread createWorkerThread() {
       return new Thread(
               new Runnable () {
                   public void run() {
                       File file = null;
                       FileWriter wri = null;
                       try {
                           file = new File("test.out");
                           System.out.println("Started with file at ["
                                   + file.getAbsolutePath() + "].");
                           file.createNewFile();
                           wri = new FileWriter(file);
                           while(shouldRun) {
                               wri.write("Funny Text >> "+ new Date());
                               wri.write("\n");
                               wri.flush();
                               Thread.sleep(1000);
                           }
                       } catch(Exception e) {
                           e.printStackTrace();
                       } finally {
                           if (file !=null && wri!=null) {
                               try {
                                   wri.flush();
                                   wri.close();
                               } catch (Exception e) {
                                   e.printStackTrace();
                               }
                           }
                       }
                   }
                  
               }, "Worker Thread"
       );
   }

   /**
    * Creates a thread that will keep track of the commands issued on the command line.
    * @return
    */
   private Thread createCommandTrackerThread() {
       return new Thread(
               new Runnable () {
                       public void run() {
                           System.out.println("Listening to commands from terminal.");
                           while(true) {
                               InputStreamReader isr = new InputStreamReader(System.in);
                               BufferedReader reader =  new BufferedReader(isr);
                               String command = null;
                               try {
                                   command = reader.readLine();
                               } catch (IOException e) {
                                   e.printStackTrace();
                                   break;
                               }
                               if (command.equalsIgnoreCase("stop")) {
                                   stop();
                                   break;
                               } else {
                                   System.out.println("invalid command.");
                               }
                           }
                       }
               }, "Command Tracker"
       );
   }
}


2. Daemon

This is the class that actually gets detached from the terminal and becomes a daemon, while keeping a reference of the TargetApp. So basically it has to deal with 3 functionality:
a. Launch the TargetApp and keep its reference.
The TargetApp can be launched as a thread or as a process. The preferred way should be as a thread. For launching as a process, the TargetApp should have a either a start method or a regular main method. For launching TargetApp as a thread, invoke methods needed to start the TargetApp, if no such specific method is there, just invoke the main method.
//Starting as a Thread (reflecting on the main method)
Class payload = getPayloadStartClass(); // determine the payload class via configurations
Object payloadInstance = payload.newInstance();
Class[] paramTypes = new Class[1];
paramTypes[0] = arguments.getClass();
Object[] param = new Object[1];
param[0] = getPayloadArguments();
Method method = payload.getMethod("main", paramTypes);
method.invoke(payloadInstance, param);


Note: This snippet uses reflection. You may use an interface to make things better, using a startup method.
// Starting as a Process
Class payload = getPayloadStartClass(); // determine the payload class via configurations
String javaOptions = //get it via configurations
String payloadArg = //get it via configurations
String command = "java " + javaOptions + " "
 + payload.getCanonicalName() + " "
 + payloadArg;
log.debug("Starting command line: " + command);
Runtime.getRuntime().exec(command, null);


b. Starts up a shutdown port and waits for a shutdown command to be issued.
ServerSocket serverSocket = ServerSocket(getShutdownPort());
Socket socket = serverSocket.accept();
boolean isLocalConnect = false;
InputStream stream = null;
PrintWriter out = null;
try {
   isLocalConnect = socket.getInetAddress().isLoopbackAddress();
   socket.setSoTimeout(getNetworkTimeout());
   stream = socket.getInputStream();
   out = new PrintWriter(socket.getOutputStream(), true);
} catch (AccessControlException ace) {
   log.warn("StandardServer.accept security exception: "
                      + ace.getMessage(), ace);
   return;
} catch (IOException e) {
   log.error("StandardServer.await: accept: ", e);
   return;
}


c. Stop the TargetApp and then itself for a clean shutdown on receiving valid shutdown command on the shutdown port.
String command = readStream(stream); // reads command from the input stream
boolean match = command.toString().equals(shutdown);
if (match) {
   log.debug("Shutdown command recieved via Telnet.");
   shutdown(); // calling the shutdown procedures
   return;
} else {
   log.warn("Invalid command.");
}


You can telnet to the port and provide shutdown command. The Launcher's stop routine also does the same thing (the code will follow soon).

Remember to register shutdown hooks on the application startup, and remove it explicitly, if the shutdown was not initiated by the shutdown hook. Shutdown hook is a Thread that is not started, and registered with the JVM. When the JVM goes down, this thread will be started. So we can write some cleanup code in this thread. Remember to return from the run of the shutdown hook quickly.

3. Launcher

This class will provide us hooks to start up the Daemon, and stop it based on arguments (from the command line). To launch the Daemon as a Java process and detach it from the command line, we will use the Runtime.exec(). This will be similar to the process mode launching of TargetApp.
String strCmd = "java " + "-classpath " + classpath + javaOptions
   + " org.ops4j.laboratory.users.open4thomas.jdaemon.Carrier";
log.debug("Starting command line: " + strCmd);

Process proc = Runtime.getRuntime().exec(strCmd, builder.getArguments());
Thread hook =  createShutdownHook(proc);// create shutdown hook thread
Runtime.getRuntime().addShutdownHook(hook);


To stop the running Daemon cleanly, the class will set up a socket connection to the Daemon's shutdown port, and supply a shutdown command. This will trigger a shutdown procedure in Daemon, which in turn can trigger shutdown on the TargetApp.
Socket socket = null;
PrintWriter out = null;
try {
   socket = new Socket("localhost", getShutdownPort());
   out = new PrintWriter(socket.getOutputStream(), true);
} catch (UnknownHostException e) {
   log.error("Unknown address: localhost.");
  return;
} catch (IOException e) {
   log.error("Couldn't connect to: localhost.");
   return;
}

out.write("shutdown\n");
out.flush();


Getting around with security holes.

There is one important security hole in this approach. When the Daemon is listening on the shutdown port for shutdown command, it is possible for a hacker to compromise on this feature.
This can be eliminated with few approaches:

1. Provide password protection

The application could ask for a password that could be stored in encrypted format, and before accepting shutdown command, the user will be needed to provide a password.

2. Preventing DoS attack

A malicious user can connect to the shutdown port, and provide junk command continuously effectively doing a DoS attack. This can be dealt by accepting limited amount of failed password, or wrong command attempts. Limit the length of command/password.

More improvements

In the Launcher's stop, after issuing shutdown command to the Daemon, we could check for the existence of a PID file to know if the application is still running. The PID file can be created by the Daemon with "deleteOnExit" flag.

Another improvement area is to provide an interface that the TargetApp should implement, so that it will be easy to call startup and shutdown routines on the TargetApp.

There could be more, but not right now in my mind.

Want more code?

Sure, the code snippet I provided here is just to explain the core concept. There are other things required to get it working. I worked out quickly on this, and the code is available in my lab at OPS4J here. Please note that I may change the code over time. I am writing this article as of revision #14752.

This is a regular SVN repository. So you can get the source code with
svn co https://scm.ops4j.org/repos/ops4j/laboratory/users/open4thomas/jdaemon/jdaemon-main

Wednesday, April 8, 2009

Exciting future for the web

The future of web applications seem to be exiting for every web user. With the drafting and evolution of HTML 5 , and browsers like Firefox coming up with plans on Prism and Weave , its plugins, and evolving nature of supporting applications such as Google Sync (mobile and browser ) and Google Gears , the experience of a web user seems to be in a good light. Some of the fantastic experiences would be:
  • Inherent support for offline browsing and using web applications by browsers.
  • Inherent support for rich web UI.
  • Enhanced experiences with audio and video on web browsers.
  • Seamless and centralized browser profiles. Browser will seamlessly synchronize your profile including bookmarks, browsing history, saved passwords and tabs across multiple browsers, so that your desktop, laptop, and mobile phone can all work together.
Many of them are already available for a beta preview. These seem to not only enhance the experience of a web user, but also developers like me will find an interesting turn in developing web applications.

Monday, April 6, 2009

Beating the Recession with Free Software

Few weeks back, one of my friends working with a big enterprise asked for my help in getting his personal PC to the right health. He would use computers only for browsing, spreadsheets, preparing presentations, and for word processing. His PC had Windows installed all infested with viruses, that was making his daily working trouble. He told that he had installed a free Antivirus , but still was having such problems. Even in the past that he had to get his PC with a fresh installation of Windows, and now it was barely a month after that.

I was more than happy to introduce him to Linux. But he was not readily accepting the call since he had only used Windows, and had not even had a glance of Linux. But because I told him that Linux won't catch a virus, he was ready to go for a dual boot. Just then I took out my Laptop and showed him my Ubuntu installation, and asked him to use it for a while. After sometime he was comfortable to use all of its features that he wanted - the Firefox and the Open-Office met all his needs. To make him further convinced, showed him my installation of Sun's Virtual-Box, in which I had a Windows XP image running. He was so happy at this that he asked me to teach him on how to get on to Linux from the boot.

The next question he asked me was - "Why are we using Windows XP in office? We have to spend a good amount for its licensing every year. Added to that, we also have to spend on purchasing Antivirus and its licensing, while we can actually do it all with Linux."

Spending on the Stack?

The question is not only on spending on one or two paid and closed software, but rather the stack of such software that is often built up. Let us not think about geeks who are able to modify the FOSS software such as the Linux to their use, but even from the perspective of end users like my friend, who use computers only to get their daily tasks done. For them, getting a functionality on their system means searching on the net for a software, finding the cheapest of all (many a times a shareware, and if they are not lucky - a malware), and installing them with a questionable security of your personal data. By just considering the Operating System, Antivirus software and Internet Security solutions now seem as an inevitable part of paid and closed software stack. Does anyone know, what is inside a closed software?

Compare this with a Free OS such as Linux. First, you are legally free to copy and redistribute the software. And the software comes for free "as in free beer" most of the time. Even if you spend, compare this to the cost you spend on the closed source OS.

This is true not just for simple end users, but also advanced users of computers. Those software companies who are providing solutions with FOSS stack, can provide their customers with reasonably cheaper solutions (stability - I am avoiding here, because it depends on the solution provider).

Leveraging business with FOSS

Even if considering this on an enterprise scale, I am sure there should be enough testimonials to say how much one should have saved when using a Free Software stack. In either case you need a support staff to manage all the software and its maintenance, so then why not spend on a software that are know to contain comparatively lesser security holes, and fewer headaches on licensing, and of course saving on the expenses.

One thing however, is that it becomes necessary to choose a stack wisely that can reduce your overall costs. You have to weigh a lot of factors, but definitely, in your considerations, give a good space to the FOSS stack. I am sure every business can workout a good solution to beat such recessions with an optimal strategy with FOSS stack.

Some questions can come up in your mind when reading through this. I have left those for you to think. I can't cover every aspect of this discussion in a blog post like this. But feel free to post your comment on here.

Tuesday, March 31, 2009

Domain Driven Design in 10 minutes

I had prepared some notes, when I was reading through the DDD book from InfoQ - "Domain Driven Design Quickly " which is just summarizes the essence of what DDD is, drawing mostly Eric Evans' book.

I thought, this note might be helpful to somebody who want to get the idea very quickly. This is not a complete description of what DDD is. Infact DDD is a very big topic that mostly comes into your grasp as you start practicing it. That is why probably books were written (and are written even now!) on it.

So here it goes...

1, 2, 3...

  • DDD introduces nothing new, DDD is all about bringing Object Oriented paradigms to the Enterprise development
  • Focus on business domain instead of technology hype

What is DDD

  • Software exist to solve real world problems - automate processes. Software is deeply entangled to the domain it is built for.
  • To create a good software, one must deeply understand the domain for which the software is built.
  • The knowledge of the domain is known to the people inside it - the domain experts. The software should always start from this domain knowledge.
  • Software should be a reflection of the domain.
  • One even without a knowledge in the domain should be able to learn about it, by just reading the code.
  • Software that does not reflect the domain cannot react well to the changes over time.

Building Domain Knowledge

  • Software development process starts with understanding the domain. This starts with discussions between Software Designers/Developers and the Domain Experts.
  • Knowledge about the domain from the domain experts may be disorganized and unpolished. Their organization of knowledge is very specific for their usage and not always suited to building a software system. We should extract the right information from them by asking the right questions towards transforming it to a useful form - the domain model.
  • Early prototyping may be required for clarifying on the understanding between the two. However, the domain model is the place where the expertise of the two meet.
  • Sometimes the discussions brings out some of the unearthed concepts from the domain to picture.
  • There may be too many rounds of two and fro between domain expert, designer, and the developer.
  • The process may seem to be time consuming - it is, and it should because, at the end, the software's purpose is to solve business problems in a real life domain.

Building the Domain Model

  • Software process starts with Software Design - by talking to the domain experts.
  • Knowledge from the domain experts is organized, systematized, divided into logical modules and abstracted as the Domain Model.
  • Domain Model is very necessary throughout the design and development process as we will make lot of references to it later. We will need it to communicate with domain-experts, fellow designers, and developers.
  • Domain model is an essential part of Software Design. Domain model is the domain knowledge abstracted for the software. All the thinking about the domain is synthesized into this model.
  • Domain model can be expressed as a diagram (UML, DFD, FreeMind etc), in writing or a mix of both.

Software Designing

  • As a part of Software Design we have the Domain model expressed. Now we can start doing the Code Design.
  • Code design is different from Software Design. Software Design is like creating the architecture of a house, while Code Design is working on its details - deciding the location of a painting.
  • A code design mistake is usually more easily corrected, while software design errors are a lot more costly to repair. Its easy moving the paining from one wall to the other, but a different thing to tear down one side of the house.
  • Code design is where the design patterns comes to play. Good coding help create clean , maintainable code.

The Ubiquitous Language - The need for a common vocabulary.

  • The Software process begins with building a domain model that happens by communication between the Domain expert and the Software Designer/Developer. Better Software needs better communication, that needs a common vocabulary between the parties - expert, designer and the developer.
  • The project faces serious problem for lack of a common vocabulary to discuss the domain. There will be serious communication gap.
  • Developers talk about classes, methods, algorithms, design patterns, databases, about which the domain expert hardly knows anything. But since we are solving the problems at the domain with the software, we should use the vocabulary that domain expert will understand, and normally uses.
  • Domain experts on the other hand use their own jargon that is not easy to understand for designers and developers. We bridge this gap of common vocabulary by creating the domain model, where all involved in the team arrive at a common vocabulary mainly driven from the actual domain. This is where the software meets the domain.
  • The vocabulary developed with the domain model must be used consistently in all communications without any ambiguity.
  • The language should evolve with the consent from both the parties. Domain experts should object to awkward terms, while developers should watch for terms that may cause any problems at their side.

Model Driven Development

  • Correctly expressing the model does not necessarily mean that the model will be correctly translated to a proper code. A good model producing a bad code model is not good anyway.
  • Any domain can be expressed with many models, and any model can be expressed in various ways in code. For each particular problem there can be more than one solution. It is then important to choose a model which can be easily and accurately put into code. Again, the code should be able to communicate the business domain.
  • The model chosen should allow the code to be accurately mapped to the model. This mapping should not be lost as the code and/or the domain evolves over time.
  • The transition from model to code should not lose any information, and even worse, should not convey the wrong meanings. This happens primarily when the model chosen is modeled without taking the coding aspect.
  • Analyst and domain expert meeting should not be closed room meetings. It would be good and productive to include the developers in the analyst meetings, so that they have a clear and complete view of the domain and the model before they start designing the code. The domain model should be constructed with an eye towards the software and design consideration, while not heavily relying on the code design.
  • Everyone on the team must be aware of the model as well as the code aspect of the system. A refractoring in code should not break away from the model, similarly, a model constructed should be practical to program.
  • Mapping between the model and code should be very simple and obvious. Complex mapping is a warning indicator. Revisit the model and modify it to be implemented more naturally in software, even as you seek to make it reflect deeper insight into the domain. Demand a single model that serves both purposes well, in addition to supporting a fluent Ubiquitous Language.
  • Code and the model are two aspects of the project. A change in one aspect should ripple across the other aspect as well. This should be a conscious decision.
  • Object-oriented programming is suitable for model implementation. They make it possible to create direct mappings between model objects with their relationships, and their programming counterparts.

Building blocks of Model Driven Development

Layered Architecture

  • We should plan for layered architecture to allow different concerns - UI, Application, Domain, Infrastructure

Entity

  • When we are interested about, which object it is - eg a bank account

Value Object

  • When we are more concerned about the attributes of the object rather than its identity. e.g - the points in a line.
  • It is recommended to select as entities only those objects which conform to the entity definition. And make the rest of the objects Value Objects.
  • Having no identity, Value Objects can be easily created and discarded. Nobody cares about creating an identity, and the garbage collector takes care of the object when is no longer referenced by any other object. This simplifies the design a lot.
  • Value objects should be immutable so that it can be shared. Make copies of it and share it with other objects.
  • Value objects may contain - attributes, other value objects, reference to entities.

Services

  • Service encapsulate the behavior of the domain. The behavior does not seem to belong to any of the objects- entity or value objects. Service can group related functionality which serves the Entities and the Value Objects.
  • Such domain behaviors are attached to the Service objects. Such an object does not have an internal state, and its purpose is to simply provide functionality for the domain.
  • Services act as interfaces which provide operations. A Service usually becomes a point of connection for many objects. This is one of the reasons why behavior which naturally belongs to a Service should not be included into domain objects.
  • Services promote loose coupling between domain objects
  • A Service should not replace the operation which normally belongs on domain objects. We should not create a Service for every operation needed. But when such an operation stands out as an important concept in the domain, a Service should be created for it.
  • There are three characteristics of a Service: 1. The operation performed by the Service refers to a domain concept which does not naturally belong to an Entity or Value Object. 2. The operation performed refers to other objects in the domain. 3. The operation is stateless.
  • Services can belong to 1. domain layer, 2. Application layer, 3. Infrastructure. These services should not be confused, and be kept separate:
  • Both application and domain Services are usually built on top of domain Entities and Values providing required functionality directly related to those objects.
  • If the operation performed conceptually belongs to the application layer, then the Service should be placed there. If the operation is about domain objects, and is strictly related to the domain, serving a domain need, then it should belong to the domain layer.
  • The application layer is a thin layer which stands between the user interface, the domain and the infrastructure. (application layer = JSF actions written inside beans ?)
  • Application layer does not talk to the domain object, but rather only to the service object, that in-turn work with domain objects.

Modules

  • Modules allow to split application into logically related items, and call them as modules. It reduces complexity and cohesion between modules. Modules should be made up of elements which functionally or logically belong together assuring cohesion.
  • Modules should have well defined interfaces which are accessed by other modules. Instead of calling three objects of a module, it is better to access one interface, because it reduces coupling. Low coupling reduces complexity, and increases maintainability.
  • Give the Modules names that become part of the Ubiquitous Language. Modules and their names should reflect insight into the domain.
  • We have he module concept naturally in place with OSGi's bundles. Every bundle (pair) forms a module for the system.
  • Well defined interfaces to the Core bundles are provided by the OSGi service interfaces that we write.
  • About refractoring modules: It is true that module refractoring may be more expensive than a class refractoring, but when a module design mistake is found, it is better to address it by changing the module then by finding ways around it.

Domain Design Patterns

Managing the life cycle of a domain object constitutes a challenge in itself, and if it is not done properly, it may have a negative impact on the domain model. Domain design patterns will help us deal with this.

Aggregates

  • Aggregate is a domain pattern used to define object ownership and boundaries.
  • Aggregates come to play when domain objects are associated with one another, creating a complex net of relationships. Associations can be one-to-one, one-to-many, and many-to-many.
  • Many-to-many, bi-directional associations increases complexity. Reducing the complexity is very important. Tips:
    • associations which are not essential for the model should be removed. They may exist in the domain, but they are not necessary in our model, so take them out.
    • multiplicity can be reduced by adding a constraint. If many objects satisfy a relationship, it is possible that only one will do it if the right constraint is imposed on the relationship
    • many times bidirectional associations can be transformed in unidirectional ones.
  • An Aggregate is a group of associated objects which are considered as one unit with regard to data changes.
  • Each Aggregate has one root. The root is an Entity, and it is the only object accessible from outside. The root can hold references to any of the aggregate objects, and the other objects can hold references to each other, but an outside object can hold references only to the root object. External objects cannot hold direct references to the objects in the aggregate for it to be directly changed, thus enforcing integrity.
  • It is possible for the root to pass transient references of internal objects to external ones, with the condition that the external objects do not hold the reference after the operation is finished. The external objects should not be able to modify the original aggregate objects.
    • One simple way to do that is to pass copies of the Value Objects to external objects.
    • Pass on interface references that does not have object modifiers.
  • If objects of an Aggregate are stored in a database, only the root should be obtainable through queries. The other objects should be obtained through traversal associations.
  • Objects inside an Aggregate should be allowed to hold references to roots of other Aggregates.

Factories

  • Factories gain importance when:
    • creation of object is a major operation in itself.
    • Knowledge needed to create an object needs to be encapsulated from the client objects that need them.
  • Especially useful to create aggregates. When the root of the Aggregate is created, all the objects contained by the Aggregate are created along with it, and all the invariants are enforced.
  • Good to have separate factories for each kind of objects. But related objects should be created in one factory.
  • Object creation process needs to be atomic. Object creation should enforce all its invariants (rules), and should only produce valid objects (its states). If an object cannot be created properly, an exception should be raised, making sure that an invalid value is not returned.
  • Factory: Therefore, shift the responsibility for creating instances of complex objects and Aggregates to a separate object, which may itself have no responsibility in the domain model but is still part of the domain design. Provide an interface that encapsulates all complex assembly and that does not require the client to reference the concrete classes of the objects being instantiated. Create entire Aggregates as a unit, enforcing their invariants.
  • Patterns: e.g,. Abstract Factory, Factory Method
  • Warning!:When creating a Factory, we are forced to violate an object's encapsulation, which must be done carefully. Whenever something changes in the object that has an impact on construction rules or on some of the invariants, we need to make sure the Factory is updated to support the new condition.
  • Types:Entity Factories and Value Object Factories needs to be differentiated.
    • Values objects are usually immutable, and all the necessary attributes need to be produced at the time of creation. When the object is created, it has to be valid and final. It won't change. Value objects don't need identity.
    • Entities are not immutable. They can be changed later, by setting some of the attributes with their invariants respected. Entities need identity.
  • Use Constructor instead of factory when:
    • The construction is not complicated.
    • The creation of an object does not involve the creation of others, and all the attributes needed are passed via the constructor.
    • The client is interested in the implementation, perhaps wants to choose the Strategy used.
    • The class is the type. There is no hierarchy involved, so no need to choose between a list of concrete implementations.

Repositories

  • Need:
    • When client objects need references to pre-existing domain objects, it can obtained by accessing the database, where they can be stored. But directly accessing it from the database meas muddling with the infrastructure, and the such access logic scattered throughout the entire domain, compromising the domain model.
    • If infrastructure is simple to pull out data from the database (Hibernate), domain logic moves into queries and client code, and the Entities and Value Objects become mere data containers. The overall effect is that the domain focus is lost and the design is compromised.
  • Repository:
    • Encapsulate all the logic needed to obtain object references.
    • The domain objects won't have to deal with the infrastructure to get the needed references to other objects of the domain. They will just get them from the Repository and the model is regaining its clarity and focus.
    • Encapsulates the persistence mechanism.
  • Repositories may:
    • Cache domain object references
    • use a strategy to switch between various persistence storage mechanism.
    • on request from client, reconstitute the domain object from - cache, various repositories, or from a factory.
  • Repository should have a well known interface that will provide access to add, remove and searching objects based on criteria. This will encapsulate actual add, remove, and querying techniques. This will keep the client focused on the model, delegating all object storage and access to the Repositories.
  • An Entity can be easily specified by passing its identity. Other selection criteria can be made up of a set of object attributes.
  • The implementation of a repository can be closely liked to the infrastructure, but that the repository interface will be pure domain model.

Factory vs Repository

  • They are both patterns of the model-driven design, and they both help us to manage the life cycle of domain objects.
  • While the Factory is concerned with the creation of new objects, the Repository takes care of already existing objects.
  • When a new object is to be added to the Repository, it should be created first using the Factory, and then it should be given to the Repository which will store it like in the example below.

Refractoring Towards Deeper Insight

Continuous Re-factoring

  • Re-factoring the code is necessary and should be a continuous process.
  • Re-factoring is the process of redesigning the code to make it better without changing application behavior.
  • Re-factoring is usually done in small, controllable steps, with great care so we don't break functionality or introduce some bugs. Automated tests are of great help to ensure that we haven't broken anything.
  • Apart from code re-factoring for technical reasons, there should be emphasis on re-factoring for domain model.
  • Sometimes there is new insight into the domain, something becomes clearer, or a relationship between two elements is discovered. All that should be included in the design through re-factoring.
  • Usually initial models from specification are shallow model, without much clarity on the domain. But the design should be flexible enough to incorporate changes as more clarity on the domain is obtained.

Bring Key Concepts to Light

  • When re-factoring, try to bring the implicit concepts as explicit ones.
  • Looking for implicit concepts: The language we are using during modeling and design contains a lot of information about the domain. At the beginning it may not be so much, or some of the information may not be correctly used. Some of the concepts may not be fully understood, or even completely misunderstood. This is all part of learning a new domain. But as we build our Ubiquitous Language, the key concepts make their way into it. That is where we should start looking for implicit concepts.
  • Watch out for hidden concepts and contradictions in the concepts when doing re-factoring.

Explicit Constraints, Processes, and Specifications

  • A Constraint is a simple way to express an invariant (business rule). This should be put as an explicit method so that its clearly visible. There is also room for growth adding more logic to the methods if the constraint becomes more complex.
  • Processes are usually expressed in code with procedures. The best way to implement processes is to use a Service. If there are different ways to carry out the process, then we can encapsulate the algorithm in an object and use a Strategy. Make only those process as explicit that are mentioned in the Ubiquitous language.
  • Specification is used to test an object to see if it satisfies a certain criteria. These are again, business rules, that operate against the domain data (from entity or value objects)
Rules
  • The domain layer contains business rules which are applied to Entities and Value Objects. Those rules are usually incorporated into the objects they apply to.
  • When business rules become complex to be stored into the objects they apply to, the rule should be encapsulated into an object of its own, which becomes the Specification for the object to which it applies, and should be kept in the domain layer.
  • Often a single Specification checks if a simple rule is satisfied, and then a number of such specifications are combined into a composite one expressing the complex rule.
             Specification customerEligibleForRefund = new
                             Specification(
                   new CustomerPaidHisDebtsInThePast(),
                   new CustomerHasNoOutstandingBalances());
             if(customerEligibleForRefund.isSatisfiedBy(customer)
             {
                   refundService.issueRefundTo(customer);
             }
      

Preserving Model Integrity

Bounded Context

  • Splitting the domain into multiple models, each having its own context, and each model having a well defined contract with other models.
  • Good for large teams working on a project. Instead of one stepping on others toe, bring in the models such that independent teams can work on their own model, thereby splitting the team to work independently on the modules.
  • Still, everyone should have the big picture in mind.

Continuous Integration

  • Even when a team works in a Bounded Context there is room for error.
  • If one does not understand the relationships between objects, they may modify the code in such a way that comes in contradiction with the original intent. It is easy to make such a mistake when we do not keep 100% focus on the purity of the model.
    • One member of the team might add code which duplicates existing code without knowing it, or they might add duplicate code instead of changing the current code, afraid of breaking existing functionality.
  • A model is not fully defined from the beginning. It is created, then it evolves continuously based on new insight in the domain and feedback from the development process. That means that new concepts may enter the model, and new elements are added to the code. This is why we need continuous integration.
  • We need a process of integration to make sure that all the new elements which are added fit harmoniously into the rest of the model, and are implemented correctly in code.
    • The sooner we merge the code the better. For a single small team, daily merges are recommended.
    • We also need to have a build process in place. The merged code needs to be automatically built so it can be tested.
    • Another necessary requirement is to perform automated tests. If the team has a test tool, and has created a test suite, the test can be run upon each build, and any errors are signaled. The code can be easily changed to fix the reported errors, because they are caught early, and the merge, build, and test process is started again.

Anti Corruption Layer

  • A layer between two separate but related systems. The two systems depend on each other closely. This layer will ensure that the two systems evolve independently of each other, while the ACL layer will manage the complexities that may come when the two systems evolve.
  • Sort of a two way adapter?

Separate Ways Pattern

  • The Separate Ways pattern addresses the case when an enterprise application can be made up of several smaller applications which have little or nothing in common from a modeling perspective.

Thursday, March 26, 2009

OSGi Discontent

Nice post on OSGi and its current problems on enterprise world.

http://java.dzone.com/news/osgi-discontent-no-migration

This was something that I was planning to make a post from a long time, that I' ve been facing such problems myself. But I was pushing this on from a long time for lack of time.