Forking Classloader

From EggeWiki

The other day we were having an issue with a JBoss container, which has about 30 war & ear files deployed (not counting the bundled deployments like Tomcat). One of the ear's occasionally consumes 100% of the containers memory, and we have to bounce the server. We are able to make some educated guesses as to the cause, and have looked into added instrumentation type logging to the container. We may be able to find this problem, but next week a different component could be the cause. We have a medium sized development team, and different teams develop and release components into a set of shared containers.

So when something goes wrong at the container level, it can affect multiple services and be difficult to track down. The obvious solution would seem to be to run fewer components in each container, or run one component per container.

Say we decide to create 30 JBoss instances. Now, the difficulty is managing ports and ensuring that every service can locate every other. This would make one think, maybe we run a pair of HA-JNDI servers, and programmatically create a container for each war/ear file we want to deploy. Unfortunately, not everything can make use of HA-JNDI, so you still have issues like web service and http calls needing to be services by one port. Another solution might be to create 30 VMs, each one running JBoss on the same ports. A dynamic DNS system could allow services to be located. The would require the ability to programmatically create and destroy VMs. 30 sits right between barely manageable manually, and not enough to built an automated system. Something like Amazon's EC2 infrastructure is great, but currently there aren't similar setups which one can manage VMs in house that I know of.

So, when one of our containers goes on the brink, I'm quick to point out that our JBoss has similar isolation characteristics as did Windows 95. I've also said it's just like your browser when you have thirty tabs open, except that you can't play whack-a-mole with your JBoss components.

Now, as of this week, I can't use the browser example, as Google Chrome runs tabs in multiple processes. The unix has had fully isolated processes for years. In Microsoft Windows, processes are much more expensive, so one tries to create fewer of them, but run multiple threads in each process. Because of the overhead which processes have, Windows services are often bundled into the same process space, and can be seen as 'svchost.exe' in one's process manager.

The Java world certainly has been in the one process camp for a while now. Since Java has to run on multiple platforms, it doesn't have the concept of the unix fork, which creates a copy of ones process. About the only place you see forking and java mentioned together is in the Ant tool. Ant tasks can either run in the same process space as Ant, or in a child process. This is especially important for things like unit tests, or tasks which require a lot of memory. Ant 1.6+ has further gone to allow multiple classloaders for specific tasks. This is great when you have one target like Antlr which requires certain libraries in the classpath, and other targets like Checkstyle which require different conflicting resources. The multiple classloaders allow them to exist safely in the same process space.

Even on the faster Solaris or Windows machine, creating a new Java VM takes a noticeable amount of time. In running unit tests you have the option to run each one in it's own forked VM. If you have a reasonable amount of unit tests, this will take a very long time compare to running them in the same process.

So Java can create child Java processes, and even give them a similar environment to the parent. In a J2EE container, each deployed component runs in it's own class loader, and J2EE specs require that data be marshaled and not directly shared between components. My question, is why not write a forked classloader. Instead of a component running in a unique classloader, it could run in it's own process space. It could only use up to the amount of memory of it's own JVM, and could, if misbehaving be killed or frozen.

JBoss already has a 'call-by-value' flag which you can turn off for performance during your install. Since deployment can run 'call-by-value', they should be able to run out of process. It would be great if the JBoss installer asked you if you wanted each deployment to run in it's own process.

install_isolation.png

A forked classloader would make a copy of the parent environment, add the deployment ear/war, and launch itself. It would then need some careful marshaling to be able to get resources from the parent process and to be able to handle requests properly. The forking classloader would likely need to have two parts - an intelligent classloader on the server, which would fork the child process, and a stub classloader on the client, which could get resources from the parent.

The Erlang crowd has long said that shared memory is dead. A forking classloader, although certainly a challenge, would provide additional stability and scalability to the Java platform.