This project has been slowly gathering momentum on my laptop, as both my notes and stories at clients sites fill up my hard drive. I think it's about time someone shed some light on the 'secret' part of the product stack: the embedded application server. Sure, we could treat the server as a 'black box', but if you're like me you want to reach your hand inside and feel around a bit to see what's there. I'm pretty sure Schrodinger wouldn't have wanted me hanging around his cat. It seems funny that something which can both single handedly make or break your portal performance numbers and has so much room for configuration seems to have received so little attention in the BEA documentation.
In order to address some of the questions I generally get in the field, as well as earn some ALUI street cred, I'll try to give a decent picture of what actually happens under the covers when you start up almost any ALUI product, and what you can do to manipulate that startup. Oh yeah, I guess now would be a good time to mention now that none of this should be taken as an official recommendation from BEA. Proceed at your own risk.
ALUI Application Arichtecture
Below you'll find a very rough outline of how the ALUI embedded application server works. In the rest of this post, I'll deconstruct the application server piece by piece. For specific examples, I'm going to choose the ALUI API Service (located in $ALUI_HOME/ptws/). Also, since this example is taken from my WinXP laptop, some of the terminology will be in Microsoft-ese. Translating it to a Solaris/Linux environment should be a snap.
The Tanuki Software Wrapper
At the end of the day, the initial bootstrap for any ALUI embedded application server is the Tanuki Software application wrapper. The wrapper itself is an open source java service wrapper that allows application developers to rapidly embed logging, support for services, and some other features into their applications. Knowing this, of course, we can start investigating the Tanuki Software website for more information on the wrapper: http://wrapper.tanukisoftware.org/.
We can use what we know about the wrapper to establish a foothold in the mountain of configuration files in our alui directory. Looking at the apiserviced.bat file yields the wrapper executable, as well as the location of the wrapper configuration files. In the case of the ws server, we're looking at the following:
set _WRAPPER_CONF="%_REALPATH%..\settings\config\wrapper.conf
Thus, the bulk of the wrapper's startup configuration comes from the wrapper.conf file, and if we look at the bottom line of wrapper.conf, the wrapper_base.conf file (also found in the settings/config directory of the API web service.
Using this knowledge, we'll look at some of the wrapper's configuration settings to figure out what's going on and how we can manipulate it to our advantage.
Wrapper Logging
The first thing any good administrator will wonder about an application is: how can I figure out what it's doing? In the case of the wrapper, we can amp up the its log settings by changing the wrapper.console.loglevel, wrapper.logfile.loglevel, and wrapper.syslog.loglevel configuration parameters (valid settings for these parameters are enumerated in the file). What this will do is allow us to see what the console and error output of the embedded application server is, as well as monitor what happens to the wrapper itself. Unfortunately, we won't be able to see much more than that, since the loglevel settings only control the wrapper itself, not the embedded application.
That said, startup problems with the embedded application server, as well as the wrapper's JVM running periodic pings to make sure the application is still running, will all show up here. Cranking the loglevels to DEBUG/INFO and doing a tail -f on the log file specified by wrapper.logfile during startup is always a good idea.
Which brings us to controlling logfile location and sizes. As I stated, wrapper.logfile will allow you to control where logfiles are created. This, along with the loglevel settings, wrapper.logfile.maxsize, and wrapper.logfile.maxfiles should allow you to keep your logs under control (again, see comments in the batch file for more detail).
But Where is the Server?
You may have noticed that we have a whole bunch of configuration files for each product, but no server. What gives? You may have noticed some lines in the wrapper_base.conf file that start like this:
%COMMON_PATH%/container/
That path can usually be found in the $ALUI_HOME/common directory. As you will see if you look at this directory on a few ALUI servers, there is more than one embedded application server. Most products actually use Tomcat, but Publisher uses JBoss (again, see the config files for more details). For the purposes of this post, I'm going to shy away from JBoss configuration and focus on Tomcat.
ALUI's Tomcat Bootstrap
Our next stop on this tour de force of the wrapper configuration is the wrapper.app.additional parameters located in the wrapper_base.conf file. These parameters are the most important of all the wrapper parameters, since they are the arguments passed to the java instance executing the application server. In essence, they determine what program will actually be launched by the application server. These parameters are listed sequentially, with a number appended to each additional argument.
Those familiar with the ALUI portal products for any length of time will know that they came from a little company called Plumtree Software. Back in the day, *everything* was called Plumtree. Nowadays, we only get to see plumtree branding when we look under the covers. I bring this up because you'll notice a bit of Plumtree branding in application parameter 1: com.plumtree.container.Bootstrap.
What does this do? Rather than go right out and launch an application server with the Tanukisoft wrapper, ALUI developers created their own boostrap class to handle the heavy lifting for us. One of the commented arguments you'll notice in the batch file is the original Tomcat startup bootstrap: org.apache.catalina.startup.Bootstrap.
What this bootstrap really does, that the Tomcat bootstrap does not, is read a very simple configuration file and generate a server.xml instance to pass to the Tomcat bootstrap (for those not familiar with Tomcat, server.xml is the configuration file used to configure Tomcat). In other words, Tomcat has been dumbed down a bit. The simple configuration file I'm talking about is actually set up as a java system property in the wrapper.java.additional parameters specified in wrapper_base.conf:
wrapper.java.additional.5=-Dplumtree.application="%APPLICATION_PATH%"
wrapper.java.additional.6=-Dplumtree.application.config="%APPLICATION_PATH%/settings/config"
These parameters tell the bootstrap loader where to look for its configuration. The loader then grabs application.xml, parses it to create a server.xml file, writes the server.xml to the same directory as application.xml and starts the tomcat bootstrap with a parameter indicating where the server.xml file is located. If you want to see this in action, delete any server.xml in the application.xml directory and restart your ALUI application of choice. Magically, a new server.xml will appear (provided the wrapper started up okay).
Why this is good: It's simple. All you have to do to configure the application server is edit application.xml and restart the server.
Why this is not so good: Now we are locked into a server.xml configuration for our Tomcat instance. Every time we restart, we are forced to accept the default generated server.xml with a few minor customizations.
What if, for example, we want to bind our Tomcat server to a specific network adapter, rather than the same port across all IP's (for instance, if there is a port conflict with another application)? Tomcat allows this functionality, but the way the application server works, this particular scenario would seem all but impossible to accommodate.
Of course, if that were true then why would I have mentioned it? By being a little sneaky, and knowing a bit about how the bootstrap loader works, we can create and customize our own server.xml. Before you go about this, remember my at your own risk provision at the top of this post.
In order to customize, we first have to start the application server once, so that we can work with a 'stock' server.xml. This is the one that has the big ***THIS FILE IS AUTOMATICALLY CREATED DO NOT MODIFY IT***. Did I mention my disclaimer at the top of this article?
Once we have this file generated, we have to figure out two things:
- How do we stop the bootstrap from generating this file?
- How do we tell the application server where our modified server.xml is?
First let's deal with #1: preventing the bootstrap loader from generating the server.xml. It turns out this is surprisingly easy. Simply don't tell the bootstrap where the application settings are. In other words, just leave off the last java additional parameter. In our example, I took out the following line:
wrapper.java.additional.6=-Dplumtree.application.config="%APPLICATION_PATH%/settings/config"
Now for the second part of the equation: telling tomcat where to find our modified server configuration. Since this is Tomcat, it turns out to be a snap as well. All we have to do is change our wrapper_base.conf arguments to pass the server.xml as a configuration parameter, in much the same way the bootstrap loader would have. Here are the relevant lines from my wrapper_base.conf file:
wrapper.app.parameter.1=com.plumtree.container.Bootstrap
wrapper.app.parameter.2=3
wrapper.app.parameter.3=-config
wrapper.app.parameter.4=%APPLICATION_PATH%/settings/config/server.xml
wrapper.app.parameter.5=start
wrapper.app.parameter.6=org.apache.catalina.startup.Bootstrap
wrapper.app.parameter.7=true
wrapper.app.parameter.8=1
wrapper.app.parameter.9=stop
Notice a few things here:
- The -config parameter is passed as the third argument, the actual configuration location is the fourth.
- I had to modify parameter 2, which tells the Tanuki Software wrapper how many arguments I'm passing to the bootstrap (it was 1 in the original file).
That's it. Our next step is to test the configuration by changing changing something in server.xml, leaving application.xml alone and seeing what happens when we start the application server. I chose to change the port attribute of the Connector element. After I started my application server I simply checked for the server listening on the new port (11900 in my case) by running:
netstat -a | grep 11900 (if you don't have grep, you can try telnet localhost 11900)
My results were satisfactory:
TCP xxxxxxxx:11900 0.0.0.0:0 LISTENING
Solaris Users, Read On...
As a consequence of this blog post, I was fortunate enough to get an opportunity to try this on Solaris. Unfortunately, my results were a little less satisfactory than with the Windows version of the embedded application server. It turns out there is a nasty check based on the os.name java system property that attempts to remove quote characters from all paths passed to the bootstrap. Unfortunately, the check can't handle null strings. This means that leaving out wrapper.java.additional.6 is not really an option.
Thanks to some dialog with Brian Hak, and a bit of playing around, there is a solution. Simply leave the configuration as-is and pass an additional server config to the bootstrap (with a different path than the server.xml that is generated automatically). The catch here is that you must pass the -config argument before the start flag in the wrapper.additional parameters (otherwise you'll get classpath errors). In a nutshell, wrapper.java.additional doesn't change from the original, but wrapper.app.parameters now become:
wrapper.app.parameter.1=com.plumtree.container.Bootstrapwrapper.app.parameter.2=3
wrapper.app.parameter.3=-config
wrapper.app.parameter.4=%APPLICATION_PATH%/settings/config/newServer.xml
wrapper.app.parameter.5=start
wrapper.app.parameter.6=org.apache.catalina.startup.Bootstrap
wrapper.app.parameter.7=true
wrapper.app.parameter.8=1
wrapper.app.parameter.9=stop
Using the Wrapper to Tune the JVM
At one point in this article I mentioned that the embedded application server was also a gold mine of performance improvement. I could fill an entire post up with Tomcat performance tuning advice, but I think it might be more beneficial just to mention a few things to follow up with here. More specifically, there are a few arguments we can pass to the JVM, using the wrapper.java.additional arguments which can help us tune its garbage collection to optimal performance. More specifically, we can do things like log garbage collection via -Xloggc, add the -server flag to our JVM, etc.. For more information, here's an interesting article about Java Garbage Collection Tuning.
Hints, Allegations and Things Left Unsaid
As you can tell, there is a lot we can do with the embedded application server that isn't described in the instruction manual. As such, here are a few more things to think about before I sign off: it might be possible to change the entire embedded application server, JVM (major or minor revision), or even host out of a non-embedded environment. These are all exercises that, of course, would be unsupported by your friendly BEA helpdesk, but they may be of interest for the more enterprising, or more curious, reader.
Leave a comment