Chapter 8. Tomcat Quirks

Table of Contents

Typical run-time failures
Deployment failures
Manager webapp Quirks
Administrative App Quirks
Manual deployment failures
Tomcat is stuck!
Tomcat won't shut down!
Email Config Problems

Typical run-time failures

Whenever Tomcat fails, always check your application logfiles and the Tomcat log files at $CATALINA_BASE/logs. One easy-to-recognize kind of error which shows up in the Tomcat logs are out-of-memory errors. These may be due to the cumulative expenditures of memory due to bad deployments, etc., since your last Tomcat startup. In that case, just restart Tomcat (often, in this situation, you will need to kill the lead pid yourself, and sometimes all of the children pids). If the problem occurs after a clean restart and just running your apps normally, then your web apps require more RAM to run. In that case, set the environmental variable JAVA_OPTS (like to -Xmx512m to increase from the default of 64M to 512M). Where to set this env. variable depends on how you start Tomcat. If you use the Suse init script, you would set it in /etc/sysconfig/j2ee. If you use my init script, you can set the default for all Tomcat instances in the init script itself, or for specific Tomcat instances in $CATALINA_BASE/instance.env.

Deployment failures

The most annoying kind of deployment failures are partial failures. Tomcat is really stupid about items in the webapps directory which it doesn't think are webapps. It will entirely ignore them. If you are a remote user, you will have no idea that the item is even there. If there happens to be a remnant from a previous incomplete undeployment, or if a previous deployment with that name partially failed for you or somebody else, then it will be in the webapp directory but Tomcat won't see it. When you try to deploy a webapp with the same name, it will choke because it will try to create the file or directory when there is already one there with the same name. It will either not tell you there is any problem at all, or it will just say that the deployment failed.

The fix is to manually remove the unrecognized file or directory from the webapps directory.

If your webapps consume more than the max JVM RAM (which defaults to 64M), that sometimes looks like a deployment problem. Check the Tomcat logs and read the previous section.

Another problem that can prevent webapps from starting is if your webapp requires a Graphics context. There are many third party libraries that require a graphics context, most notably JasperReports (and probably JFreeChart). I forget the exact error message that shows up in the logs (it is not intuitive). I'll add that here the next time I encouhter it. But the fix is just to add the JAVA_OPTS -Djava.awt.headless=true. See the previous section about where to set JAVA_OPTS. (For ancient JVM's, you used to run graphics emulators like Xvfb. Luckily the headless option works better and is much easier to set up.).

Tomcat on Windows has a terrible time releasing Jar files. Make sure that you are not "tailing", "moring", or otherwise accessing log files or anything else to do with a web app when you attept to stop or deploy it. What will frequently happen is that resources within the web app will get completely stuck and you won't be able to remove them. Attempting to remove them while Tomcat is still running can make things get so bad that even shutting down Tomcat won't help and you'll have to reboot. The problem with not being able to remove these files is that you can't redeploy the application.

Manager webapp Quirks

The Manager webapp does very inadequate validation of the names that you supply. If you accidentally paste a space along with a war file name, the deployment will fail with no error message anywhere near to close to indicating the real cause. Similarly with other naming problems. If you suspect this kind of problem, you really need to get on the server and see what is really in the webapps directory.

"Reload" does not re-read the web.xml file from applications. You must stop and start the web app to do that.

Administrative App Quirks

The Admin app realy has a crappy persistence implementation. Often you will say to commit, and you will get an error message. I've found that things work much better if you take your time and make sure that operations complete before doing something else (including Committing).

Commits in the Admin web app often cause restarts of effected web apps, including the Admin app itself! After you hit Commit, just wait a couple minutes for the Commit to complete. It's likely that the Admin web app will have restarted and you will get an authorizatin failure when you try to do anything else. Just log back in again.

Manual deployment failures

If you do local deployments by copying or moving war files or webapp directory branches into the webapps directory, then you are susceptible to directory polling race condition problems... and you probably will encounter them. The solution is to use a Webapp Staging Directory.

Tomcat is stuck!

By stuck, I mean that the Tomcat processes are running, but you can't get it to do anything for any web app. (Sometimes you get empty pages back, but usually the server will accept requests but will not reply at all). This is an extremely common problem with Tomcat. I have never encountered any app server that will not sometimes get stuck when running bad applications, but Tomcat is pretty bad in this respect. Over many years of seeing this behavior, it seems to happen either (a) upon deploying a problematic web application (or at the next Tomcat startup after said deployment), or (b) when a webapp runs and throws exceptions (which you can see in the Tomcat log files). I have always been able to recover from these problems by completely clearing the problematic webapp(s) and re-deploying them, as described here.

If you are running Tomcat 5.5.x, and it chronically gets stuck upon startup, then switch to Tomcat 5.0.x. By chronically, I mean, you follow the procedures below, but the problem keeps recurring. I know that this can happen with Tomcat 5.5.x and Java 1.4 with large or troublesome webapps. (With the Java 1.4 patch, of course). I don't know about the reliability of Tomcat 5.5.x with Java 1.5.

Note

This section assumes that you have previously set up a Webapp Staging Directory.

The first recovery strategy is to restart Tomcat. There's about a 50% chance that Tomcat won't shut down properly. See the Tomcat won't shut down! section if that happens.

If things are still stuck after a restart, you need to identify the problematic webapp. If Tomcat gets stuck immediately upon startup, then you can find the last webapp start Tomcat attempted to start up in the Tomcat logs. If Tomcat gets stuck when you actually hit a webapp, then that's the one.

Shut down Tomcat completely again. Wipe out the work directory for the problematic webapp. The work directory resides at a location like $CATALINA_BASE/work/Catalina/<HOSTNAME>/<APPCONTEXTROOT>. If you deployed a war, wipe out the exploded directory branch $CATALINA_BASE/<APPCONTEXTROOT>. Move your deployed item (a war file, or a webapp directory) from $CATALINA_BASE/webapps/<APPCONTEXTROOT> to inside of your webapp stage directory $CATALINA_BASE/webapps/stage/<APPCONTEXTROOT>.

Start up Tomcat and verify that everything runs ok (besides the webapp which you have manually undeployed). If another webapp causes problems, repeat the procedure above for other problematic webapps. Once things are happy, manually redeploy the webapp by moving it back from the stage directory to the main webapps directory. I.e., from $CATALINA_BASE/webapps/stage/<APPCONTEXTROOT> to $CATALINA_BASE/webapps/<APPCONTEXTROOT>. Restart Tomcat.

If things fail after you redeploy the webapp, then this webapp has serious internal problems which need to be fixed before it will run on Tomcat. There should be exception stack traces in the Tomcat web logs and/or clues in your application logs.

Tomcat won't shut down!

After you shut down Tomcat in any way, do a ps and see if there are remaining "java" processes owned by your Tomcat owner. If so then telnet to the SHUTDOWN port on localhost, type "shutdown" and hit ENTER. (The shutdown port is listed in the file $CATALINA_BASE/conf/server.xml. Check the processes again. You're on your own doing this in Windows. In UNIX, use ps (or a variant) to find the parent "java" process owned by your Tomcat owner. If you are starting Tomcat with an init script, the ppid of this process leader will be 1. Send the TERM signal (not KILL or INT or anything else) to this process. The processes should all quit in about a second. (Depending on your operating system, your JVM, and even your JVM settings, you may see one process per thread, or just one for each JVM).

Warning

LAST RESORT. If the TERM signal doesn't stop Tomcat, then send the KILL signal to the process leader. You want to avoid this because it doesn't give the threads a chance to gracefully shut down, but more importantly, because it's more likely that the process leader will not shut down the children. In that case, you will have to hunt down all of the children and kill them with TERM or KILL (the former if it works, otherwise the latter).

Email Config Problems

When sending SMTP email, somehow the Properties that you supply to the Session are ignored. To get around this, set Java sys properties (like mail.smtp.host) via $CATALINA_OPTS.