Table of Contents
Whenever Tomcat fails, always check your application logfiles and
the Tomcat log files at $CATALINA_BASE/logs
.
One easy-to-recognize kind of error which shows up in the Tomcat
logs are out-of-memory errors.
These may be due to the cumulative expenditures of memory due to
bad deployments, etc., since your last Tomcat startup.
In that case, just restart Tomcat (often, in this situation, you
will need to kill the lead pid yourself, and sometimes all of the
children pids).
If the problem occurs after a clean restart and just running your
apps normally, then your web apps require more RAM to run.
In that case, set the environmental variable
JAVA_OPTS
(like to
-Xmx512m
to increase from the default of 64M
to 512M).
Where to set this env. variable depends on how you start Tomcat.
If you use the Suse init script, you would set it in
/etc/sysconfig/j2ee
.
If you use my init script, you can set the default for all
Tomcat instances in the init script itself, or for specific
Tomcat instances in
$CATALINA_BASE/instance.env
.
The most annoying kind of deployment failures are
partial failures.
Tomcat is really stupid about items in the
webapps
directory which it doesn't think
are webapps.
It will entirely ignore them.
If you are a remote user, you will have no idea that the item
is even there.
If there happens to be a remnant from a previous incomplete
undeployment, or if a previous deployment with that name
partially failed for you or somebody else, then it will be in
the webapp directory but Tomcat won't see it.
When you try to deploy a webapp with the same name, it will
choke because it will try to create the file or directory when
there is already one there with the same name.
It will either not tell you there is any problem at all, or it
will just say that the deployment failed.
The fix is to manually remove the unrecognized file or directory
from the webapps
directory.
If your webapps consume more than the max JVM RAM (which defaults to 64M), that sometimes looks like a deployment problem. Check the Tomcat logs and read the previous section.
Another problem that can prevent webapps from starting is if your
webapp requires a Graphics context.
There are many third party libraries that require a graphics
context, most notably JasperReports (and probably JFreeChart).
I forget the exact error message that shows up in the logs
(it is not intuitive). I'll add that here the next time I
encouhter it.
But the fix is just to add the JAVA_OPTS
-Djava.awt.headless=true
.
See the previous section about where to set
JAVA_OPTS
.
(For ancient JVM's, you used to run graphics emulators like
Xvfb. Luckily the headless option works better and is much
easier to set up.).
Tomcat on Windows has a terrible time releasing Jar files. Make sure that you are not "tailing", "moring", or otherwise accessing log files or anything else to do with a web app when you attept to stop or deploy it. What will frequently happen is that resources within the web app will get completely stuck and you won't be able to remove them. Attempting to remove them while Tomcat is still running can make things get so bad that even shutting down Tomcat won't help and you'll have to reboot. The problem with not being able to remove these files is that you can't redeploy the application.
The Manager webapp does very inadequate validation of the
names that you supply.
If you accidentally paste a space along with a war file name,
the deployment will fail with no error message anywhere near
to close to indicating the real cause.
Similarly with other naming problems.
If you suspect this kind of problem, you really need to get
on the server and see what is really in the
webapps
directory.
"Reload" does not re-read the web.xml file from applications. You must stop and start the web app to do that.
The Admin app realy has a crappy persistence implementation. Often you will say to commit, and you will get an error message. I've found that things work much better if you take your time and make sure that operations complete before doing something else (including Committing).
Commits in the Admin web app often cause restarts of effected web apps, including the Admin app itself! After you hit Commit, just wait a couple minutes for the Commit to complete. It's likely that the Admin web app will have restarted and you will get an authorizatin failure when you try to do anything else. Just log back in again.
If you do local deployments by copying or moving war files or
webapp directory branches into the webapps
directory, then you are susceptible to directory polling race
condition problems... and you probably will encounter them.
The solution is to use a
Webapp Staging Directory.
By stuck, I mean that the Tomcat processes are running, but you can't get it to do anything for any web app. (Sometimes you get empty pages back, but usually the server will accept requests but will not reply at all). This is an extremely common problem with Tomcat. I have never encountered any app server that will not sometimes get stuck when running bad applications, but Tomcat is pretty bad in this respect. Over many years of seeing this behavior, it seems to happen either (a) upon deploying a problematic web application (or at the next Tomcat startup after said deployment), or (b) when a webapp runs and throws exceptions (which you can see in the Tomcat log files). I have always been able to recover from these problems by completely clearing the problematic webapp(s) and re-deploying them, as described here.
If you are running Tomcat 5.5.x, and it chronically gets stuck upon startup, then switch to Tomcat 5.0.x. By chronically, I mean, you follow the procedures below, but the problem keeps recurring. I know that this can happen with Tomcat 5.5.x and Java 1.4 with large or troublesome webapps. (With the Java 1.4 patch, of course). I don't know about the reliability of Tomcat 5.5.x with Java 1.5.
This section assumes that you have previously set up a Webapp Staging Directory.
The first recovery strategy is to restart Tomcat. There's about a 50% chance that Tomcat won't shut down properly. See the Tomcat won't shut down! section if that happens.
If things are still stuck after a restart, you need to identify the problematic webapp. If Tomcat gets stuck immediately upon startup, then you can find the last webapp start Tomcat attempted to start up in the Tomcat logs. If Tomcat gets stuck when you actually hit a webapp, then that's the one.
Shut down Tomcat completely again.
Wipe out the work directory for the problematic webapp.
The work directory resides at a location like
$CATALINA_BASE/work/Catalina/<HOSTNAME>/<APPCONTEXTROOT>
.
If you deployed a war, wipe out the exploded directory branch
$CATALINA_BASE/<APPCONTEXTROOT>
.
Move your deployed item (a war file, or a webapp directory) from
$CATALINA_BASE/webapps/<APPCONTEXTROOT>
to
inside of your webapp stage directory
$CATALINA_BASE/webapps/stage/<APPCONTEXTROOT>
.
Start up Tomcat and verify that everything runs ok (besides the
webapp which you have manually undeployed).
If another webapp causes problems, repeat the procedure above for
other problematic webapps.
Once things are happy, manually redeploy the webapp by moving it
back from the stage directory to the main webapps directory.
I.e., from
$CATALINA_BASE/webapps/stage/<APPCONTEXTROOT>
to $CATALINA_BASE/webapps/<APPCONTEXTROOT>
.
Restart Tomcat.
If things fail after you redeploy the webapp, then this webapp has serious internal problems which need to be fixed before it will run on Tomcat. There should be exception stack traces in the Tomcat web logs and/or clues in your application logs.
After you shut down Tomcat in any way, do a ps and see if
there are remaining "java" processes owned by your Tomcat
owner.
If so then telnet to the SHUTDOWN port on localhost, type
"shutdown" and hit ENTER.
(The shutdown port is listed in the file
$CATALINA_BASE/conf/server.xml
.
Check the processes again.
You're on your own doing this in Windows.
In UNIX, use ps (or a variant) to find the parent "java" process
owned by your Tomcat owner.
If you are starting Tomcat with an init script, the ppid
of this process leader will be 1.
Send the TERM signal (not KILL or INT or anything else) to this
process.
The processes should all quit in about a second.
(Depending on your operating system, your JVM, and even your
JVM settings, you may see one process per thread, or just
one for each JVM).
LAST RESORT. If the TERM signal doesn't stop Tomcat, then send the KILL signal to the process leader. You want to avoid this because it doesn't give the threads a chance to gracefully shut down, but more importantly, because it's more likely that the process leader will not shut down the children. In that case, you will have to hunt down all of the children and kill them with TERM or KILL (the former if it works, otherwise the latter).