A.J.

For the past week or so we have had a company (which will remain nameless) reporting that their application server (Oracle Application Service aka oc4j) has been reporting loads of 404 hits in our application.

They are seeing this inside of Enterprise Manager’s top servlets page. The effected servlet is com.evermind.servlet.http.NotFoundServlet. The first thing I think of is.. crap, we have some broken links somewhere. Yet, looking at our application with Firebug doesn’t show any 404s at all.

At this point I decide to decompile oracle’s NotFoundServlet and see what it is doing. All it actual does is a sendError 404, like the name implies it should. I decide to add some more debugging info, compile, and shove it back into oc4j. I have to override the existing oc4j-internal.jar to do this. The only thing NotFoundServlet reports are urls that I enter en by hand that are invalid. We give the information to the client so they can see that there is indeed no missing resources in our app even though the top servlets page shows NotFoundServlet being hit. The client decides to go live anyway.

A week goes by, the client’s server is having some performance issues. Though our support department claims it is not physically possible that the NotFoundServlet reporting could be the issue and mentions a few other fixes, the client insists that it is. The client later goes on to cuss out one of our sales representatives, but that is a different story.

Anyways, after a couple of conversions with support manager (Kevin) and the vp of engineering (Bill) I begin to think that the problem may be that the oc4j admin screen must always report a servlet or file being hit. This isn’t the case in clearspace. Clearspace uses webwork. Webwork as of 2.2 is implemented on top of a Filter instead of a Servlet. Many requests in webwork never hit a servlet, because we use freemarker instead of JSPs. With freemarker the filter outputs the content directly instead of fowarding on to a JSP.

To test my theory I create a webapp that has one filter, and one servlet. The filter is mapped to /filtertest/*, the servlet to /servlettest/*. The filter simply grabs the response’s output stream and prints “You hit the filter”, the servlet behaves in a similiar fashion.

When hitting the servlet Enterprise Manager reports my test servlet being hit, this is ok and good. Theoretically Enterprise Manager should not display anything in the top servets when my filter is hit. When I hit the filter and I see request count against com.evermind.servlet.http.NotFoundServlet increase, thus proving my hypothesis to be correct (wtf!).

Bill then has the idea of actually sticking something something physically in the war file of my tests app’s path for a specific filter request. We add the jsp /filtertest/foo.jspa (note that it is indeed under the filter mapping). The file only contains the content “You should not see this”. We hit the url, the filter displays correctly, but this time NotFoundServlet’s count does not increment.

So apparently, oc4j likes you to have one of the following:
1. A Servlet.
2. A real file inside the war file.

btw, We a are seriously thinking of dropping support for oc4j