It seems there were a lot of points in our code base at work that are ripe for deadlocking on database connections. We are acquiring a connection, and then calling some other methods, and a few frames down the stack, we acquire another connection (before releasing the first). This, of course, can result in a deadlock, and it will only be detectable during PERF testing, which is always a bad time to discover a whole category of bugs in the code.
These deadlocks manifest themselves as
ConnectionWaitTimeoutExceptions, with stack traces like this:
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180003 at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1615) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2422) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:937) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:611) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:449)
So, my new trick, which I will try to use on a regular basis going forward in my everyday development, is to set a conditional breakpoint in a wrapper class that my project uses anytime it acquires a connection, so that any time a connection is opened, and there is already an open connection, it will suspend. Of course, this means that only one user can be using the app at a time, but for a local development environment, that is usually okay. You can also just reduce the size of your connection pool to 1, and see if anything blows up, but (IMHO) the breakpoint makes it easier to detect where each of the connections are coming from so you can figure out how to fix the problem.