Commons dbcp creating connection issues, CLOSE_WAIT and java.lang.Thread.State: WAITING

Recently I was working on a application that was using dbcp  version 1.4 for connection pools and was running in tomcat 7. And after sometime of working with the application we found out that lots of connections related to the tomcat were ending up in CLOSE_WAIT.Here is a sample of the connections with tomcat pid as 1234:

[email protected]tem:/$ sudo netstat -anp|grep  1234
tcp6       0      0 :::8080                 :::*                    LISTEN      1234/java
tcp6       1      0 127.0.0.1:8080          127.0.0.1:6233         CLOSE_WAIT  1234/java
tcp6       1      0 127.0.0.1:8080          127.0.0.1:6233         CLOSE_WAIT  1234/java

After some time the situation was as worse as below, all the connection in CLOSE_WIAT

[email protected]:/$ sudo netstat -anp|grep 1234|grep CLOSE_WAIT|wc -l
199

"ajp-bio-8080-exec-50" - Thread [email protected]
   java.lang.Thread.State: WAITING
 at java.lang.Object.wait(Native Method)
 - waiting on <a2658bd> (a org.apache.commons.pool.impl.GenericObjectPool)
 at java.lang.Object.wait(Object.java:503)
 at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:793)
 at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
 at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)

initialSize=5
maxTotal=50
maxIdle=10

minIdle=5
maxWait=10000

We set maxWait property to 10 seconds. So no more hanging connections. And since then we didn’t have any of CLOSE_WAIT connections.  But we now had a problem that lots of new connections were not able get the connection at all. This was the next error:

 org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection

So we started to look at the root cause of the issue, which was pointing to a connection pool exhaustion. The developer that wrote the code didn’t close the connections properly. On every error the connection was not closed and hence leading to connection pool running out of connections after some days. So the solution was to close the connection in the finally block. It was as simple as that.

public void saveMyData(ListList<String> mydata)  throws MyException { 
 Session session = null;
 try {
  session = this.sessionFactory.openSession();
  session.beginTransaction();
  // your code goes here
  session.getTransaction().commit();
 } catch (Exception e) {
  logger.error(e);
  rollbackQuietly(session);
  throw new MyException(e);
 } finally{
  closeQuietly(session);
 }
}

// the exception what got it here has the precedence so preserving that exception by rolling back quietly
private void rollbackQuietly(Session session) {
 if (session != null) {
  try {
   session.getTransaction().rollback();
  } catch (Exception e) {
   logger.error(e);
  }
 }
}
// the exception what got it here has the precedence so preserving that exception by closing connection quietly
private void closeQuietly(Session session) {
 if (session != null) {
  try {
   session.close();
  } catch (Exception e) {
   logger.error(e);
  }
 }
}

So the whole scenario was like this.

1. DB connections were leaked by the ill written code.
2. After some time all the db connections were leaked
3. Then any new attempt to access the connection led to the indefinite waiting on the server leading to CLOSE_WAIT connections to the tomcat and hence making all tomcat connections exhausted.
4. Using maxWait helped to atleast release the tomcat connections and tomcat was responsive again.
5. And fixing the root cause of connection leakage let the application run again in stable mode.