Recently I was working on a application that was using dbcp version 1.4 for connection pools and was running in tomcat 7. And after sometime of working with the application we found out that lots of connections related to the tomcat were ending up in CLOSE_WAIT.Here is a sample of the connections with tomcat pid as 1234:
[email protected]:/$ sudo netstat -anp|grep 1234 tcp6 0 0 :::8080 :::* LISTEN 1234/java tcp6 1 0 127.0.0.1:8080 127.0.0.1:6233 CLOSE_WAIT 1234/java tcp6 1 0 127.0.0.1:8080 127.0.0.1:6233 CLOSE_WAIT 1234/java
After some time the situation was as worse as below, all the connection in CLOSE_WIAT
[email protected]:/$ sudo netstat -anp|grep 1234|grep CLOSE_WAIT|wc -l 199
"ajp-bio-8080-exec-50" - Thread [email protected] java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <a2658bd> (a org.apache.commons.pool.impl.GenericObjectPool) at java.lang.Object.wait(Object.java:503) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:793) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
initialSize=5 maxTotal=50 maxIdle=10
minIdle=5 maxWait=10000
We set maxWait property to 10 seconds. So no more hanging connections. And since then we didn’t have any of CLOSE_WAIT connections. But we now had a problem that lots of new connections were not able get the connection at all. This was the next error:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
So we started to look at the root cause of the issue, which was pointing to a connection pool exhaustion. The developer that wrote the code didn’t close the connections properly. On every error the connection was not closed and hence leading to connection pool running out of connections after some days. So the solution was to close the connection in the finally block. It was as simple as that.
public void saveMyData(ListList<String> mydata) throws MyException { Session session = null; try { session = this.sessionFactory.openSession(); session.beginTransaction(); // your code goes here session.getTransaction().commit(); } catch (Exception e) { logger.error(e); rollbackQuietly(session); throw new MyException(e); } finally{ closeQuietly(session); } } // the exception what got it here has the precedence so preserving that exception by rolling back quietly private void rollbackQuietly(Session session) { if (session != null) { try { session.getTransaction().rollback(); } catch (Exception e) { logger.error(e); } } } // the exception what got it here has the precedence so preserving that exception by closing connection quietly private void closeQuietly(Session session) { if (session != null) { try { session.close(); } catch (Exception e) { logger.error(e); } } }
So the whole scenario was like this.
- DB connections were leaked by the ill written code.
- After some time all the db connections were leaked
- Then any new attempt to access the connection led to the indefinite waiting on the server leading to CLOSE_WAIT connections to the tomcat and hence making all tomcat connections exhausted.
- Using maxWait helped to atleast release the tomcat connections and tomcat was responsive again.
- And fixing the root cause of connection leakage let the application run again in stable mode.