A few days ago I had a strange problem with an Oracle database together with the current 1.7 version o fOpen JDK. The problems occurred on a Jenkins build slave running on CentOS 6.6.
The connection to the Oracle database was done with Oracle’s JDBC driver. The script doing the database stuff runs without error. But not every time. So I got the error message „failing randomly“. Failing randomly, what the heck …
I started to create a small shell script which executes the script in question several times. The first run of the script always succeeds. The second run needs a lot more time to connect to the database or fails. The third run always fails. The error message on the client says „IO Error: End of TNS data channel“. The Oracle logs do not have a lot of usable output for this error. Google doesn’t help me very well also.
Having a deeper look at the JDBC driver I noticed the Oracle JDBC driver is not the same version as the database is. So changed the JDBC driver to the same version the database has as the DBA mentioned that the current configuration can cause connection problems. Being very optimistic to have solved the problem I started my test script again. Same strange behaviour as before. Ok go on investigating the issue …
After a lot of investigation I found the root cause of the problem. The Jenkins slave was just created and only one job is running on the slave. In fact this slave has nothing to do except running the database script. The JDBC driver tries to setup a secure connection to the database and needs entropy for the encryption. And that was the mess. Unfortunately /dev/random was used. Checking java.security file which comes directly from the CentOS openJDK package I found the following entry:
securerandom.source=file:/dev/urandom
The securerandom.source parameter is interpreted as URL. So the spelling was wrong which makes Java fall back to /dev/random. The proper setting must be
securerandom.source=file:///dev/urandom
After fixing the java.security file the database script runs without failure. And there was no delay in connecting to the database.
If you think you have an entropy problem with Java you can overwrite the java.security settings giving the following Java option:
-Djava.security.egd=file:/dev/urandom
Hope this little blog entry helps.