I was configuring a new build on Linux machine and it kept on failing with error
java.io.FileNotFoundException: /home/build/jenkins/workspace/OnPrem.AA-OnPrem.6.0.2.1-SP3-P4-HF50/configurations/target/classes/log4j.properties (Too many open files)
This error was occurring because our build is trying to utilize number of open File Descriptor beyond the limit.
By default in our RHEL 5.8 machine, the open Hard file limit was 1024. It can be checked with the command
# ulimit -a|grep open
open files (-n) 1024
or
# ulimit -Hn
1024
#ulimit -Sn
1024
Solution: Increase the Open file limit.
Step 1: vi /etc/security/limits.conf
Step 2: Add the below line in the end
* - nofile 4096
# End of file
It will increase the open file limit to 4096 for all the users.
Step 3: Verify it by running the below command in a new shell or session.
# ulimit -a|grep open
How to witness in a running build, a process is trying to overuse the file open limit?
Step 1: Start the build / Command. In my case I started the build from Jenkins
Step 2: Find out the process ID (PID) of the running task and note it down. In my case it is ps -aef|grep java
Step 3: Then check the number of files under /proc/<PID>/fd folder
ls /proc/7455/fd|wc -l
As the build progressed, I noticed the number of files keep on increasing from 20 to 1042 and immediately build failed with the above mentioned error.
java.io.FileNotFoundException: /home/build/jenkins/workspace/OnPrem.AA-OnPrem.6.0.2.1-SP3-P4-HF50/configurations/target/classes/log4j.properties (Too many open files)
This error was occurring because our build is trying to utilize number of open File Descriptor beyond the limit.
By default in our RHEL 5.8 machine, the open Hard file limit was 1024. It can be checked with the command
# ulimit -a|grep open
open files (-n) 1024
or
# ulimit -Hn
1024
#ulimit -Sn
1024
Solution: Increase the Open file limit.
Step 1: vi /etc/security/limits.conf
Step 2: Add the below line in the end
* - nofile 4096
# End of file
It will increase the open file limit to 4096 for all the users.
Step 3: Verify it by running the below command in a new shell or session.
# ulimit -a|grep open
How to witness in a running build, a process is trying to overuse the file open limit?
Step 1: Start the build / Command. In my case I started the build from Jenkins
Step 2: Find out the process ID (PID) of the running task and note it down. In my case it is ps -aef|grep java
Step 3: Then check the number of files under /proc/<PID>/fd folder
ls /proc/7455/fd|wc -l
As the build progressed, I noticed the number of files keep on increasing from 20 to 1042 and immediately build failed with the above mentioned error.