Yesterday I was trying bring up my application server which loads a lots of data into the memory for that purpose I had provided 110GB heap space to my JVM. After starting the application server I used to start the data loading job. After some time in middle of data loading the application server used to exit without any error message. After checking server logs in /var/log/messages I found that my application server was being killed by the kernel because it was running low on memory.
It is wrong on the part of kernel to kill application with no fault on their part. If kernel is running low on memory then it should deny requests for allocating more memory by applications. But kernel was killing processes instead.
The problem lies with overcommit feature of Linux. This feature should have never been developed. I don't see any valid use case for this feature. It relies on assumption that 30% of the memory which is allocated by application is never written to by application. Whenever application request for memory kernel zero out the page and map that page to process address space. Kernel instead of providing a new memory page every time uses a single read only ZERO_PAGE and map it to all application requesting memory allocations. When applications start writing to allocated pages it generate page-fault which kernel handles by allocating a new page and replacing ZERO_PAGE in application address space. So if application never writes to allocated memory pages kernel don't need to allocate real memory page. It is OK if it is done to reduce overhead of zeroing out memory pages. Kernel need to keep as many pages allocated to applications in buffer as treat them as allocated because applications can start allocated memory any time.
But with feature of overcommit kernel allocates more number of pages to applications than it is having in free pool. When applications start writing to allocated memory and kernel is unable to find free pages to replace the ZERO_PAGE it kills the application. It can't fail the request because it is not a memory allocation request but memory write instruction of the processor which can't return failure.
I disabled the overcommit feature of kernel by setting vm.overcommit_memory=2 using following command
http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/
It is wrong on the part of kernel to kill application with no fault on their part. If kernel is running low on memory then it should deny requests for allocating more memory by applications. But kernel was killing processes instead.
The problem lies with overcommit feature of Linux. This feature should have never been developed. I don't see any valid use case for this feature. It relies on assumption that 30% of the memory which is allocated by application is never written to by application. Whenever application request for memory kernel zero out the page and map that page to process address space. Kernel instead of providing a new memory page every time uses a single read only ZERO_PAGE and map it to all application requesting memory allocations. When applications start writing to allocated pages it generate page-fault which kernel handles by allocating a new page and replacing ZERO_PAGE in application address space. So if application never writes to allocated memory pages kernel don't need to allocate real memory page. It is OK if it is done to reduce overhead of zeroing out memory pages. Kernel need to keep as many pages allocated to applications in buffer as treat them as allocated because applications can start allocated memory any time.
But with feature of overcommit kernel allocates more number of pages to applications than it is having in free pool. When applications start writing to allocated memory and kernel is unable to find free pages to replace the ZERO_PAGE it kills the application. It can't fail the request because it is not a memory allocation request but memory write instruction of the processor which can't return failure.
I disabled the overcommit feature of kernel by setting vm.overcommit_memory=2 using following command
For making this effect permanent across reboots you need to add following line to /etc/sysctl.confsysctl vm.overcommit_memory=2
You can find more analysis on this at following page:vm.overcommit_memory=2
http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/