Monday, September 23, 2013

Java : OutOfMemoryError

java.lang.OutOfMemoryError Exception

Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.


  • -Xmx, is the JVM max Heap Size
  • -Xss, is the Java Thread Stack Size, the default is OS and JVM dependent, and it can range 256k-to-1MB. 
  • The default should be tuned down to a range that doesn’t cause StackOverflow.  I often use 128k-192k.  Since the default –Xss is high, tuning it down can help save on memory used and given back to the Guest OS.
  • Perm Size is an area additional to the –Xmx (Max Heap) value and is not GC-ed because it contains class-level information.
  • “other mem” is additional mem required for NIO buffers, JIT code cache, classloaders, Socket Buffers (receive/send), JNI, GC internal info
These are the list of OutOfMemoryErrors a typical java application can see:

java.lang.OutOfMemoryError: Perm Space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: requested xxxx bytes for Chunk::new. Out of swap space

java.lang.OutOfMemoryError: Perm Space
This indicates JVM ran out of PermSpace.
permspace is specific to jvm implementations.
For Eg, SUN JVM has one but Jrockit doesn't have an allocated permspace, i.e its part of heap.
To understand what information is stored in permspace, an example would be:
classA obj = new classA()
obj is an object i.e instance of classA.

Now JVM maintains the structure for "obj" on heap where as stucture for classA will be on permspace.
This is the reason applications that has lots of classes(libraries, JSPs) tend to occupy more permspace.
Also, we would see this space used when classloading (dynamic) is done.
Typical example would be application deployments on appservers like weblogic.
There is other kinds of information stored on permspace like information for JVM optimizations etc.

java.lang.OutOfMemoryError: Java heap space
This is a typical java out of memory from application.
This message simply says, for this particular request the response from JVM was "OutOfMemoryError".
Typically we see this message if JVM heap is full and FullGC is not able to reclaim any objects.
This can be an expected memory shortage or a memory leak.
Typically out of memory from memory leaks occur over a period of time, where as out of memory from load would occur under peak usage and JVM would recover (continue servicing requests even if some request fails with OutOfMemory error.
Obviously for load based OOM, increasing heap would help.
For memory leak scenario, we would have to use profiler tools to get to root cause. JDK6 provides a utility jmap that would serve the basic needs. We can have JVM generate a heap dump on OOM error. The heap dumps can be read by tools like jhat.
There are lot of commercial tools available like Jprofiler etc to analyze memory leaks.

java.lang.OutOfMemoryError: unable to create new native thread
This would mean, the java process size has reached its limit.
When an application creates a java thread object, correspondingly an OS thread has to be created and to create this thread OS tries to reserve space for its stack (thread stack to store local variables) within the process address space.
In most cases, we will see this error if the application spawns too many threads.

java.lang.OutOfMemoryError: requested xxx bytes for Chunk::new. Out of swap space
This would mean the process could not allocate memory as the virtual memory was full. This can happen if there are too many process running on the machine or there is some other process colocated on same machine that consuming most of the process memory.
If neither is the case, the the current process might be having a native memory leak. This can some from native code within the application and rarely can be from JVM itself which has JNI code. 

Sunday, September 22, 2013

Understanding Java Thread Dumps


A java thread dump is snapshot of JVM at that point in time.
More specifically it’s a snapshot of all the java threads at that point in time. 


A typical thread dump comprises of JVM spawned and application spawned threads. 





Some of the JVM spawned threads are (Sun JVM1.5.x) 

Low Memory Detector (Daemon thread to detect and report low memory conditions)
CompilerThreadn (Daemon thread/s to compile & optimize byte code)
Signal Dispatcher (Daemon thread to respond OS signals)
Finalizer (Daemon thread to handle objects in finalizer queue)
Reference Handler (Daemon thread to handle objects in reference queue)
VM Thread (Main thread from JVM)
GC task thread#n (Thread/s to perform garbage collection) 


Application threads are spawned from application (can be a combination of application server/web server and java application deployed).  

A typical thread dump snapshot would have most of the threads in one of the following states (Terminology might differ depending on JVM implementation)



R - Runnabl

MW - Monitor wait
CW - Conditional wait


                                


                                                                                                                                                                CW - conditional wait:
A thread in conditional wait would mean its waiting on a certain condition to be fullfilled. Typical Scenarios would include:
i) An idle thread waiting for work to be assigned.
ii) Thread waiting to get an EJB instance from pool
iii) Thread waiting to get a connection from connection pool.
Eg1:
"ExecuteThread: '16' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x010dc720 nid=0x29 in Object.wait() [a1a7f000..a1a7fc30]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:154) - locked <0xb1593fa0> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:174)

The above thread is an idle thread in weblogic server and is in conditional wait within ExecuteThread.waitForRequest() i.e waiting on a condition to fullfill before it starts executing. Condition being, socket reader thread would accept the request and delegate the call to notify() on this waiting thread which then resumes the request execution.

Eg2:
http-8080-Processor25" daemon prio=1 tid=0x09298688 nid=0x2781 in Object.wait() [0xa82f8000..0xa82f8fc0]
at java.lang.Object.wait(Native Method)- waiting on <0xab436a68> (a org.apache.commons.pool.impl.GenericObjectPool)

at java.lang.Object.wait(Object.java:474)

at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:810)- locked <0xab436a68> (a org.apache.commons.pool.impl.GenericObjectPool)

at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:95)

at org.apache.commons.dbcp.BasicDataSource.getConnect ion(BasicDataSource.java:540)

at org.springframework.orm.hibernate3.LocalDataSource ConnectionProvider.getConnection(LocalDataSourceCo nnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnectio n(ConnectionManager.java:417)

at org.hibernate.jdbc.ConnectionManager.getConnection (ConnectionManager.java:144

This is from tomcat/hibernate framework.
The thread is waiting on pool object, but looks like all the pool objects are in use. When one of the executing threads is done using the poolobject, it issues a notify() on this thread so that it can resume and pick the poolobject (JDBC connection) Any time a thread goes to wait(), it releases the current lock.

MW - Waiting for Monitor entry
MW is waiting for monitor entry i.e the thread is blocked and waiting to enter a synchronized block/method to get a java monitor. From the thread dump we can determine the monitor that is blocking the thread. If too many threads are in MW state across multiple snapshots, we know there is a contention for the lock.

Eg:
TP-Processor633" daemon prio=5 tid=0x004921d0 nid=0x5ba waiting for monitor entry [872fe000..872ffc28]
at oracle.jdbc.pool.OracleConnectionCacheImpl.closeSingleConnection(OracleConnectionCacheImpl.java:677) - waiting to lock <0xbf4e98f8> (a oracle.jdbc.pool.OracleConnectionCacheImpl) at oracle.jdbc.pool.OracleConnectionCacheImpl.reusePo oledConnection(OracleConnectionCacheImpl.java:592)
at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:130) - locked <0xbf4e9e60> (a oracle.jdbc.pool.OracleConnectionEventListener)
at oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:482)
at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:445) - locked <0xcc9af820> (a oracle.jdbc.pool.OraclePooledConnection)
at oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:2936) - locked <0xcc9af8c8> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:1454) - locked <0xcc9af8c8> (a oracle.jdbc.driver.OracleConnection)
..........
In the example above, the thread is waiting to lock "0xbf4e98f8" (object ID for "oracle.jdbc.pool.OracleConnectionCacheImpl") Note this thread is already holding locks for "0xbf4e9e60"(OracleConnectionEventListener),
"0xcc9af820" (OraclePooledConnection) and "0xcc9af8c8" (OracleConnection)
Having several threads in same state (waiting for same object), indicates there is a contention for the object.
In the current example, thread dump would reveal the thread holding the lock (Look for the thread with entry: - locked <0xbf4e98f8>)

R- Runnable
This state would mean thread is executing/utilizing the CPU or OS ready state for utilizing the CPU.

Eg:
"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x005af448 nid=0x20 runnable [0x6847d000..0x6847f8f0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
........................

at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3361) - locked <0xe225f140> (a oracle.jdbc.driver.T4CPreparedStatement) - locked <0xe221b390> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:100)
at com.determine.log.FQLogListImpl.collect(FQLogListImpl.java:80)
at com.determine.persistence.DBFacade.collect(DBFacade.java:75)
......
The above thread is connected to DB (opened socket on server and reading response from DB server). Now if the thread is in same state across multiple thread dumps, we know either the query is taking long time to execute or there is some issue with DB server that’s causing the DB to respond slowly.

There are good tools that parse and provide graphical representation of threads dumps like
Samurai , Thread Dump Analyzer.



Note : These details are based on the personal knowledge and other related details available over the internet. 

  

Java : OutOfMemoryError

java.lang.OutOfMemoryError Exception Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no ...