Caucho maker of Resin Server | Application Server (Java EE Certified) and Web Server


home / admin/configuration / health / report
 

Resin Documentation

Aug 2012: Resin outscales C-based web server nginx in AutoBench benchmark
Feb 2012: NetCraft survey says Resin experiencing strong growth in last year and used in number of the Million Busiest Sites.
home company blog wiki docs 
app server web server 
health cloud java ee pro 		 Resin Server | Application Server (Java EE Certified) and Web Server
resin 4.0
admin/configuration
starting
migrating
administration
clustering
command line
rest interface
configuration
database
deployment
health
health checking
meters
report
watchdog
installation
logging
scheduled tasks
security
url rewrite
web server
details
development
changes
reference
 

health report

meters
watchdog

Resin Professional includes a PDF report of your system's health.

Getting a Health Report

test

  1. Getting a Health Report
  2. Getting a Health Report
    1. resin.xml automatic pdf report generation
  3. Report Overview
  4. Heap Dump
    1. ClassLoader Heap Dump
  5. Thread Dump
  6. CPU Profile
  7. Logging
  8. JMX Dump

Getting a Health Report

The quickest method to get a health report is to use the command-line "pdf-report". The pdf-report will ask the server to generate a report immediately.

Example: generating a PDF report
unix> resinctl pdf-report

generated /var/resin/log/default-Summary-20110921T1218.pdf

resin.xml automatic pdf report generation

PDFs can also be configured in the resin.xml to be generated weekly, or on events like a restart.

Example: PDF weekly summary generation
<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">
<cluster id="">

  <resin:import path="${__DIR__}/health.xml"/>
  
  <health:PdfReport>
     <path>${resin.root}/doc/admin/pdf-gen.php</path>
     <report>Summary</report>
     <snapshot/>
     <mailto>user@example.com</mailto>
     <profile-time>60s</profile-time>
     
     <health:IfCron value="0 0 * * 0"/>
   </health:PdfReport>

   ...
</cluster>   
</resin>

The previous example generates weekly report by creating a snapshot (heap, threads, jmx, and profile), generating the PDF, and mailing the report to user@example.com.

The next example generates a PDF on a restart by the watchdog system.

Example: PDF watchdogreport
<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">
<cluster id="">

  <resin:import path="${__DIR__}/health.xml"/>
  
  <health:PdfReport>
     <path>${resin.root}/doc/admin/pdf-gen.php</path>
     <title>Restart</title>
     <watchdog/>
     <mailto>user@example.com</mailto>
     
     <health:OnRestart/>
   </health:PdfReport>

   ...
</cluster>   
</resin>

Report Overview

  • Summary: The key JVM, OS and Resin versions.
  • Meter Graphs: Graphs of statistics over the reported time.
  • Heap Dump: The top heap memory usage.
  • CPU Profile: A CPU profile of the system, showing the most active locations.
  • Thread Dump: All the threads in Resin at the time the snapshot was taken.
  • Log Report: The most recent "warning" level messages from java.util.logging.
  • JMX Dump: Data from all the JMX mbeans at the time the snapshot was taken.

The reports are designed be used in several situations:

  • bug report/support: for Resin's own support, it's much easier to debug a problem when we have a full snapshot to work from.
  • restart/crach analysis: if the JVM crashes for some reason, the watchdog system report can help track down the cause.
  • CPU and performance: if the JVM shows unexpected CPU use or sluggish performance, the report can help show the main causes.
  • thread spikes and locking: the thread dump and CPU can be used to track down locked code.

Heap Dump

The heap dump gives a quick overview of the memory allocation of the system. It's generally useful as a check for any unusual allocations.

The heap dump is sorted by "self+desc", which is the objects own size plus its descendant size. A java.lang.String for example would include the char[] as part of its "self+desc".

The following is an example of a basic idle Resin heap dump.

Example: Heap Dump
 Class Name                                self+desc  self  count

 byte[]                                     22.23M  22.23M   14741
 com.caucho.db.block.Block                  20.28M  134.9K    2410
 char[]                                     13.89M  13.89M  122606
 com.caucho.util.LruCache                    7.52M   30.4K     317
 java.lang.String                            7.15M   1.97M   61426
 ...

The first items, the byte[] and Block are primarily Resin's internal proxy cache and distributed cache database. Notice that the "self" for the Block is much smaller than its. "self+desc", because each Block has a large byte[] buffer.

Similarly, the String "self+desc" is much large than its "self" because it includes the char[] buffer.

ClassLoader Heap Dump

The Heap Dump section has a separate ClassLoader heap dump section which just displays the ClassLoader usage. You can use this report to check for class-based memory leaks.

Thread Dump

For CPU problems and stuck threads, the Thread Dump will show what each thread is doing in the system. You can use the report to see if many threads are piled up in an unusual location, like and unexpected lock, or track down a spinning thread.

The thread dump report merges threads which share the same stack trace. The merged threads will all be listed together, followed by their stack trace.

Blocked threads and the lock's owning thread are grouped together, so it's easier to see which thread is preventing many threads from continuing.

The following example shows a normal blocking situation. The JDK's SSL implementation only allows one thread to accept a connection at a time. All other threads will wait for the first thread. In this case, the thread named "http://*:8444-17" owns the SocksSocketImpl. Three threads are waiting in line for the lock: "http://*:8444-1", "http://*:8444-1", and "http://*:8444-1".

Example: Thread Dump
http://*:8444-17
    java.net.PlainSocketImpl.socketAccept
    -- locked java.net.SocksSocketImpl@1199747469
    java.net.PlainSocketImpl.accept
    java.net.ServerSocket.implAccept
    ...
    com.caucho.env.thread.ResinThread.runTasks
    com.caucho.env.thread.ResinThread.run
  
http://*:8444-1
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
http://*:8444-10
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
http://*:8444-11
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
    java.net.PlainSocketImpl.accept
    java.net.ServerSocket.implAccept
    com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept
    ...
    com.caucho.env.thread.ResinThread.runTasks
    com.caucho.env.thread.ResinThread.run

...

CPU Profile

Because the CPU profile is calculated by repeated thread dumps, it's possible for a single stack trace to have more than 100% of the time when multiple threads are waiting at the same place.

You may need to skip the first set of waiting threads to see the profile traces you're interested in.

Example: CPU Profile
CPU Profile

Time:    60.1s
GC-Time: 0.303s
Ticks:   601
Sample-Period: 100
End: 2011-09-21 16:33

5000.00% 3005.00s  com.caucho.vfs.JniSocketImpl.nativeAccept()  RUNNABLE (JNI)
                   com.caucho.vfs.JniSocketImpl.nativeAccept()
                   com.caucho.vfs.JniSocketImpl.accept()
                   com.caucho.vfs.JniServerSocketImpl.accept()
                   com.caucho.network.listen.TcpSocketLinkListener.accept()
                   com.caucho.network.listen.TcpSocketLink.accept()
                   com.caucho.network.listen.TcpSocketLink.handleAcceptTask()

3564.23% 2142.10s  com.caucho.env.thread.ResinThread.waitForTask()   WAITING
                   sun.misc.Unsafe.park()
                   java.util.concurrent.locks.LockSupport.park()
                   com.caucho.env.thread.ResinThread.waitForTask()
                   com.caucho.env.thread.ResinThread.runTasks()
                   com.caucho.env.thread.ResinThread.run()

...

Logging

The most recent warning logs are reported as part of the heap dump.

Example: Warning Logs
Log(Warning)

2011-09-21 11:06:07 warning  WarningService: Resin restarting due to
                                    configuration change

JMX Dump

The JMX dump includes the full report of all the JMX MBeans in the system along with their values. Using this part of the report is somewhat specialized, either checking configured values against expectations or looking at statistics that aren't graphed as part of the metering system.

The JMX beans are sorted alphabetically.

Example: JMX Dump
JMX Dump

JMImplementation:type=MBeanServerDelegate
   ImplementationName     Resin-JMX
   ImplementationVendor   Caucho Technology
   ImplementationVersion  Resin-4.0.s110921
   MBeanServerId          Resin-JMX
   SpecificationName      Java Management Extensions
   SpecificationVendor    Sun Microsystems
   SpecificationVersion  1.4
    
com.sun.management:type=HotSpotDiagnostic
  ...
meters
health
watchdog
Copyright © 1998-2012 Caucho Technology, Inc. All rights reserved. Resin ® is a registered trademark. Quercustm, and Hessiantm are trademarks of Caucho Technology.