One Pager for:

Java applications which are critical to the business often suffer from performance problems. The magnitude of the problem increases when the business transactions are at its peak causing huge losses. Administrators who are trying to find the root cause of the problem spend a lot of time in trying to diagnose the problem with the tools available in the market. These tools are not viable to be used in a production environment due to the high overhead they impose, in effect mandating them to use these tools in either test or development environment. Many times, the problems are either not reproducible or it takes too long reproduce the problem.

Administrators need a lightweight tool to identify the problem, so it can be used with little overhead in the production environment. Apart from this, they need to have enough details to be able to identify the root cause of the problems in production environment without impacting the businesses. Application server would be the right product to build this intelligence of generating the lightweight data in production environment and analyzing them to present it in a meaningful way, so the administrators can get to the root cause of the problem. This will also result in giving value to the users by giving the right set of tools along with App Server product.

We will build it on the prelude monitoring infrastructure and provide extended capabilities in the areas of JDBC, JPA, EJB and JRuby container. Expose monitoring data through JMX and REST. Also will provide for ad hoc monitoring using popular scripting languages.

2.2. Risks and Assumptions

Risks: Availability of resources to complete the planned tasks.
Assumptions: gfProbes Infrastructure will work on JDK 6.

3. Problem Summary

3.1. Problem Area

Administrators need a lightweight production monitoring that can give enough details when the problems occur.

There is a need for lightweight monitoring mechanism that will allow monitoring to be turned on in production environment with minimal impact. There should be no overhead when there is no monitoring.

There is a need for an infrastructure that allows clients to decide what should be monitored with an ability to monitor specific issues (e.g.: Only HTTP 200 OK response). There is also a need to allow clients to conditionally monitor some activity (e.g.: Track EJB methods ONLY when called within a Tx). Additionally, there should be capability to monitor components beyond EJB, Web etc (Thread Pools, Replication etc)

The monitoring data exposed in prelude was limited as most time was spent on providing infrastructure and monitoring for Web Container & JVM components. In this release we plan to provide monitoring data for the modules JDBC, JPA, EJB and JRuby container. Also to maintain v2 compatability, we plan to expose monitoring data thru JMX. Also planning to expose the monitoring data through light weight REST API and enable ad hoc monitoring through client scripting.

3.2. Justification

The monitoring in Sun Appserver has been poor throughout the product life cycle and it did not evolve as much as the other parts of GlassFish. Today GlassFish is well known for its Enterprise features like Clustering, Load Balancing, Administration, Web Services, JBI etc., and monitoring isn't mentioned there. We want to change that and would like to add value to the product by providing a good monitoring solution for v3.

GlassFish v2 monitoring infrastructure is heavy weight with three monitoring levels:

off,
low and
high

In the low mode, simple statistics are captured (e.g.: statistics like createCount for all EJBs or all apps). In the high mode, in addition to the simple statistics, method statistics are also calculated (e.g.: method count, duration etc). Users can also call CallFlow to do further monitoring.
The current infrastructure is not powerful enough. The only way to find out the number of times a method is called requires the monitoring level to be at “High”. However, turning on monitoring to high, turns on monitoring for all components impacting the system performance. What is required is a very lightweight monitoring that allows monitoring of only what is needed by the client. In addition when no client is interested in any monitoring data the system should operate without any overhead.

Also, the set of monitorable data is fixed. There is no easy/extensible way to add new monitoring data to the system.

To start with, provide for monitoring data and expose the same through JMX to be v2 compatible. Provide for flexibility by exposing the monitoring data also through REST API which may be preferred by certain customers who are familiar with REST. Also it gives the flexibility to turn on/off the JMX or REST interfaces. It provides for enhanced experience if we provide for ad hoc monitoring through client scripting as more and more developers are using scripting languages.

4. Technical Description

4.1. Details

4.1.0 Architectural Block Diagram

Gmbal Class Diagram

4.1.1. Monitorable Components for v3

Following are the list of components that will be made monitorable for this release

a. JPA

b. EJB

c. JDBC

d. JRuby Container

e. (Grizzly) Thread Pool

f. JVM Thread Pool

The Monitoring for Web and JVM have already been in place for v3 Prelude, we will carry them forward for v3 release.

4.1.2. Monitoring Infrastructure

The architectural block diagram gives a high level design on how the monitoring in v3 will be implemented. Telemetry is the core of Monitoring layer which is responsible for listening, collecting and analyzing the data that is generated through the gfProbes infrastructure explained in the v3 Prelude Monitoring one pager.

4.1.2.1 Probe Providers

For GlassFish v3 release the Monitoring infrastructure will depend on the Probe Provider implementation by the Components described in Section 4.1.1. The Probe providers would have to provide certain probe points as listed by gfProbes Probe Provider contract explained in the v3 Prelude Monitoring one pager.

4.1.2.2 Telemetry (Built-in ProbeListeners)

4.1.2.2.1 Probe Listener

Telemetry component is responsible for collecting the monitoring information at the lowest level. The data collection can happen in two ways:

By listening to the data through registering the listeners for Probe points in the provider module.
By pulling the information on need basis (sampling) from certain components of App Server which doesn't provide the data through Probe infrastructure (ex. JVM information like CPU utilization). This component will collect the data which is specific to the Application Server monitoring.

The Telemetry component listeners for a provider will be registered or unregistered based the configuration parameter of that component (monitoring-level to be specific, either turned ‘ON’ or ‘OFF’) and on the lifecycle event from the provider (when provider is coming up or going down), thus making the Telemetry component agnostic to whichever modules it’s associated with (i.e. no dependencies). Once the listeners are registered, the data is collected from probe points, where each record can encompass of several events from several listeners (ex., getting the response time data from 'Request start' and 'Request end' probes/listeners). Analyzer is a built-in facility of the Telemetry module which will massage the data collected from listeners to expose it using Monitoring Registry.

4.1.2.2.2 Configuration

The user will be allowed to configure to turn the monitoring to 'ON' or 'OFF' which will in turn register/unregister the listeners at the Telemetry level for certain monitorable components and its attributes. The configuration changes can be done from either GUI or CLI commands. The interface provided from GUI will be basic for this release (turn it at a very high level, say for each provider component or a sub component, ex. turn off the Request statistic gathering), and can be more advanced (say, at the monitorable attribute level or turn off based on a feature) for next release. The CLI will expose the 'asadmin set' command to allow the configuration (enable/disable) of monitorable components.

The CLI will expose the 'asadmin set' command to allow the configuration (enable/disable) of monitorable components. This will be in addition to what we would do in V2 (see below).

The existing domain.xml elements and the child nodes will be preserved to turn the monitoring levels at the very high level for a module (for ex. turn off the monitoring for all the components of web-container). The levels 'Low' and 'High' will have no difference for v3 release, we will expose them from UI as either turning them 'On' or 'Off'.

We will also provide the configuration means to enable/disable the monitoring at a more granular level, say at the ‘Monitorable element’ level. This will be like a tree hierarchy starting at the Component level and exploding till the leaf element level. For example,

JVM

 --class-loading-system

 --compilation-system

 --garbage-collectors

 --memory

 --operating-system

 --runtime

*We are not sure if we want to extend this functionality to the attribute level, as it will increase the configuration file size to a great extent, thus incurring a lag in startup performance.

4.1.2.2.3 Monitoring Registry

Monitoring Registry is the interface with which the monitoring infrastructure exposes its information for the clients. Consider this as a tree where the root is located by the clients using the HK2 habitat. It consists of interfaces for the Telemetry objects to register/unregister themselves as the tree nodes. Once part of the tree they are exposed to any client (REST/CLI). The tree nodes are dynamically changing either when a new module comes in, or when the configuration is changed (monitoring for a node/attribute turned on/off). The requests for monitoring data coming in to the Monitoring Registry are served by retrieving the data from the Telemetry objects (which consists of the latest and greatest information).

Thus the information gathering happens at a much granular level only for the requested nodes (or attributes for next release). For example the Request Information gathering which can be a very expensive operation during the peak time of a business, one can choose to turn it off and this object will be omitted from the hierarchy and also the probe listeners are unregistered to stop collecting the information.

The CLI will traverse the Monitoring Registry tree for a given dotted name, which results in the retrieval of the data corresponding to the dotted name. We will be backward compatible with regards to the dotted name.

Above is done similar to prelude.

4.1.2.3 Supported Interfaces (JMX/AMX & REST)

The user should be able to access the Monitoring data with either the REST or JMX interfaces, thus allowing all the relevant clients to be able to use the exposed Appserver monitoring data. Note that the monitoring is also accessible using the GlassFish asadmin CLI and the Admin console applications. We will allow the user to turn ON/OFF either of the interfaces (REST/JMX) using the CLI ‘set’ command or with the admin GUI screen.

4.1.2.3.1 JMX/AMX

We will use the JMXA framework to expose the (Monitoring data) Objects in Monitoring Registry using the JMX interface. JMXA is a runtime annotation processor that creates Open MBeans. This is useful for creating a management API for existing code with minimal effort. You can see more details about JMXA here.

The way this is done is by providing the JMXA annotations in the Telemetry object, which in turn are processed by the JMXA and registered in the MBeanServer thus exposing the monitoring objects as JMX. The likely JMXA annotations we will use in Telemetry objects are @ManagedObject, @ManagedAttribute, @ManagedOperation, @ManagedDescription, and @ObjectNameKey.

JMXA will automatically expose the AMX MBeans for all the JMXA annotated Telemetry objects, thus making it accessible with the regular dotted names as part of the ‘asadmin get’ command. You can see more details about AMX and CLI here

The configuration saved in domain.xml will allow the user to turn the JMX to ON/OFF for monitoring if he chooses to, for performance reasons.

4.1.2.3.2 REST

REST strictly refers to a collection of network architecture principles which outline how resources are defined and addressed. The term is often used in a looser sense to describe any simple interface which transmits domain-specific data over HTTP without an additional messaging layer such as SOAP or session tracking via HTTP cookies.

The REST interface proposed will enable GlassFish to add monitoring for additional types of technology clients such as browsers, JSON, PERL, Ruby, etc., in addition to the Java clients it currently supports. The goal of this project is to expose objects in Monitoring Registry as REST Uri’s using Jersey 1.0 (JAX-RS implementation). This project will use Jersey 1.0.x for its implementation, wherever applicable. Jersey is standards-based, pluggable, and provides common default representations.

There will be REST adapter which will intercept the REST calls, parses the URI and redirects the requests to the appropriate handler based on the annotations in Telemetry objects and the configuration parameters for REST & JMX

As in JMX, there will be a configuration parameter in domain.xml which will allow the users to turn the REST to ON/OFF for monitoring. The monitoring through REST is exposed in three ways.

REST annotations

In this approach the Telemetry objects which were annotated with JAX-RS and JAXB annotations will be registered directly with the Jersey (assuming the Configuration parameter for REST says enabled). Any incoming requests through REST are intercepted by the REST Adapter and will be redirected to the Jersey.

JMXA annotations

Telemetry objects with JMXA annotation in the absence of JAX-RS (& JAXB) annotations will be used to map the incoming URI to the declared JMX ObjectName by the REST Adapter, thus retrieving and returning the Monitoring data from the Monitoring Registry.

No annotations (uses the CLI way)

When there are no JMXA or REST annotations in the Telemetry Objects, the incoming requests intercepted by the REST adapter will be redirected directly to the Objects in the Monitoring Registry, the same way the CLI dotted name ‘get –monitor=true’ works.

The following table explains the different scenarios which were described above.

ID	REST Annotations	JMXA Annotations	JMXA generates MBeanInfo	Comments
1	True	True/False	True/False	Register the object with Jersey and use Rest Adapter to redirect the calls to URI’s in Jersey. If Jersey runtime does not exist then fall back to the next implementation model.
2	False	True	True	Use Rest Adapter to Map the URI’s to Object Names (on the fly) and invoke the appropriate object
3	False	True	False	Use the Rest Adapter to Map the URI’s to the appropriate object in the Monitoring Registry
4	False	False	False	Use the Rest Adapter to Map the URI’s to the appropriate object in the Monitoring Registry

4.1.2.4 CLI commands to configure and display

Following are the list of asadmin commands that will be provided as part of monitoring infrastructure

asadmin monitor command: Exposes data from the Monitorable attributes. The output of the monitor command models Unix's vmstat.

The syntax of the monitor command is as follows:

monitor --type monitor_type --interval 30 --filter filter --filename filename target

To monitor http-listener1 in server instance:

>asadmin monitor --type web-container.http-listener.http-listener1 --interval=5 server

Where type denotes the dotted name value referring to the sub component which you would like to monitor (http-listener1 in this case). Note that the dotted name would be able to identify any third party component also, provided they are implemented with the right contracts (annotations) and interfaces.

The filename option allows user to save the monitoring attributes to a file in comma separated format.

The interval option is the duration of time when it refreshes the screen with new data from the server.

We will also support the keywords like httplistener, jvm etc., for --type option, to be backward compatible with V2.

asadmin get --monitor command: This command will expose the monitorable attributes in the dotted name format. Following is the syntax

asadmin get --monitor dotted-name

set: This command will enable/disable the monitoring of Monitorable attributes. This will in turn call the interface enable/disable which is exposed as part of the Object view hierarchy tree at different nodes. The syntax is as follows

set <dotted-name=true|false> [target=server]

{{ex: asadmin set server.web-container.thread-pool.thread-pool-1.enabled=false

Note that the dotted name exposed should be able to accommodate any third party components also.

· asadmin get --monitor=true --rest=true command: This command will expose the monitorable objects from the Monitoring Registry given a REST URI. The syntax is as follows

asadmin get --monitor=true --rest=true <URI>

· asadmin get-uri --monitor=true command: This command will return the URI of the monitorable objects in the Monitoring Registry given a dotted-name. The syntax is as follows

asadmin get-uri --monitor=true [--target=server] <dotted-name>

4.1.2.5 GUI

A basic monitoring UI page for each of the Monitorable Component listed in section 4.1.1 will be shown in the Admin Console. Please refer to Section 4.1 of the Admin GUI Functional Spec for more details.

4.1.2.6 Ad hoc monitoring using Scripting

Scripting is the functionality which seems to be of great interest among the users of GlassFish. Allowing the user to deploy a script which will listen to the Monitoring Probe events and report the data back to the client is a very useful feature that we are planning to do at least in a minimal way. The scripting support we are looking at is for the following languages (one of the following will be finalized)

JavaScript
Groovy
JRuby
Java (as a source)
Jython etc.,

To enable ad hoc monitoring through popular scripting languages (java script, groovy, ruby) we need support from both client and server. The client will send the script to server for execution and await the response for display. It may be necessary for the client to establish a dedicated communication stream so that the response can be received and displayed in real time. The asadmin framework needs to be augmented accordingly.

While on the server, the script needs to be bound to the underlying monitoring infrastructure for two way communication, first to register itself as a a client and second to get called whenever there is a response. We need to figure out what is this binding mechanism for different scripts, for ex. it could be DSL for groovy or JRuby and java binding for java script, etc.

4.1.2.7 Pluggability

Pluggability in general is described in a separate spec. Here we will discuss the pluggability aspects which are specific to monitoring. Any module (either third party or built-in) will be able to use our monitoring infrastructure to expose the monitoring functionality for its component. A module owner would need to come up with Telemetry objects (listeners), Monitoring Registry tree (sub) nodes, Probe Points, CLI and optionally GUI interface for their module. We will provide a way for the module writer to be able to seamlessly integrate his module with ours to expose the monitoring capabilities for his module.

Probe Points - Please refer to Probes section on how to do this
Telemetry - User needs to come up with his set of listeners to listen to the exposed probes either by his module or from other modules. The probes infrastructure will give an exposed interface to build this.
Monitoring Registry tree hierarchy - This is an important interface that a module owner needs to expose for him to be able to integrate with our REST/CLI interfaces. The user has to use the Monitoring Registry API to add his monitorable node to the Monitoring Registry for the Monitoring infrastructure to recognize them being part of the larger hierarchy.
CLI - 'asadmin monitor' command will take a --type option. The --type option accepts any component exposed in the Monitoring Registry, thus exposing all the monitoring attributes within with this generic command. Also exposed is the 'asadmin get --monitor ' command, to expose the data from Monitoring Registry in a generic way. Same is true for 'set' and 'list' commands. A third party module owner thus will be able to integrate the monitoring functionality of his module with the existing CLI commands.
REST - The nodes and its attributes in the Monitoring Registry are exposed by URI's mapped directly to the Objects in Monitoring Registry. See more details in the REST one pager.
GUI - Please refer to the Pluggability one pager for details on how to integrate the third party GUI pages into the Appserver Admin Console. User should be able to use the hierarchy of Objects in Monitoring Registry and REST interfaces to write their own full blown UI functionality into the GUI pages and integrate them as part of Admin Console.

4.2. Bug/RFE Number(s)

None.

4.3. In Scope

The Probe Providers that express intent to provide probe points can provide higher level artifacts that are container specific to clients. This can provide richer data for clients to monitor. This is not provided through existing solutions like profilers or debuggers.

4.4. Out of Scope

// Aspects that are out of scope if not obvious from above.

4.5. Interfaces

// Interfaces are a major part of Architectural review.
// Commands, Files, Directory Layout, Ports, DTD/Schema, admin tools,
// config files, APIs, CLIs, and almost anything that is externally
// observable is an interface. In 1-Pager it is necessary to document
// any interface that can be used by external projects and products.
// Documented public interfaces must be assigned a stability level.
// Some commonly used Stability levels in prior projects are:
//
// Stable : Widely used public interface. changed very rarely.
//
// Standard : Defined by a standards body (e.g: JDBCv3). Rare but
// incompatible clarifications and changes could occur
// in a standard. Product will specify version of std
// supported. J2SE, J2EE and WS* are classified
// as Standard.
//
// Evolving : Subject to carefully controlled but possibly
// incompatible change at a major or minor release.
// When a change is made all efforts will be made
// to address incompatiblity and migration. All
// incompatibilities will need to be reviewed
// and approved by as-ccc@sun.com.
//
// Unstable : Early access, subject to unrestricted degree of
// change. A few App Server interfaces are classified
// as Unstable. Docs must call out exported unstable
// interfaces. Be wary of importing Unstable interfaces.
//
// External : Defined external to GlassFish Application Server,
// but not by a Standards body. Suitable for describing
// other freeware, open source interfaces.
//
// http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/
// describes the permitted interface taxonomy.

4.5.1 Exported Interfaces

// Disclose all interfaces that this project exports.

Interface	Stability	Former Stability (if changing)	Comments

4.5.2 Imported interfaces

// Disclose interfaces this project imports.

Interface	Stability	Exporting Project: Name, Specification or other Link.	Comments

4.5.3 Other interfaces (Optional)

// Any private interfaces that may be of interest?

Interface	Stability	Exporting Project: Name, Specification or other Link.	Comments

4.6. Doc Impact

New CLI commands and GUI screens need to be documented. Also need to describe how the customer can implement monitoring in his (3rd party) module using our monitoring infrastructure. Documentation will need to write manuals that describe the probe provider interfaces and must include:

when each of the individual probe methods are invoked
the parameter name for each parameter that are available to the client
the set of computed parameters that are available

4.7. Admin/Config Impact

There will be configuration information for the monitoring elements that needs to be persisted in the domain.xml. The CLI and GUI clients are written by monitoring and GUI teams for such configuration.

4.8. HA Impact

None

4.9. I18N/L10N Impact

None

4.10. Packaging & Delivery

Monitoring infrastructure utilites will go as part of admin/monitor package.

4.11. Security Impact

Clients run in the same VM and have access to container artifacts. Clients however will not be able to alter the runtime state of the container. Clients will be deployed on the Probes container and they will be subject to the same restrictions as other JavaEE applications.

4.12. Compatibility Impact

We think the compatibility won’t be an issue with JMX/AMX and Dotted names for the final release of v3.

JSR77 could be an issue as we are trying to incorporate the REST functionality. REST has some limitations in terms of the objects exposed which need to have specific interfaces implemented in each of those objects. We need to think of a way to overcome this for the final release.

JMX/AMX could expose the data in Monitoring Registry using the JMXA framework. See JMXA functional specification for more details.

Dotted names should be able to support and enhance the monitoring and configuration data from the previous release.

4.13. Dependencies

JMXA

REST (Jersey)

Probe Providers

a. JPA

b. EJB

c. JDBC

d. JRuby Container

e. (Grizzly) Thread Pool

5. Reference Documents

Pluggability Funtional specification
JMXA Function Specification
JPA One Pager
EJB One Pager
JDBC One Pager
JRuby One Pager
Web Tier One Pager (for Grizzly thread pool monitoring)
CLI and AMX specification
A presentation on gfProbes and Monitoring infrastructure

6. Schedule

6.1. Projected Availability

For details on the Monitoring schedule – refer monitoring specification.