Making a module monitorable in
GlassFish V3
Table of Contents
Author: Prashanth Abbagani
Email: abbagani@dev.glassfish.java.net
Last Modified: 6/30/09
1.0 Introduction
Salient features of module based monitoring infrastructure are
- Container(s) / sub-system only emit events/probes
- Example: onWebRequest(...), onWebResponse(...)
- Interesting values are passed as paramters
- Containers do NOT
maintain any stats
- We use the term "providers"
- If there are no listeners registered for this, then there is no
overhead
- Listeners receive the event that are emitted by providers
- Listeners can examine the parameters passed
- Listeners maintain the stats
- The Monitoring framework connects “providers” and “listeners”
- Supports various clients, like asadmin, JMX/AMX, REST, Admin GUI, Client-Scripting (JavaScript, JRuby*)
- Monitoring events are exposed as probes in the DTrace world
- Allows the Turning on/off, (configuring) the monitoring for
components
2.0 Monitoring Pluggability
As a subsystem owner, you would bring in your own monitoring
implementation into the GlassFish. The monitoring infrastructure will
provide you API to do the following
- Define the Probe points from your subsystem as part of your
Provider implementation. Probe points are the
monitoring events, emitted when glassfish goes through that code path.
The Providers and its probes are discovered and registered
automatically and thus exposed to the DTrace, Client-Scripting and
Probe Listener clients. This is explained in detail in the next sub
section.
- You implement the StatsProvider (Listener) objects, which listen
to a specific set
of probes, collects the data into the Statistic (StatsProvider member
variables) objects. You then instantiate and publish
the StatsProvider object using the Montiroing Framework API. As part of
publishing these objects, the statistics are automatically
exposed
through
different clients (asadmin, GUI, JMX/AMX, REST).
Explained in subsequent sections in detail
- The Monitoring configuration (turning the
monitoring flag to ON/OFF) for your module and the actions to turn
ON/OFF the listeners (and the collection of data in the statistcis
inside your StatsProvider objects) accordingly, is handled
automatically by
the Monitoring infrastructure. Explained in Section
5.0 in detail.
3.0 Probe Provider
You will use the provider API to expose the probes for your subsystem.
A provider is either a Class or an XML file, defined by the
contianer/module.
It defines the set of events that are emitted by the provider.
Providers are discovered and registered automatically, as and when the
modules
are loaded into the glassfish.
3.1 Defining the
Probes
You will need to define the
probe points from where the probes
(monitoring events) are fired. There are essentially two ways to do
this.
3.1.1 Provider as class with
annotation
Using
these annotations before your (new) class & method headers and
specifying
the list of
parameters to
expose, you would define the Provider class. You will need to
instantiate this class and call the (empty) method from where you want
to emit this probe (see the second example).
*Note that there is a restriction on the parameter types of your Probe,
which needs to be only primitive types (for now). This restriction is
placed because of DTrace. We plan to relax this restriction going
forward, by accomodating the composite types in some fashion and
exposing to the DTrace clients.
Defining the Provider Class
@ProbeProvider(moduleProviderName="glassfish", moduleName="web",
probeProviderName="jsp")
Class JspProbeProvider {
...
@Probe(name="jspLoadedEvent")
public void jspLoadedEvent(
@ProbeParam("jsp")
String jsp,
@ProbeParam("hostName")
String hostName) { }
....
}
Emitting the probe
Class
JspProbeEmitterImpl.java {
...
public JspProbeEmitterImpl() {
JspProbeProvider jspProvider = new
JspProbeProvider();
......
}
public jspLoaded (String jsp, String vsId) {
jspProvider.jspLoadedEvent(jsp, vsId);
}
...
}
3.1.2 Provider as an XML
This
will allow you to define probes in a non-intrusive way. Just feed the
XML file to the Monitoring infrastructure and make the probes active.
Your module doesn't have to depend on any Monitoring Framework API
if you do this way.
Defining the Provider XML
<probe-provider
moduleProviderName="glassfish" moduleName="web" probeProviderName="jsp"
>
<probe name="jspLoadedEvent"
<class>com.sun.enterprise.web.jsp.JspProbeEmitterImpl
</class>
<method>jspLoaded</method>
<signature>void
(String,String)</signature>
<probe-param
type="String" name="jsp"/>
<probe-param type="String"
name="hostName"/>
<return-param type="void" />
<location>RETURN/ENTRY/EXCEPTION</location>
</probe>
</probe-provider>
The Container class's method being
called
Class JspProbeEmitterImpl.java{
...
public jspLoaded (String jsp, String vsId) {
....
}
}
3.2
Emitting
Probes
by Providers
This section explains how to emit probes (events) from the Providers
(Section 3.1.1). This scenario is
applicable when you are using the @ProbeProvider annotation approach of
writing the
provider. For XML approach, you dont have to do anything, the
probes are automatically emitted when it visits the code path described
in your XML.
Section 3.1.1 explains with example on
how to do this.
3.3
Registering the Provider
The Monitoring infrastructure will discover and register the providers
automatically when you follow the instructions on packaging the
providers, which are explained in section 3.4.
3.4 Packaging the Probe Provider
This section explains on how to package your Probe Provider. As
explained there are two approaches to write a provider, either by
annotating a class or by writing an XML. When the provider is written
as a class (Section
3.1.1) or as an XML (Section
3.1.2),
you will need
to do the following step so the providers can be discovered by the
Monitoring Infrastructure automatically.
You will have to list all the provider class names and xml file names
in your jar's
META-INF/MANIFEST.MF file.
Following is the format.
probe-provider-class-names :
<probe-provider-class-name[,probe-provider-class-name]>
probe-provider-xml-file-names :
<path of xml file name w.r.t root [, xml file name]>
You will need to put one of the above entries, not both (mutually
exclusive). You could automate this by putting the following in your
modules pom.xml
<plugins>
<plugin>
<groupId>com.sun.enterprise</groupId>
<artifactId>hk2-maven-plugin</artifactId>
<configuration>
<archive>
<manifestEntries>
<probe-provider-class-names>
org.glassfish.web.admin.monitor.RequestProbeProvider,org.glassfish.web.admin.monitor.JspProbeProvider
</probe-provider-class-names>
</manifestEntries>
</archive>
</configuration>
</plugin>
</plugins>
3.5 Dependencies for Provider
Your module when doing the 'Class' approach (Section
3.1.1),
you need
to have the runtime and build-time dependencies on the following Jars
(which will be external'ized). Apart from this, your module doesn't
have
any other dependency for your probe-provider to work in the GF
environment.
management-api.jar (This Jar consists of the annotation
definitions, ex. @ProbeProvider, @Probe and @ProbeParam).
<dependency>
<groupId>org.glassfish.external</groupId>
<artifactId>management-api</artifactId>
<version>3.0.0-b003</version> {// Do not specify version if
your module is under glassfish v3 project. The correct version
will be picked up from v3/pom.xml}
</dependency>
4.0 StatsProvider objects
A Probe listener (part of the functionality of StatsProvider Object) is
basically an object that receives callbacks from
the
providers. A method in the listener can be "marked" as callback method
for a particular event from a specific provider.
StatsProvider are the objects which gather and expose the monitoring
data
of various components. They
do the following functionality
- Allows configuration of a component for Monitoring i.e,
Enable/Disable
- Gathers statistics
- Defines the Statistics Objects
- May Listen (Via. Probe Listeners) to Probe events emitted by
Provider
- May collect statistics based on the probe events or however way
it wants (for ex. using some specific container events)
- Also may collect statistics from non-Probe Provider, Ex. JVM
statistics
- The statistics objects will be publised to the Monitoring Registry
- They are the target objects for AMX/JMX and REST clients
4.1 Writing StatsProvider Object
This section describes on how to write a StatsProvider object.
*Note: We need to make sure the ObjectNames are compatible with V2
style for existing components.
import
org.glassfish.external.statistics.CountStatistic;
import
org.glassfish.external.statistics.impl.CountStatisticImpl;
import
org.glassfish.external.probe.provider.annotations.ProbeListener;
import
org.glassfish.external.probe.provider.annotations.ProbeParam;
import org.glassfish.gmbal.ManagedAttribute;
import org.glassfish.gmbal.ManagedObject;
@ManagedObject
class JspStatsProvider() {
private
CountStatisticImpl activeJspsLoadedCount = new
CountStatisticImpl("ActiveJspsLoaded", "count", "Number of currently
loaded JSPs");
private CountStatisticImpl totalJspsLoadedCount = new
CountStatisticImpl("TotalJspsLoaded",
"count", "Cumulative number of JSP pages that have been loaded into the
web module");
^--- Statistic
name - capitalize first letter of each word, no spaces
....
@ManagedAttribute(id="activejspsloadedcount")
<---- id should be all
lowercase
public CountStatistic
getActiveJspsLoadedCount(){
return
activeJspsLoadedCount;
}
@ManagedAttribute(id="totaljspsloadedcount")
public
CountStatistic getTotalJspsLoadedCount(){
return totalJspsLoadedCount;
}
@ProbeListener("glassfish:web:jsp:jspLoadedEvent")
public void
jspLoadedEvent(
@ProbeParam("jsp")
String jsp,
@ProbeParam("hostName")
String hostName) {
activeJspsLoadedCount.increment();
totalJspsLoadedCount.increment();
}
....
}
The annotations are explained below
@ManagedObject - This is a Gmbal annotation telling that this is a
managed object and is exposed via. JMX/AMX.
@ManagedAttribute(<attribute-name>) - The name of the attribute
when exposed through JMX/AMX, also will be used to publish it as a leaf
node in the Monitoring Registry
@ProbeListener(<probe-provider-identifier>) - This
annotation specifies
the callback (listener) method when a probe (mentioned as parameter) is
fired.
@ProbeParam(<param-name>)
- This annotation allows you to mention only the param-names that you
are interested in, others wont be passed along.
Naming Convention
For consistency between v2
and v3, between MBean attributes, dotted names, REST, and Admin
Console, as well as across all statistics, please use the following
naming conventions for attribute ids and statistic names.
- Attribute ids - all lowercase (ie @ManagedAttribute(id="activejspsloadedcount"))
- Statistic names - capitalize first letter of each word, no spaces
(ie new CountStatisticImpl("ActiveJspsLoaded",)
Attribute ids and Statistic names are displayed in the CLI dotted name
from the asadmin get command, REST, Admin Console, and monitoring
MBeans.
CLI dotted-name (asadmin get
--monitor=true server.web.jsp.*)
server.web.jsp.activejspsloadedcount-count
= 1
server.web.jsp.activejspsloadedcount-description = Number of currently
loaded JSPs
server.web.jsp.activejspsloadedcount-lastsampletime = 1249556014500
server.web.jsp.activejspsloadedcount-name = ActiveJspsLoaded
server.web.jsp.activejspsloadedcount-starttime = 1249556014500
server.web.jsp.activejspsloadedcount-unit = count
REST
(http://localhost:4848/monitoring/domain/server/web/jsp)
activejspsloadedcount :
Count : 1
Admin Console
(http://localhost:4848)
Jconsole
4.2 Registering StatsProvider object
To enable monitoring for your Module, you first write a Probe Provider
(Section 3.0), then you would write the
StatsProvider Objects (Section 4.1). You
would
then control the registering and unregistering of these objects using
Monitoring Framework, either at statup/shutdown or may be based on some
config-change events (like when a new jdbc-connection-pool is created).
When registring the StatsProvider object, the Monitoring Framework will
automatically register them for JMX/AMX, REST and Monitoring Registry
(which will be referred to by asadmin/Admin GUI clients), while also
registering the listeners for associated Probes. Following is the ex.
StatsProviderManager.register("jsp-container",
PluginPoint.SERVER, "applications/app1/jsp1", new JspStatsProvider());
The StatsProvider object will be registered under the “server” node in
the Monitoring registry. Third parameter gives the intermediary
sub-tree (followed after the 'server' node) in the Monitoring registry
under which the Statistics objects (defined in the StatsProvider
object) would be created as leaf nodes. The fourth parameter is the
StatsProvider object itself.
>asadmin get server.applications.app1.jsp1.* //will
return these stats objects
4.3 Unregistering the StatsProvider Object
When unregistering the
StatProvider object, the Monitoring Framework will automatically
unregister them from JMX/AMX, REST clients and MonitoringRegistry
(which will be referred to by asadmin/Admin GUI clients), while also
unregistering the listeners for associated Probes. Following is the ex.
StatsProviderManager.unregister(jspStatsProvider);
4.4 Packaging & Dependency of
StatsProvider
The StatsProvider of a Module can be packaged as a separate OSGI jar or
It can be part of the Provider/Container. There is no restriction on
how you would package the StatsProvider objects.
StatsProvider Module will have dependency on the following jars
management-api.jar
(definitions for @ProbeListener, the StatsProviderManager
API, and the Statistic API)
gmbal.jar
(definitions for @ManagedObject, @ManagedAttribute, and @AMXMetadata)
Put the
following dependencies in your pom.xml
<dependency>
<groupId>org.glassfish.external</groupId>
<artifactId>management-api</artifactId>
<version>3.0.0-b003</version> {// Do not specify version if
your module is under glassfish v3 project. The correct version
will be
picked up from v3/pom.xml}
</dependency>
<dependency>
<groupId>org.glassfish.gmbal</groupId>
<artifactId>gmbal</artifactId>
<version>3.0.0-b009</version> {// Do not specify version if
your module is under glassfish v3 project. The correct version
will be
picked up from v3/pom.xml}
</dependency>
5.0 Configuration
The configuration of monitoring for your module is automatically
handled by the Monitoring Infrastructure. The first parameter that you
gave as part of the MonitoringFramework.register API, will be
used to construct the
config sub-element and saved in the domain.xml (an attribute under the
<monitoring-service>/<monitoring-level> element).
The management of this configuration is supported by
asadmin (get/set/list)/GUI/JMX/REST (all standard clients of config).
The
enable/disable of the monitoring flag for the component is
handled automatically, by turning on/off the associated listner(s) to
the
probes.
asadmin set command
glassfish/bin>asadmin get
"*"|grep
monitoring-level
configs.config.server-config.monitoring-service.module-monitoring-levels.web-container=HIGH
configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=OFF
configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH
glassfish/bin>asadmin set
configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
6.0 Monitoring Clients
There are several ways the monitoring is exposed to the end user.
These are done in an uniform and seamless manner, so the Monitoring
providers would not have to do anything from their part. Following are
the different ways.
6.1 DTrace
When the Probe Provider is registered with the monitoring
infrastructure (Note that StatsProvider objects have nothing to do with
this), the probes from glassfish components are automatically
exposed (for Solaris) so the DTrace clients can write their scripts to
listen to these probes. We use the BTrace as the underlying
infrastructure to do this. Note that as a provider, you wont have to do
anything for this to work.
6.2 Scripting Client
We expose all our probes to a
scripting client that would allow GF user to enable/disable the probes
and get the data they want. This is similar to DTrace philosophy
but will work on any OS for GlassFish runtime. We will have the
infrastructure with a Scripting Container, also the asadmin and GUI
clients for
deploying and running the scripts written in JavaScript and JRuby (next
release) and opening up a Comet channel to display the data as and when
the probe is fired (a Push model, does it asynchronously). Note that
the script is something similar to a
StatsProvider object in this case, the difference being, the user could
write adhoc script (no compilation is necessary) and just run it on the
fly to see the data instantly.
Javascript example
var
nServlets=0;
function getCallbackMethod() {
return
"requestStart";
}
function requestStart(hostName)
{
nServlets=nServlets+1;
ScriptContainer.println("js:requestStart: called servlet count = " +
nServlets);
}
println('Javascript> Invoking
testMethod method from JavaScript...');
params =
java.lang.reflect.Array.newInstance(java.lang.String, 1);
params[0]="hostName";
ScriptContainer.registerListener('glassfish.web.requestStart',
params, getCallbackMethod());
asadmin deploy --type clientscript <javascript file-location>....
asadmin run-script --script <script-name>
Apart from the adhoc scripting support, we will also provide pre-canned
script capability, which user would be able to invoke directly. The
output from the scripting container is name=value pairs with the value
coming from the Object.toString(). Following is the example
asadmin run-script --probe "glassfish.web.requestStart" --paramnames
"moduleName,hostName"
The module owner doesn't have to do anything to get this functionality.
Its available by default, when the container (or module) comes up with
its probe provider.
6.3 JMX/AMX
We use Gmbal infrastructure to expose the Statistics objects as
JMX/AMX. As part of the MonitoringFramework.register API, we
are exposing these Statistics objects as JMX/AMX automatically.
Example: view jvm AMX mbeans with
jconsole:
1. asadmin start-domain
2. asadmin set
server.monitoring-service.module-monitoring-levels.jvm=HIGH
3. Start jconsole by typing 'jconsole' on command line
4. In jconsole connection window, select remote process and type
'localhost:8686' (leave username and password blank), and Connect
5. Goto Mbeans tab.
6. Expand 'amx' folder
7. Look for class-loading-system-mon, compilation-system-mon,
garbage-collector-mon, memory-mon, operating-system-mon, runtime-mon
folder.
To view web container mbeans
1. asadmin set
server.monitoring-service.module-monitoring-levels.web-container=HIGH
2. deploy web app
3. Look for jsp-mon, servlet-mon, session-mon, web-request-mon folder.
AMX Monitoriong MBean Object Names
The object name is made
up of 4 parts:
[1] :
[2]
,
[3]
, [4]
amx:pp=/mon/server-mon[server],type=class-loading-system-mon,name=jvm/class-loading-system
[1] amx is the domain name for all the amx mbeans
[2] pp is the parent path, 'server' monitoring
[3] type is the 'interface' type which comes from annotation
@AMXMetadata(type="class-loading-system-mon"). If @AMXMetadata is
not specified, the type defaults to the classname, such as
JVMClassLoadingStatsProvider.
[4] name is used to uniquely identify mbeans of the same type
value. name is derived from the sub-tree (third param) used in StatsProviderManager.register("jvm",
PluginPoint.SERVER, "jvm/class-loading-system", new
JVMClassLoadingStatsProvider());
6.4 REST
We use the REST infrastructure to expose the Statistics objects as REST
API. As part of the MonitoringFramework.register API we
notify the REST infrastructure to expose these Statistics objects as
REST
automatically. Please see REST One-pager on how this happens and the
URI to access these objects.
6.5 asadmin get/list
As explained earlier, the asadmin get and list commands (with
--monitor=true option to differentiate from the config get/list
commands) will
automatically expose the monitoring data, once you publish your
statistic objects into the monitoring registry. It would look something
like below
6.6 asadmin monitor
To be able to plug in your own monitoring type and view its related
output, you should be able to use the pluggability feature of the CLI
monitor command. Please follow the link
for an explanation on how this works.
This
command
displays commonly monitored statistics for
Enterprise Server components and services. Monitoring output is
displayed continuously in a tabular format; the --interval option can
be used to display output at a particular interval (default 30 seconds).
C:\glassfishV3\v3-prelude\v3_prelude_release\glassfish\bin>asadmin
monitor --type=webmodule server
asc ast rst st
ajlc mjlc tjlc aslc
mslc tslc
0 0
0 1
0 0
0 8
8 8
C:\glassfishV3\v3-prelude\v3_prelude_release\glassfish\bin>asadmin
monitor --type=jvm server
JVM
Monitoring
UpTime(ms)
Heap
and NonHeap Memory(bytes)
current
min
max
low
high count
11623062
8585216 167313408
0
0 38531072
7.0 References
Monitoring
One-Pager
REST
One-Pager
