Initialization of the MBeanServer in Glassfish—
Fixing the SunoneInterceptor initialization

December 3, 2006, Lloyd.Chambers@sun.com

Current initialization sequence

Classnames used in this document

Most of these classes are all in the appserv-core module.

com.sun.enterprise.admin.server.core.AdminService
com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor
com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilderFactory
com.sun.enterprise.admin.server.core.ConfigInterceptor

com.sun.enterprise.util.FeatureAvailability
com.sun.enterprise.util.RunnableBase

Problematic initialization

The main player is SunoneInterceptor, the MBeanServer replacement used by GlassFish for some not-so-good reasons. The SunoneInterceptor wraps the javax.management.MBeanServer.

The current initialization sequence is problematic for multiple reasons:

It is possible for certain types of code to run even before the JVM calls main() [PEMain]. This creates an MBeanServer, but all future AppServer code will fail, because it will locate the wrong MBeanServer (see bug #xxxx). The code is sprinkled heavily with the idiom getMBeanServers().get(0). Short of finding and changing every single call site, it is critical to make sure the correct MBeanServer is #0.
Implementation dependencies of SunoneInterceptor make it impossible to initialize the MBeanServer prior to initialization of various other facilities, such as the ServerContext, ConfigContext and AdminContext.
Building of the SunoneInterceptor is done by AppServerMBeanServerBuilder, which expects a boolean to be set by AppServerMBeanServerBuilderFactory, which indicates that the SunoneInterceptor is to be created. It also expects this to be synchronized.
The SunOneInterceptor exists as an MBeanServer in a state of incomplete initialization for a period of time (until setJmxMBeanServer() is called). There is no guarantee that intervening code wouldn’t attempt to use it during that period—only sheer luck and the desire of all developers for QuirkLook to pass before comitting changes.
Thread-unsafe code (see Correctness Problems below).

Correctness problems of SunoneInterceptor

SunoneInterceptor inserts a large amount of unnecessary performance-degrading code. Some of it is literally run for no reason at all—code that should have been stripped out long ago. Some of it has an apparent purpose, which at least for PE seems to be mostly unnecessary. And some of it isn’t even correct code — it’s not thread-safe.

Thread-unsafe variables

The following variables are not protected by synchronized or volatile. Though the server appears to work, it is actually broken, because multiple threads can access these variables.

adminContext — static variable set by one thread (main thread), but accessed by any number of threads.
realMBeanServer — instance variable initialized by setJmxMBeanServer() via main thread, but accessed by any number of threads.

initialization of ConfigInterceptor

When SunoneInterceptor.setJmxMBeanServer() is called, it calls com.sun.enterprise.admin.util.proxy.ProxyFactory to create an MBeanServer proxy, implemented by ProxyClass, which wraps ConfigInterceptor:

void setJmxMBeanServer(MBeanServer jmxMBS) throws InitException {
    realMBeanServer = (MBeanServer)ProxyFactory.createProxy(
        MBeanServer.class, jmxMBS, adminContext.getMBeanServerInterceptor());
    logMBeanServerInfo();
    initialize(); 
}

Performance problems caused by SunoneInterceptor

Performance degradation calling defunct code

The following in SunoneInterceptor are all code that should not exist and/or can be easily removed:

checkHotConfigChanges — ends up calling new InstanceEnvironment() for every call on every MBean. This creates hundreds and possibly thousands of objects, only to return a constant 'true', for every get/setAttribute(s) and invoke call.
registerWithPersistenceCheck — dynamic loading of MBeans, none of which need to be dynamically loaded, and very few of which actually are. All but one of these are “ias” MBeans, which are duplicate registrations of com.sun.appserv MBeans (see below). The remaining MBean is type=server-instance, and it’s unclear that it’s even used. It can be handled in a far better manner (by a NotificationListener).
Duplicate registration — Some MBeans are registered twice; once in the “ias” domain and once in the “com.sun.appserv” domain. Changing the constant kDefaultIASDomainName in ObjectNames.java to “com.sun.appserv” instead of “ias” solves this duplicate registration.

Performance degradation by ConfigInterceptor

The use of ConfigInterceptor (see Initialization of ConfigInterceptor) routes all calls through it. This is inefficient, because very few calls are relevant—only those that change configuration. These include setAttribute(), setAttributes() and invoke(). Of course, invoke() can be called on many operations that do not modify configuration (getAbc(), listAbc(), etc).

It is also troublesome that for each relevant setAttribute(s)/invoke call, there are probably 100X or 1000X as many get/list/invoke calls that are read only. In fact, once the server has been configured, the penalty is paid forever, even if configuration is never changed again.

Compounding the transgression (of impacting all MBeanServer operations), all MBeans in all JMX domains are impacted, when in fact only certain ones are relevant to ConfigInterceptor—namely the com.sun.appserv:category=config MBeans. For example, the java.lang, amx, category=monitor, etc MBeans are all routed through ConfigInterceptor. AMX MBeans route all config operations through their Delegate, which is a com.sun.appserv:category=config MBean.

It is not just a waste of clock-cycles—multiple new objects are instantiated for every call. These objects must be garbage-collected at a future time.

The name says it all—“Config” and “Interceptor”. Impacting things that aren’t related to config modifications has not only immediate clock-cycle performance impacts, but possible unknown side-effects, and future garbage-collection impacts.

Inability to concurrently start services/modules

SunoneInterceptor is initialized by the AdminService, a heavyweight service which can take 2-3 seconds to initialize even on the very fastest machines. AdminService initializes a large number of admin-related services, MBeans, etc.

As AdminService is the first ServerLifecycle to be run, all subsequent ServerLifecycle modules must wait until it has finished. Since most of these need only the MBeanServer and/or a few specific MBeans (and in some cases nothing at all), this has the effect of precluding concurrent initialization of other services.

Solution

Recode the initialization of the MBeanServer such that it can be done immediately upon entry to PEMain.main() (in its own thread), or even before PEMain is called. This allows any dependent code to immediately have access to the MBeanServer without waiting for the AdminService. Such code might depend on other features; see com.sun.enterprise.util.FeatureAvailability.
Defer the insertion of the ConfigInterceptor until the time at which it can be created eg when the AdminService is initialized.
Internally thread the initialialization and startup sequence of the AdminService. Very little of what it initializes has a serialization dependency on prior items. See com.sun.enterprise.util.RunnableBase; 5 threads are used within the AdminService in the author’s non-committed code base.

Threading example

Below are two examples of a threaded sub-part of the AdminService; the FORCE_SERIAL boolean can be flipped to allow true asynchronous initialization. For the 1st checkin phased, the boolean will be set to true to mimic existing behavior, though this appears to be unnecessary.

    // initialize AdminChannel
        _adminChannelIniter   = new RunnableBase( "AdminService _adminChannelIniter") {
            protected void doRun()  throws Exception {
                adminChannel = new AdminChannelLifecycle();
                adminChannel.onInitialization(context);
            }
        };
        _adminChannelIniter.submit( FORCE_SERIAL );

        _callflowAndJKSIniter = new RunnableBase( "AdminService _callflowAndJKSIniter" ) {
            protected void doRun()  throws Exception {
                initCallFlow(); 
                setupJKS();
            }
        };
        _callflowAndJKSIniter.submit( FORCE_SERIAL );

Fixing the MBeanServer initialization sequence

Overview

It appears that the functionality of ConfigInterceptor is still required, namely to flush the Config when a change is made (though even that is not a clear necessity—further research will look into it). For this reason, an interceptor must still be used until it is established that the config-flushing behavior is not needed, or that it can be done via other means.

Instantiating a new interceptor

The new interceptor DynamicInterceptor will be instantiated from within PEMain in a separate thread (“Dynamic” because its behavior can be modified at a later time) . This entails the following:

modifying the javax.management.builder.initial property in several domain.xml templates;
kicking off a thread to instantiate the MBeanServer (about 110ms on a fast machine).
Modifying com.sun.enterprise.admin.common.MBeanServerFactory to call FeatureAvailability to return the MBeanServer.

The code below initializes the MBeanServer. Note that it is critical to establish that the MBeanServer can be started immediately because there are means by which code can ask for the MBeanServer even before main() is called (see bug#1409 for an example).

    static private final String BUILDER_SYSTEM_PROPERTY =
			"javax.management.builder.initial";
    static private final String MBEAN_SERVER_BUILDER =
        "com.sun.enterprise.interceptor.InterceptorMBeanServerBuilder";
    
    private static class MBeanServerIniter extends RunnableBase {
        MBeanServerIniter() {
            final String builder    = System.getProperty( BUILDER_SYSTEM_PROPERTY );
            if ( ! builder.equals( MBEAN_SERVER_BUILDER ) ) {
                throw new Error( "initializeMBeanServer: incorrect MBeanServer specified" );
            }
        }
        protected void doRun() {
            final long  start   = System.currentTimeMillis();
            
            final MBeanServer server =
                java.lang.management.ManagementFactory.getPlatformMBeanServer();
            
            if ( ! (server instanceof DynamicInterceptor) ) {
                // Don't use a Logger; it has not even been initialized yet
                // this message indicates a development bug and so doesn't need I18N
                System.err.println( "initializeMBeanServer: wrong MBeanServer was created" );
                throw new Error( "initializeMBeanServer: incorrect MBeanServer was created" );
            }
            FeatureAvailability.getInstance().registerFeature(
                FeatureAvailability.MBEAN_SERVER_FEATURE, server );
            
            final long elapsed = System.currentTimeMillis() - start;
            _Debug.println( "MBeanServer created successfully in "  + elapsed + "ms");
        }
    };
	
	public static void main(String[] args) {
	    final MBeanServerIniter initer  = new MBeanServerIniter();
		initer.submit();
		...

Note the use of FeatureAvailability to record the availability of the MBEAN_SERVER_FEATURE. This is the only mechanism by which other dependent code should obtain the MBeanServer, though the current idioms will continue to work.

Maintaining the existing ConfigInterceptor in the call chain

The DynamicInterceptor accepts a “hook” eg:

public synchronized void addHook( String jmxDomain, DynamicInterceptorHook hook )

The AdminService uses this hook to insert the ConfigInterceptor into the call chain:

FeatureAvailability.getInstance().waitForFeature(
    FeatureAvailability.MBEAN_SERVER_FEATURE, "AdminService.init" );
final MBeanServer mbs = getMBeanServer();
try {
    adminContext.setMBeanServer( mbs );
    final FlushConfigHook hook  = new FlushConfigHook( adminContext );
    ((DynamicInterceptor)mbs).addHook( "com.sun.appserv", hook );
    sLogger.info( "core.mbs_init_ok");
...

The FlushConfigHook optimizes the calling path such that only relevant MBeans are passed along to ConfigInterceptor . Thus, MBeans in other JMX domains do not suffer a performance hit from insertion of the FlushConfigHook (com.sun.jbi, amx, java.lang, amx-support, com.sun.apspserv:category=monitor/runtime, etc).

Conclusions

This change will make a major improvement in the way the MBeanServer is instantiated, setting the stage for far more efficient startup of the server (eg concurrent initialization with today’s ServerLifecycle approach, which can cut server startup time by about 40% on a dual-core machine).