Administrative Credentials for Application Server 8.x

Author: Kedar Mhaswade. All comments to Kedar.Mhaswade@Sun.Com
Table of Contents

0. Introduction

This chapter deals with administrative credentials for application server. As you know, the basic boundary that is protected by the administrator for application server is the domain. An application server administrator manages one or domains, each of which can have distinct administrative credentials. By managing a domain an administrator effectively manages various resources like server instances, server clusters, libraries etc. that are required by the enterprise Java applications.

Most of this text applies to both the Platform Edition and Enterprise Edition of application server software.

1. Application Server Domain: Administrative Credentials and their Use

1.1 The Background

The application server domain (or just the domain) is your entry point into using the application server. A domain provides the essential runtime required for administration. Each domain has its own configuration stored in a set of defined files on disk. These files are stored in the config subfolder inside the domain folder. The config folder is secured by providing read-only access exclusively to the administrative user. Note that this is the operating system user that is being referred to. Usually an application server administrator is same as the system administrator of the machine where the application server software is installed (this is not required however). When that is not the case, the application server administrator should make sure that there are enough privileges available so that a domain could be created on a particular system (machine).

It is important to note that it is not required that the user who installs the application server software is the same as the user who creates administrative domains on a given system. This is one of the strengths of Sun's application server software. Usually a system administrator installs the application server software using a suitable mechanism, creates operating system users with sufficient privileges and then these users create the domains with their own configurations. Thus domains are administrative namespaces utilizing the same binaries installed elsewhere. All domains are mutually exclusive sandboxes capable of residing on a given system, provided that enough physical resources are available.

When an operating system user creates a domain, this user is the owner of this domain. To be precise, this user is the owner of the domain admin server process, process being the well-known operating system artifact. Thus, invoking the asadmin create-domain command is how a user creates the domain. If you carefully observe the behavior of this command, following is evident:

  • You must provide the admin user name for this domain.
  • You must provide the admin password for this domain.
  • You must provide the master password for this domain, in some form. This has been dealt with in detail in the section on master password.

The admin user name, admin password and master password together define what is called as the admin 3-tuple. Anyone who knows the admin-3-tuple is an administrator for the application server domain.

[Note that the cases of hardware accelerator, crypto card passwords etc. for additional security have not been considered.]

1.2 Admin Operations and Use of Admin Credentials

An Admin Operation is defined as any operation that results in changing the domain's configuration in some defined way. There are various ways in which an admin operation can be performed. In other words, there are various entry points into the domain administration. Important thing to note is that all of the control flow through all these entry points must satisfy a security constraint defined by the admin user name and admin password. Thus the admin user name and password represent the single sentinel that guards all of administrative access.

         ________________                _____________________________________________________     
        |asadmin/NetBeans|--HTTP/S------|----|   |--------------|asadmin-web-app|             |                    
        |________________|              |    | A |                                            |
         _______________                |    | D |                                            |
        |   Pure JMX/AMX|--JMX/SSL------|----| M |--------------|system-jmx-connector-server| |
        |_______________|               |    | I |                                            |
         _____________                  |    | N |                                            |
        |   Browser   |----HTTP/S-------|----| R |--------------|admingui-web-app|            |
        |_____________|                 |    | E |                                            |
       ________________                 |    | A |                                            |
      |JSR-77 Client   |--RMI/IIOP/S----|----| L |--------------|MEJB Implementation|         |
      |________________|                |    | M |                                            |
                                        |_____________________________________________________|
                                                         Domain Admin Server Runtime

Figure 1: Schematic showing admin operations

The admin user name and admin password is required by what is called an admin security realm. This is a security realm provided exclusively for administrative access. The realm implementation decides whether the given credentials (on each admin request) are sufficient to carry out the given request. If the realm rejects a particular user name and password, the entire operation fails and domain configuration is protected. This is the very reason why the Domain Admin Server should be running in order to perform an administrative operation.

The admin realm is realized as follows:

  • Almost every remote asadmin operation (e.g. deploy, create-jdbc-resource) communicates with a servlet based web application that is deployed onto the domain itself, as a system application. This web application authenticates the user name and password using the HTTP Basic Auth. Thus, every remote asadmin request has to be accompanied by an appropriate HTTP request header comprising of admin user name and password.
  • A browser accessing the admin URL (e.g. http://das-host:admin-port on Platform Edition and https://DAS-host:admin-port on Enterprise Edition, by default), always accesses another web application that is deployed onto domain itself, as a system application. This web application uses a login form and provides the content to the authenticated user.
  • A pure JMX Client (e.g. JConsole in Java SE 5.0 onwards) and an administrative client using AMX (Management API for the application server) usually sends the user name and password as a two-string array. The implementation of the server end utilizes the same realm to authenticate the received user name and password.
  • A standard JSR-77 client is nothing but an EJB client, the EJB being the so-called MEJB. The access to MEJB is authenticated using the same user name and password in the admin realm. The MEJB is deployed as a standard system EJB module onto the domain itself.

The true security can only be achieved when all these remote connections have transport layer security using SSL. In Platform Edition, the admin user name and password might be sent in clear over the network, whereas in Enterprise Edition, care is taken such that only secure connections are attempted in the case of asadmin and browser clients (this is the default behavior).

In addition to the above, note that node-agents and server instances communicate with the DAS for internal application server tasks, using the same admin user name and password.

1.3 Admin Security Realm and Multiple Admin Users and Groups

The default admin security realm implementation is called the FileRealm. When a domain is created, following two things happen in this regard:

  1. A file named admin-keyfile is created in the config subfolder of domain folder. This file contains a line containing the provided admin user name, a secure one-way hash of the provided admin password and a group name (fixed name: asadmin). What this entry means is that there is a logical group (of administrators) named asadmin, in which the given admin user is added. All the users in this group have administrative access to the domain.
  2. An entry is made into the domain.xml such that the security realm named admin-realm is mapped to the admin-keyfile.

Thus, out-of-the-box, the admin user created is authenticated against the file created this way. It is possible that an application server administrator for a large enterprise chooses to authenticate against a corporate LDAP using the LDAPRelam. The way to do this is mapping admin-realm onto an appropriately configured LDAP.

An administrative user can add or delete other users from the asadmin group. All the users in the group asadmin have identical privileges and it is impossible to distinguish one administrative user from another unless auditing is enabled. In other words, there is no role-based administration in application server for this release.

The asadmin commands that modify the users in admin-realm, while using the FileRealm are:

  • create-file-user --authrealmname admin-realm ... : creates a new administrative user.
  • delete-file-user -- authrealmname admin-realm ...: deletes an existing administrative user. Don't attempt to delete the last administrative user this way.
  • update-file-user --authrealmname admin-realm ...: updates the password, group name etc. for an existing admin user. You'd not need to use this command for admin user, in general.

There are Admin GUI (Browser based) equivalents for the same. Refer to admin reference and manpages for the exact details of these operations.

Note the following regarding these commands:

  • These commands or GUI operations are available only for FileRealm. If you have configured admin-realm to associate with the LDAP Realm, you will need to manage these things at the LDAP layer.
  • The selected auth-realm-name should be admin-realm. If you forget to specify it, the default realm selected is the standard J2EE security realm that manages accesses to the user application. These users are stored in a file called keyfile in the config subfolder of the domain folder. Don't confuse admin-keyfile or admin-realm with keyfile or user-realms.


2. The Master Password

Like it is said above, an administrator who creates a domain has to provide and remember the admin user name, admin password and master password for a domain. Really speaking, the master password is a misnomer, it is the SSL store password with which the keystore and truststore for the domain are locked.

When a domain is created, a self-signed certificate is created (anew every single time). This certificate (nicknamed s1as) is stored in X509 format in the trust-store in config subfolder of domain folder. The master password is the password which protects this store and the keystore.

To enhance the developer experience, the developers or the users of Platform Edition are shielded from master password. In case of Platform Edition, the master password is defaulted to "changeit" and user does not have to provide it. In fact, the developers can completely be agnostic that there is such a password.

Enterprise Administrators on the other hand will be prompted for master password while creating the domain in Enterprise Edition. For Platform Edition, JKS (Java KeyStore) and relevant tools (e.g. JDK keytool) are used, whereas for Enterprise Edition NSS (Network Security Services) and relevant tools (e.g. certutil) are used.

When you start a particular domain and/or node-agent, the security subsystem needs to be initialized with the certificates and keys and in order to unlock the stores, the domain and/or node-agent startup needs access to an identical master password. Note that for Enterprise Edition, all the node-agents in a given domain must have the same master password as that of the domain itself. It is also important to note that master password is never put on the network. It has to be provided every time it is required by either node-agent startup or the domain startup process.

There is a way to store the master password on disk in an encrypted manner and the way to realize it is by using --savemasterpassword flags on create-domain and create-node-agent commands. It is also available on the change-master-password command. The idea of --savemasterpassword is that, when this option is used at the time of creation of domain and node-agent, the startup of these entities will read it from there instead of prompting the users for it.


3. Application Server Administrative Credentials. and Operating System Users

It is possible to get confused with the relationship between admin user name in the admin 3-tuple and the operating system user created by the system administrator. Simply put, there is no relationship between these two. These are two independent entities. From the application server standpoint, any operating system user who has access to the admin user name and password (and master password) is the administrator. Note the following in this regard:

  • Only an operating system user (or superuser on Unix systems and Administrator on Windows systems) who has created a domain can start the domain.
  • Once a domain has been started by a system user "joe", it can well be managed by another system user "blo" as long as the latter knows the admin user name and admin password.
  • The master password is needed only by the domain and/or node-agent startup. To manage a domain that's already running, you just need admin user and admin password.
  • In an enterprise scenario, imagine that a domain is owned by a system user "Joe" on a system "foo". Usually, node agents and server instances will be hosted on another system "bar". The node-agent and server instances can be owned by any other user "blo" on system "bar". While in operation, it does not matter what users own the running domain/node-agent/server instance processes on given systems.


4. Changing Administrative Credentials

It is obvious that administrators would want to change the admin password and master password from time to time. There are supported ways to perform these operations. One care to exercise in this regard is to plan the activity of changing the passwords. Note the following in this regard:

  • It is not possible to change the admin user name for a domain.
  • If you are using the default FileRealm implementation for admin-realm, you should use asadmin update-file-user --authrealmname admin-realm command to change the admin password. Once you change the password on the DAS, it is immediately effective on the DAS Thus all subsequent logins to the Admin GUI or asadmin commands would need new admin password. Thus, if you are using the --passwordfile on asadmin commands with AS_ADMIN_PASSWORD in it, you need to modify that file in order to perform the commands.
  • In an enterprise scenario, after the admin password has been changed, you should restart all the node-agents and server instances in order for the new admin password to take effect domain-wide. The way to restart the node-agents and server instances is as follows after changing the admin password:
    • Restart the DAS This is asadmin stop-domain followed by start-domain.
    • Restart the node-agent only by providing the new admin password. This is asadmin stop-node-agent followed by start-node-agent --startinstances=false. It is important to not start the instances along with the node-agent startup.
    • Restart the instances and/or clusters by stopping them followed by starting them.
  • The master password can only be changed by the command asadmin change-master-password. The master password has to modified for the domain and each node-agent explicitly. The change in the master-password is effective only after a restart of DAS and node-agent(s).

4.1 The Case of Forgotten Passwords

It is possible that administrators/developers forget the admin user name, admin password or master password for a domain. In such situation, the administrator cannot really manage the domain. Hence you should be careful with the credentials. If you have compromised the security inadvertently, you should consider abandoning the domain to reduce the damage further.

In some extreme situations however, it is desirable to recover credentials so that the administrator knowingly wants to recover the credentials by undertaking the associated risk. The following text pertains to recovery of credentials under such situations and constraints.

Since it is evident from the above that admin security revolves around the admin-realm, it is the configuration of admin-realm that dictates the ability to recover the admin credentials. The following applies to recoverability of admin credentials, if the admin-realm is configured to be the default FileRealm. If you have configured it to be any other realm, it is up to you to recover the credentials. Note that this file admin-keyfile is readable only by the owner of the domain. Thus, you will be able to view the contents of this file if you've created this domain.

4.1.1 Recovering admin user name

Follow these steps:

  • See the contents of domain-dir/config/admin-keyfile. The contents are semicolon separated lines. Each line contains the first field as the admin user name in clear text. In case there are multiple admin users created, you can choose any one of them.

4.1.2 Recovering admin password

Assuming that you either know or have recovered the admin user name, following is a workaround to recover the admin password for the domain. Note that merely looking the contents of the admin-keyfile won't reveal the password in clear. This file only has a secure one-way hash of the admin password. Assume that the admin user name is "admin" and domain name is "mydomain" for the domain whose admin password is forgotten.

  • Create another domain "foo" with admin user name "admin" and desired password (that will not be forgotten!) in a temporary location, thus: asadmin create-domain --user admin --passwordfile passfile --adminport 1234 --domaindir /tmp foo (note that passfile contains AS_ADMIN_PASSWORD=desired-admin-password)
  • A file /tmp/foo/config/admin-keyfile would be created by default.
  • Copy this file on to mydomain/config/admin-keyfile.

The admin passwords for both mydomain and foo are now the same. You can now safely delete the domain foo.

In reality, this is not the recovery of the old password, but changing it indirectly.

4.1.3 Recovering master password

It is not possible to recover or indirectly modify the master password. If you have forgotten the master password, you cannot start or restart a given domain or node-agent(s). The domain will need to be recreated.

5. Storing the Credentials

Like any other secure system, the application server does not store the administrative credentials. in clear text anytime. Here is how the domain stores the passwords:

  • With FileRealm, the admin password is stored as a secure one-way hash in admin-keyfile. It has to be presented for almost all administrative operations by the user, in clear-text.
  • By default, the master password is not stored anywhere on the file system in domain's configuration. It has to be provided at the time of domain/node-agent startup. If the domain or node-agent has been created with --savemasterpassword option, then the master password is saved in an encrypted form in a special JCEKS store named "master-password", so that it does not have to be provided at the time of domain/node-agent startup. This facility is provided for the unattended boots of the domain and server instances. Details about this are covered in password aliasing section.

6. Admin Credentials. and asadmin Commands

All the remote asadmin commands do their job by communicating with a running Domain Admin Server (DAS). Examples of remote asadmin commands are: deploy, create-jdbc-resource etc. Other kind of asadmin commands is local. These commands do not communicate with the DAS All remote asadmin commands need admin user name, admin password, DAS's admin port and DAS's host name. These commands also need to know if the admin http port needs plain text connection or SSL. Out-of-the-box, Platform Edition has all the domains with admin port listening for plain text connections, whereas Enterprise Edition has all the domains with admin port listening for secure connections. These options are named:

  • --host (localhost being the default value)
  • --port (4848 being the default value for Platform Edition, whereas 4849 being the default value for Enterprise Edition)
  • --user (admin user name, no default value)
  • --password (admin password, no default value)
  • --secure (false by default for Platform Edition, whereas true by default for Enterprise Edition)

Remote commands is a misnomer, really speaking. It does not really mean that for these commands, the asadmin script and DAS have to be running on different machines. All it means is that these commands need a running DAS (with compatible versions) defined by the above parameters. While specifying the admin user name and admin password on asadmin commands, following should be noted:

  • The admin user name can be specified by --user command line option, or environment variable AS_ADMIN_USER.
  • The admin password can be specified in the following ways in order of precedence:
    • If a value is specified for the option --password, it is used. Note however that this option is deprecated. Do not use this option as it is likely to be removed from future releases and moreover, it is unsafe to use clear text passwords on command lines.
    • If a value is specified for the environment variable AS_ADMIN_PASSWORD. Note however that this support too is deprecated. Do not use this option as it is likely to be removed from future releases and moreover, it is unsafe to use clear text passwords in memory.
    • If a value (in clear) is specified like AS_ADMIN_PASSWORD in a properties file and the path of that file is specified on --passwordfile option, this value is used. It is safer than the above. But security of this file then becomes a responsibility of the administrator.
    • If a file named ".asadminprefs" is present in the user's home directory and that file contains a property named AS_ADMIN_PASSWORD, then this password is used. This file is created during a Platform Edition installation, when a developer chooses not to be prompted for admin user and password for asadmin commands.
    • If all of the above fails, asadmin prompts the user for the admin password.
  • The master password can be specified in a passwordfile with a property named AS_ADMIN_MASTERPASSWORD. If it is not specified this way, then user gets prompted for it (unless --savemasterpassword is not used on start-node-agent/start-domain commands).

7. Advanced Topic: Password Aliasing

Please see the document: aliased passwords.

8. Other Credentials

In addition to the above credentials, there are other credentials that an application server domain administrator should be aware of. Here is a brief account of those:

  • Message Queue Broker Credentials: A separate MQ broker instance is started by default, with Domain Admin Server. There is a fixed admin user name and password for this JMS implementation that is integrated with the application server. It can be changed after the domain is created. Refer to Sun Java System MQ admin guide for the same. Usually, you don't need to change these credentials.
  • The HADB software Credentials: HADB software has a multitude of passwords used for specific purposes. It is important that you understand them. Refer to HA guide for the same.
  • Various Database Credentials.: These are stored in domain.xml in clear, by default. These are used by the connection pooling infrastructure with application server. Refer to the password aliasing section for facilities provided to protect these.

9. Credential Best Practices

Here are some of the best practices to exercise while using application server domain credentials:

  • Choose good passwords that are easy to remember for you. Follow standard procedures that are part of your organization.
  • Prefer using --passwordfile to --password on asadmin command lines.
  • Protect the passwordfile all the time.
  • Consider using password aliasing for database passwords.
  • Use --savemasterpassword on the create-domain/create-node-agent commands judiciously. This should be used only when you have stringent requirements for the unattended boots. Protect the master-password file in that case (it is protected by default).


Table of Contents

0. Introduction
1. Application Server Domain: Administrative Credentials
2. The Master Password
3. Application Server Administrative Credentials. and Operating System Users
4. Changing Administrative Credentials
5. Storing the Credentials
6. Admin Credentials and asadmin Commands
7. Advanced Topic: Password Aliasing
8. Other Credentials
9. Credential Best Practices