WebLogic Admin Server Shows Stuck Threads Due To A Misconfiguration Of The Node Manager

The Admin Server goes to "Warning" state (stuck thread error) after a couple of hours, even though there are no activities in the deployed application Oracle Enterprise Repository (OER). When OER is shutdown, the Admin Server works fine. The Admin Server goes in warning state when the managed server is started and logs show the following error:


SERVER = AdminServer MESSAGE = [STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "610" seconds working on the request "weblogic.kernel.WorkManagerWrapper$1@1f9c1ae0", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-88 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@1f9dbb78[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:250)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:289)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:125)
^-- Holding lock: java.io.InputStreamReader@1f9c05d7[thin lock]
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:105)
java.io.BufferedReader.readLine(BufferedReader.java:288)
^-- Holding lock: java.io.InputStreamReader@1f9c05d7[thin lock]
java.io.BufferedReader.readLine(BufferedReader.java:362)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:287)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:312)
weblogic.nodemanager.client.NMServerClient.start(NMServerClient.java:93)
^-- Holding lock: weblogic.nodemanager.client.SSLClient@1f9c0e0d[thin lock]
weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:45)
weblogic.kernel.WorkManagerWrapper$1.run(WorkManagerWrapper.java:63)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}

NodeManager not Reachable: java.io.IOException: Invalid State File Format

Weblogic managed Servers cannot be started via the WLS console as Node Manager is not reachable.
Also, the nodemanager.log file is showing the following messages:
domainName\servers\serveName\data\nodemanager\startup.properties">
FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\serveName\data\nodemanager\startup.properties">
<Feb 9, 2011 11:06:01 AM> <WARNING> <There was a problem initializing the domain 'steffworld' at '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'. Please make sure that this domainName: 'domainName' is registered and is fully enrolled for this nodemanager at: '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'.>
<Feb 9, 2011 11:06:01 AM> <WARNING> <I/O error while reading domain directory>
java.io.IOException: Invalid state file format. State file contents:
at weblogic.nodemanager.common.StateInfo.load(StateInfo.java:135)
at weblogic.nodemanager.server.ServerMonitor.loadStateInfo(ServerMonitor.java:475)
at weblogic.nodemanager.server.ServerMonitor.isCleanupAfterCrashNeeded(ServerMonitor.java:139)
at weblogic.nodemanager.server.ServerManager.recoverServer(ServerManager.java:255)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:103)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:55)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:257)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:224)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:108)
at weblogic.nodemanager.server.Handler.run(Handler.java:70)
at java.lang.Thread.run(Thread.java:619)

Changes

After a power failure, the server machine (running as a Windows service) restarted automatically.

Cause

The state file of the managed server is in an invalid state: for example, it may be empty. Under each managed server directory, there is a NodeManager directory containing a state file <managed_server_name>.state. If this file is empty or corrupt, then the described errors occur.
For example, under \FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\<serverName>\data\nodemanager, the <serverName>.state file is empty.
The cause of the empty file could be one of the following:

WebLogic Server: Procedure for configuring Node Manager with SSL

Steps for configuring Node Manager with SSL for WebLogic Server.

First create custom certificates using the keytool command utility:

keytool -genkey -alias mykey -keyalg RSA -keysize 1024 -dname "CN=Tariq, OU=Customer Support, O=BEA Systems Inc, L=Denver, ST=Colorado, C=US" -keypass password -keystore identity.jks -storepass password
keytool -selfcert -v -alias mykey -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -selfcert -v -alias mykey -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -import -v -trustcacerts -alias mykey -file rootCA.der -keystore trust.jks -storepass password


Now configure "Custom Identity and Custom Trust" for admin and managed servers from console
Now enter the Key Alias and Private Key Passphrase under the SSL tab for both the servers from console