Showing posts with label EXCEPTION. Show all posts
Showing posts with label EXCEPTION. Show all posts

Java.Lang.OutOfMemoryError In weblogic.WLST Remote Deploy Command


Symptoms


Failure to remotely deploy a very large enterprise application using WLST or the Admin console and getting out of memory error while loading the ear file.
WLSTException: 'Error occured while performing deploy : Target exception thrown while deploying application: Error occured while performing deploy :
Deployment Failed. Error occurred while performing deploy : Deployment Failed.

Use dumpStack() to view the full stacktrace.
The load of a remotely located ear file also fails via the Administration console with the following message:

An unexpected exception has occurred processing your request Message:
  Stack Trace: java.lang.NullPointerException
  at com.bea.console.actions.app.install.Flow.uploadApp(Flow.java:256)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Cause


The issue was addressed in bug 9456759. As part of the fix, a system property flag weblogic.deploy.UploadLargeFile was added. In WLS 10.3.4 and higher, this flag should be added to the Java startup options for the deployment client if large files are to be deployed. For earlier versions, apply the patch below: no flag is needed.


Solution


Patches are available for Bug 9456759:
PATCH INFORMATION
WLS VersionPatch Number
9.2.2Patch 9456759
9.2.3Patch 9456759
10.0.1Patch 9456759
10.0.2Patch 9456759
10.3.0Patch 9456759
10.3.1Patch 9456759
10.3.2Patch 9456759

Fixed in: 9.2.4, 10.3.4
To apply one of these patches, click on the link for your WLS version and download the appropriate patch from My Oracle Support. You can also search in My Oracle Support for the patch number for your WLS version:


NOTE: Patches are applied per WLS installation and not per domain. That is, if you apply this patch on one WLS installation, then all of the servers from all the domains in that installation will have this patch. On the other hand, if you have a managed server in another machine in a domain (that is, set up with its own WLS installation), you need to install this patch on that other machine as well. Generally, patches can only be applied while the server is not running because WLS locks the needed files while it is running. If, however, you are able to apply a patch while WLS is running, you must restart WLS before the patch will take effect.

WLST Failed Authentication at connect() When Stopping WebLogic Server


Symptoms

Reports have occurred that users are unable to stop WebLogic Server using stopWebLogic.sh with the following error:

Connecting to t3://hostname.domainname:2001 with userid {AES}qFA3rQexfsasfgjdf560srqyGQ15UNglcziS0uR1yJQw= ...
This Exception occurred at Wed Feb 02 16:31:32 PST 2011.
javax.naming.AuthenticationException [Root exception is java.lang.SecurityException: User:
{AES}qFA3rQexfsasfgjdf560srqyGQ15UNglcziS0uR1yJQw, failed to be authenticated.]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:42)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:787)
at weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:681)
at weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:469)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:376)
at weblogic.jndi.Environment.getContext(Environment.java:315)

Oracle Weblogic Server : How to Disable the Exception Stack Trace in a WLST Script

This document describes how to disable an exception stack trace from writing out to the screen from a WebLogic Scripting Tool (WLST) Python script file. A small demonstration is provided to show how the parameter works.
A Python script file, with try/catch code implemented, will display a stack trace to the screen when the script is executed with WLST.


Here is an example. The abc.py script file is using an invalid password:

try:
  connect('weblogic','invalid_password','t3://localhost:7001')
   print "Successfully connected"
except:
  print "Error while trying to connect!!!"
exit()
Execute the script file as follows, and observe the output shown:

/WLS/bea1030/wlserver_10.3/common/bin>wlst.sh abc.py
Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...
This Exception occurred at Fri Apr 22 05:34:13 IST 2011.
javax.naming.AuthenticationException [Root exception is java.lang.SecurityExcept
ion: User: weblogic, failed to be authenticated.]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(Exceptio
nTranslator.java:42)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLIni
tialContextFactoryDelegate.java:783)
at weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialCo
ntextFactoryDelegate.java:677)
.
.
.
.
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.jav
a:473)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.jav
a:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Error while trying to connect!!!

Exiting WebLogic Scripting Tool.

Solution

To resolve the issue, use the following script which redirects the stack trace to a temp file and then deletes the temp file in the end:

import tempfile
import os
wlstOut = tempfile.mktemp(suffix="_wlst.txt")
redirect(wlstOut,"false")
try:
   connect('weblogic','weblogic','t3://localhost:7001')
   stopRedirect()
   print "Successfully connected"
except:
   stopRedirect()
   print "Error while trying to connect!!!"
os.remove(wlstOut)
exit()
Thus running the script now will give following output (excluding the stack trace for the exception), which is the goal:
/WLS/bea1030/wlserver_10.3/common/bin>wlst.sh abc.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...

Error while trying to connect!!!

Exiting WebLogic Scripting Tool.

Oracle WebLogic Server - Apache Proxy log Throws READ_ERROR_FROM_FILE


There are reports of requests failing and the Apache Proxy log throwing "READ_ERROR_FROM_FILE" and "Cannot read 0 bytes of postData from tmp file" exceptions in WebLogic Server.
Below is the exception in Apache proxy logs at the time of the issue:
READ_ERROR_FROM_FILE [os error=0, line 144 of ap_proxy.cpp]: Cannot read 0 bytes of postData from tmp file '/tmp/_wl_proxy/_post_916_xxx

Cause

The problem is related to network latency and file caching. When the POST data in a request is greater than 2048 bytes and when FileCaching is set to ON, the POST data is first read into a temporary file on disk and then forwarded to the WebLogic Server in chunks of 8192 bytes.

Solution

To resolve this issue disable File Caching on the Apache web server.
To disable File Caching, in the Apache configuration under <IfModule mod_weblogic.c>, add the setting below:
FileCaching OFF

Can I Install or Restore a Patch Now Appearing As Invalidated In Smart Update?

I'm installing a set of new patches. One of them is supposed to replace an older patch which is now marked in Smart Update as "invalidated."

In the event there is a problem with the application after the new patches are installed, I would like to restore the original combination of patches. Is it possible to restore the original patch even though it is now invalidated?


Cause


I'm installing a set of new patches. One of them is supposed to replace an older patch which is now marked in Smart Update as "invalidated."

In the event there is a problem with the application after the new patches are installed, I would like to restore the original combination of patches. Is it possible to restore the original patch even though it is now invalidated?

Solution

If a patch is invalid, that means that there is a problem with it, and we're specifically preventing you from using it. You don't want to install that patch because you'd be introducing errors into your system.

Attempting to install or restore an invalidated patch is thus strongly discouraged, for your own safety.

You should be sure to test the new patches thoroughly in your test environment before putting them into production. Work out any problems with the new patches there. Then when you're ready to put the new patches into production, you can do so with confidence that you won't need to revert to an older, problematic version.

Smart Update Fails to Apply Patch: java.io.IOException: Unable to backup file


When attempting to install a patch using Smart Update (BSU), the following error occurs:
java.io.IOException: Unable to backup file <path>/weblogic_patch.jar to <path>/weblogic_patch.jar.bak
Verbose logging gives some more details:
java.util.concurrent.ExecutionException: com;bea.plateng.patch.PatchInstallationException: java.io.IOException: Unable to backup file <path>/weblogic_patch.jar

Cause

This is a common problem customers experience when they try to apply patches while the WLS domain is running. Because the domain is running, the jar which Smart Update needs to write to is locked, and this error occurs.

Solution

Please stop the WLS domain, and make sure there are no processes or services running which would be holding a lock on that jar file. If Node Manager is running, you should also stop that.
Once all those processes and/or services are stopped, please run Smart Update and apply the patch. It should work.

WebLogic Server: Smart Update Throwing OutOfMemoryError


An issue has been reported of the Smart Update utility throwing an OutOfMemoryError when using bsu.cmd or bsu.sh script inside WL_HOME/utils/bsu directory. Below is the exception trace in bsu.log at the time of the issue.

JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "main" java.lang.OutOfMemoryError
at org.apache.xmlbeans.impl.store.Cur.createElementXobj(Cur.java:257)
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.startElement(Cur.java:2992)
at org.apache.xmlbeans.impl.store.Locale$SaxHandler.startElement(Locale.java:3194)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportStartTag(Piccolo.java:1082)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseOpenTagNS(PiccoloLexer.java:1473)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseTagNS(PiccoloLexer.java:1362)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXMLNS(PiccoloLexer.java:1293)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXML(PiccoloLexer.java:1261)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.yylex(PiccoloLexer.java:4808)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yylex(Piccolo.java:1290)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yyparse(Piccolo.java:1400)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:714)
at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3435)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1270)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1257)
at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345)
at com.bea.cie.common.dao.xbean.XBeanDataHandlerSpiImpl.create(XBeanDataHandlerSpiImpl.java:130)
at com.bea.cie.common.dao.xbean.XBeanDataHandlerSpiImpl.create(XBeanDataHandlerSpiImpl.java:111)
at com.bea.plateng.patch.dao.cat.PatchCatalogFactory.create(PatchCatalogFactory.java:163)
at com.bea.plateng.patch.dao.cat.PatchCatalogFactory.create(PatchCatalogFactory.java:85)
at com.bea.plateng.patch.dao.cat.PatchCatalogFactory.create(PatchCatalogFactory.java:71)
at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchCatalog(PatchCatalogHelper.java:57)
at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:363)
at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
at com.bea.plateng.patch.PatchSystem.setCacheDir(PatchSystem.java:201)
at com.bea.plateng.patch.Patch.main(Patch.java:281)

NodeManager not Reachable: java.io.IOException: Invalid State File Format

Weblogic managed Servers cannot be started via the WLS console as Node Manager is not reachable.
Also, the nodemanager.log file is showing the following messages:
domainName\servers\serveName\data\nodemanager\startup.properties">
FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\serveName\data\nodemanager\startup.properties">
<Feb 9, 2011 11:06:01 AM> <WARNING> <There was a problem initializing the domain 'steffworld' at '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'. Please make sure that this domainName: 'domainName' is registered and is fully enrolled for this nodemanager at: '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'.>
<Feb 9, 2011 11:06:01 AM> <WARNING> <I/O error while reading domain directory>
java.io.IOException: Invalid state file format. State file contents:
at weblogic.nodemanager.common.StateInfo.load(StateInfo.java:135)
at weblogic.nodemanager.server.ServerMonitor.loadStateInfo(ServerMonitor.java:475)
at weblogic.nodemanager.server.ServerMonitor.isCleanupAfterCrashNeeded(ServerMonitor.java:139)
at weblogic.nodemanager.server.ServerManager.recoverServer(ServerManager.java:255)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:103)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:55)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:257)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:224)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:108)
at weblogic.nodemanager.server.Handler.run(Handler.java:70)
at java.lang.Thread.run(Thread.java:619)

Changes

After a power failure, the server machine (running as a Windows service) restarted automatically.

Cause

The state file of the managed server is in an invalid state: for example, it may be empty. Under each managed server directory, there is a NodeManager directory containing a state file <managed_server_name>.state. If this file is empty or corrupt, then the described errors occur.
For example, under \FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\<serverName>\data\nodemanager, the <serverName>.state file is empty.
The cause of the empty file could be one of the following: