Showing posts with label TROUBLESHOOT. Show all posts
Showing posts with label TROUBLESHOOT. Show all posts

Jdeveloper composite deployment error : java.lang.NoClassDefFoundError: oracle/integration/platform/common/MDSSessionBuilder

Error 500--Internal Server Error
java.lang.NoClassDefFoundError: oracle/integration/platform/common/MDSSessionBuilder at oracle.integration.platform.blocks.deploy.servlet.MDSManagerUtils.getMDSSession(MDSManagerUtils.java:203) at oracle.integration.platform.blocks.deploy.servlet.MDSManager.transferCompositeData(MDSManager.java:494) at oracle.integration.platform.blocks.deploy.servlet.BaseDeployProcessor.deploySARs(BaseDeployProcessor.java:243) at oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWork(DeployProcessor.java:203) at oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeployWork(DeployProcessor.java:147) at oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.doDeploy(DeployProcessor.java:134) at oracle.integration.platform.blocks.deploy.servlet.DeployProcessor.process(DeployProcessor.java:100) at oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.doPostInsideLoggingSession(CompositeDeployerServlet.java:221) at oracle.integration.platform.blocks.deploy.servlet.CompositeDeployerServlet.doPost(CompositeDeployerServlet.java:130) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138) at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324) at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464) at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121) at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211) at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138) at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324) at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464) at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121) at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211) at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Admin Server Fails To Start weblogic.security /w SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception Caused By: java.lang.NoSuchMethodError: javax/xml/transform/TransformerFactory.newInstance

The AdminServer.log shows:
#### <> <> <> weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: javax/xml/transform/TransformerFactory.newInstance(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/transform/TransformerFactory;
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.NoSuchMethodError: javax/xml/transform/TransformerFactory.newInstance(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/transform/TransformerFactory;
at oracle.security.jps.service.policystore.info.RuleExpressionEntry.(RuleExpressionEntry.java:1708)
at oracle.security.jps.internal.policystore.entitymanager.impl.PolicyRuleManagerImpl.(PolicyRuleManagerImpl.java:212)
at oracle.security.jps.internal.policystore.info.PolicyDomainEntryImpl.(PolicyDomainEntryImpl.java:111)
at oracle.security.jps.internal.policystore.JpsAppPolicy.(JpsAppPolicy.java:474)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.init(XmlPolicyStore.java:355)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.initDataStoreEntry(XmlPolicyStore.java:484)
at oracle.security.jps.internal.policystore.xml.XmlPolicyStore.buildFromFile(XmlPolicyStore.java:525)

JDBC Error Causes in WebLogic Server and troubleshoot

Even a transaction timeout will not kill or timeout any action that is done by the resources that are enlisted in this transaction. The actions will run as long as they take, without interruption. A transaction timeout will set a flag on the transaction that will mark it as rollback only, so that any subsequent request to commit this transaction will fail with a TimedOutException or RollbackException. However, as mentioned above, the long running JDBC calls can lead to blocked WebLogic Server execute threads, which can finally lead to a hanging instance, if all threads are blocked and no execute thread remains available for handling incoming requests.

More recent WebLogic Server versions have a health check functionality that regularly checks if a thread does not react for a certain period of time (the default is 600 seconds). If this happens, an error message is printed to your log file similar to following:

The time interval for the health check functionality is configurable. Please check the StuckThreadMaxTime property in the <Server> tag of your config.xml file or the "Detecting stuck threads" section in the WebLogic Server administration console help

The following are some different possible reasons that can cause JDBC calls to lead to a hanging WebLogic Server instance:

WebLogic Server : Binary Core File Analysis

Problem Description

An application gets a binary core file produced when the WebLogic Server process terminates due to some invalid native core (machine specific code). A server crash, JVM crash, machine crash, or HotSpot error may also be associated with this occurrence. This article will describe what steps are needed to gather information from a core file on various platforms.

Problem Troubleshooting

Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.

Troubleshooting Steps

Why does the problem occur?

In order to determine the cause of such an error you need to determine all potential sources of native code used by the WebLogic Server process. The places to focus on are:
  1. The WebLogic Server performance pack. The WebLogic Server performance pack is native code and when enabled could potentially produce such an error. Disable this feature to determine if that is the cause. You can do this via the console or via the command line. Using the console look under the Server tab by setting NativeIOEnabled to false. See the documentation section Enabling Performance Packs to get the exact sequence of steps under the Server tab in the console. The steps are:
    1. Start the Administration Server if it is not already running.
    2. Access the Administration Console for the domain.
    3. Expand the Servers node in the left pane to display the servers configured in your domain.
    4. Click the name of the server instance that you want to configure.
    5. Select the Configuration --> Tuning tab.
    6. If the Enable Native IO check box is selected, please deselect it in the check box so it is now *not* enabled.
    7. Click Apply.
    8. Restart the server.

Investigating ORA-01000: maximum open cursors exceeded WebLogic Server

Problem Description

Oracle uses the OPEN_CURSORS parameter to specify the maximum number of open cursors a session can have at once. When this number is exceeded, Oracle reports an ORA-01000 error. When this error is propagated to WebLogic Server, a SQLException is thrown.
java.sql.SQLException: ORA-01000: maximum open cursors exceeded
This pattern discusses possible causes and solutions of this error when using WebLogic Server.

Problem Troubleshooting

Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.

NoClassDefFoundError When Deploying JSF Application Using Commons-Logging And Log4j to Oracle WebLogic

Symptoms

When a single JSF page application with a backing bean that contains an apache commons logging statement using Log4j.jar as the logging provider is deployed,Weblogic always throws an exception.
Including commons-logging.jar and log4j.jar on the classpath causes the exception.


com.sun.faces.mgbean.ManagedBeanCreationException: Cant instantiate class: view.backing.Hello.
at com.sun.faces.mgbean.BeanBuilder.newBeanInstance(BeanBuilder.java:193)
at com.sun.faces.mgbean.BeanBuilder.build(BeanBuilder.java:105)
at com.sun.faces.mgbean.BeanManager.createAndPush(BeanManager.java:369)
at com.sun.faces.mgbean.BeanManager.create(BeanManager.java:230)
at com.sun.faces.el.ManagedBeanELResolver.getValue(ManagedBeanELResolver.java:88)
Truncated. see log file for complete stacktrace
org.apache.commons.logging.LogConfigurationException:
org.apache.commons.logging.LogConfigurationException: No suitable Log constructor
[Ljava.lang.Class;@16b7343 for org.apache.commons.logging.impl.Log4JLogger (Caused by
java.lang.NoClassDefFoundError: org/apache/log4j/Logger) (Caused by
org.apache.commons.logging.LogConfigurationException: No suitable Log constructor
[Ljava.lang.Class;@16b7343 for org.apache.commons.logging.impl.Log4JLogger (Caused by
java.lang.NoClassDefFoundError: org/apache/log4j/Logger))

at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at view.backing.Hello.<init>(Hello.java:14)
Truncated. see log file for complete stacktrace
org.apache.commons.logging.LogConfigurationException: No suitable Log constructor
[Ljava.lang.Class;@16b7343 for org.apache.commons.logging.impl.Log4JLogger (Caused by
java.lang.NoClassDefFoundError: org/apache/log4j/Logger)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:413)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
Truncated. see log file for complete stacktrace
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.getConstructor(Class.java:1657)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:410)
Truncated. see log file for complete stacktrace
java.lang.ClassNotFoundException: org.apache.log4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
Truncated. see log file for complete stacktrace


Cause

The webapp classes are not overriding the system classes.

Solution

Add the parameter "prefer-web-inf-classes" to your web-app's weblogic.xml:

   <container-descriptor>
      <prefer-web-inf-classes>true</prefer-web-inf-classes>
   </container-descriptor>


To create a weblogic.xml descriptor:
  1. Right click your ViewController Project in JDeveloper, select New.
  2. Go to Deployment Descriptors > WebLogic Deployment Descriptors > Click OK.
  3. Choose weblogic.xml and follow through.

Deploying Application on Oracle Weblogic 10.3.x throws java.lang.NoClassDefFoundError: org/apache/log4j/Priority


Symptoms

Deploying an ear application onto a WLS 10.3.x domain throws the following error:
Web application: "DataExtractWeb2".
java.lang.NoClassDefFoundError: org/apache/log4j/Priority
  at org.sagph.dataextract.web.DxWebStartup.loadProperties(DxWebStartup.java:101)
  at org.sagph.dataextract.web.DxWebStartup.init(DxWebStartup.java:60)
  at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
Caused By: java.lang.NoClassDefFoundError: org/apache/log4j/Priority
  at org.sagph.dataextract.web.DxWebStartup.loadProperties(DxWebStartup.java:101)
  at org.sagph.dataextract.web.DxWebStartup.init(DxWebStartup.java:60)
  at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
  at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
The same application works fine on WLS 10MP1 but it is not working on WLS 10.3.x.



Cause


In WLS versions prior to 10.3.0, the log4j.jar is present in the class path and it gets loaded automatically at the time of server start. But this is not the case for WLS 10.3.0 and later releases. In WLS 10.3.0 and higher, the log4j file is no longer automatically loaded. In this case, the customer application had a dependency on log4j, and so it failed when log4j.jar was no longer automatically loaded with the WLS installation.


Solution


There are two ways to resolve, as below:

The first solution is to load the log4j jar file on startup. To accomplish this, change the value of load-on-startup tag to 1 from -1in web.xml for the servlet code that is referring to log4j. For example, in web.xml, make this change:
<load-on-startup>1</load-on-startup>

The second solution is to place the jar file in the APP-INF/lib directory of the ear file, i.e., <ear_file>/APP-INF/lib. The jar file will be loaded with the ear file and provide the jar file when the application needs it. After adding the jar file to the ear, redeploy the application and restart.

Oracle Weblogic: Deployment of Applications Failing with java.lang.ClassCastException

Sometime weblogic throws ClassCastException while application deployment. Below is one of the scenario and steps how go through with this issue.

Symptoms


WebLogic Server deployment of applications using stax classes is failing with a ClassCastException:

java.lang.ClassCastException: com.ctc.wstx.stax.WstxInputFactory

Cause

This issue was because the customer application uses a different stax jar file version as compared to jar file used by system class loader. Since both have "com.ctc.wstx.stax.WstxInputFactory" as a member, the class is loaded by the system class loader rather than the application class loader, and that is incompatible with its use by the application code.

Solution


To resolve this issue, use the FilteringClassLoader in weblogic.xml (or) weblogic-application.xml descriptor files to load particular classes/packages (in this case, "com.ctc.wstx.stax.*") from the application jar file instead of the system classloader. Below is a snippet of weblogic-application.xml file which has FilteringClassLoader implementation (<prefer-application-packages>) for reference.
UTF-8" ?>
<wls:weblogic-application xmlns:wls="http://www.bea.com/ns/weblogic/90"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/j2ee_1_4.xsd">
<wls:application-param>
<wls:param-name>webapp.encoding.default</wls:param-name>
<wls:param-value>UTF-8</wls:param-value>
</wls:application-param>

<prefer-application-packages>
<package-name>javax.jws.*</package-name>
<package-name>org.apache.xerces.*</package-name>
<package-name>com.sun.xml.messaging.saaj.*</package-name>
<package-name>com.ctc.wstx.stax.*</package-name>
<package-name>javax.xml.stream.*</package-name>
</prefer-application-packages>


JVM crash: java.lang.OutOfMemoryError

The JVM crashed. The head of the fatal error log file shows the error message in bold below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
#
Internal Error (allocation.cpp:166), pid=14032, tid=22
Error: ChunkPool::allocate
#
# JRE version: 6.0_23-b05
# Java VM: Java HotSpot(TM) Server VM (19.0-b09 mixed mode solaris-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Cause

The malloc() call in the ChunkPool::allocate() function failed to allocate new memory, causing the JVM process to exit with the error "ChunkPool:allocate".

In other words, the HotSpot JVM ran out of native memory in the space reserved in native heap (C-heap) for compiling java methods.

WLST Failed Authentication at connect() When Stopping WebLogic Server


Symptoms

Reports have occurred that users are unable to stop WebLogic Server using stopWebLogic.sh with the following error:

Connecting to t3://hostname.domainname:2001 with userid {AES}qFA3rQexfsasfgjdf560srqyGQ15UNglcziS0uR1yJQw= ...
This Exception occurred at Wed Feb 02 16:31:32 PST 2011.
javax.naming.AuthenticationException [Root exception is java.lang.SecurityException: User:
{AES}qFA3rQexfsasfgjdf560srqyGQ15UNglcziS0uR1yJQw, failed to be authenticated.]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:42)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:787)
at weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:681)
at weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:469)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:376)
at weblogic.jndi.Environment.getContext(Environment.java:315)

Oracle WebLogic Server Plug-Ins and SSL

Introduction

This document was created to help users understand their needs when using the WebLogic plugin and SSL. It describes in detail questions to ask when setting up the architecture of the environment. The three web servers that will be used as examples are: Apache, iPlanet (SunOne), and Microsoft IIS.

Prerequisites

Before you start, it is important to understand the handshake process. Refer to the Understanding and Investigating SSL Issues  for information.
Before you start, ask yourself the following questions:
  1. Will I have SSL set up between the client and the web server hosting the proxy (Apache, Sun One. IIS)?

    If the answer is yes, will it need to be 2-way SSL? This design has the advantage of offering the possibility to propagate client certificates to the back-end WebLogic Server (e.g., for authentication).
     
  2. Will I have SSL set up between the plugin and the WebLogic Server?

    If the answer is yes, will I need to "intercept" a client certificate from the first front-end handshake?
     
  3. Is it only 1-way SSL that I need? Is it only to encrypt the data between the plugin and the WebLogic Server?

Oracle Weblogic Server: How To Take Thread Dumps With WLST

The first step is to establish a connection to the Admin Server. From there, the script will perform the threadDump() function in a particular server. The number of thread dumps to take can be defined and the sleep time between them as well.

Every thread dumps will be saved as a file with the following name structure "dump_<server name>_<thread dump time>.dmp" in the same directory where the script was launched. After that, the connection with the Admin Server will be closed.

The following script will take 5 thread dumps on the server "MS1", 15 seconds apart form each other.

Linux version



connect('weblogic','weblogic1','localhost:7001')

from time import strftime
from java.text import SimpleDateFormat
serverName = 'MS1'
counter = 0
sleepTime = 15000
threadNumber = 5

for counter in range(threadNumber):
        currentDate = java.util.Date().toString()
        myDate = currentDate.split(' ');
        finalDate = myDate[3]
        java.lang.Thread.sleep(sleepTime)
        fileName = 'dump' + '_' + serverName + '_' + finalDate + '.dmp'
        threadDump('true', fileName, serverName)

disconnect()

Windows version

This particular version of the script introduces some changes in the way how thread dump files are being saved.

connect('weblogic','weblogic1','localhost:7001')

from java.lang import *
from java.util import Date

serverName = 'MS1'
counter = 0
sleepTime = 15000
threadNumber = 10

d= Date()

for counter in range(threadNumber):
        currentFile = 'Thread_Dump_%s_%s_%s.log' % (serverName, d.time,counter)
        threadDump(writeToFile='true', fileName=currentFile,serverName=serverName)
        currentFileRead = open(currentFile, 'r')
        currentFileRead.close()
        Thread.sleep(sleepTime)

disconnect()

Run the script as follow: java weblogic.WLST ThreadDumps.py

Different ways to take thread dumps in Oracle WebLogic Server

Tux, the Linux penguin
Tux, the Linux penguin (Photo credit: Wikipedia)

Thread dumps are essential diagnosis information used to analyze and troubleshoot performance related issues such as server hangs, deadlocks, slow running, idle or stuck applications, slow database interactions etc...

Different ways to take thread dumps in WebLogic Server

WebLogic Server (WLS) and Java offer several ways to generate thread dumps, they are detailed below.  It is always recommended to obtain the thread dumps by using operating system (OS) commands rather than by using Java classes or the Administration Console, because if the console is hanging, users won't be able to connect to it to issue thread dumps.

  1. Use operating system commands to get the thread dumps when WLS starts up from a command-line script:
    • On Windows OSes, thread dumps can be created by
      <ctrl>+<break> -- the thread dumps are generated in the server stdout
    • On POSIX-compliant platforms (e.g. Solaris and Linux), first identify the process ID (pid) using the command ps -ef | grep java, then run
      kill -3 <pid> 2>&1
      Signal 3 is equivalent to SIGQUIT. Note that in Solaris, the thread dump is generated in the current shell, but in Linux, the thread dump is generated in the shell which started the java process specified by the pid.
  2. Using beasvc (up to WLS 10.3.5 included):
    beasvc -dump -svcname:<service_name>
    • service_name is the Windows service that is running the server instance (e.g. mydomain_myserver)
  3. Using wlsve (from 10.3.6/12.1.1):
    wlsve -dump -svcname:<service_name>
  4. Using weblogic.WLST:
    setDomain.cmd or setDomain.sh depending on the OS
    java weblogic.WLST
    connect("<username>","<password>","t3://<url>:<port>")
    threadDump()
    The thread dump will be generated in Thread_Dump_AdminServer.txt.   WLST thread dump in more details with examples on how to define sleep time between each dump and number of dumps to take.
  5. From a command line or shell, a thread dump can be generated via the following command (deprecated from WLS 9.0):
    setDomain.cmd or setDomain.sh depending on the OS
    java weblogic.Admin <url>:<port> -username <username> -password <password> THREAD_DUMP
    The thread dump will be generated in the defined server stdout.
  6. From the WLS Administration Console, a thread dump can be created by navigating to Server -> <server_name> -> Monitoring -> Dump threads stack. This method could lead to truncated or incomplete thread dumps.
  7. Java VisualVM can also be used to take thread dumps while applications are running, see http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/applications_local.html for more details
  8. With jstack
    jstack <pid> or jstack -l <pid> to print additional information about locks
  9. From the JRockit command line:
    jrcmd <pid> print_threads
  10. From Java Mission Control with JDK 7:
    jcmd <pid> Thread.print 

Oracle Weblogic : Troubleshooting Out of Memory and Memory Leak Problems


Out Of Memory (OOM): An Out of Memory error occurs due to memory exhaustion, either in java heap or native memory. In the JVM, OOM errors are thrown when the JVM cannot allocate an object because it is out of heap memory, and no more heap memory could be made available by the garbage collector.
Memory Leak: A memory leak occurs if memory is used by an application and not released by the application when it is finished with it. A memory leak can occur in either java heap or native memory, and either will eventually cause an out of memory situation.


Problem Troubleshooting

Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.

Troubleshooting Steps

Java heap, Native memory and Process size

Java heap: This is the memory that the JVM uses to allocate java objects. The maximum value of java heap memory is specified using the -Xmx flag in the java command line. If the maximum heap size is not specified, then the limit is decided by the JVM considering factors like the amount of physical memory in the machine and the amount of free memory available at that moment. It is always recommended to specify the max java heap value.
Native memory: This is the memory that the JVM uses for its own internal operations. The amount of native memory heap that will be used by the JVM depends on the amount of code generated, threads created, memory used during GC for keeping java object information and temporary space used during code generation, optimization etc.
If there is a third party native module, it could also use the native memory. For example, native JDBC drivers allocate native memory.
The max amount of native memory is limited by the virtual process size limitation on any given OS and the amount of memory already committed for the java heap with -Xmxflag. For example, if the application can allocate a total of 3 GB and if the max java heap is 1 GB, then the max possible native memory is approximately 2 GB.
Process size: Process size will be the sum of the java heap, native memory and the memory occupied by the loaded executables and libraries. On 32-bit operating systems, the virtual address space of a process can go up to 4 GB. Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 - 2 GB). The remaining is available for the application.
Windows: Different versions of Windows support different process sizes. See http://msdn.microsoft.com/en-us/library/windows/desktop/aa366914%28v=VS.85%29.aspx for more details.
RedHat Linux: Different kernels are available on RH Linux, and these differnet kernels support different process sizes. See https://blogs.oracle.com/gverma/entry/redhat_linux_kernels_and_proce_1 for more details.
For other operating systems, please refer to the OS documentation for your configuration.
For more information on configuring all of these for WebLogic Server, please see Tuning Java Virtual Machines (JVMs).

Node Manager: Common Problems and Resolutions

Following are some of the exceptions/errors and resolutions.

Host name verification (Node Manager log)

Following problem is due to Node manager Setup and seen in the node manager log:

<May 3, 2005 1:00:45 PM EDT> <Error> <NodeManager@xxxx11:5559> <NodeManager is not configured to receive commands from host : /10.62.3.215. Please update the trusted hosts file : /home/rbabu/nodemanager.hosts of the node manager by adding the hostname or ip address of /10.62.3.215>

Resolution: Add the host name or IP address to nodemanager.hosts and restart the node manager.
If, after adding the entry to the nodemanager.hostsfile you still see the error, add the following to the node manager start script and admin server.


Node Manager:

-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
Admin Server:

-Dweblogic.security.SSL.ignoreHostnameVerification=true
Or you can do the same using console as shown below:
Under Keystores & SSL tab, click on "Advanced Options." Change the Hostname verification to None.

Checklist for Troubleshooting Node Manager SSL Problems

  1. Check what certificates are being used. Demo, Commercial, self-signed?
  2. In the case of demo certificates make sure none of the settings are changed. You don't need any entries in the nodemanager.propertiesfile. Nor you do not need to make any changes to the settings in the admin or managed server.
  3. In the case of commercial certificates (Verisign, Thawte, Comodo, etc.) make sure that the certificate chain is complete and the root and intermediate certificates are configured properly.
  4. If the certificates are self-signed, make sure you have followed the sequence mentioned earlier in this document.
  5. Make sure that the validation dates are correct.
  6. Turn on the debug flags on both admin server and managed server to get all possible information.
  7. Node manager debug flag. This flag needs to be added to the start script
  8. (The log files are located under <NodeManagerHome>/nodemanager.log)

  • -Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true
  • -Dweblogic.StdoutDebugEnabled=true -Dweblogic.nodemanager.debugEnabled=true -Dweblogic.nodemanager.debugLevel=90