Incidently, I've noticed that many clients invoke the _is_non_existent() method
on the servants before invoking an operation.

From: Rhythm Tyagi 

We are using Jacorb version 2.2.1 on RedHat Linux and facing a very similar 
issue.


We see that over time, the number of Jacorb threads keep on increasing. In 
particular I see two types of threads:-

 

"Thread-183" daemon prio=1 tid=0x08cb3cb8 nid=0xa73 runnable 
[15276000..1527686c]

            at java.net.SocketInputStream.socketRead0(Native Method)

            at java.net.SocketInputStream.read(SocketInputStream.java:129)

            at org.jacorb.orb.iiop.IIOPConnection.read(IIOPConnection.java:93)

            at org.jacorb.orb.giop.GIOPConnection.getMessage
(GIOPConnection.java:249)

            at org.jacorb.orb.giop.GIOPConnection.receiveMessages
(GIOPConnection.java:339)

            at org.jacorb.orb.giop.MessageReceptor.doWork
(MessageReceptor.java:53)

            at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:68)

            at java.lang.Thread.run(Thread.java:534)

 

"Thread-898924" daemon prio=1 tid=0x17a8cc38 nid=0xa73 in Object.wait() 
[bc535000..bc53586c]

            at java.lang.Object.wait(Native Method)

            at java.lang.Object.wait(Object.java:429)

            at org.jacorb.util.threadpool.ThreadPool.getJob(ThreadPool.java:125)

            - locked <0x92c69ec8> (a org.jacorb.util.threadpool.ThreadPool)

            at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:56)

            at java.lang.Thread.run(Thread.java:534)

 

I believe the second kind of thread issue has been fixed and was tracked by 
bugs 577 and 626. However I am more concerned

about the former. It seems like the thread hangs during the client read. Could 
you check and let us know more details on this.

 

Also I have a heap dump and I see a large number of jacorb objects:-

 

num   #instances    #bytes  class name

--------------------------------------

  1:     52131    11860384  [B

  2:     65913    11526712  <methodKlass>

  3:    112673    10426904  [C

  4:     14709     6657840  [I

  5:    109065     4921168  <symbolKlass>

  6:      5352     3643992  <constantPoolKlass>

  7:    113752     2730048  java.lang.String

  8:      4692     2361632  <constantPoolCacheKlass>

  9:      5352     1816152  <instanceKlassKlass>

 10:     19793     1602464  [Ljava.lang.Object;

 11:     13734     1318464  org.jacorb.orb.giop.ServerGIOPConnection

 12:     13864     1109120  java.net.PlainSocketImpl

 13:     16224     1095984  [Ljava.util.Hashtable$Entry;

 14:     29542      709008  java.util.Hashtable$Entry

 15:     83474      667792  java.lang.Object

 16:     13734      659232  org.jacorb.orb.iiop.ServerIIOPConnection

 17:     15322      612880  java.util.Hashtable

 

Could you help us resolve this issue? Let us know in case you need any other 
information from our side.

 

I would really appreciate your help on this.

 
Regards

Rhythm

Could someone please work on this? We are facing this issue very often. Is 
there any fix already available in 2.2.3 version?

Please test with the current release.

Tested with 2.2.4, and it still occurs. Is anyone going to work on this? it's
more than 2 years old. 

The current release is 2.3.0 - please reproduce the issue with that.

What properties (e.g. timeout) do you have configured? What is your
architecture? Do you have code to reproduce the issue?

You should try to run your server under a current JVM and inspect the running
JVM using jconsole to check what threads are there.

Have you seen bug 801?

Following is a sample thread dump:-

Thread Dump starts here

***************************

***************************

 

Name: Thread-104

State: RUNNABLE

Total blocked: 0  Total waited: 0

 

Stack trace: 

java.net.PlainSocketImpl.socketAccept(Native Method)

java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

java.net.ServerSocket.implAccept(ServerSocket.java:450)

java.net.ServerSocket.accept(ServerSocket.java:421)

org.jacorb.orb.iiop.IIOPListener$Acceptor.run(IIOPListener.java:491)

 

************************

 

Name: Thread-112

State: RUNNABLE

Total blocked: 1  Total waited: 0

 

Stack trace: 

java.net.SocketInputStream.socketRead0(Native Method)

java.net.SocketInputStream.read(SocketInputStream.java:129)

org.jacorb.orb.iiop.IIOPConnection.read(IIOPConnection.java:93)

org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:249)

org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:339)

org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:53)

org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:68)

java.lang.Thread.run(Thread.java:595)

 

******************************

 

Name: Thread-153

State: WAITING on java.lang.Object@141928f

Total blocked: 8  Total waited: 17

 

Stack trace: 

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:474)

org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:212)

org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:242)

org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:339)

org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:53)

org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:68)

java.lang.Thread.run(Thread.java:595)

 

***********************************

 

Name: Thread-2977

State: WAITING on org.jacorb.util.threadpool.ThreadPool@b92dc2

Total blocked: 222  Total waited: 552

 

Stack trace: 

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:474)

org.jacorb.util.threadpool.ThreadPool.getJob(ThreadPool.java:125)

org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:56)

java.lang.Thread.run(Thread.java:595)

Also in my last update to this bug, i had attached a heap dump.

is there any update on this? 
We seems to be struck by this issue out of sudden, and would love to see a solution.

Leon, is it the ClientMessageReceptor threads that you have blocked, or others?
Provide a thread dump please - jstack output from a 1.5/1.6 JVM, or sigquit on 
1.4 will do it.

What timeout properties do you have configured?

and also please confirm that you are using JacORB 2.3.0

Hi,

My system is also having the similar problem. The thread count keeps increasing
and eventually taking up all the resources (no threads can be created).

I'm using Debian Etch and jacORB version 2.3.0.  I had the same problem with
jacORB 2.2.3 also.

Wrt. to the note of wenlong 2007-12-22:
Our mistake was that we had initialized the ORB for each client connection to a
CORBA server but never called shutdown() on the ORB. Thus the client reader
threads kept waiting for input from the CORBA server.

The fix consists of 2 parts:
1) Store the ORB returned by org.omg.CORBA.ORB.init(...) in a class
2) Handle HTTP session unbinding in that class (we are using JacOrb together
with Tomcat): 

public class OsbClientUser implements HttpSessionBindingListener {
    ...
    // HttpSessionBindingListener interface
    public void valueBound(HttpSessionBindingEvent event)
    {
        // nothing 2do
    }

    public void valueUnbound(HttpSessionBindingEvent event)
    {
        if (null == orb_) return;

        log.debug("Shutdown the ORB...");
        orb_.shutdown(true);
        orb_ = null;
        log.info("ORB shut down.");
    }
    ...
}

In my understanding this issue is invalid in JacOrb 2.3.0.

Submitter needs to retest with current version. Stephan Br�nnimann can't reproduce with 2.3.0

hi All

we are using jacorb 2.3 and running our application on Linux OS. when pump large burst of alarms on Notification service. Corba is blocked.


here is the stack trace collected.it seems like the issue is not resolved.

"ServerMessageReceptor9" daemon prio=10 tid=0x00000000473b6000 nid=0x1afc runnable [0x00002b82e30b7000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.jacorb.orb.etf.StreamConnectionBase.read(StreamConnectionBase.java:110)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:380)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor18" daemon prio=10 tid=0x00002b82e4250000 nid=0x60af in Object.wait() [0x00002b82e9412000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000007034254f8> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor17" daemon prio=10 tid=0x00002b82e43b2800 nid=0xb99 in Object.wait() [0x00002b82e32b9000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x0000000702d021e0> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor16" daemon prio=10 tid=0x00002b82e4382000 nid=0x3a3d in Object.wait() [0x00002b82e2fb6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x000000070254d540> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ServerMessageReceptor8" daemon prio=10 tid=0x0000000047204000 nid=0x2963 runnable [0x00002b82e9c1a000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.jacorb.orb.etf.StreamConnectionBase.read(StreamConnectionBase.java:110)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:380)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"RequestProcessor-27" daemon prio=10 tid=0x000000004720d000 nid=0x26c7 in Object.wait() [0x00002b82ea01e000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:792)
	- locked <0x00000007015fa850> (a org.jacorb.poa.RequestProcessor)

"RequestProcessor-26" daemon prio=10 tid=0x000000004720b000 nid=0x26c6 in Object.wait() [0x00002b82e9f1d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:792)
	- locked <0x00000007015fa948> (a org.jacorb.poa.RequestProcessor)

"RequestProcessor-25" daemon prio=10 tid=0x0000000047208800 nid=0x26c5 in Object.wait() [0x00002b82e9e1c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:792)
	- locked <0x00000007015faa40> (a org.jacorb.poa.RequestProcessor)

"RequestProcessor-14" daemon prio=10 tid=0x0000000046b1b000 nid=0x7cc3 in Object.wait() [0x00002b82e31b8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:792)
	- locked <0x000000070126f550> (a org.jacorb.poa.RequestProcessor)

"ClientMessageReceptor15" daemon prio=10 tid=0x00000000472ce000 nid=0x25e6 in Object.wait() [0x00002b82e9311000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000006f97bba98> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor14" daemon prio=10 tid=0x00002b82e4134000 nid=0x56a8 in Object.wait() [0x00002b82e9210000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000006f8ba6d90> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor13" daemon prio=10 tid=0x00002b82e4550000 nid=0x4dad in Object.wait() [0x00002b82e910f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000006f8ba1d88> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor12" daemon prio=10 tid=0x00000000472d3000 nid=0x31a5 in Object.wait() [0x00002b82e900e000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000006f88a4680> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

"ClientMessageReceptor11" daemon prio=10 tid=0x00002b82e4547800 nid=0x1ae9 in Object.wait() [0x00002b82e8f0d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jacorb.orb.giop.GIOPConnection.waitUntilConnected(GIOPConnection.java:333)
	- locked <0x00000006f88a1b30> (a java.lang.Object)
	at org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:373)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:538)
	at org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:527)
	at org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
	at org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
	at java.lang.Thread.run(Thread.java:662)

Can you reproduce with the current version?

Can you provide a standalone runnable test case?

Sorry Currently we have restriction like we need to work the same release.

and one more thing we noticed is there is DB pool connection dead lock in our Code also. after fixing that we have not seen the issue. Since closing the issue.,