Bug 1002

Currently we are testing our system on Java 8 and are facing a mysterious problem. In an automated performance-test after some hours and several thousand connections (sequential, not parallel) jacorb gets locked in an infinite loop. We are using JacORB 3.4, but with 3.5 the problem persists. On Java 7 everything works well. I tried to debug the problem, but it isn't easy because it happens only after hours and JacORBs code is complex. Therefore I can't provide a test case. 

My analysis of the problem:
The deadly event seems to be a SocketException thrown when GIOPConnection.close() calls transport.close() (see line 1140) where the transport is a ServerIIOPConnection:
(line numbers for jacorb 3.4)

org.omg.CORBA.COMM_FAILURE: IOException: java.net.SocketException: Socket is closed  vmcid: 0x0  minor code: 0  completed: No
org.jacorb.orb.etf.ConnectionBase.to_COMM_FAILURE(ConnectionBase.java:152)
org.jacorb.orb.iiop.IIOPConnection.handleCommFailure(IIOPConnection.java:78)
org.jacorb.orb.iiop.ServerIIOPConnection.close(ServerIIOPConnection.java:111)
org.jacorb.orb.giop.GIOPConnection.close(GIOPConnection.java:1140)
org.jacorb.orb.giop.ServerGIOPConnection.close(ServerGIOPConnection.java:237)
org.jacorb.orb.giop.ServerGIOPConnection.streamClosed(ServerGIOPConnection.java:229)
org.jacorb.orb.giop.GIOPConnection.getMessage(GIOPConnection.java:398)
org.jacorb.orb.giop.GIOPConnection.receiveMessagesLoop(GIOPConnection.java:540)
org.jacorb.orb.giop.GIOPConnection.receiveMessages(GIOPConnection.java:529)
org.jacorb.orb.giop.MessageReceptor.doWork(MessageReceptor.java:69)
org.jacorb.util.threadpool.ConsumerTie.run(ConsumerTie.java:60)
java.lang.Thread.run(Thread.java:745)

Because of this exception the do_close flag isn't set to true in line 1141 of GIOPConnection. Afterwards the loop in GIOPConnection.receiveMessages() never ends, although it gets a SocketException in every iteration. It is catching it and only logging "Unexpected error during receiveMessages. Lost a message!" - then it enters the next iteration, infinitely.

I have no idea about the cause of the closed socket and why it doesn't seem to be closed when running on Java 7. But the infinite loop is a serious issue preventing our client from switching to Java 8, which is planned for the next weeks.

Our config looks like this:

## set to false to resolve some instability we were suffering after migrating to jacorb 3.4 
jacorb.connection.client.disconnect_after_systemexception=false

ORBInitRef.NameService=http://www.x.y.z/~user/NS_Ref
jacorb.orb.print_version=on
jacorb.log.default.verbosity=2
jacorb.logfile.maxLogSize=4
jacorb.logfile=<not shown for privacy reason>
jacorb.logfile.append=false
jacorb.debug.dump_outgoing_messages=off
jacorb.debug.dump_incoming_messages=off
jacorb.giop_minor_version=2
jacorb.retries=5
jacorb.retry_interval=500
jacorb.maxManagedBufSize=28
jacorb.deferredArrayQueue=8
jacorb.hashtable_class=java.util.Hashtable
jacorb.use_bom=off
jacorb.giop.add_1_0_profiles=off
jacorb.dns.enable=off
jacorb.dns.eager_resolve=true
jacorb.compactTypecodes=off
jacorb.cacheTypecodes=off
jacorb.cachePoaNames=off
jacorb.disableClientOrbPolicies=off
jacorb.acceptor_exception_listener=org.jacorb.orb.listener.DefaultAcceptorExceptionListener
jacorb.enhanced_thread_name=off
jacorb.interop.indirection_encoding_disable=off
jacorb.interop.comet=off
jacorb.interop.lax_boolean_encoding=off
jacorb.interop.strict_check_on_tc_creation=on
jacorb.interop.chunk_custom_rmi_valuetypes=on
jacorb.interop.sun=off
jacorb.interop.null_string_encoding=off
OAIAddr=<not shown for privacy reason>
OASSLPort=<not shown for privacy reason>
org.omg.PortableInterceptor.ORBInitializerClass.standard_init=org.jacorb.orb.standardInterceptors.IORInterceptorInitializer
jacorb.use_imr=off
jacorb.use_imr_endpoint=on
jacorb.imr.allow_auto_register=off
jacorb.imr.check_object_liveness=off
ORBInitRef.ImplementationRepository=http://www.x.y.z/~user/ImR_Ref
jacorb.imr.table_file=<not shown for privacy reason>
jacorb.imr.backup_file=<not shown for privacy reason>
jacorb.imr.ior_file=<not shown for privacy reason>
jacorb.imr.timeout=
jacorb.imr.endpoint_host=
jacorb.imr.endpoint_port_number=
jacorb.imr.connection_timeout=2000
jacorb.imr.object_activation_retries=5
jacorb.imr.object_activation_sleep=50
jacorb.implname=StandardImplName
jacorb.java_exec=java -Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB -Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton
jacorb.security.support_ssl=on
jacorb.ssl.socket_factory=org.jacorb.security.ssl.sun_jsse.SSLSocketFactory
jacorb.ssl.server_socket_factory=org.jacorb.security.ssl.sun_jsse.SSLServerSocketFactory
jacorb.security.ssl.client.supported_options=20
jacorb.security.ssl.client.required_options=20
jacorb.security.ssl.server.supported_options=20
jacorb.security.ssl.server.required_options=20
jacorb.security.keystore=<not shown for privacy reason>
jacorb.security.keystore_password=<not shown for privacy reason>
jacorb.security.jsse.trustees_from_ks=off
jacorb.security.ssl.server.cipher_suites=SSL_DH_anon_WITH_RC4_128_MD5
jacorb.security.ssl.client.cipher_suites=SSL_DH_anon_WITH_RC4_128_MD5
jacorb.security.randomClassPlugin=
jacorb.security.ssl.ssl_listener=
jacorb.poa.monitoring=off
jacorb.poa.log.verbosity=2
jacorb.poa.thread_pool_max=20
jacorb.poa.thread_pool_min=5
jacorb.poa.threadtimeout=0
jacorb.poa.queue_wait=off
jacorb.poa.queue_max=100
jacorb.poa.queue_min=10
jacorb.poa.useServantClassLoader=off
jacorb.naming.log.verbosity=2
jacorb.naming.purge=on
jacorb.naming.ping=on
jacorb.naming.ior_filename=c:/NS_Ref
jacorb.notification.filter.thread_pool_size = 2
jacorb.notification.proxyconsumer.thread_pool_size = 2
jacorb.notification.proxysupplier.thread_pool_size = 4
jacorb.notification.supplier.poll_intervall = 1000
jacorb.notification.max_batch_size = 1
jacorb.notification.max_events_per_consumer = 100
jacorb.notification.order_policy = PriorityOrder
jacorb.notification.discard_policy = PriorityOrder
jacorb.notification.consumer.backout_interval = 5000
jacorb.notification.consumer.error_threshold = 3
jacorb.notification.proxysupplier.threadpolicy = ThreadPool
jacorb.notification.default_filter_factory = builtin
jacorb.notification.stop_time_supported = on
jacorb.notification.proxy.destroy_causes_disconnect = on
jacorb.security.sas.log.verbosity=3
jacorb.security.sas.TSS.log.verbosity=3
jacorb.security.sas.CSS.log.verbosity=3