Deadlocks making CORBA calls. Threads/Ace Service Configurator.

[SoftwareGeek01]

Hi,
I am porting a large software framework to TAO from another ORB, and I am getting deadlocks. The attached (contrived) example demonstrates this.

In words, we are using the ACE Service Configuration framework to bring up objects. They attempt to make CORBA calls (resolving other names, binding their own names etc) in separate threads. One thread holds the Service Registry recursive thread mutex in Service_Gestalt.cpp , in the initialize() method. Other threads attempting to make CORBA calls end up attempting to find ORB related services, and block waiting on that mutex. For "reasons", the first thread is waiting on the others to complete, and hence never releases that mutex.

Is this a case of "Just don't do that" ? Is there some way to preload the ORB so that later calls to bind, resolve_initial_references and other such calls don't need to find services?

Currently I am pulling things apart and trying to put them together so that all initialisation happens in one thread but it is a major undertaking, just wanting to check that I am overlooking something simple. The framework I am working with was originally written using ACE/TAO so unless it mutated a lot in recent years I am assuming things worked somehow back in history. Unfortunately it is so old everyone involved has retired/moved on.

deadlock.tar.gz

[mitza-oci]

In this case init can't complete until the bind succeeds. Because bind is a remote call, it could take a long time or require explicit retries (or just fail). Is it feasible to change it so that the service object can init without bind?

[SoftwareGeek01]

Yes, my options are to re-write our homegrown framework, but it is not going to be an easy task :-).
Just note, the bind() cannot return, as it is blocked waiting for the recursive thread mutex in ACE Service_Registry code that is being held by a different thread (the one calling init()). (And no, I don't know why our framework spawns threads willy nilly to do things... that is something I will have to address).

As a second data point, this is code modified so that some of the CORBA is happening in the main thread as well as the spawned thread. If I run this as normal (with just an -ORBInitRef NameService=) , it will deadlock , with thread 1 in init() , and thread 2 trying to find "TAO_ORB_Core_Static_Resources" service.

However , if I run it with an additional -ORBGestalt Local it will work. I am not sure what the ramifications of that are. This actually helps with some of our business code, but some still deadlocks looking for other (don't have the details here) TAO_ORB_xxxx services.

I was hoping there was some mix of ORB flags, svc.conf settings , and "priming" calls that I could make in the main thread before getting into the nitty gritty that would cause all the finding/initializing of TAO ORB services to happen in main, but if there isn't anything obvious I will just have to dive into changing our framework. (It is just strange that this must have worked in the past, but it could be that we've changed too much stuff since those days).

deadlock2.tar.gz

[SoftwareGeek01]

Oh and thank you for having a look at this.

[mitza-oci]

Could you clarify the statements ... "with an additional -ORBGestalt Local it will work" and "some still deadlocks"? Does that mean it gets past the problem shown in the previous attachment but it doesn't actually bring up the system correctly?

There are details about -ORBGestalt in section 17.13.29 of the TAO Developer's Guide. If there is only one ORB in the process it doesn't seem like it should matter.

The point I was trying to make in my previous message is that even if the ORB internals like the use of ACE Svc Config weren't blocking bind(), calling bind() from init instead of svc may be a problem.

The overall direction you're going makes sense -- trying to get as much set up in the main thread before other threads are spawned. I haven't dug in deeply enough to know if this should be sufficient to fix it. Another tool that may be useful is that if there's one thread calling orb->run() you can post arbitrary work to that thread using orb->orb_core()->reactor()->notify().

[SoftwareGeek01]

Hi , sorry about the delay replying, I was performing some code archeology (see further below).

Firstly, to answer your questions:
We have multiple applications. Some applications will get to a state where they look like they are ready to do work if I run them up as
./xxxxx -ORBInitRef NameService=yyyy -ORBGestalt Local
if I have put the additional steps of resolving the RootPOA and the NamingService etc in the main thread. Other applications don't, because they deadlock for different services. I think the -ORBGestalt Local triggers the use of thread specific storage for some things (not sure what, I think I saw something about that in the change logs).

Anyway, I've built ACE 5.2.2 / TAO 1.2.2 (on an old RedHat 6 VM) and this code does not deadlock using that version of ACE. I think the behavior changed with the rewrite of the Service Configuration Repository / introduction of the Service Gestalt code, as described in ChangeLog-2006b , dated Wed Apr 26 20:21:49 UTC 2006.

So, I don't know if this should be regarded as a bug in ACE or not... I assume that there isn't too many people with this exact use case or this would have been noticed in the last 16 years :-).

I am happy to understand why this is happening for me now, when it presumably wasn't in the past.

Cheers

[SoftwareGeek01]

Sorry for the delayed reply, I have been doing software archeology. (Also apologies if this appears twice, I thought I had already commented but my comments seem to have disappeared).
Firstly, I have established this deadlock does not occur with ACE 5.2.2/TAO 1.2.2. I think the behavior changed with the refactoring mentioned in ChangeLog-2006b , Wed Apr 26 20:21:49. This appears to be a major rewrite of the service configuration stuff, introduced the Service_Gestalt stuff.

To answer your questions
We have multiple applications. Some of them appear to get to a good running state if I put some of the CORBA stuff in the main thread (as in deadlock2.tar.gz) and run them as
./xxxx -ORBInitRef NameService=... -ORBGestalt Local
Other applications still deadlock on that mutex that is shared between Service_Gestalt and Service_Repository.

Anyhow, I am pleased to know why things used to work, and now don't. I am not sure if this should be considered a bug in ACE or not. I assume there are not too many people with the same use cases, seeing the change is 16 years old!.

I need to read up on the Gestalt stuff/look for examples. In particular the comments in that change log entry talk of dynamically loaded services using their own sets of service objects, in contrast to the ACE_Service_Config class that is dealing with a process-wide configuration state, which sounds like it might be something to look at.

Cheers

[SoftwareGeek01]

I found some examples of using ACE_Service_Gestalt, instead of the (remaining) static methods in ACE_Service_Config class, based on some examples in the ACE/TAO code base. This works as expected, no deadlocks. I am evaluating this change in our business code.

Cheers

[jwillemsen]

Did you attach the correct code, don't see service gestalt usage in the nodeadlock gz file

[SoftwareGeek01]

Sorry about that, jumping between airgapped, VMs and my home system , must have gotten that mixed up. This (should) be correct.
Is there somewhere to read about the "newer" features? My library for ACE/CORBA/TAO are the usual 20 year old volumes... (Advanced CORBA Programming using C++, the ACE User Guide and the two volumes of C++ Networking with ACE, sorry the exact names escape me at the moment).

nodeadlock.tar.gz

[jwillemsen]

There are no newer books, the amount of effort to write books is extreme, some years try to get into contact with some of the original authors to see if there would be interest in some updates but got no responses.

Newer features are CORBA Compression (ZIOP), that is now part of the CORBA standard. There is now a IDL to C++11 language mapping which greatly simplifies the API and at the same moment makes it more safe and performant. See TAOX11 for an implementation of this mapping, we have some presentations on the website for comparison.