Modify the releasing Zap endpoints from IO thread process

[nichamon]

The patch separates the process of reducing the number of Zap endpoints associated with an IO thread from the process of releasing the endpoint from the IO thread. Additionally, it defers the nullification of zap->thread until the endpoint reference reaches zero.

zap_sock and zap_ugni need to remove their events from epoll before delivering the disconnected event to applications, preventing the reception of additional epoll events. However, without this patch, this results in resetting zap->thread to NULL. Thus, the applications cannot retrive Zap thread information in the disconnected path. The patch addresses this limitation.

[nichamon]

The patch is a part of the bigger patch to improve the thread_stats command.

[narategithub]

The explanation on zap_io_thread_ep_release() and zap_io_thread_ep_remove() are confusing to me. Not sure how to make it better though ... maybe just saying release means the endpoint won't be processed by the thread any more (i.e. no more event callbacks to the application) but the endpoint still keep a reference to the thread for further stat data access, while the remove means disassociate the endpoint from the thread and reducing thead's ep load? I'm not sure ...

[narategithub]

@nichamon I think the code looks OK to me. Just a nit on the function documentation.

[tom95858]

I think this creates a race between disconnect and ldms_xprt_update. With this change isn't it possible to have an ldms_xprt_update succeed after a disconnect?

[nichamon]

I think this creates a race between disconnect and ldms_xprt_update. With this change isn't it possible to have an ldms_xprt_update succeed after a disconnect?

The patch keeps the existing logic for event removal from the epoll by zap_sock, zap_ugni, and zap_fabric, so it should not cause a race between the disconnected and update complete events. I also went through (my code inspection) every if-condition that checks for ep->thread whether it would cause any race condition by the patch but I didn't see any problems.

I made this patch so that Zap's applications can obtain the IO thread statistics of a Zap endpoint in the disconnected path.

@narategithub has suggested another change to facilitate this. The suggestion was to add a pointer to the zap_thrstat structure to the zap_ep structure so that Zap could retrieve the thread stats when ep->thread is NULL. However, this could mess up the reference count when an endpoint has been released from the IO thread and t->_n_eps and t->stats->n_ep have been decremented; although, the endpoint is still able to access the thread statistics, which is a part of the thread structure. Note that t->_n_eps and t->stats->n_ep are always in sync.

[tom95858]

@nichamon thank you for looking into this...

[tom95858]

Is everyone happy with this??

[narategithub]

@tom95858 I think this patch looks OK to me.