I am excited to report that a very significant and lingering bug in the Zaptel channel driver of the Asterisk PBX has been resolved. With hardware from The VoIP Connection and development assistance from Matthew Fredrickson, a long time Digium Employee, we were able to specifically recreate and ultimately eliminate this bug using non-production hardware.
We have been receiving complaints from a few call center-type customers, as well as my VoIP Provider had experienced this issue for quite a while, but we could never specifically or reliably reproduce the situation. We initially blamed the telco, since the PRI debug we acquired showed they had already been notified of a pending outbound call, yet they still sent an inbound call to that same channel.
After The VoIP Connection had received similar complaints and could not resolve this issue for their customers, we decided it was in everyone’s best interest to dedicate time and resources to solve this bug once and for all.
Olaf Bellstedt, Senior Partner at The VoIP Connection, and I setup two systems. One was a current Intel P4, while the other was a 1ghz mini-itx system. We added one TE2XXP card into each machine with T-1 Crossover cables into each span. Using a simple shell script that generated Asterisk .Call files, we slammed both T-1 Spans with more call setups than each could deal with – specifically causing a significant amount of Glare, which in Telecom terms is the condition for a call collision where the local and network elements request a call to be setup on the same channel at nearly the same time.
Usually within 3 to 5 minutes we would experience the channel/thread locked situation that production systems took 3 to 5 days or longer to experience, thus very effectively providing the necessary debug and motivation to fix this bug.
Not knowing all that much about the inner-workings of chan_zap.c, we were dependent on others to come up with the solution for us. After a few rounds of questions/comments from us and other Asterisk Community members, MattF provided a patch for us to try. His patch utilizes the B-Channel transfer concept that was somewhat recently added to chan_zap. When a call comes in on a channel that Asterisk/Zaptel is otherwise using or hasn’t totally finished with instead of hanging up the new inbound call, which caused that channel to be locked and otherwise unusable, chan_zap requests the channel be moved to another, known idle channel. This new behavior very effectively works around even the possibility of having a locked channel hang around until we can restart Asterisk – which in production usage sometimes can be quite a few hours or even the better part of a day.
I would like to personally thank Matthew Fredrickson, The VoIP Connection, Digium, jmls, bweschke, one47 and serge-v for their assistance in solving this long time bug in Asterisk.
This once again proves to me that Open Source is absolutely a viable method for software and business development.
Comments on this entry are closed.