Monitor Availability, Performance with Proactive Measures
Top 5 Failures that Cripple IP Voice

Dec. 3, 2009 (Vol. 30, No. 23)

End users take dial tone and high call quality for granted.

But enterprise voice pros know the work that goes into maintaining that high performance. The obstacles to 99.999% availability multiply when your enterprise moves to IP telephony. In fact, most IP service level agreements will only guarantee up to 99.9%, says IP expert Gary Audin, president of consulting firm Delphi Inc., in Arlington, Va.

Imagine not getting dial tone when you need to make that 911 call. “If it doesn’t work then, you’re in big trouble,” Audin says.

So, how can you realistically measure your enterprise’s network performance? The FCAPS model, developed by the International Telecommunications Union (ITU), emphasizes fault management, configuration, accounting, performance and security as the main tenets for effective IT and telecom resource management.

Not surprisingly, the weakest area for most enterprises is performance, Audin says.

Consider this list of five common causes of IP telephony performance problems and tips for tackling them proactively, compiled with help from Jeff Parker, CEO of St. Charles, Ill.-based Monolith Software, and other IP voice experts.

#1: Too Little Bandwidth; Voice Traffic a Low Priority

Latency, jitter and packet loss signal something is amiss in your network, Parker explains. All three can result from not having enough bandwidth, or a network that doesn’t prioritize voice traffic appropriately.

As voice traffic increases, latency, jitter and packet loss increase together, because all three are caused by the same factors, Audin explains.

In fact, the inability of the network infrastructure to support high-quality VoIP might be the single largest contributor to poor IP voice quality, says Parker, whose company sells software that screens networks in real time.

But only about one in 10 enterprises focuses on monitoring network performance in addition to availability.

You can measure latency, jitter and packet loss with IP SLA and mean opinion score, or MOS (which gives a numerical value to the perceived quality of a voice transmission). Most Cisco phones today feature IP SLA embedded in the IOS code, creating a built-in ability to send synthetic “test” voice packets across the network on schedule. This software then uses an algorithm to generate a dynamic MOS.

These measurements are “an absolute must,” Parker emphasizes. Your IP infrastructure isn’t providing good quality service if you’re not regularly running synthetic VoIP quality tests on your network.

As your network clogs up with traffic, it eventually hits a high enough level of congestion to cause voice quality to plummet, called the “congestion cliff,” Audin explains.

You can prevent latency, jitter and packet loss by buying enough bandwidth to eliminate congestion, or by instituting a quality of service that gives preferred treatment to VoIP, Audin says.

#2: Malfunction in Administrative Components

A malfunction in your network’s trivial file transfer protocol (TFTP), domain name system (DNS) or dynamic host configuration protocol (DHCP) will render your IP voice system unusable, Parker says.

TFTP is a common VoIP server used to download software directly to phones and gateways. The two main problems with TFTP are its relatively slow speed and insecurity, Audin says. As a result, many enterprises choose a hypertext transfer protocol (HTTP) server, which offers faster and more secure service, he says.

DNS functions as a type of Internet directory, assigning IP addresses to domain names. 

DHCP is an administration box that assigns IP addresses to devices and functions with little or no manual intervention. 

These services are either separate or built into one communications manager, Audin explains. While each service is administrative and doesn’t directly affect voice quality or operations, it is crucial to keep the overall structure up and running reliably.

Be careful when making changes to the system and avoid administrative errors, Audin cautions, citing a recent incident where a major airline failed to correct an error in its backup system, causing serious problems when it suddenly needed to switch from primary to backup. “Tiny errors can have magnificent consequences,” he says.

Without these essential administrative services in place, your network will be as ineffective as a road would be if you didn’t have the gas or wheels to drive to your destination, Parker says.

#3: Switch and Server Failures

VoIP requires an always-on network infrastructure in order to pass calls along, and its functionality depends on vigilant monitoring of the network’s available assets. 

LAN switches are generally a stable and uncomplicated system, and can even monitor themselves. In the last few months switches have been manufactured with software inside them to monitor what’s going on internally and how much power they’re consuming, Audin explains.

When monitoring Ethernet switches, failure tops performance as a main concern, Audin explains. Power failure is extremely important to monitor – it’s caused by power outages or errors during reconfiguration.

Power outages have even greater ramifications when it comes to server failures. More and more enterprises virtualize various applications onto one server to minimize costs. But the risk of losing that many apps during a single-minute outage is significant.

“Power becomes important because a short outage produces a long boot-up time,” Audin explains. For example, a five- to 15-minute power outage could set your enterprise back much longer, since phones must then reregister sequentially, not simultaneously.

To safeguard against failures, consider employing active/active server configuration. Active/active servers are fully synchronized and are mirror images of each other, so that if one server fails the other can immediately kick in and minimize downtime, Audin explains.

Active/active configuration allows for seamless continuation of service in the event of a failure, while active/passive configuration means some service interruption will occur.
 
#4: Running Out of Disk Space

Call servers manage the setup and connection of telephone calls, and create and send the messages necessary to process call requests.

Running out of disk space on the call processing server is a common occurrence that can bring down an entire IP telephony environment, Parker explains. No application can run on a full disk.

Although actually running a network doesn’t take a lot of disk space, the real issue is archiving information, Audin explains. If you don’t have an archiving system in place, communications will quickly eat up your disk space.

That’s why it’s so critical to monitor the availability and performance of the servers running your IP infrastructure.

A good monitoring system will track utilization of all your resources, including disks and servers, and how busy the network is, Audin says. Ensure you have sufficient resources, but that you don’t overspend.

Pay special attention to your WAN network, Audin advises. Although LAN space is leased, you pay a monthly bill for WAN access and can easily miscalculate how much space you really need. Don’t just estimate what you need now, but predict a three-year trend and take seasonal trends into account.

For example, L.L. Bean disables 75% of its toll-free numbers during nine months of the year, turning them on only during the busiest winter months. Likewise, the IRS disables many lines that go idle after the April 15 tax deadline, Audin reports.

#5: No Monitoring

Service monitoring is essentially “the ability to tie together all of the required components that your IPT infrastructure is reliant upon to perform service management and to provide consolidated, real-time IPT dashboards,” Parker explains.

Create a proactive plan for monitoring and support that will deal with unanticipated emergences as well as with everyday occurrences, suggests Matt Brunk, president of Monrovia, Md.-based Telecomworx. Crises can be avoided by staying on top of issues and addressing potential network problems end to end.

Investing in good monitoring tools isn’t cheap. Service and software costs can climb into the tens of thousands or even hundreds of thousands of dollars, Audin says. You’ll face a pricey initial bill, but consider the dividends: A one-time software expenditure instead of taking on more staff will put an effective monitoring system in place, and you’ll save in the long run by not having to cut those extra salary checks.

Audin recommends a capital investment in software and other monitoring tools instead of an ongoing staff expense. “Your staff will stay the same size, but you can have much more control,” Audin says.

Network monitoring should be proactive, Audin emphasizes. A service manager’s job description extends to far more than just troubleshooting, allowing him or her to manage resources and make effective predictions.

Audin and Parker agree that managers should never wait for an end user to call with a complaint. When the manager calls the user instead, explaining that he or she noticed a problem and is taking steps to fix it, the user’s loyalty and trust in the manager will greatly increase, Audin explains. “The goal is to take management from reactive to proactive,” he says. (