A new report from a device testing company found Long Term Evolution (LTE) devices for public safety have speech intelligibility concerns.
The findings in the report show that phrase misunderstandings can occur in mission-critical communications scenarios up to almost 50 percent of the time, even when using devices that are marketed specifically for use by the public-safety community. The tests included two devices marketed specifically to the public-safety sector and two consumer-grade devices.
Spirent conducted testing of a representative sample of public-safety LTE handsets in its Maryland-based performance center, a testing facility that specializes in evaluating the communications performance of voice, video and data devices. The company evaluated speech intelligibility in four commercially available devices. Testing emulated use within a quiet environment, followed by six emulated first-responder noise environments including a nightclub, emergency siren and a fire-rescue situation using a saw.
Speech intelligibility is the ability of a device to transmit all the critical components of speech in the presence of background noises so that words and phrases can be easily understood.
“The testing that was completed was based on our ongoing work with a major North American wireless operator,” said Eric Sinclair, senior manager, business development public safety and mission-critical services for Spirent.
Spirent did not release the names or models of the devices it tested. In the findings, the company said that in some cases, consumer-grade devices outperformed devices purpose-built for public-safety use. The top-performing device outperformed the others tested by at least a six-to-one margin in certain emulated environments. No device proved perfect; intelligibility failure rates ranged from 1% to 49%, depending on the environment. Great performance in one environment did not necessarily correlate with similar performance in other emulations.
“Our testing looked at certain devices through the lens of how well they were able to transmit the essentials of speech while suppressing most background noise because that’s where the largest performance gaps emerge in public-safety applications,” said Spirent Vice President of Product Marketing Saul Einbinder. “The significant performance discrepancies we uncovered indicate that device makers must further re-evaluate designs and components to assure performance on the mission-critical LTE networks being rolled out around the world.”
Spirent applied an algorithm designed by the U.S. National Telecommunications and Information Administration (NTIA) and used by the National Institute of Standards and Technology (NIST) Public Safety Communications Research Division (PSCR). The ABC-MRT16 algorithm models the human auditory system and, in conjunction with a head and torso simulator in a 3D sound reproduction lab with automated testing, was used to assess over 80,000 distinct speech points to determine the quality of speech intelligibility.
Achieving a higher bar for device performance in public-safety environments begins in the development process, Spirent said. Software algorithms determine which parts of the audio are actually speech and yet still allow some background sounds to be transmitted to help the listener better understand situational context. Independently testing elements of the handset voice path, together with fine-tuning microphone circuitry and audio processing software, are some of the ways device designs can better accommodate first responder scenarios. Testing for a range of common public-safety scenarios is critical to establishing that a device will perform every time a first responder relies on it.
Spirent shared the performance results with the manufacturers whose devices were tested to offer guidance on where necessary adjustments can be made on future devices to improve performance. “One device manufacturer has implemented fixes to their handset design after our recommendations,” said Sinclair. “This is a service we offer to any device maker and are often brought into the process ahead of launch to help identify and address performance challenges before they hit the market.”
“While our testing found substantial room for improvement in some cases, the good news is that there is a pathway to better performance,” said Einbinder. “This is absolutely a solvable problem. First responders need to trust that the devices they use for life-saving measures will perform when it matters most, and we believe pinpointing performance challenges that exist early in the development process will help the entire industry take a major step toward that goal.”
Device speech intelligibility tests are nothing new in the public-safety sector. A 2017 NTIA report summarized the results of speech intelligibility tests on five speech codec operating modes that might be chosen to provide mission-critical voice services to public-safety users over LTE networks. In 2015, NTIA described an effort to identify which digital speech and audio technologies are best suited for mission-critical voice communications 4G commercial networks.
PSCR led tests around the audio of Project 25 (P25) digital radios beginning in 2008, paving the way for current public-safety LTE speech intelligibility testing. Spirent plans to conduct additional public-safety LTE device testing in its lab but does not have a confirmed timeframe.
“When we showed this report to other device manufacturers, they all wanted to see how their devices compared to the ones we tested already,” Sinclair said. “In just a short period of time, we’ve already had a new device in our lab for these purposes and more are coming shortly.”