for devices is critical whether it’s for Wi-Fi 6/6E or 7, which is on the horizon. As a result, there are new test plan approaches being developed for the new challenges that are being introduced as these technologies advance.
Wi-Fi Alliance members, including Spirent, have collaborated to produce a Wi-Fi performance test plan describing a relatively simple testbed that provides all the means necessary to perform a wide variety of Wi-Fi tests. The testbed provides a foundation to ensure repeatable and reproduceable RF testing and will likely be the cornerstone of further revisions, enhancements, and expansion of the test plan.
While small chamber over-the-air (OTA) testing is becoming the de facto method for testing, theintroduces approaches that we believe are unique from other test plans in the industry.
The testing methodology limits the number of test parameters and avoids needless nested parameter loops by specifying fixed “external” parameters like topology, number of devices etc., and allows the tester to save time and money by focusing on “internal” parameters like security, NSS, and others, as appropriate for their organization. Since more and more consumer devices are pre-, or self-configuring, Wi-Fi Device Metrics encourages out-of-box testing to emulate a customer’s experience.
Real-world customer environments produce results with wide variation. Wi-Fi Device Metrics recognizes this and recommends presentation of results in a statistical manner, rather than just a single number. Each test produces a wealth of data and Wi-Fi Device Metrics details ways that this test data can be efficiently analyzed to provide statistical analysis that more accurately illustrates the customer’s experience.
In the beginning
In the early days of Wi-Fi, RF performance was characterized in the lab using RF cabled setups usually housed in shielded RF rooms. Testing tended to be very specifically PHY related, concentrating on TX power, RX sensitivity, spectral mask, EVM and so on. MAC/PHY testing was covered by Wi-Fi Alliance interoperability testing but testing of radio performance in the real world was limited.
OTA RF performance testing is complicated. Often, it was performed in shielded RF rooms, but generally the results were unreliable because standing waves and movement of people within the chamber caused signal variations as much as 70 dB. Open air testing was often performed in a dedicated test home, which gave some degree of real-world assessment, but again reflections and interference almost guaranteed Never The Same Result Twice (NTSRT).
Nearly all of these legacy test methodologies were aimed at point-to-point connections, which is in line with the goals of all Wi-Fi generations up to Wi-Fi 5.
Wi-Fi 6 was designed to give better user performance to a community of concurrent users in the presence of overlapping basic service set (OBSS) interference. This broke the legacy mold for testing.
State of the art testing today
Now, a testbed needs to contain multiple devices, up to 37 in the case of OFDMA. The testbed must reproduce spatial diversity for MU-MIMO to be tested, there must be facility to generate OBSS traffic, and there is a need for multiple sniffer devices. Cabling such a setup is impractical and all the same NTSRT problems still remain in an OTA environment.
Theis built using relatively small anechoic chambers and uses directional antennas to “couple” RF energy into, and out of, the chamber. This approach limits the signal variability due to standing waves, avoids the need to disassemble the device to make a physical connection, and provides good spatial diversity to support MU-MIMO operation, as explained further in and . The chambers are interconnected with RF cables through variable attenuators to simulate distance.
This type of testbed is relatively new in the industry and has been shown to give results that are repeatable (on the same system), and reproduceable (on a remote system). It is rapidly becoming accepted as the de-facto way to test Wi-Fi as evidenced by other standards bodies such as Broadband Forum, ETSI, and others.
Along with the testbed, the test measurement methodology also needs to change. Methodologies like RFC 2544, which rely on the system to be in equilibrium for a measurement to be performed, are no longer useful because of the continuous variation of the measured parameters caused by all the other devices also contending for the channel.
Better performance indicators
Performance indicators of interest need to be more sophisticated. The traditional raw throughput with one device is less useful in the context of real-world testing where we need to assess the combined experience of a community of users. Instead, we see aggregated throughput, individual device latency, and roaming performance becoming more important.
Wi-Fi Device Metrics use modern traffic generation tools such as multiPerf that provide second-by-second key performance indicator data so that the variability of results can be captured and analyzed. MultiPerf also provides more sophisticated analysis capabilities, for example, measuring the packet-by-packet distribution of one-way delay (OWD).
MultiPerf also has more sophisticated traffic generation modes that help the tester more accurately mimic real-life applications. For example, multiPerf allows one to generate traffic with a specified data rate mean and variance to mimic a user. Indeed, it goes further to produce isochronous traffic at a specified frame rate to mimic voice or video.
Comprehensive presentation of results
As mentioned earlier, the presentation of the metrics needs to be revised. A single throughput number, or a single delay number is not that useful because, in practice there is always a distribution of results, and it is often the spread of those results that affect users more adversely.
Wi-Fi Device Metrics recognizes this fact and proposes a much more statistical analysis of the results, which are presented in various levels of detail.
The first level will typically be in tabular form presenting the mean, standard deviation, coefficient of variance, and performance at certain percentiles. This is useful for simple reports, perhaps for regression testing, where the numbers can be compared, or matched up with the testers’ own criteria for pass/fail.
The next level will be to generate Probability Distribution Functions (PDF) and Cumulative Distribution Functions (CDF) to visualize the spread of results so that anomalies can be illustrated. Depending upon the test case, other detailed analysis and visualization is performed, as appropriate.
An example of packet delay results
An example of this analysis is shown in the image below where the latency test produces a PDF of packet delays on a packet-by-packet basis. Here we mimic multimedia by generating isochronous, variable bit rate traffic.
Generally, real time video or voice is fairly tolerant to a reasonable mean delay, but the tails of the delay spread are the parts that irritate the customer.
Explicit examination of the spread is important to gauge customer experience. This graph shows a mean delay of about 8 ms. Delays around 20 ms are unlikely to cause problems, but there is a finite probability of delays at 50 ms, 55 ms, and beyond, which might disrupt a voice or video stream occasionally and would be irritating to the user.
Spirent Wi-Fi testing
As a leader in Wi-Fi performance testing for devices and access points, Spirent is a key member of the Wi-Fi Alliance task group and regularly contributes to their evolving standards. In doing so, we continually gain insights for developing our automated testbeds to meet the demand for the ever-changing Wi-Fi technology industry.
Each generation of Wi-Fi has delivered higher data rates with a focus on improving performance for end users. It’s an increasingly tall order—consider the average family surrounded by devices all contending for the same airwaves. Or the same scenario faced by an enterprise user.
Wi-Fi 6 aimed to solve this dynamic in home and at the office. It was a paradigm shift that introduced new functionality and mechanisms to better support multiple users. Close on Wi-Fi 6’s heels came Wi-Fi 6E, which incorporated use of the 6 GHz spectrum.
Anticipating what’s next, the industry has its sights set on Wi-Fi 7, which promises to refine and expand Wi-Fi 6 functionality in the 6 GHz spectrum. It adds new features and mechanisms aimed at finally tackling issues that have persistently snarled certain Wi-Fi use cases.
While higher throughput—up to 12 Gbps—is the main benefit of Wi-Fi 7, it is not achieved easily.
Let’s explore why, and implications, starting with a look at Wi-Fi 7’s core capabilities and benefits.
320 MHz bandwidth for much more data
Similar to Wi-Fi 6E, Wi-Fi 7 uses the 6 GHz spectrum that can support channels as wide as 320 MHz—twice what is supported by Wi-Fi 6E and four times Wi-Fi 6. In fact, with Wi-Fi 7, you can get three 320 MHz channels in the 6 GHz band. Because a wider channel can transmit more data, Wi-Fi 7’s pipe is larger than ever.
But is that spectrum actually usable? Such a wide 320 MHz channel is likely to have an interferer in the band which means sections of the channel might be unusable. Wi-Fi 7 deals with this by using a mechanism to puncture out that part of the spectrum so that it is partitioned, but the rest of the 320 MHz channel can be used.
Elevated order modulation for 20% higher speed
Quadrature amplitude modulation (QAM) conveys data over radio waves using discrete points in the constellation diagram. Each discrete point represents a number of bits of data. The more allowable discrete points, the more data that can be transmitted. Wi-Fi 6 provides 1024 (point) QAM, a 25% increase data rate from Wi-Fi 5. Wi-Fi 7 has increased it another 20% to 4096 QAM, which is 12 data bits per symbol.
The problem with this high order modulation is the impact of channel noise, which makes demodulation difficult. Although 4096 QAM is fast, it needs a high signal-to-noise ratio (SNR) to work properly. That limits its use to short operating distances of about 18 feet—inferior for some applications, but excellent for others, such as virtual reality.
Multiple Resource Units provide better spectrum efficiency
OFDMA improves performance by allowing simultaneous transmissions between multiple clients. With Wi-Fi 6 and LTE, a channel can be divided into Resource Units (RUs), which are frequency groupings. Each device is allocated one RU. To provide better spectrum efficiency, Wi-Fi 7 allows multiple RUs to be allocated to each device, making use of otherwise potentially unused spectrum.
Multi-link operation increases link and channel efficiency
In traditional Wi-Fi mesh networks, each mesh node communicates with the devices close to it on a single band and the mesh nodes communicate with each other. This is sometimes not an efficient approach for device-to-device traffic. Instead, with Wi-Fi 7, multi-link operation (MLO) enables multiple simultaneous links to operate in separate channels, with each link operating independently. For example, 2.4 GHz, 5 GHz and 6GHz radios can all be used as though they were one.
MLO is an important new feature in Wi-Fi 7. It is a unified and consistent framework to consistently manage multiple links, reducing management overhead. By aggregating links on different channels, MLO increases throughput. It also improves latency by using multiple links in parallel for flexible channel access. Reliability can be increased by sending duplicated data on multiple links, and quality of service (QoS) can be improved by assigning traffic to appropriate links.
Enhanced QoS management for priority access
Normally, all devices contend for a single channel on a first come first served basis. This is a non-starter for applications like voice calls, where timing is critical.
Wi-Fi 7 introduced enhanced QoS Management so that devices can request guaranteed time. For example, they would inform the access point that a voice call will need 5 ms every 20 ms. The access point will then pre-allocate the channel if possible. This guarantees access to the channel when the voice packet is transmitted. Enhanced QoS management provides smoother channel access management than previous first come, first served methods.
Restricted service periods for deterministic latency
Latency is important for enhanced reality. Wi-Fi 6 improved latency with OFDMA but, depending on the number of gamers in the house, the latency could fluctuate significantly. Wi-Fi 7 can provide deterministic latency that reserves what you need when you need it.
Wi-Fi 7 testing considerations
Wi-Fi 6’s new features had major testing implications which need to be further refined for Wi-Fi 7. The biggest impact is multi-link operation, which will need new testing methodology and test plans for its more consistent, but a different approach compared to standard mesh device testing. Similarly, testing methodologies need to be enhanced for targeted QoS wait time.
As always, test planning will leverage Wi-Fi Alliance test plans when they become available. In the meanwhile, maintaining an up-to-date testing approach for Wi-Fi 6 and Wi-Fi 6E is essential to keep pace with evolving Wi-Fi technology.
Learn about Spirent’sand read the eBook .
Unique TaaS option enables Wi-Fi manufacturers to access the most sophisticated test procedures on an as-required basis
Available to customers globally, the service is based at Spirent’s Massachusetts research facility where OCTOBOX testbeds are developed and manufactured. This allows Spirent to easily scale up and down testing services for customers based on their ever-changing needs and leverage the company’s unrivalled Wi-Fi emulation and testing expertise. The Spirent lab also provides test environments that are engineered with optimized hardware to ensure all devices are validated against the highest quality standards accepted for deployment by carriers and enterprises, such as TR-398 and RFC 2544.
“While many test facilities use software-based testbeds, these are unable to perform realistic test scenarios that model actual deployments,” said Roberts. “They often lack the ability to produce deterministic results which is essential for repeatability. By utilizing our OCTOBOX testbeds along with our automation framework, we emulate real world scenarios such as congestion, interference, distance, and movement – all of which affect the quality of the user experience.”
For more information about Spirent’s Test as a Service for Wi-Fi Devices, visit.
Going on two decades, Wi-Fi has delivered simple, inexpensive wireless connectivity for the masses. It was never perfect, but the convenience and generally fine performance for most applications made it good enough for most.
Now, the stakes are higher.
Consumers are requiring more demanding use cases and more powerful apps. Despite its promises, 5G connectivity remains limited. This has meant everyone from personal to enterprise users have been leaning heavier than ever on Wi-Fi to meet connectivity needs.
This has been a driving force behind a whole new generation of Wi-Fi tech and adoption. Whether working or playing from home, we expect Wi-Fi to operate flawlessly with high performance and reliability. This is especially true as Wi-Fi cements a role in mission-critical private networks that support emerging industry applications.
Wi-Fi 6 and 6E are being positioned to meet these new roles and requirements. The new Wi-Fi standards provide distributed connectivity, high throughput, and low latency. In particular, the radio interface has received an overhaul, with modulation and spectral efficiency on a completely new level.
Wi-Fi 6 routers, extenders, and mesh networks are being installed in homes and offices to provide wider coverage, support more users, and accommodate a growing range of devices. Each brings pros and cons, but whichever is chosen, it is safe to say Wi-Fi networks have evolved far beyond initial roles as simple routers.
In fact, the changes seen in Wi-Fi 6/6E introduce challenges that are in some ways more significant than the migration from 4G to 5G.
New Wi-Fi complexity means device testing has grown in complexity, too
As expectations of Wi-Fi surge, so too does the need for precise, comprehensive . As always, testing must address conformance to standards, interoperability, and performance. With Wi-Fi, many factors can impact performance, such as walls, electrical interference, , signal reflection, and other Wi-Fi and wireless usage in the area. All Wi-Fi 6/6E devices must be tested against these factors in controlled, repeatable ways.
Wi-Fi 6/6E sees multiple standards in play, including TR-398 performance for routers and access points, RFC-2544 network device benchmarking specification, and others in the offing. Still, as this next generation of Wi-Fi comes to market quickly, comprehensive test and interoperability standards have yet to be developed.
As such, testing Wi-Fi 6/6E devices has become more complicated than traditional Wi-Fi device testing. Trying to test manually not only makes little economic sense, but it has become nearly impossible.
Automated testing of Wi-Fi 6/6E devices
To overcome the complexity of testing a growing number of Wi-Fi 6/6E devices, automated device hardware and software testing is essential for cost-efficient, scalable, accurate, timely, and repeatable results.
Traditionally, test teams must research and keep up to date with evolving standards to create and execute tests. Alternatively, automated Wi-Fi automation packages from test experts like Spirent can be leveraged. Such automated test packages reflect existing standards and, where standards don’t yet exist such as for mesh networks, expertly define appropriate tests. Executing automated test software on your testbed enables fast, 24×7 testing. And all those resources that were grinding through test scenarios can instead put energies into evaluating results.
Spirent Wi-Fi 6/6E automated test packages
Spirent, the Wi-Fi testing market leader, offers a wireless test bed that validates Wi-Fi networks and devices. Its software automation packages automate standards-based Wi-Fi test plans to assess conformance, interoperability, and performance.
To accelerate Wi-Fi 6/6E testing, test automation packages spanning mesh interoperability performance testing of devices and device performance evaluation through RFC 2544 benchmark tests have been developed.
Learn more about howcan enable you to quickly assess Wi-Fi device conformance to standards, interoperability, and performance.
The advancement of technology has done great things for society. Today, innovations from autonomous vehicles to advanced pacemakers rely on more than microprocessors and software but the underlying network that seamlessly connects everything together. Yet issues with connectivity – from poor latency to instability under certain scenarios – can have a massive impact on the viability of many critical use cases. For both device manufacturers and service providers, network assurance testing is of paramount importance – but it’s a task that is becoming increasingly complex due to the highly varied array of network types and usage scenarios that must be validated.
This need to simplify and improve these assurance processes is one of the reasons whybecame a single company last March. Everybody in the industry knows that OCTOBOX is a leading testbed for automated validation of Wi-Fi networks and devices. Even when we were rivals, from an engineer’s standpoint, we could not help admiring the compact yet stackable approach that made it easy to create a robust testbed suited to a wide variety of scenarios.
However, as Spirent, before we joined forces, we still had many joint customers that needed to perform more specialist assurance scenarios that were not well suited to OCTOBOX. Especially for granular channel emulation or for use cases where complex mobility was involved. For these customers, it meant potentially setting up dual test beds running Spirent and OCTOBOX products for very different workloads. This also required integrating these disparate solutions and then trying to tie the data together into a coherent set of results. Not an impossible task but far from ideal.
In a world getting more complex, this highlights one of the fundamental benefits of our union – the ability to create new products that combine our expertise and technologies into a better overall solution. A coherent approach that is not only easier to use but also more suited to the diversity of use cases that we are all seeing across our modern society. It has been nearly a year of intense engineering work, but the first fruits of that vision have ripened. We are proud to announce the launch of an integrated solution that combines the modular nature of OCTOBOX with, our channel emulator able to replicate the comprehensive noise and spatial conditions of even the most complex wireless channels.
In simple terms, we now have a solution that combines best in class traffic emulation, channel emulation and performance evaluation into an integrated platform. And this arrives at a time where we are seeing an explosion in new devices like wearables and industrial IoT along with a major shift due to the emergence of 5G that will spawn new use cases that will really push legacy testing methodologies. No matter the vertical market, we are all being forced to answer new questions: How will this device work on an airplane’s Wi-Fi network from take-off to landing? What happens if a medical device loses connection as it’s being moved between hospital wards? What impact will load have on network latency for a particular sensor used on a busy factory production line? Can an autonomous vehicle seamlessly move between 5G and Wi-Fi based control without incident? We could ask pages and pages of questions like these that need to be answered before innovation can go from the drawing board to real world application!
Keep it simple
There are literally hundreds of scenarios where the answers will not just impact physical product design, but also have a ripple effect on device placement and day-to-day operating procedures. Making it easier to simulate these scenarios, gain answers and then adapt to change is fundamentally important for everybody concerned with test and assurance.
Even with our integrated OCTOBOX / Vertex solution now heading out into the field for real world use, this ongoing strategy to overcome complexity will continue. Other ways in which we can integrate the best of octoScope and third-party technologies into better solutions continues within our R&D teams. It must, because we know that the world will always find new ways to blend technology and connectivity to create amazing ideas. Hopefully, we can equip the people who make sure these things work in the real world with the right tools to get this vital job done efficiently and accurately with simple-to-use platforms.
As the demand for 5G and Wi-Fi 6 continues to grow at a rapid pace and their standards become increasingly entwined, there has never been a better time to look at the possibility of convergence between the two technologies.
There are more than 16 billion wireless devices in the world today, driving $3.3 trillion in global economic value. The industry will ship an additional 4 billion Wi-Fi devices in 2021 alone.
This incredible growth is driven not only by existing use cases, but also emerging ones. From shopping malls and office buildings to factories and hospitals, reliance on Wi-Fi is pervasive and there’s a heightened priority to rigorously test Wi-Fi products before they ship.
Tech thought leader Diana Adams recently joined me to talk about key insights from Spirent’s new eBook, Testing Wi-Fi for High-Performance Use Cases. Watch the video below as we discuss how Wi-Fi testing today is critical for tomorrow’s wireless world.
To learn more about Wi-Fi testing,.
High-performance use cases such as Work From Home, industrial IoT, telehealth and 4K streaming video are driving adoption of a new generation of Wi-Fi connectivity. These use cases come with high expectations even as the complexity of Wi-Fi networks and the interoperability challenges surge. How can service providers and vendors ensure Wi-Fi 6/6E and beyond deliver the performance their customers demand? They must thoroughly test new Wi-Fi products and services in controlled, repeatable conditions that mimic the real-world.
Learn how to validate 802.11 WLAN AP and device performance, ensure network security, maximize service coverage.
You could say there’s an inverse correlation between how easy Wi-Fi has made broadband connectivity and how complicated mass market Wi-Fi device testing has become. Consider the plight of service providers and device makers tasked with testing Wi-Fi in recent years: constantly acquiring multiple pieces of testing equipment from multiple vendors, managing integrations, adding more and more test chambers with more antennas, running substantial cabling between antennas and chambers, and figuring out how to properly isolate components for repeatable testing. That was challenging enough when Wi-Fi was a mostly standalone, consumer-focused technology. But with Wi-Fi 6 underpinning new high-performance service offerings and expanding 5G convergence on the horizon, “complicated” can no longer be tolerated when it comes to testing.
This escalating challenge served as the backdrop for Spirent’s recent acquisition of. When I founded octoScope to streamline Wi-Fi testing, the burdens on the teams actually doing the testing were already mounting. Now, with Wi-Fi 6 and 5G poised to join forces to address trends across #WFH, industry 4.0, healthcare anywhere, fixed wireless access, and far beyond, stakeholders simply don’t have time for testing headaches. With that in mind, it’s no longer feasible for service providers and device makers to continue trying to make do with homegrown testbed solutions. Not when profitability, smooth customer experiences and time-to-market are on the line.
“It’s no longer feasible for service providers and device makers to continue trying to make do with homegrown testbed solutions. Not when profitability, smooth customer experiences and time-to-market are on the line.”
Wi-Fi 6 driving new testing needs
The complexity of traditional Wi-Fi testing environments has steadily grown over time. There were roaming, handoffs and mesh networks to be tested across access points and devices using dozens of pieces of testing equipment and hardware from multiple providers. All those devices under test and test systems had to be isolated in an array of chambers and interconnected via hundreds of cable connections, as expanding needs like traffic loading and ever-increasing numbers of antennas were added to the equation.
Wi-Fi 6 and now Wi-Fi 6E, fueled by more than a gigahertz of new spectrum in several countries, introduces even more testing requirements. More capacity, longer battery life, latency improvements, extended range, throughput increases, QoS guarantees, even more users – and it all has to be repeatable across a range of environments.
Service providers and device makers can’t keep running to multiple vendors for standards-based and high-performance testing and they can’t be consumed with managing a complex array of chambers and antennas and the cabling between them. Really, there’s only one way forward: a complete solution that can be deployed quickly and configured for precisely the new testing needs at hand. With Wi-Fi 6/6E, a turnkey, modular approach to testing has emerged not as a nice-to-have, but a necessity.
“There’s only one way forward: a complete solution that can be deployed quickly and configured for precisely the new testing needs at hand. With Wi-Fi 6/6E, a turnkey, modular approach to testing has emerged not as a nice-to-have, but a necessity.”
Defining a complete Wi-Fi test platform, on-demand
A modern, unified Wi-Fi testing platform built to accommodate traditional and emerging requirements will be capable of delivering channel emulation, testbed automation, full layer 2-7 testing and load testing – but only as needed in the moment. A modular approach addresses the full range of needs any service provider or device maker would bring, from standards-based and pre-certification testing, all the way through to advanced options for high-performance testing. It is supported by highly-realistic channel and network traffic emulation that allows stress testing based on real-world conditions.
Streamlining Wi-F 6 testing requires a modular approach that allows automated testbeds consisting of isolation chambers, antennas, emulators and test instruments to be rapidly configured and deployed.
Today, Wi-Fi chipset, access point, residential gateway and connected device vendors rightly place a premium on automation and ease of integration to existing systems when selecting Wi-Fi testbeds. In response, a comprehensive testing solution must support automated benchmarking of product performance and validation of functionality and scalability, while presenting simple interfaces and options for integration with existing lab automation.
On the service provider side, operators offering a Wi-Fi service place a high value on automation and ease of use when selecting Wi-Fi testbeds. Their focus is on home and business network environments that “just work” and provide high levels of service that reduce call center inquiries, truck rolls and churn. Delivering this experience at scale to upwards of millions of customers for a range of environments means automated test and assurance processes are a must. In enterprise environments, SLAs for Wi-Fi will increasingly become a reality and pre-deployment testing will go a long way toward ensuring service providers meet promises. From device performance validation and vendor selection to pre-deployment testing, software upgrade testing and recreation of field issues for problem solving, automation will be the common denominator amongst successful Wi-Fi 6 deployments.
The path ahead for better Wi-Fi testing
Modular solutions capable of generating highly-realistic traffic, authentically replicating a range of deployment environments, reducing traditional complexity headaches and supporting automated, repeatable testing scenarios represent the new must-haves for Wi-Fi testing. With octoScope now part of the Spirent family of test solutions, we’re better positioned than ever to offer the industry a one-stop shop for these must-haves, while pursuing octoScope’s original mission of streamlining Wi-Fi testing with a renewed sense of purpose and urgency.