Pre-10G Path to High-Quality UX Cuts Costs of Shifting to New Service Paradigms
For anyone wondering why network service providers should feel some urgency about offering 10G connectivity to broadband households, here’s a startling heads-up from Comcast Cable’s president of technology Tony Werner: “Holographic displays are coming sooner than people think.”
Speaking at the Cable-Tec Expo in October, Werner listed many other reasons for taking bandwidth to levels of symmetrical throughput he once thought would never be necessary. But the timeline he posited for including holographic content in the bitstream was especially meaningful for service providers who’ve been lulled into complacency about supporting various permutations of extended reality (XR) content amid lackluster market response to the experiences offered by virtual reality (VR) and other headgear-based technologies.
“I believe we’ll see another change in display technology with true holographic displays in another 12 to 16 months,” Werner said. “Volumetric capture is coming.”
Indeed, the unrelenting advancement of technologies supporting various approaches to XR, including light-field display technology used to render headgear-free holographic experiences, is gradually pushing XR into the cultural mainstream. It will take a long time for the market to sort out what resonates and what doesn’t, but at this juncture in early 2020, it’s clear we’re speeding toward a vastly altered video display environment that’s likely to foster demand for a wide range of new services, many of which can be delivered utilizing existing network capacity.
VR Tech Advances
Where VR is concerned, some of the latest progress was widely on view at this year’s CES in Las Vegas – as was ample evidence that there’s still a long way to go not just with consumer-priced headgear but also with development of content compelling enough to drive mass market acceptance.
Perhaps the biggest sign of progress in the content domain, which had nothing to do with entertainment, was the fact that VR is gaining traction as a marketing tool. People lined up to don headsets and enter various car manufacturers’ visions of the future, such as Honda’s take on advances that will shape the driving experience over the next 15 years and Hyundai’s version of a ride in an air taxi, a prototype of which hung above the company’s booth.
While VR gaming demos were abundant, the primary CES focus was on advances in headsets, which addressed to some extent key issues plaguing both VR and AR (augmented reality), albeit at price points that were well above any mass market baseline. The developments in VR technology were especially meaningful for network service providers, given the bandwidth implications of VR.
Initially, these breakthroughs in form factor reductions, higher display resolution and field-of-view (FOV) dimensions will be offered at price points that are only viable for industrial and institutional applications, which, as reported elsewhere, have come on strong as VR seedbeds. But they suggest what’s in store for consumers as technology matures and prices fall.
Where reductions in form factor are concerned, Panasonic generated some buzz with an industrial use-targeted prototype which it described as “the world’s first compact High Dynamic Range (HDR) capable VR eyeglasses.” The company used micro OLED technology and other techniques to create “natural and distortion-free images in super single focus,” albeit with a reduced FOV compared to conventional head-mounted devices (HMDs).
For people looking for a commercially available product capable of delivering performance at levels stipulated by NASA, VRgineers is offering the latest version of its XTAL HMD for about $8,000. Pimax’s similarly priced Vision 8K X was another HMD targeted to aerospace applications. Both HMDs touted full 4K resolution for each eye.
More down to earth but still targeted to the enterprise, an HMD offered by China’s Pico Interactive showed some major performance gains over competing devices at a relatively low price of $900. With a form factor that approaches the eyewear dimensions set by Panasonic’s prototype, Pico’s standalone (computer free) Neo 2 Eye delivers what the company calls best-in-class 4K resolution, a better-than-average 1010 FOV and other benefits, including eye-tracking technology from Tobil, which, as previously reported, triggers changes in views of a scene in response to eye movement.
While these devices are opening a track to higher performance that ultimately will be attainable at consumer price levels, announcements from major suppliers of consumer HMDs or, more accurately, the absence thereof, did little to alter the perception that current generation consumer products remain burdened by issues that have slowed adoption. One of the rare counterpoints to that view came with reports that the recently introduced Oculus Quest standalone HMD, which runs on the Qualcomm Snapdragon 835 system-on-a-chip and features two six-degrees-of-freedom (6DOF) controllers, sold out over the holidays at prices ranging from $399 to $499.
Fast Track to High Quality UX
But, notwithstanding HMD issues, there’s another track where things are moving very fast toward enabling network delivery of superior 3600 VR and 2D viewing experiences to the mass market. That it’s now possible to do this at global scale with extraordinary levels of bandwidth efficiency was the point of a remarkable demonstration at the IBC show in Amsterdam last summer.
There a coalition of vendors including Intel, Tiledmedia, Akamai, Google, Iconic Engine, Oculus and Voysys, with a local backbone link assist from KPN, provided 3600 VR coverage of Intel’s Visual Cloud and IABM’s Future Trends Theater conferences captured at 8K resolution and delivered worldwide at bitrates averaging 12-15 Mbps. The demonstration utilized the ClearVR technology developed by Tiledmedia, which, as previously described, reduces VR bandwidth consumption by delivering without discernable delay “tiles” containing just the content that comprises the viewer’s “viewport” as it changes with every turn of the head.*
The purpose of the IBC project was to demonstrate to industry decision makers throughout the world that this approach to delivering this type of XR content works at global scale. People could access the Internet-streamed content in VR mode using HMDs or scan 2D renderings of the 3600 space with devices running on either iOS or Android operating systems.
“This was a collaborative effort that allowed us to do some really significant new things,” said Tiledmedia co-founder and chief business officer Rob Koenen. “One of the interesting aspects is we broadcast worldwide in 8K resolution.”
Live capture at 8K resolution solves one of the big issues with VR displays, which, because the viewer’s eyes are so close to the screen, produce the “screen-door” effect that delineates individual pixels when content is rendered in 4K resolution. With reliance on video captured by 8K cameras, the ClearVR Cloud Live software was able to pack more pixels into the tiles that were HEVC encoded for distribution through the Akamai ecosystem to users’ devices, where the decoding was performed by the native codecs. Since the tile viewport represents only about 20 percent of the full field of view, the encoded bitstream was well within average bandwidth parameters.
While it’s practical to encode in 8K, the ClearVR technique entails some adjustments to the encoding process, which were accommodated by Intel. “We require certain constraints in the motion vectors on the encoder,” Koenen said, noting this was done with Intel’s open-source SVT-HEVC software encoder.
The conference proceedings were captured by two Kandao Obsidian R cameras, each of which produced three individual feeds for compilation into separate files, providing viewers two looks into that space that they could switch between at will. VR production software from Voysys was used to compile each of the camera feeds at 8092×4046 resolution into an equirectangular projection, analogous to the flat 360o rendering of a global map, which was then formatted to fill the six sides of a cubeknown as a cubemap.
The live cubemap-formatted bitstream was fed via the KPN network to nearby Google Cloud Platform facilities in Groningen, the Netherlands. There ClearVR Cloud Live software decoded and processed the cubemap stream into two sets of 96 tiles, 16 for each cube face, from which the viewports of all viewers were fashioned in response to prompts from ClearVR client software running on each viewer’s device. Along with reading which way the viewer is looking as determined by HMD trackers, the ClearVR client has the logic to request, decode and arrange tiles relevant to rendering the viewport.
As explained by Koenen in a blog following IBC, the encoding of two sets of 96 tiles for each cubemap allowed the platform to choose tiles comprised of short group-of-picture (GOP) segments when the user switched viewports while otherwise relying on longer GOPs, which are more bandwidth efficient. The ClearVR-directed tiling encoding process also entailed creation of six larger tiles at lower resolution, one for each cube face, to provide background coverage of the user’s FOV beyond the immediate focus represented by the viewport.
The ClearVR-packaged bit streams then had to be ingested from the GCP into the Akamai CDN for worldwide distribution. This required use of six parallel Akamai Media Services Live (MSL) ingest points per composite feed, insofar as Akamai has set a 45 Mbps ingest limit per MSL. That limit is enough to handle any 2D video stream, but in this case where more were needed to handle the 120 Mbps volumetric streams, six were chosen to ensure there would be ample capacity on each MSL to accommodate traffic spikes and other flows. Once ingested, the IBC feeds were recombined and uploaded onto Akamai’s EU region origins for distribution worldwide.
All of these processes from camera outputs through delivery were executed under direction of the ClearVR Orchestrator by encoders, packagers and other software components running on many hundreds of Intel Xeon Platinum 9200 processors. It took about 30 seconds from the instant of capture in real time to the on-screen rendering of that instant on each display, Koenen said. He noted that, going forward, end-to-end delivery of real-time events would be reduced to less than 10 seconds with use of Chunked Transfer Encoding, an option available with transmissions based on MPEG’s Common Media Application Format (CMAF).
Once each viewer acquired the stream, the platform had to react to each shift of that viewer’s gaze by sending a revised tile compilation matched to the new viewport quickly enough to be experienced as virtually instantaneous by the viewer. This requires very low latency in the roundtrip from the instant the user’s device communicates a change in viewport to when the content is rendered on the device.
According to Koenen, in the case of a video encoded at 30 frames per second, the new scene must be rendered within about three frames from the point when the viewer shifts perspective. That translates to something on the order of 100 milliseconds. This is easily doable in a closed, tightly managed network environment, but to accomplish this from a single point of origin communicating across the ends of the earth requires another innovation introduced with this project.
“We worked with Akamai to implement pre-fetching of tiles at the edge,” Koenen told ScreenPlays. That entailed implementation of a VR-specific configuration in the CDN, devised by Tilemedia and Akamai, that uses edge caches to queue up tiles comprising the viewports deemed most likely to be requested by users.
An Expanding Testbed
The IBC project occurred in train with other trials and early commercial projects employing tiling and some of the other processes described here, including strictly “planar” (2D) 3600 viewing of live sports events at 8K. For example, BT sport tested the 8K experience with delivery of live content from the 2019 FA Cup Final at Wembley Stadium in London for 3600 viewing on 100 testers’ Android and iOS smartphones.
BT began offering 4K 3600 viewing with soccer events in 2017. “4K gets you so far but we wanted to really work out how to improve that experience and take it to the next level,” said BT Sport chief engineer Andy Beale.
Commenting on the FA Cup test last May, Beal said, “The results are remarkable. You get super high-resolution images that you can then pinch and zoom on as well as move around. You really do feel like you’re at the stadium.”
Use of the zooming capability with planar viewing is not yet available for mass market applications, Koenen noted. “Right now, we’re looking at this for in-venue use where there’s no need to transmit out to the cloud and back,” he said.
But the ability to zoom in is an attractive component for potential mass-market delivery of 3600 viewing with sports and other events. In fact, enabling zooming (and panning) was the purpose behind including tiling as part of HEVC Main 10 for traditional 2D streaming. The real question is how strong market support will be for enabling the capability with the 3600 2D experience.
There’s been some discussion about enabling zooming with 3600 viewing on TV sets, possibly using remotes or hand gestures. “We’ve talked with service providers about doing this,” Koenen said.
But, he stressed, the company’s focus has been on supporting a superior network-delivered immersive VR experience where zooming is not an issue, which was the primary aim of the IBC project. “Sports will be a major driver,” he said, noting the company is working with MediaKind to enable sports and music VR services under development at Deutsche Telekom.
Tiledmedia is working with many other entities such as Viaccess-Orca, Harmonic and Intel on VR delivery projects around the world. “There’s a lot going on in China with VR,” Koenen said. “We’re working with Intel there and starting to do videos involving the 12K headsets that are now on their roadmap.”
Getting to high-res display quality is vital to building mass market support, Koenen added. “To date it hasn’t been possible to do good quality at scale,” he said. “At IBC we proved it’s now possible to do great quality at scale.”
Ultimately, as Tony Werner noted, 10G connectivity will change the game when it comes to enabling volumetric content, even when the bandwidth requirements reach 500 Mbps, as would be the case using current compression techniques for transmission of holographic content. The thought that holographic projections suited for mass market consumption might be just a year or so away, echoed by others at Expo, puts an entirely new spin on what might be doable with network support for volumetric content.
But, given the capabilities shown with Tiledmedia, the emergence of a mass market base for 3600 VR content doesn’t require bandwidth expansion. The question service providers will soon be addressing isn’t whether there will ever be a serious market interest in immersive experiences. It will be about how to make sure their networks can deliver whatever consumers and businesses decide they want.
* While the relatively static motion environment of a conference minimized the moment-to- moment transmission bitrates, the point that the tiling mode of delivering 3600 VR delivery content serves to lower bandwidth consumption to levels comparable to 2D transmission was well made, because the type of scene captured live has no impact on the effectiveness of tiling. Thus, the proof-of-concept pursued with this project would have been made even if this had been a live sports event requiring twice as much bandwidth per stream.
