U.S. patent application number 15/156069 was filed with the patent office on 2017-11-02 for method and system for defining a virtual reality resolution distribution.
The applicant listed for this patent is Immersive Enterprises, LLC. Invention is credited to Husam Aldalaq, Taher Baderkhan, Moshe Ben-Zacharia, Charles Dunn, Juan Ferreira, Vincent Machacek, Daniel Morton.
Application Number | 20170316549 15/156069 |
Document ID | / |
Family ID | 59382646 |
Filed Date | 2017-11-02 |
United States Patent
Application |
20170316549 |
Kind Code |
A1 |
Dunn; Charles ; et
al. |
November 2, 2017 |
METHOD AND SYSTEM FOR DEFINING A VIRTUAL REALITY RESOLUTION
DISTRIBUTION
Abstract
Aspects of the present disclosure include a method and system
for the delivery of virtual reality content that enables a more
immersive experience to the user by adaptively changing desired
resolution distributions according to a variety of parameters and
to create resolution-defined projections. The resolution-defined
projections can be two-dimensional and virtual reality content
agnostic. According to some aspects, the projections can be used to
generate virtual reality content images and video that can be
encoded and/or compressed for processing and transmission with
lower or without increasing bandwidth requirements, and to be
perceived of a higher resolution. Additional aspects may include,
the adaptive focus via the resolution-defined projections using the
user's viewing direction and/or orientation, system limitations,
user vision's constraints, optimization data from a feedback loop
and/or crowdsourcing data.
Inventors: |
Dunn; Charles; (Astoria,
NY) ; Baderkhan; Taher; (New York, NY) ;
Machacek; Vincent; (Brooklyn, NY) ; Aldalaq;
Husam; (New York, NY) ; Ferreira; Juan;
(Richmond Hill, NY) ; Morton; Daniel; (Elmsford,
NY) ; Ben-Zacharia; Moshe; (New York, NY) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Immersive Enterprises, LLC |
New York |
NY |
US |
|
|
Family ID: |
59382646 |
Appl. No.: |
15/156069 |
Filed: |
May 16, 2016 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15143424 |
Apr 29, 2016 |
9721393 |
|
|
15156069 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G09G 2350/00 20130101;
H04L 67/18 20130101; G06F 3/011 20130101; G06T 3/4092 20130101;
G06T 2200/04 20130101; G06T 15/205 20130101; G06T 2200/08 20130101;
G06T 3/005 20130101; G06T 19/20 20130101; G09G 2340/02 20130101;
G09G 2370/16 20130101; G09G 5/005 20130101; G06T 19/006 20130101;
G09G 5/395 20130101; G06F 3/013 20130101; G09G 2340/0407 20130101;
G06F 3/14 20130101; G06F 3/012 20130101; G06T 2219/2016 20130101;
G06T 15/20 20130101; H04L 67/38 20130101 |
International
Class: |
G06T 3/40 20060101
G06T003/40; G06T 3/00 20060101 G06T003/00; G06F 3/01 20060101
G06F003/01; G06T 19/00 20110101 G06T019/00; G06T 19/20 20110101
G06T019/20; G06F 3/01 20060101 G06F003/01 |
Claims
1. A system for processing and delivering virtual reality content,
the system comprising: one or more computer servers comprising one
or more processors and one or more storage devices; and executable
software stored on at least one of the one or more storage devices
and executable on demand, the software operative with at least one
of the one or more processors to cause the one or more computer
servers to: select one or more sets of parameters affecting the
virtual reality content's resolution when inversely projected to a
three-dimensional shape; define a desired resolution distribution
using at least some of the one or more sets of parameters selected,
wherein the desired resolution distribution is able to counteract
all or some constraints corresponding to the selected one or more
sets of parameters; create a two-dimensional resolution-defined
projection according to the desired resolution distribution; map
the virtual reality content to the two-dimensional
resolution-defined projection spaces; derive one or more
two-dimensional renditions from the mapped virtual reality content
to the two-dimensional resolution-defined projection spaces; and
access the one or more two-dimensional renditions for display by a
device configured to inversely project the virtual reality image
data to a sphere or other three-dimensional shape.
2. The system of claim 1, wherein at least one of the sets of
parameters correspond to a virtual reality content capturing set up
and the software is additionally operative with at least one of the
one or more processors to cause the one or more computer servers
to: match the virtual reality content's pixel resolution resulting
from the parameters corresponding to the virtual content capturing
set up to a spatial resolution that can be used to define the
desired resolution distribution.
3. The system of claim 2, wherein the parameters corresponding to
the virtual content capturing set up can include one or more of a
physical arrangement of image capturing devices, lens properties,
and stitching methods.
4. The system of claim 1, wherein at least one of the sets of
parameters correspond to a user's virtual reality device and the
software is additionally operative with at least one of the one or
more processors to cause the one or more computer servers to: match
the virtual reality content's pixel resolution resulting from the
parameters corresponding to the user's virtual reality device to a
spatial resolution that can be used to define the desired
resolution distribution.
5. The system of claim 4, wherein the parameters corresponding to
user's virtual reality device can include one or more of a
display's specifications, power consumption, connection properties,
and processing speeds.
6. The system of claim 1, wherein at least one of the sets of
parameters correspond to a user's visual capacity and the software
is additionally operative with at least one of the one or more
processors to cause the one or more computer servers to: match the
virtual reality content's pixel resolution resulting from the
parameters corresponding to the user's visual capacity to a spatial
resolution that can be used to define the desired resolution
distribution.
7. The system of claim 6, wherein the parameters corresponding to
the user's visual capacity can include one or more of a viewing
direction, a fovea field of view, eyeball movement, and visual
impairments.
8. The system of claim 6, wherein the user's visual capacity
parameters include specific user's visual impairments and the
software is additionally operative with at least one of the one or
more processors to cause the one or more computer servers to: match
the virtual reality content's pixel resolution resulting from the
parameters corresponding to the specific user's visual impairments
to a spatial resolution that can be used to define the desired
resolution distribution, the desired resolution distribution
implemented to correct or provide a therapeutic visual effect to
the user.
9. The system of claim 1, wherein at least one of the sets of
parameters correspond to crowdsourced data and the software is
additionally operative with at least one of the one or more
processors to cause the one or more computer servers to: match the
virtual reality content's pixel resolution resulting from the
parameters corresponding to the crowdsourced data to a spatial
resolution that can be used to define the desired resolution
distribution.
10. The system of claim 1, wherein at least one of the sets of
parameters correspond to feedback loop data defining one or more
user's preferences and the software is additionally operative with
at least one of the one or more processors to cause the one or more
computer servers to: match the virtual reality content's pixel
resolution resulting from the parameters corresponding to the
feedback loop data defining one or more user's preferences to a
spatial resolution that can be used to define the desired
resolution distribution.
11. The system of claim 1, wherein at least one of the sets of
parameters correspond to content metrics and the software is
additionally operative with at least one of the one or more
processors to cause the one or more computer servers to: match the
virtual reality content's pixel resolution resulting from the
parameters corresponding to the content metrics to a spatial
resolution that can be used to define the desired resolution
distribution.
12. The system of claim 11, wherein the parameters corresponding to
the content metrics can include one or more of virtual reality
content contrast, virtual reality content detected movement, and
colors contained in the virtual reality content.
13. A method for processing and delivering virtual reality content,
the method comprising: selecting one or more sets of parameters
affecting the virtual reality content's resolution when inversely
projected to a three-dimensional shape; defining a desired
resolution distribution using at least some of the one or more sets
of parameters selected, wherein the desired resolution distribution
is able to counteract all or some constraints corresponding to the
selected one or more sets of parameters; creating a two-dimensional
resolution-defined projection according to the desired resolution
distribution; mapping the virtual reality content to the
two-dimensional resolution-defined projection spaces; deriving one
or more two-dimensional renditions from the mapped virtual reality
content to the two-dimensional resolution-defined projection
spaces; and accessing the one or more two-dimensional renditions
for display by a device configured to inversely project the virtual
reality image data to a sphere or other three-dimensional
shape.
14. The method of claim 13, wherein at least one of the sets of
parameters correspond to a virtual reality content capturing set up
and additionally comprising: matching the virtual reality content's
pixel resolution resulting from the parameters corresponding to the
virtual content capturing set up to a spatial resolution that can
be used to define the desired resolution distribution.
15. The method of claim 14, wherein the parameters corresponding to
the virtual content capturing set up can include one or more of a
physical arrangement of image capturing devices, lens properties,
and stitching methods.
16. The method of claim 13, wherein at least one of the sets of
parameters correspond to a user's virtual reality device and
additionally comprising: matching the virtual reality content's
pixel resolution resulting from the parameters corresponding to the
user's virtual reality device to a spatial resolution that can be
used to define the desired resolution distribution.
17. The method of claim 16, wherein the parameters corresponding to
user's virtual reality device can include one or more of a
display's specifications, power consumption, connection properties,
and processing speeds.
18. The method of claim 13, wherein at least one of the sets of
parameters correspond to a user's visual capacity and additionally
comprising: matching the virtual reality content's pixel resolution
resulting from the parameters corresponding to the user's visual
capacity to a spatial resolution that can be used to define the
desired resolution distribution.
19. The method of claim 18, wherein the parameters corresponding to
the user's visual capacity can include one or more of a viewing
direction, a fovea field of view, eyeball movement, and visual
impairments.
20. The method of claim 18, wherein the user's visual capacity
parameters include specific user's visual impairments and
additionally comprising: matching the virtual reality content's
pixel resolution resulting from the parameters corresponding to the
specific user's visual impairments to a spatial resolution that can
be used to define the desired resolution distribution, the desired
resolution distribution implemented to correct or provide a
therapeutic visual effect to the user.
21. The method of claim 13, wherein at least one of the sets of
parameters correspond to crowdsourced data and additionally
comprising: matching the virtual reality content's pixel resolution
resulting from the parameters corresponding to the crowdsourced
data to a spatial resolution that can be used to define the desired
resolution distribution.
22. The method of claim 13, wherein at least one of the sets of
parameters correspond to feedback loop data defining one or more
user's preferences and additionally comprising: matching the
virtual reality content's pixel resolution resulting from the
parameters corresponding to the feedback loop data defining one or
more user's preferences to a spatial resolution that can be used to
define the desired resolution distribution.
23. The method of claim 13, wherein at least one of the sets of
parameters correspond to content metrics and additionally
comprising: match the virtual reality content's pixel resolution
resulting from the parameters corresponding to the content metrics
to a spatial resolution that can be used to define the desired
resolution distribution.
24. The method of claim 23, wherein the parameters corresponding to
the content metrics can include one or more of virtual reality
content contrast, virtual reality content detected movement, and
colors contained in the virtual reality content.
Description
RELATED APPLICATIONS
[0001] This application claims priority as a continuation
application to U.S. patent application Ser. No. 15/143,424, titled
"METHOD FOR PROCESSING AND DELIVERING VIRTUAL REALITY CONTENT TOA
USER" filed on Apr. 29, 2016, the contents of which are
incorporated herein by reference.
FIELD OF THE INVENTION
[0002] The present invention relates generally to the field of
image processing in the virtual reality space and more particularly
to an improved method for processing and delivering virtual reality
content using spherical projections.
BACKGROUND OF THE INVENTION
[0003] The development of virtual reality technology has rapidly
increased over the last few years. As virtual reality technology
improves, new virtual reality applications in an array of fields
including, for example, education and training, art, architectural
design, movie production, advertisement, entertainment, video
games, therapy, and medical fields, have been adopted or shown
potential interest. The implementation of virtual reality in each
field however often presents new challenges and requirements for
the virtual reality application to be useful and functional. In
resolving those challenges to meet each application's requirements,
data content exponentially increases, compromising delivery
reliability, feasibility, speeds and resolution.
[0004] Virtual reality experiences require high resolution content
that is reliably delivered to the user for the experience to by
fully immersive. Without a fully immersive experience, users of
virtual reality are unlikely to adopt and implement the virtual
reality technology for their particular application. In nearly all
current virtual reality applications, the constraint on resolution
generally results from the bandwidth available, as opposed to any
limit on content generation. Depending on the playback device,
whether it is streaming content or accessing locally stored
content, and the application, additional constraints also result
from the image processing and resolution due to limitations of the
central processing unit (CPU), memory, screen maximum resolution,
graphics processing unit (GPU) and storage capacity, power
consumption, and overheating of the electronic device. These
constraints in most cases decrease the level of immersion for the
user and greatly compromise the reliability needed for some
applications.
[0005] Recent image and video encoding and compression efforts have
been explored. None of these however have resulted in a widely
adopted solution due to fundamental limitations of their own. For
example, various virtual reality content compression algorithms and
methods specifically designed for 360-degree video have been
experimented with. None have gained widespread use, nor have they
adequately demonstrated their improvement over, or compatibility
with, traditional video compression methods. U.S. Pat. No.
6,005,611 to Gullichsen et al., for example, describes a method
that seeks to provide efficient delivery of 360-degree content
where warped fish-eye lens video is stored and locally distorted
depending on viewing geometry. This method by its own solution does
not encode the full 360.degree. of content, but rather only
transmits and dewarps a rectangular subset of the data,
irrecoverably limiting the delivered content. Because the content
is irrevocably limited to only the subset transmitted, the viewer
will experience dislocated views outside the transmitted region,
disrupting immersion.
[0006] Alternative methods have been described in "Data compression
on the sphere" http://arxiv.org/pdf/1108.3900.pdf and "Low bit-rate
compression of omnidirectional images"
http://infoscience.epfl.ch/record/130365/files/omni_cr.pdf both
which seek to compress data still in its spherical form. These too
however have failed to be widely adopted in virtual reality
applications due to the lack of flexibility between applications
and projection devices. As another example of an unsuccessful
compression idea for virtual reality content, U.S. Pat. Pub. No.
2016/0021373 A1 to Queru describes a method that only works with
JPEG images, excluding all other forms of virtual reality content.
In particular, because the method changes the number of macroblocks
spatially, it must be fully integrated with the compression
algorithm, rather than being an independent preprocessing step.
This method also introduces compression artifacts which appear
blocky and greatly hinder immersion, especially when present in
stereoscopic content, as well as restricting the final resolution
to be constant within all individual macroblocks. A common
fundamental problem of the aforementioned compression methods is
that traditional compression algorithms have hardware support for
encoding and decoding on appropriate devices, so trying to force
adoption of new compression algorithms is challenging.
[0007] Due to the aforementioned requirements and limitations, the
prior art solutions thus suffer in their ability to enable
processing and/or transmitting of high resolution virtual reality
content to a user while also reducing the amount of content
processed and bandwidth without compromising reliability.
Accordingly:
[0008] There is a need for encoding and compression agnostic
virtual reality content image processing solutions;
[0009] There is a need for image processing solutions that are
flexible as to the projection functions that can be used to process
and deliver virtual reality content:
[0010] There is a need for image processing solutions that result
in useful distributions of resolution, such that data is uniformly
and maximally useful;
[0011] There is a need for a virtual reality content processing and
delivery method that can provide content flexibility and lower
bandwidth requirements without compromising the virtual reality
experience, for example, due to lower resolution and/or playback
problems;
[0012] There is a need for a virtual reality content processing and
delivery method that lowers the hardware and software requirements
of processing and display devices; and
[0013] There is a need for image/video processing solutions that
can reduce the amount of data processed by the devices to improve
battery life and prevent overheating of the device.
[0014] Accordingly, improved virtual reality content processing and
delivery methods and systems capable of overcoming the
aforementioned needs are desired.
SUMMARY OF THE INVENTION
[0015] The foregoing needs are met, to a great extent, by the
present invention, wherein in some aspects of embodiments of the
invention are intended to address one or more of the above noted
fundamental problems rooted in image processing for the processing
and/or delivery of virtual reality content due to streaming, image
processing and resolution limitations. The inventive aspects
disclosed enable a more immersive virtual reality experience while
providing flexibility in the compression method used and reducing
both the amount of content processed and bandwidth requirements. In
some embodiments, for example, this can be accomplished by enabling
one or more of: using desired resolution distribution of
projections in conjunction with adaptive focus, selectively
transmitting content according to the view direction, converting
spherical virtual reality content to rectangular content for
encoding and/or compression, dynamically sending resolution that
does not exceed the max DPI for the screen, using crowd sourced
data to control the projections and/or transmission of data and
lowering bandwidth requirements.
[0016] According to some aspects, an exemplary system for
processing and delivering virtual reality content is disclosed. The
system including: one or more computer servers comprising one or
more processors and one or more storage devices and executable
software stored on at least one of the one or more storage devices
and executable on demand. The software operative with at least one
of the one or more processors to cause the one or more computer
servers to: select one or more sets of parameters affecting the
virtual reality content's resolution when inversely projected to a
three-dimensional shape; define a desired resolution distribution
using at least some of the one or more sets of parameters selected,
wherein the desired resolution distribution is able to counteract
all or some constraints corresponding to the selected one or more
sets of parameters; create a two-dimensional resolution-defined
projection according to the desired resolution distribution; map
the virtual reality content to the two-dimensional
resolution-defined projection spaces; derive one or more
two-dimensional renditions from the mapped virtual reality content
to the two-dimensional resolution-defined projection spaces; and
access the one or more two-dimensional renditions for display by a
device configured to inversely project the virtual reality image
data to a sphere or other three-dimensional shape.
[0017] In some embodiments, at least one of the sets of parameters
can correspond to a virtual reality content capturing set up and
the software can be additionally operative with at least one of the
one or more processors to cause the one or more computer servers
to: match the virtual reality content's pixel resolution resulting
from the parameters corresponding to the virtual content capturing
set up to a spatial resolution that can be used to define the
desired, resolution distribution. The parameters corresponding to
the virtual content capturing set up can include, for example, one
or more of a physical arrangement of image capturing devices, lens
properties, and stitching, methods, in additional embodiments, at
least one of the sets of parameters may, additionally or
alternatively, correspond to a user's virtual reality device and
the software is additionally operative with at least one of the one
or more processors to cause the one or more computer servers to:
match the virtual reality content's pixel resolution resulting from
the parameters corresponding to the user's virtual reality device
to a spatial resolution that can be used to define the desired
resolution distribution. The parameters corresponding to user's
virtual reality device can include one or more of a display's
specifications, power consumption, connection properties, and
processing speeds.
[0018] In yet additional embodiments, at least one of the sets of
parameters may, additionally or alternatively, correspond to a
user's visual capacity and the software can be additionally
operative with at least one of the one or more processors to cause
the one or more computer servers to: match the virtual reality
content's pixel resolution resulting from the parameters
corresponding to the user's visual capacity to a spatial resolution
that can be used to define the desired resolution distribution. The
parameters corresponding to the user's visual capacity can include,
for example, one or more of a viewing direction, a fovea field of
view, eyeball movement, and visual impairments. Where the visual
impairments are accounted for, the desired resolution distribution
implemented can correct or provide a therapeutic visual effect to
the user, for example.
[0019] In yet additional embodiments, at least one of the sets of
parameters may, additionally or alternatively, correspond to
crowdsourced data and the software can be additionally operative
with at least one of the one or more processors to cause the one or
more computer servers to: match the virtual reality content's pixel
resolution resulting from the parameters corresponding to the
crowdsourced data to a spatial resolution that can be used to
define the desired resolution distribution. The sets of parameters
corresponding to feedback loop data defining one or more user's
preferences and the software can be additionally operative with at
least one of the one or more processors to cause the one or more
computer servers to: match the virtual reality content's pixel
resolution resulting from the parameters corresponding to the
feedback loop data defining one or more user's preferences to a
spatial resolution that can be used to define the desired
resolution distribution. Similarly, at least one of the sets of
parameters correspond to content metrics and the software may be
additionally operative with at least one of the one or more
processors to cause the one or more computer servers to: match the
virtual reality content's pixel resolution resulting from the
parameters corresponding to the content metrics to a spatial
resolution that can be used to define the desired resolution
distribution. The parameters corresponding to the content metrics
can include one or more of virtual reality content contrast,
virtual reality content detected movement, and colors contained in
the virtual reality content.
[0020] The one or more image capturing devices may be any image
capturing device that is enabled to capture all or a portion of the
immersive image content. For example, in some embodiments, the
image capturing devices can include 360.degree. image capturing
devices. Alternatively or in addition to, in some embodiments, one
or more of the capturing devices can include computers enabled to
capture immersive content from a computer generated environment
(i.e., in virtual space). In some embodiments, the generating of
the one or more renditions can include creating a two-dimensional
resolution-defined projection according to a desired resolution
distribution that is modified during playback of the virtual
reality content according to the retrieved parameters. Moreover,
one or both of lowering the resolution and changing parameters of
at least one of the one or more renditions is a function of the
modified desired resolution distribution.
[0021] According to additional aspects, a method corresponding to
the exemplary system for processing and delivering virtual reality
content is disclosed. The method including: selecting one or more
sets of parameters affecting the virtual reality content's
resolution when inversely projected to a three-dimensional shape;
defining a desired resolution distribution using at least some of
the one or more sets of parameters selected, wherein the desired
resolution distribution is able to counteract all or some
constraints corresponding to the selected one or more sets of
parameters; creating a two-dimensional resolution-defined
projection according to the desired resolution distribution;
mapping the virtual reality content to the two-dimensional
resolution-defined projection spaces; deriving one or more
two-dimensional renditions from the mapped virtual reality content
to the two-dimensional resolution-defined projection spaces; and
accessing the one or more two-dimensional renditions for display by
a device configured to inversely project the virtual reality image
data to a sphere or other three-dimensional shape.
[0022] Additional general optional features can additionally
include, for example, predicting and/or identifying resolution
distributions to control processing according to a change to take
place, or be encouraged, in the viewing direction and/or
orientation of the user. Other aspects of the invention will be
understood by those of skill in the art upon review of the
teachings herein. Other aspects of the invention may involve
combinations of the above noted aspects of the invention. These
other aspects of the invention may provide various combinations of
the aspects presented above as well as provide other
configurations, systems, functional relationships, and processes
that have not been specifically set forth above.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings, which are included to provide a
further understanding of the invention, are incorporated in and
constitute a part of this specification, illustrate embodiments of
the invention and together with the detailed description serve to
explain the principles of the invention.
[0024] FIG. 1 is a schematic of an exemplary system that can be
used according to aspects of the present disclosure;
[0025] FIG. 2 is a schematic of an exemplary arrangement of
functional components that can be used according to aspects of the
present disclosure;
[0026] FIG. 3 is a flowchart illustrating method steps that may be
implemented for the processing and delivery of virtual reality
content according to some aspects of the disclosure;
[0027] FIG. 4 is a flowchart illustrating method steps that may be
implemented to generate a resolution-defined projection according
to some aspects of the disclosure;
[0028] FIG. 5 is a schematic representation illustrating the method
steps of FIG. 4;
[0029] FIG. 6 is a flowchart illustrating method steps that may be
implemented to transmit virtual reality content according to
additional aspects of the disclosure;
[0030] FIG. 7 is a schematic representation illustrating the method
steps of FIG. 6;
[0031] FIG. 8 is a flowchart illustrating method steps that may be
implemented to process and deliver virtual reality content
according to yet additional aspects of the disclosure; and
[0032] FIG. 9 is a schematic representation illustrating the method
steps of FIG. 8.
[0033] The present invention is further described in the detailed
description that follows.
DETAILED DESCRIPTION OF THE INVENTION
[0034] Going forward, various aspects of the inventions of the
present disclosure may be illustrated by describing method steps or
combination of method steps which overcome various limitations and
problems rooted in streaming virtual reality content technology.
The inventions are not limited to a particular application/field
implementing virtual reality content. Various aspects of the
steering device may be illustrated with reference to one or more
exemplary embodiments. Moreover, as used herein, the terms "in some
embodiments" or "exemplary" mean serving as an example, optional
aspect, instance, or illustration, and should not necessarily be
construed as preferred or advantageous over other embodiments
disclosed herein.
Glossary
[0035] In this description and claims directed to the disclosure,
various terms may be used for which the following definitions will
apply:
[0036] "Adaptive focus", as used herein can refer to the optimized
processing and/or transmission of a selected different virtual
reality image(s) that may at least partially dependent on a desired
resolution distribution. In some embodiments, the desired
resolution distribution may depend on a fovea field of view for the
corresponding viewer's direction so that the resolution can be best
for whichever direction the viewer is looking, contrast of the
virtual reality image data, movement of the virtual reality image
data content, capturing camera arrangement and geometry, capturing
camera lenses, display parameters, and crowd sourced data relating
to the virtual reality image content.
[0037] "Crowd sourced data", as used herein, can refer to a
collection of data collected from at least one other virtual
reality user or person associated with the virtual reality content.
The data, collected from at least one other virtual reality user
may be in relation to him/her viewing the virtual reality content
and include, for example, data descriptive of head movement
patterns (specific or non-specific to displayed content),
connection, geographic location, time duration, frequency between
action patterns, and the such. The data collected from at least one
other person associated with the virtual reality content may
include, for example, data collected from the at least one other
person that is part of or has directly viewed the virtual reality
content (either live or in the past).
[0038] "Desired resolution distribution", as used herein can refer
to a spatial resolution function that can be defined by one or more
of the fovea field of view for a particular viewing direction,
contrast of the virtual reality image data, movement of the virtual
reality image data content, camera arrangement, camera lenses, and
crowd sourced data.
[0039] "Fovea field of view", as used herein can refer to a
correlation of spherical virtual reality image content and the
human's field of view that is of higher visual acuity/sensitivity
near the center of the retina. More specifically, about 200 degrees
horizontally, of the 360 degrees of content, and 135 degrees
vertically, of the 180 degrees of content, with higher resolutions,
for example, at about 10 degrees from direction of focus to
gradually degraded towards the periphery.
[0040] "Image content" or "image data" as used herein can include
360 content in either one or both of three-dimensional and
two-dimensional formats for processing and display by virtual
reality devices and, in some embodiments, immersive displays of
wireless devices, smartphones, tablets, PCs, desktops, smart TVs
and the such. The virtual reality devices or immersive displays are
capable of processing selected portions of the 360 content in still
images and/or video clips. Accordingly, image content or image data
can include both captured still images or portions thereof and/or
video.
[0041] "Projection", as used herein can refer to the mapping of
virtual reality spherical image data to flat rectangular data.
According to aspects of the disclosure, the projection can be, a
resolution-defined projection as to enable compression and higher
resolution virtual reality experiences with lower virtual reality
content transmission bandwidths.
[0042] "Resolution-defined projection", as used herein can refer to
a projection that is defined by a resolution distribution. In some
embodiments, the projection may be able to adapt to a separable
resolution distribution. The separable resolution distribution
function may be, for example, a product of two different resolution
functions, one for each of azimuth and elevation. Accordingly, the
basis for a resolution-defined projection may be equirectangular or
any bijective projection that has a separable Jacobian.
[0043] "User specific learned behavior", as used herein may refer
to the customized delivery of virtual reality content to a user
according at least in part to feedback data from one or more
previous virtual reality content experiences (i.e. feedback loop
data) of the user.
[0044] "Virtual reality content" or "virtual reality image data",
as used herein refer to any one or more of "image content"/"image
data" recorded, received, processed (including during encoding,
storing, and compression), and transmitted, in one of a 3
dimensional spherical format or any other 3 dimensional shape, 2
dimensional rectangular format or projected (e.g., as video onto a
spherical display) according to the different aspects of the
process disclosed.
[0045] Furthermore, it shall be understood that a projection into a
sphere or any other three dimensional shape can refer to the way
the image is processed to generate an immersive image/video that
can be displayed and not to the shape of the screen of the device.
I.e., the spherical or 3D shape projecting aspect is a function of
the software in generating the rendering and not of the display
shape.
[0046] The embodiments of the invention and the various features
and advantageous details thereof are explained more fully with
reference to the non-limiting embodiments and examples that are
described and/or illustrated in the accompanying drawings and
detailed in the following description. It should be noted that the
features of one embodiment may be employed with other embodiments
as one skilled in the art would recognize, even if not explicitly
stated herein. Descriptions of well-known components and processing
techniques may be omitted, so as to not unnecessarily obscure the
embodiments of the disclosure. The examples used herein are
intended merely to facilitate an understanding of ways in which the
disclosure may be practiced in the transmission and processing of
virtual reality content and to further enable those of skill in the
art to practice the embodiments of the disclosure. Accordingly, the
examples and embodiments herein should not be construed as limiting
the scope of the disclosure, which is defined solely by the
appended claims and applicable law. Moreover, it is noted that like
reference numerals can represent common steps/parts throughout the
drawings.
[0047] Virtual reality content can be immersive because a user can
be entirely surrounded by a virtual environment. As opposed to
traditional image or video displayed on a screen, virtual reality
immersive experiences fill every possible viewing direction. A
virtual reality experience thus requires processing of a full
sphere of image content. This sphere of content often must be
delivered both in high resolution and over a bandwidth-constrained
network(s). Compression of the content can be essential, according
to aspects of the disclosure to optimally utilize existing
algorithms for image and video data compression, the virtual
reality input can be mapped to be flat and rectangular.
Accordingly, as further described in the exemplary embodiments
herein, a projection can be used along with desired resolution
distributions to convert the three-dimensional (3D) spherical
content in virtual reality space to the two-dimensional (2D)
rectangular representation for input to encoding methods. In this
description, various exemplary projections and resolution functions
are further disclosed to demonstrate the inventive aspects of the
resolution-defined projections useful for virtual reality content
processing and delivery.
[0048] To one skilled in the art, this disclosure enables at least:
(a) encoding and compression agnostic image content processing
solutions; (b) process flexibility as to the projection functions
that can be used to process and deliver image content; (c)
solutions that can work using arbitrary resolution functions; (d) a
virtual reality content processing and delivery method that can
provide compression and content flexibility and lower bandwidth
requirements without compromising the virtual resolution
experience, for example, due to lower resolution and/or playback
problems; (e) a solution to can lower the amount of image data
processed to reduce power consumption and overheating of the
device; (f) method that enables useful distributions of resolution,
such that data is uniformly and maximally useful; (g) a method that
can enable the changing of the resolution sent based on display
and/or processing parameters; and (h) a virtual reality content
processing and delivery method that lowers the hardware and
software requirements of display devices.
[0049] Referring now to FIG. 1, a schematic of an exemplary system
1000 that can be used according to aspects of the present
disclosure is depicted. In particular, the system 1000 which can
better process and transmit image content for providing an
immersive experience to users. According to some aspects, image
content 1005 can be transmitted to a computer server 1010.
Transmission may take place via-wireless or wired connections
automatically or upon request. For example, image content 1005 may
be downloaded or streamed from one or an array of 360 image content
capturing device(s). In some embodiments, the system may be a
distributed system with a main computer server 1010 handling the
loading of the image content inputted and scheduling and/or
transferring content to various processors 1010a that can work in
parallel, for example, to independently re-sample the image content
and map it to new projection spaces according to the methods
described herein. Processed image content 1010b can then include
variously projected versions in the variety of new projection
spaces. Processors 1010a and/or additional distributed processors
1010c which may also work in parallel can downsample and/or encode
the image content to a variety of video sizes, bitrates, and
settings. The variously projected and variously encoded versions
1010d, including image content in a variety of projection spaces
and encoded at a variety of settings, may be further processed
and/or arranged to be stored in database 1015 as renditions
aggregated and made ready for playback on demand.
[0050] Referring now to FIG. 2, a schematic diagram 2000 including
an exemplary arrangement of functional components that can be used
according to aspects of the present disclosure is depicted. In
particular, the schematic diagram 2000 showing interaction between
the system components for carrying out the improved methods of
processing and transmission of image content. According to some
embodiments, variously projected and variously encoded versions of
the image content 2005 can, be saved in a database managed by the
provider. The image content 2005 can be transmitted via a network,
for example, a wireless network 2010, to an antenna 2015 capable of
transmitting requests for content, along with related information
for the request of specific renditions, and receiving the image
content. The antenna 2015 can be in communication with a central
processing unit 2025 that can receive the image content 2005 and
include or be in connection with a video decoder 2020 enabled to
process the image content and get pixel values. In alternative
embodiments, the content may be downloaded, for example, via a
direct connection, or the system may be fully integrated.
[0051] According to aspects of the disclosure, the central
processing unit can receive and process one or more input(s) 2030
including, for example, from an accelerometer, gyroscope,
crowdsource data, viewing parameters, or other sensor. Input(s)
2030 may come from sensors associated with the display device 2040
for the user 2045 or from stored parameters and preferences or data
relating to prior experiences from other users (i.e. crowdsourced
data). The display device 2040 can receive the data from a graphics
processing unit 2035 in, direct or indirect, communication with the
video encoder 2020.
[0052] Referring now to FIG. 3, a flowchart 100 illustrating method
steps that may be implemented for the processing and delivery of
virtual reality content according to some aspects of the disclosure
is shown. More particularly, the method steps which enable the use
of existing two-dimensional image compression methods for virtual
reality content, at least in part, by optimizing the conversion
from spherical virtual reality content to rectangular content so
that can be easily compressed through any two-dimensional encoding
methods, including but not limited to the JPEG or H.264. This
optimization may be done in conjunction with adaptive focus aspects
as taught by this disclosure. Adaptive focus as used herein can
refer to the optimized processing and/or transmission of a selected
different virtual reality image(s) that may at least partially
dependent on a viewer's direction. In some embodiments, the
selection of may additionally depend on a resolution distribution
based on the fovea field of view for the corresponding viewer's
direction, so that the resolution can be best for whichever
direction the viewer is looking.
[0053] By controlling the resolution distribution to match the view
direction, virtual reality content can be, transmitted in a manner
that is compressed to give the most immersive experience for a set
of view directions. This approach is contrary to current virtual
reality content processes that require virtual reality encoding be
performed for all of the same data, regardless of view direction,
thus increasing processing and transmission bandwidth requirements.
By overcoming this requirement, the method steps disclosed can
greatly reduce the required bandwidth for virtual reality content,
even while potentially boosting the observed resolution. Moreover,
the amount of image content that needs to be processed for display
is significantly reduced. By reducing this processing power
consumption can also be reduced, improving reliability of the
delivery and preventing overheating of the device.
[0054] Referring back to FIG. 3, the virtual reality content
delivery process disclosed in flowchart 100 can generally consist
of four main parts--projection generation 110, projection
conversion 120, encoding 130, processing, including accessing, or
transmission 135 and display 140. Beginning with projection
generation at 110, a mapping from a sphere to a flat rectangle can
be generated. This projection can be used to map the spherical
virtual reality content to a rectangular format, in part to enable
encoding with traditional algorithms such as JPEG or H.264.
Depending on the virtual reality content and/or the encoding
desired, at step 112, at least one existing projection may be
specified/selected to convert the 360.degree. three-dimensional
spherical content to two dimensional rectangular content. The
specified/selected projection must have a separable Jacobian.
[0055] At step 114, the two-dimensional projection(s)
specified/selected can be used to create a projection that is
defined by a desired resolution distribution. The desired
resolution distribution should describe a desired resolution
function than is separable when expressed in the projection space
specified/selected. With this a set of single-variable first-order
differential equations can be used to solve for the final
projection equation. If a solution exists, then the full projection
from sphere to new projection space can be described using the
solutions to the differential equations (further explained in
subsequent sections of this description).
[0056] In some embodiments, the desired resolution distribution can
refer to, a smooth two dimensional rectangular resolution
distribution that generally adjusts according one or more of the
fovea field of view for a particular viewing direction, contrast of
the virtual reality image data, movement of the virtual reality
image data content, and crowd sourced data. Accordingly, in some
embodiments, data from the captured 360.degree. spherical content
101, the viewing direction of the user 146, crowdsourcing 150,
user's feedback 144, may be transmitted (shown as (A), (C) and/or
(D)) to the system for use during the projection generation 110. In
accordance with aspects of the disclosure, the resolution
adjustments that define the projection can result at least in part
from the mathematical derivation process in the generation of the
new projection (the resolution-defined projection). A few examples,
including generation of the novel Gaussian and Cauchy projections
is disclosed with reference to FIGS. 2 and 3.
[0057] With approach, content mapped using this resolution-defined
projection can match the desired resolution function used to the
create the projection (i.e., by first selecting the total number of
pixels desired in the output, it is possible to then calculate the
pixel dimensions for any projection.) This additionally enabled
rounding up to the nearest multiple of a block size, typically 8 or
16 pixels a side, depending on the encoding settings and library.
Accordingly, as previously mentioned, this method could be used to
generate a resolution that matches the user's FOV, encoding
settings, and/or to highlight any region of the virtual reality
content, for example, to guide the attention of the user, or apply
warping effects to spherical content.
[0058] Projection conversion 120 can take place once a
resolution-defined projection has been created. During the
projection conversion 120 phase, the 360.degree. spherical virtual
reality content may be received at step 122. At step 124, the
360.degree. spherical virtual reality content can be applied to the
resolution-defined projection. In some embodiments, the virtual
reality content may then be stored in equirectangular format,
although the disclosed process would support any known format
regardless of the original content's projection. Taking the
equirectangular format, for example, a grid of pixel points
matching the dimensions of the output video may be used to map each
pixel to its coordinates in the equirectangular format. Lanczos
interpolation may be used to minimize projection conversion
artifacts. Accordingly, in some embodiments the coordinate mapping
can be defined by the projection equations and interpolation may be
defined by the Lanczos interpolation method. Alternatively, any
other image interpolation method could be used, including nearest
neighbor, bilinear, or bicubic. However, depending on the selected
method, it may be important to make sure to wrap the azimuth
dimension of the equirectangular image for interpolation so there
are no interpolation artifacts at the vertical edges in the new
projection space. Similar techniques that incorporate the inherent
spherical geometry of the content may be applied depending on the
original projection of the content.
[0059] This coordinate transformation and image interpolation can
be processor intensive. To account for this, in some embodiments,
the process may be parallelized, for example, by each color
dimension (RGB, YCbCr, or any other color space representation) so
that each color may be interpolated independently. Alternatively,
or in addition to, each frame of a video may also be independently
interpolated. In doing so, it can be important to ensure the
processing time to calculate coordinate mappings and interpolation
filter values runs only once for each video with specific settings.
This may be done by performing the calculation and programming the
specific settings before the first frame is processed, such that
all frames simply lookup the coordinate mapping and filter values.
In yet additional embodiments, GPU acceleration of this process may
also be implemented.
[0060] According to aspects of the disclosure, the interpolation
and method steps can enable virtual reality content generation
taking form as a plurality of renditions that can have much higher
resolutions on a subset of the original sphere, while still serving
data to cover the entire or a greater portion of the sphere. This
is different than the less sophisticated method currently known
which simply crops the spherical content to only display a subset
of the sphere, leaving the majority of the sphere content-less. For
example, it can be advantageous to serve content to cover the
entire or a greater portion of the sphere because the user could
turn her/his head quicker than the latency of displaying a new
video. By varying the resolution distribution to enable larger
content delivery without increasing the bandwidth required, if this
happens, because the disclosed method still serves content
everywhere almost everywhere on the sphere (although low
resolution), the user can maintain some level of immersion.
[0061] Encoding 130 of the plurality of renditions can then take
place. Encoding 130 can include encoding two or more renditions at
least one resolution and bitrate 132. The renditions of virtual
reality content may be encoded, for example, using a H.264 at a
bitrate that matches the pixel dimensions. This too may be
performed in a distributed computing setup to process a single
virtual reality content video input. For example, for each
rendition in terms of bitrate, dimensions, and view direction, a
different processor may speed up the task of encoding many video
renditions. Since the costliest operation can often be the
interpolation needed for projection conversion, in parallel a
downscaled version of the original input virtual reality content
may be generated and made available so a version is quickly
available for the uploader. The rest of the virtual reality content
may go through conversion to a variety of projections, projection
parameters, dimensions, bitrates, and view directions. For any
versions of the virtual reality content that is the same
projection, projection parameters, and view direction, the larger
of the versions by the standard process may be generated. It is the
possible to downsize and reduce the bitrate to generate any smaller
versions of the same video. This can greatly reduce computation
time for these smaller version as compared with generating them
directly from the original equirectangular content. In embodiments
in which the processes are distributed, notification flags can be
sent to a database when their individual task is complete so that a
centralized record of the status of the conversions is
available.
[0062] The virtual reality content renditions can be grouped in
sets according to one or more predetermined common properties
between them. For example, the renditions can be grouped in sets of
versions in which only view direction changes so that a set can
include all of the versions of the same virtual reality content
that can be served to a user depending on their view direction.
Compression of the renditions 134 may take place as part of the
encoding in 132 or separately as it may be desired.
[0063] Each of the virtual reality content sets may be split using
a video display frequency (e.g. virtual reality image content
included in a short duration virtual reality video). In some
embodiments, the sets of virtual reality content may be organized
in a standard structure folder structure database(s) that can allow
the displaying device to easily request a different set depending
on the view direction of the user detected at step 146. Head motion
recorded by the virtual reality content display device and/or
determined using data from the viewing direction detected, head
motion can be used to dynamically select at least one of the
projection versions, individually or as a data set, according to
the head motion at step 136. This dynamic selection may be done,
for example, similarly to the adaptive streaming methods in which
the bitrate of the video displayed adaptively matches available
bandwidth. However, instead of only including video quality,
viewing direction can additionally be included for the dynamic
selection.
[0064] In the display 140 phase, the dynamically selected virtual
reality content is delivered 142 to a virtual reality device of the
user. Virtual reality devices can include all head-mounted devices
(HMD) for virtual reality that include head tracking, and even
mobile devices which can use internal accelerometers to achieve
head tracking or immersive displays of PCs, tablets, smart TVs and
the such. Virtual reality devices can include but are not limited
to the Samsung GearVR.TM. system or any Google Cardboard.TM. setup.
The delivery can take into account specifications of the viewing
device, the data connection, and the view direction, to provide
content that achieves the best visible resolution for the user. In
some embodiments with integrated components, the delivery can
simply refer to the processing and selection of the specific image
content for display within the device. In addition, in accordance
with the disclosure, resolution optimization can also be a result
of the dynamic selection of pre-encoded array of videos at various
sizes, at various bitrates, at various rotations, and using
different resolution-defined projections that take into account the
small field of view of humans (i.e., fovea field of view). The
fovea field of view being relatively minor in comparison to the
full sphere of virtual reality content to allow higher resolution
where according to the eye's resolution and is also in line by the
constraints of virtual reality display devices. E.g., The Oculus
Rift.TM. has a 110.degree. field of view while the Samsung
GearVR.TM. has a 96.degree. field of view. Google Cardboard.TM.
setups have a range of field of views, but all being less than
96.degree.. The field of view constraints mean that a user will, at
any single instant, only experience resolution in a small subset of
the full sphere of content. Their eye resolution means any user's
limited ability to only appreciate extremely high resolution in
his/her direction of focus.
[0065] Depending on the virtual reality content and application, an
infinite number of video versions could be generated to match every
possible viewing direction of the user where limited virtual
reality content is at issue. Having the different versions, at
almost any instant, the video version could allow for it to be
updated to exactly match the user's field of view. i.e., taking the
user's fovea field of view resolution sensitivity and the
device-limited field of view to generate a desired resolution
profile for the virtual reality content transmitted to the virtual
reality device to achieve minimum bandwidth requirements while
still giving the viewer a perfect viewing experience. In some
embodiments, in addition or alternatively to the viewing direction
and fovea field of view, feedback data during the display of
content, including, for example, resulting data from eye movement
tracking, may be recorded at 144 for future processing and/or set
up of user preferences. Eye movement may be tracked to signal
attention diversion, discomfort, moving patterns and the such which
additionally may be used for vision condition diagnostics or
therapy treatments. For example, the feedback may be used for the
early detection of color blindness or abnormal sensitivity to
flashing light patterns, seasonal affective disorder therapy and
similar related diagnostic therapeutic applications. Similarly,
pre-recorded and/or transmitted crowdsourced data 150 may be used
to define projections, conditions, and/or parameters relating to
the delivery of the specific virtual reality content for the
implementation.
[0066] In larger virtual reality content applications, the
resolution function can be a smoothed version of the user's exact
viewing resolution distribution in order to account for deviations,
including, for example, the fact that a user could move his eyes
before the new rendition may be delivered. It can also allow for a
narrower number of view-direction renditions as every additional
view-direction rendition requires processing time to encode and
storage space, so may be incentives to reduce the total number
depending on the amount of content and application.
[0067] The aforementioned aspects may function via data
input/output by the server and virtual reality device of the user.
On the server side, processing various combinations of video
parameters, and joining videos that may only differ in view
direction in to an adaptive streaming playlist, split into
small-duration chunks, can be achieved for display 140. On the
device side, the viewing direction can be determined for a specific
rendition request. Further, an estimate the bandwidth of the
connection between the two can be determined using the limits
imposed by processing power on the size of the video that the
device can decode and a request for content that may be mapped to a
specific rendition of the video. Accordingly, all or at least some
of these different possible videos can be requested adaptively,
with the new video arriving within one to two chunks from
request.
[0068] A new video chunk may be indicated by the inclusion of a
keyframe at the beginning of every chunk. The virtual reality
device can be programmed, or run an application that is capable of
determining which frame starts a chunk from a new video parameter
set because of a flag included with this keyframe. Upon seeing a
keyframe flag, the video can be reverse-projected using the latest
parameters used to request this newest chunk.
[0069] At the virtual reality device, once the requested video
chunk arrives and is decoded, it can be ready to be displayed as a
reverse-projection. The same method as projection can be used by
creating a distribution of points in the final mapping space, usual
a three-dimensional sphere, mapping each pixel to its corresponding
coordinates in projection space, and interpolating the pixels
values near the target coordinates. The video may then be in a
standard format as a texture to be applied to a three-dimensional
shape in virtual reality space with higher resolutions in the areas
we would like.
[0070] In some embodiments, at least some aspects could be used on
350.degree. panoramas or any 360.degree. virtual reality content
device that would benefit from having a projection generation
process for customizable resolution profiles that allows us to
create virtual reality content renditions that are optimized for a
range of bandwidths, devices, and/or view directions. More
specifically, a projection generation process that can work both on
the fly, for example, to adjust to bandwidth and device parameters,
independent of viewing direction and/or according to the desired
resolution based on the viewing direction.
[0071] Referring now to FIGS. 4 and 5, a flowchart 210 and
corresponding schematic representation 310 illustrating the method
steps that may be implemented to generate a resolution-defined
projection according to some aspects of the disclosure are
depicted. Both FIGS. 4 and 5 will reference common numeral
identifiers as appropriate to help illustrate the relationship
between method steps in the generating a resolution-defined
projection process. Whilst specific mathematical equations and
additional steps are, disclosed to better enable and explain the
different novel aspects of the disclosure, the scope of those
aspects is to be defined by the claim limitations as interpreted in
view of the more simplified steps in flowchart 210 or their
corresponding equivalents.
[0072] Beginning at step 111, one or more of the fovea field of
view for a particular viewing direction, contrast of the virtual
reality image data, movement of the virtual reality image data
content, camera arrangement, camera lenses, and crowd sourced data
will inform the specification of a desired resolution distribution.
At 115, an existing bijective projection from a sphere to a
rectangle having a separable Jacobian, such as the equirectangular
projection, may be selected to function as an intermediary
projection. At 113, a desired resolution distribution informed by
one or more items in 111 that is separable in the existing
projection space of 115 is specified. Using the existing projection
as an intermediary projection and the separable resolution
distribution, at 117, a resolution-defined projection may be
generated for the content to be mapped. Moreover, at 119, the
resolution-defined projection and any associated parameters
generated can also be used for encoding and/or compression.
[0073] The following sections elaborate on the mathematical aspects
associated with the aforementioned steps used to generate
resolution-defined projections.
1. Projections
[0074] The exemplary projections considered map 3D Cartesian
coordinates, (x, y, z), to a new coordinate space, (u, v, w).
P.epsilon..sup.1.fwdarw..sup.3 is set to be any such projection and
p.sub.(.cndot.).epsilon..sup.3.fwdarw..sup.1| is the associated
projections for the individual coordinates u, v and w.
P(x,y,z)=|(u,v,w)=(p.sub.u(x,y,z),p.sub.v(x,y,z),p.sub.w(x,y,z))
(1)
[0075] As shown below, for most virtual reality content
applications, the projection can be invertible, with a defined
P.sup.-1.epsilon..sup.3.fwdarw..sup.3 and associated
p.sub.(.cndot.).sup.-1.epsilon..sup.3.fwdarw..sup.1.
(x,y,z)=P.sup.-1(u,v,w)=(p.sub.x.sup.-1(u,v,w),p.sub.y.sup.-1(u,v,w),p.s-
ub.z.sup.-1(u,v,w)) (2)
[0076] The projections considered map the unit sphere (u.s.) in
Cartesian space to a rectangle spanning only u and v in projection
space, with w=w0. This region may be referred to in projection
space as the unit-sphere rectangle (u.s.r.).
P(x,y,z)|.sub.(x,y,z).epsilon.u.s.=(u,v,w.sub.0)=(u,v)|.sub.w.sub.0
(3)
[0077] Given that the value of w0 may be calculated for each
projection, the implied |w0 notation may be omitted hereon.
P.sup.-1(u,v).ident.P.sup.-1(u,v)|.sub.w.sub.0=P.sup.-1(u,v,w.sub.0)
(4)
[0078] 2.1 Surface Integration
[0079] u.epsilon.[u.sub.a,u.sub.b), v.epsilon.[v.sub.a,v.sub.b) and
w=w.sub.0 can define the unit-sphere rectangle in projection space.
|A.sub.P| may be the area of the unit-sphere rectangle in
projection space and .SIGMA..sub.P can be a surface area element in
projection space.
.intg. .intg. u . s . r . d .SIGMA. P = .intg. v a v b .intg. u a u
b dudv = ( v b - v a ) ( u b - u a ) = A P ( 5 ) ##EQU00001##
[0080] To differentiate the coordinate bases, the subscript .sub.C
can refers to a value in Cartesian coordinates while the subscript
.sub.p refers to a value in projection coordinates. The area of the
unit sphere in Cartesian space can be known. A.sub.C can be the
area of the unit sphere in Cartesian space and let .SIGMA..sub.C be
a surface area element in Cartesian space.
.intg. .intg. u . s . d .SIGMA. C = 4 .pi. = A C ( 6 )
##EQU00002##
[0081] To calculate surface integrals across coordinate spaces, the
Jacobian of a projection is accounted for. The Jacobian can be a
function of the inverse individual coordinate projection equations
(i.e. p.sub.(.cndot.).sup.-1).
J ( u , v ) = [ .differential. p x - 1 .differential. u
.differential. p x - 1 .differential. v .differential. p x - 1
.differential. w .differential. p y - 1 .differential. u
.differential. p y - 1 .differential. v .differential. p y - 1
.differential. w .differential. p z - 1 .differential. u
.differential. p z - 1 .differential. v .differential. p z - 1
.differential. w ] ( u , v , w 0 ) . ( 7 ) ##EQU00003##
[0082] Constraints can be defined on the projections P that may be
considered valid, and the associated functions P.sup.-1,
p(.cndot.), p.sup.-1, and J, all of which may be derived from any
valid P. The surface integral of a function in Cartesian space can
be evaluated in projection space by treating f.sub.C(x, y, z) as
any function in Cartesian space.
.intg..intg.f.sub.C(x,y,z)d.SIGMA..sub.C=.intg..intg.f.sub.C(P.sup.-1(u,-
v)).parallel.J(u,v).parallel.d.SIGMA..sub.P (8)
[0083] By integrating over the unit sphere in Cartesian space, more
explicit parameters can be defined for the integration since the
integration may be done over the unit-sphere rectangle in
projection space.
.intg. .intg. u . s . f C ( x , y , z ) d .SIGMA. C = .intg. .intg.
u . s . r . f C ( P - 1 ( u , v ) ) J ( u , v ) d .SIGMA. P =
.intg. v a v b .intg. u a u b f C ( P - 1 ( u , v ) ) J ( u , v )
dudv ( 9 ) ##EQU00004##
[0084] Accordingly, the Jacobian of a projection can define the
warping of space due to a projection. This in essence may be the
same or significantly similar to the warping of space which can
affect the resolution of the spherical output content.
2. Resolution Functions
[0085] Spatial resolution r can be defined as the local density of
independent pixels, or the local ratio of independent pixels
.differential.N to the area they occupy .differential..SIGMA.. This
can be the information used as the density of an image. Spatial
resolution is not necessarily constant over an image.
r ( u , v ) .ident. .differential. N .differential. .SIGMA. ( u , v
) ( 10 ) ##EQU00005##
[0086] Pixel density .rho. can be defined as the ratio of discrete
pixels M to the area they occupy.sup.|A|. These discrete pixels are
not necessarily independent, as it is in the case of a blurred or
upsampled image. Pixel density can be treated as being constant
over an image. Even when considering the theoretical pixels over a
3D sphere of content, the pixels can all be significantly
identical/identical, as it may be from a spherical tessellation of
triangular pixels to form an icosahedron.
.rho. .ident. M A ( 11 ) ##EQU00006##
[0087] Pixel density .rho. and spatial resolution r may not
necessary be equivalent as they may only be the same when each
pixel is independent from all other pixels. Spatial resolution
cannot exceed pixel density, but pixel density could be higher than
spatial resolution, as it may be the case with a blurred or
upsampled image.
r.ltoreq..rho. (12)
[0088] .sub.ir.sub.C, .sub.i.SIGMA..sub.C, .sub.iN.sub.C, and
|.sub.iA.sub.C| can be the spatial resolution, pixel density,
independent pixel count, and total area of the spherical input
content, respectively. It may be assumed that the content could
have been generated (via cameras or computers) in a way such that
the spatial resolution can be constant and equal to the pixel
density. The spherical input data may cover a unit sphere, so the
total area of the content .sub.iA.sub.C can be taken as 4.pi..
r C i = .rho. C i = N C i A C i = N C i 4 .pi. ( 13 )
##EQU00007##
[0089] The resolution of the mapped data r.sub.P can be the same as
the pixel density .rho.P if we limit the pixel count N.sub.P such
that no oversampling of the spherical input content occurs. This
may be a reasonable assumption since the pixel count of the mapped
content can typically be much lower than the largest available
version of the spherical input content.
r P = .rho. P = N P A P ( 14 ) ##EQU00008##
[0090] The spatial resolution of the spherical output content
r.sub.C may be constrained by the mapped content resolution. The
pixel density of the spherical output data .rho..sub.C can be
determined by the field of view and screen size of the display
device. Considering current virtual reality headset resolutions and
mobile network bandwidths, the pixel density can typically be much
higher than the spatial resolution, which is often bandwidth
constrained. This can be the first case where the pixel density can
be higher than the spatial resolution, due to oversampling when
inverse projecting the mapped content to spherical output
content.
r C ( x , y , z ) = .differential. N C .differential. .SIGMA. C ( x
, y , z ) .ltoreq. .rho. C = M A C = M 4 .pi. ( 15 )
##EQU00009##
[0091] The spatial resolution may not be uniform due to the warping
of the uniform pixels of the mapped content during reverse
projection. This warping can be exactly captured by the magnitude
of the determinant of the Jacobian. We include a normalizing term
.alpha. so the integral of spatial resolution can be the total
information content, which may be equal to the total number of
mapped content pixels N.sub.P.
r C ( x , y , z ) = .differential. N C .differential. .SIGMA. C ( x
, y , z ) = N P .alpha. 1 J ( P ( x , y , z ) ) ( 16 )
##EQU00010##
[0092] Using Equations 5 and 9, we can solve for .alpha..
.alpha. = .intg. .intg. u . s . 1 J ( P ( x , y , z ) ) d .SIGMA. C
= .intg. .intg. u . s . r . 1 J ( P ( P - 1 ( u , v ) ) ) J ( u , v
) d .SIGMA. P = .intg. v a v b .intg. u a u b J ( u , v ) J ( u , v
) dudv = A P ( 17 ) ##EQU00011##
[0093] We see that the normalizing factor .alpha. for any
projection can be equal to the area of the unit-sphere rectangle in
projection space. This thus provides an equation that explicitly
relates a projection to the resulting spatial resolution of the
final virtual reality content.
r C ( x , y , z ) = N P A P 1 J ( P ( x , y , z ) ) ( 18 )
##EQU00012##
[0094] 3.1 Equirectangular Resolution Function
[0095] The Equirectangular projection can be a projection from the
unit sphere to a rectangle in latitutde-longtitude space. This may
be the spherical coordinate representation of a sphere, with
longtitude u.epsilon.[-.pi., .pi.), latitude v.epsilon.[-.pi./2,
.pi./2], and radius w(w.sub.0=1).
[0096] The Equirectangular projection can be the current standard
for virtual reality content delivery and storage.
p x - 1 ( u , v , w ) = w cos ( v ) cos ( u ) ( 19 ) p y - 1 ( u ,
v , w ) = w cos ( v ) sin ( u ) ( 20 ) p z - 1 ( u , v , w ) = w
sin ( v ) ( 21 ) p u ( x , y , z ) = tan - 1 ( y x ) ( 22 ) p v ( x
, y , z ) = sin - 1 ( z x 2 + y 2 + z 2 ) ( 23 ) p w ( x , y , z )
= x 2 + y 2 + z 2 ( 24 ) J ( u , v , w ) = [ .differential. p x - 1
.differential. u .differential. p x - 1 .differential. v
.differential. p x - 1 .differential. w .differential. p y - 1
.differential. u .differential. p y - 1 .differential. v
.differential. p y - 1 .differential. w .differential. p z - 1
.differential. u .differential. p z - 1 .differential. v
.differential. p z - 1 .differential. w ] ( u , v , w ) = [ - w cos
( v ) sin ( u ) - w sin ( v ) cos ( u ) cos ( v ) sin ( u ) w cos (
v ) cos ( u ) - w sin ( v ) sin ( u ) cos ( v ) sin ( u ) 0 w cos (
v ) sin ( v ) ] ( 25 ) c J ( u , v , w ) = w 2 cos ( v ) ( 26 )
##EQU00013##
[0097] Because there is transformation of points from the unit
sphere that can be taking place, {square root over
(x.sup.2+y.sup.2+z.sup.2)}=1 and w.sub.0=1.
.parallel.J(u,v).parallel.=cos(v) (27)
|A.sub.P|=|(v.sub.b-v.sub.a)(u.sub.b-u.sub.a)|=2.pi..sup.2 (28)
[0098] Equation 18 may be used to solve for the resolution function
resulting from the Equirectangular projection.
r C ( x , y , z ) = N P A P 1 J ( P ( x , y , z ) ) = N P 2 .pi. 2
cos ( sin - 1 ( z ) ) = N P 2 .pi. 2 1 - z 2 ( 29 )
##EQU00014##
[0099] The spatial resolution of virtual reality content delivered
via the Equirectangular projection can actually be at a minimum
along the equator (z2=0), and approaches .infin. at the poles
(z2=1). Since most virtual reality content often includes important
content along the equator, this may be nearly the opposite of what
is desired. This thus can demonstrate how the projection chosen can
be used to determine the shape of the spatial resolution of the
final content--through the projection's Jacobian. Following is an
exemplary demonstration on how to solve for a Jacobian given a
target resolution function. By implementing this, a specified
appropriate resolution function can be used to derive an
application-specific projection according to the various aspects of
the disclosure.
4. Resolution-Defined Projections
[0100] Given exemplary target resolution function rC (x, y, z),
solving for the corresponding resolution-defined projection P is
desired. If a solution exists, using this projection can lead to a
spherical output content resolution that may be identical to our
target resolution function.
r C ( x , y , z ) = N P A P 1 J ( P ( x , y , z ) ) ( 30 )
##EQU00015##
[0101] Again, Np can be the number of pixels in the unit-sphere
rectangle in the resolution-defined projection space and |A.sub.p|
can be the area of the unit-sphere rectangle in the
resolution-defined projection space. J is the Jacobian for the
resolution-defined projection P. By rearranging the terms to have
an explicit equation for J the target resolution function r.sub.C
can be defined.
J ( u , v ) = N P A P 1 r C ( P - 1 ( u , v ) ) ( 31 )
##EQU00016##
[0102] 4.1 Hybrid Projections
[0103] J includes nine degrees of freedom, so solving Equation 31
for all the elements may be open ended. In order to reduce the
degrees of freedom, a new constraint can be introduced. The
constraint can be an existing projection from the unit sphere to a
rectangle, and to which the projections can be applied to warp the
rectangle. For purposes of this disclosure this may be referred to
as an intermediate projection {tilde over (P)}. This projection can
map from the 3D unit sphere to a rectangle spanning (a, b) in (a,
b, c) coordinate space. The next projection can be the first
coordinate remapping {hacek over (P)}, which may warp the rectangle
along the a axis, preserving the b and c coordinates. This first
coordinate remapping can map from (a, b, c) coordinates to (d, e,
f) coordinates. The final projection can be the second coordinate
remapping {circumflex over (P)}, which then may warp the rectangle
in (d, e, f) space along the e-axis. This second coordinate
remapping can map from (d, e, f) space to (u, v, w) space. The
combination of these three projections may be referred to as a
hybrid projection P.
P(x,y,z)={circumflex over (P)}({hacek over (P)}({tilde over
(P)}(x,y,z))))=(u,v,w) (32)
[0104] The Jacobian of the hybrid projection J can simply be a
product of its component projections. The distributive property of
the determinant and then the constraints can also be applied to
{hacek over (P)} and {circumflex over (P)}.
J _ ( u , v , w ) = J ^ ( u , v , w ) J ( P ^ - 1 ( u , v , w ) ) J
~ ( P - 1 ( P ^ - 1 ( u , v , w ) ) ) = J ^ ( u , v , w ) J ( u , P
^ e - 1 ( v ) , w ) ) J ~ ( P d - 1 ( u ) , P ^ e - 1 ( v ) , w ) (
33 ) ##EQU00017##
[0105] Solving for {tilde over (J)}(a, b, c) may be possible using
the intermediate projection. The remapping Jacobians may also have
only one variable element in each.
J ( d , e , f ) = det [ .differential. p a - 1 .differential. d ( d
) 0 0 0 1 0 0 0 1 ] = .differential. p a - 1 .differential. d ( d )
( 34 ) J ^ ( u , v , w ) = det [ 1 0 0 0 .differential. p ^ a - 1
.differential. v ( v ) 0 0 0 1 ] = .differential. p ^ e - 1
.differential. v ( v ) ( 35 ) J _ ( u , v , w ) = .differential. p
^ e - 1 .differential. v ( v ) .differential. p a - 1
.differential. d ( u ) J ~ ( p a - 1 ( u ) , p ^ e - 1 ( v ) , w )
( 36 ) ##EQU00018##
[0106] Combining equations 31 and 36, a multivariable first-order
differential equation can be generated to solve in order to derive
the resolution-defined hybrid projection P.
.differential. p ^ d - 1 .differential. v ( v ) .differential. p a
- 1 .differential. d ( u ) J ~ ( p d - 1 ( u ) , p ^ e - 1 ( v ) ,
w ) = N P _ A P _ 1 r C ( P ~ - 1 ( p d - 1 ( u ) , p ^ e - 1 ( v )
, w ) ) ( 37 ) ##EQU00019##
[0107] Two additional constraints may be applied. By restricting
the intermediate projection to have a separable Jacobian and our
target resolution function to be separable in (a, b, c)
intermediate projection space, then all of Equation 37 can be
separable.
J ~ ( a , b , c ) = a J ~ ( a ) b J ~ ( b ) c J ~ ( c ) ( 38 ) r C
( x , y , z ) = r C a ( p ~ a ( x , y , z ) ) b r C ( p ~ b ( x , y
, z ) ) c r C ( p ~ c ( x , y , z ) ) ( 39 ) r C ( P ~ - 1 ( a , b
, c ) ) = r C a ( a ) b r C ( b ) c r C ( c ) ( 40 ) .differential.
p ^ e - 1 .differential. v ( v ) .differential. p a - 1
.differential. d ( u ) a J ~ ( p a - 1 ( u ) ) b J ~ ( p ^ e - 1 (
v ) ) c J ~ ( w ) = N P ~ A P 1 r C a ( p ^ a - 1 ( u ) ) b r C ( p
^ e - 1 ( v ) ) c r C ( w ) ( 41 ) ##EQU00020##
[0108] Separate differential equations for {hacek over
(p)}.sub.a.sup.-1 and {circumflex over (p)}.sub.e.sup.-1.
.differential. p a - 1 .differential. d ( u ) .varies. 1 a J ~ ( p
a - 1 ( u ) ) r C a ( p a - 1 ( u ) ) ( 42 ) .differential. p ^ e -
1 .differential. v ( v ) .varies. 1 b J ~ ( p ^ e - 1 ( v ) ) r C b
( p ^ e - 1 ( v ) ) ( 43 ) ##EQU00021##
[0109] The absolute value can be dropped since
.parallel.J.parallel. and r are positive values. Dropping the
absolute values can create an even stricter equation, but the
differential equations can result in a solvable form with the
following solution.
.differential. f .differential. x ( x ) .varies. 1 g ( f ( x ) ) h
( f ( x ) ) ( 44 ) f ( x ) = y s . t . x = .intg. g ( .gamma. ) h (
.gamma. ) d .gamma. .gamma. = y ( 45 ) ##EQU00022##
[0110] Using these equations, a solution for {hacek over
(p)}.sub.a.sup.-1 and {circumflex over (p)}.sub.e.sup.-1. {hacek
over (p)}.sub.a.sup.-1 that fully defines {hacek over (P)} and
{circumflex over (p)}.sub.e.sup.-1 fully defines {circumflex over
(P)}. may be obtained since an intermediate projection {tilde over
(P)} can be chosen (so long as it has the requirements above are
satisfied), all three projections can comprise the final hybrid
projection P.
[0111] 4.2 Equirectangular Intermediate Projection
[0112] The equirectangular projection can satisfy the requirements
of an intermediate projection above. Equation 27 shows that the
Equirectangular Jacobian can be separable.
{tilde over (J)}(a,b,c)=cos(b) (46)
.sub.a{tilde over (J)}(a)=1 (47)
.sub.b{tilde over (J)}(b)=cos(b) (48)
.sub.c{tilde over (J)}(c)=1 (49)
5. Examples
[0113] In the following examples, it is demonstrated how a hybrid
projection from a target resolution function can be derived.
[0114] 5.1 Uniform Resolution Function
[0115] In a uniform resolution function all directions in virtual
reality would be of equal visual quality. Because the
equirectangular projection can be used as an intermediate
projection, by employing equations 19-28 and 46-49, for example,
the target uniform resolution could be any positive value k.
r.sub.C(x,y,z)=k=r.sub.C({tilde over (P)}.sup.-1(a,b,c)) (50)
[0116] Using Equation 41.
.differential. p ^ e - 1 .differential. v ( v ) .differential. p a
- 1 .differential. d ( u ) cos ( p ^ e - 1 ( v ) ) = N P A P 1 k (
51 ) .differential. p a - 1 .differential. d ( u ) .varies. 1 ( 52
) .differential. p ^ e - 1 .differential. v ( v ) .varies. 1 cos (
p ^ e - 1 ( v ) ) ( 53 ) p a - 1 ( u ) = u ( 54 ) p ^ e - 1 ( v ) =
sin - 1 ( v ) ( 55 ) p a - 1 ( a ) = a ( 56 ) p ^ v ( e ) = sin ( e
) ( 57 ) ##EQU00023##
[0117] Combining the first and second coordinate remappings to the
Equirectangular intermediate projections can be used to solve for
the final hybrid projection of the unit sphere to a unit-sphere
rectangle spanned by (u, v).
p _ u ( x , y , z ) = tan - 1 ( y x ) ( 58 ) p _ v ( x , y , z ) =
sin ( sin - 1 ( z x 2 + y 2 + z 2 ) ) = z x 2 + y 2 + z 2 ( 59 ) p
_ w ( x , y , z ) = x 2 + y 2 + z 2 ( 60 ) ##EQU00024##
[0118] For the unit sphere w.sub.0= {square root over
(x.sup.2+y.sup.2+z.sup.2)}=1, and a simple mapping from 3D space to
2D space can be solved for.
p ~ u ( x , y , z ) = tan - 1 ( y x ) ( 61 ) p _ v ( x , y , z ) =
z ( 62 ) ##EQU00025##
[0119] The Lambert Cylindrical Equal-Area projection derived may be
used to solve for the number of pixels in projection space N.sub.p
in order to exactly match the target resolution function.
a .di-elect cons. [ - .pi. , .pi. ) ( 63 ) b .di-elect cons. [ -
.pi. 2 , .pi. 2 ] ( 64 ) u .di-elect cons. p u ( [ - .pi. , .pi. )
) = [ - .pi. , .pi. ) ( 65 ) v .di-elect cons. p ^ v ( [ - .pi. 2 ,
.pi. 2 ] ) = [ - 1 , 1 ] ( 66 ) A P _ = ( .pi. + .pi. ) ( 1 + 1 ) =
2 .pi. 2 ( 67 ) N P _ = 1 cos ( sin - 1 ( v ) ) cos ( sin - 1 ( v )
) k A P _ = 2 .pi. 2 k ( 68 ) ##EQU00026##
[0120] Therefore, if a uniform spatial resolution of k of the final
spherical output content was desired, the Lambert projection with a
total of 2.pi..sup.2k pixels.
[0121] 5.3 Gaussian Resolution Function
[0122] To concentrate the resolution in a specific direction, a
bivariate gaussian shape for the resolution function may be
desired. The resolution can then fall off smoothly in azimuth and
elevation with standard deviations .sigma.a and .sigma.b,
respectively and the equirectangular projection may again be used
as an intermediate projection.
r C ( P ~ - 1 ( a , b , c ) ) = k exp ( - a 2 2 .sigma. a 2 ) exp (
- b 2 2 .sigma. b 2 ) ( 69 ) ##EQU00027##
[0123] When unable to solve for the desired resolution function, as
presented in the preceding example, with the addition of a
complicated term, however, a better solution may result. as the
extra term can increase resolutions near the poles, but the result
may still be preferable to the Equirectangular projection.
r C ( P ~ - 1 ( a , b , c ) ) = k exp ( - a 2 2 .sigma. a 2 ) exp (
- b 2 2 .sigma. b 2 ) cos ( .pi. 2 .sigma. b erf ( b 2 .sigma. b )
) cos ( b ) ( 70 ) .differential. p a - 1 .differential. d ( u )
.varies. exp ( ( p a - 1 ( u ) ) 2 2 .sigma. a 2 ) ( 71 )
.differential. p - 1 .differential. v ( v ) .varies. exp ( exp ( p
^ a - 1 ( v ) ) 2 2 .sigma. b 2 ) cos ( .pi. 2 .sigma. b erf ( p ^
e - 1 ( v ) 2 .sigma.0 b ) ) ( 72 ) p a - 1 ( u ) = 2 .sigma. a erf
- 1 ( 2 u .pi. .sigma. a ) ( 73 ) p ~ - 1 ( v ) = 2 .sigma. b erf -
1 ( 2 sin - 1 ( v ) .pi. .sigma. b ) ( 74 ) p ~ u ( a ) = .pi.
.sigma. a 2 erf ( a 2 .sigma. a ) ( 75 ) p ^ v ( e ) = sin ( .pi.
.sigma. b 2 erf ( e 2 .sigma. b ) ) ( 76 ) p _ u ( x , y , z ) =
.pi. .sigma. a 2 erf ( tan - 1 ( y x ) 2 .sigma. a ) ( 77 ) p v _ (
x , y , z ) = sin ( .pi. .sigma. b 2 erf ( sin - 1 ( z x 2 + y 2 +
z 2 ) 2 .sigma. b ) ) ( 78 ) p _ w ( x , y , z ) = x 2 + y 2 + z 2
( 79 ) ##EQU00028##
[0124] For the unit sphere w.sub.0= {square root over
(x.sup.2+y.sup.2+z.sup.2)}=1, which may result in the final mapping
from (x, y, z) to our unit-sphere rectangle in (u, v) space.
p _ u ( x , y , z ) = .pi. .sigma. a 2 erf ( tan - 1 ( y x ) 2
.sigma. a ) ( 80 ) p _ v ( x , y , z ) = sin ( .pi. .sigma. b 2 erf
( sin - 1 ( z ) 2 .sigma. b ) ) ( 81 ) ##EQU00029##
[0125] Again solving for the number of pixels in projection space
N.sub.p may match any value of k in the original resolution
function.
a .di-elect cons. [ - .pi. , .pi. ) ( 82 ) b .di-elect cons. [ -
.pi. 2 , .pi. 2 ] ( 83 ) u .di-elect cons. p u ( [ - .pi. , .pi. )
) = [ .pi. .sigma. a 2 erf ( - .pi. 2 .sigma. a ) , .pi. .sigma. a
2 erf ( .pi. 2 .sigma. a ) ) ( 84 ) v .di-elect cons. p ^ v ( ( -
.pi. 2 , .pi. 2 ] ) = [ sin .pi. .sigma. b 2 erf ( - .pi. 2 2
.sigma. b ) ) , sin ( .pi. .sigma. b 2 erf ( .pi. 2 2 .sigma. b ) )
] ( 85 ) ##EQU00030##
[0126] The fact that sin and erf are both odd functions can be used
to simplify the calculation of the total area.
A P _ = 2 ( .pi. .sigma. a 2 erf ( .pi. 2 .sigma. a ) ) 2 ( sin (
.pi. .sigma. b 2 erf ( .pi. 2 2 .sigma. b ) ) ) = 2 2 .pi. .sigma.
a erf ( .pi. 2 .sigma. a ) sin ( .pi. .sigma. b 2 erf ( .pi. 2 2
.sigma. b ) ) ( 86 ) ##EQU00031##
[0127] Just like the uniform resolution projection aforementioned,
all the terms except k and |AP.sup.-| can cancel in the equation
for the number of pixels N.sub.p.
N P ~ = k A P ~ = k 2 2 .pi. .sigma. a erf ( .pi. 2 .sigma. a ) sin
( .pi. .sigma. b 2 erf ( .pi. 2 2 .sigma. b ) ) ( 87 )
##EQU00032##
[0128] Given in equation 70 the resolution at (a, b)=(0, 0)
(forward, center), can be equal to k, scaling of the desired
forward resolution by |A.sub.P| may be used to calculate the
required number of pixels.
[0129] 5.2 Cauchy Resolution Function
[0130] The final projection equations for the gaussian resolution
function can be fairly computationally expensive. In addition, it
may also be useful to use a function with fatter tails, so the
resolution decrease isn't quite so dramatic. Following is an
example on how the Cauchy function can be used to solve these
issues.
r C ( P ~ - 1 ( a , b , c ) ) = k 1 ( 1 + ( a .gamma. a ) 2 ) 1 ( 1
+ ( b .gamma. b ) 2 ) ( 88 ) ##EQU00033##
[0131] Whilst it may not be possible to solve for the elevational
Cauchy function exactly, with the inclusion of an extra term, it
can be possible to approximate the Cauchy function and find a valid
solution projection.
r C ( P ^ - 1 ( a , b , c ) ) = k 1 ( 1 + ( a .gamma. a ) 2 ) 1 ( 1
+ ( b .gamma. b ) 2 ) cos ( .gamma. b tan - 1 ( b .gamma. b ) ) cos
( b ) ( 89 ) .differential. p ~ a - 1 .differential. d ( u )
.varies. 1 + ( p ~ a - 1 ( u ) .gamma. a ) 2 ( 90 ) .differential.
p ^ e - 1 .differential. v ( v ) .varies. 1 + ( p ^ x - 1 ( x )
.gamma. b ) 2 cos ( .gamma. b tan - 1 ( p ^ x - 1 ( v ) .gamma. b )
) ( 91 ) p ~ a - 1 ( u ) = .gamma. a tan ( u .gamma. a ) ( 92 ) p ^
e - 1 ( v ) = .gamma. b tan ( sin - 1 ( v ) .gamma. b ) ( 93 ) p n
( a ) = .gamma. a tan - 1 ( a .gamma. a ) ( 94 ) p v ( e ) = sin (
.gamma. b tan - 1 ( e .gamma. b ) ) ( 95 ) p ~ u ( x , y , z ) =
.gamma. a tan - 1 ( tan - 1 ( y x ) .gamma. a ) ( 96 ) p ~ v ( x ,
y , z ) = sin ( .gamma. b tan - 1 ( sin - 1 ( z x 2 + y 2 + z 2 )
.gamma. b ) ) N P ^ ( 97 ) p _ w ( x , y , z ) = x 2 + y 2 + z 2 (
98 ) ##EQU00034##
[0132] For the unit sphere w.sub.0= {square root over
(x.sup.2+y.sup.2+z.sup.2)}=1, and a final mapping from (x, y, z) to
our unit-sphere rectangle in (u, v) space can result.
p _ a ( x , y , z ) = .gamma. a tan - 1 ( tan - 1 ( y x ) .gamma. a
) ( 99 ) p ~ v ( x , y , z ) = sin ( .gamma. b tan - 1 ( sin - 1 (
z ) .gamma. b ) ) ( 100 ) ##EQU00035##
[0133] Again by solving for the number of pixels in projection
space any value of k in the original resolution function may be
matched.
a .di-elect cons. [ - .pi. , .pi. ) ( 101 ) b .di-elect cons. [ -
.pi. 2 , .pi. 2 ] ( 102 ) u .di-elect cons. p u ( [ - .pi. , .pi. )
) = [ .gamma. a tan - 1 ( - .pi. .gamma. a ) , .gamma. a tan - 1 (
.pi. .gamma. a ) ) ( 103 ) v .di-elect cons. p ^ v ( [ - .pi. 2 ,
.pi. 2 ] ) = [ sin ( .gamma. b tan - 1 ( - .pi. 2 .gamma. b ) ) ,
sin ( .gamma. b tan - 1 ( .pi. 2 .gamma. b ) ) ] ( 104 )
##EQU00036##
[0134] Because sin and tan.sup.-1 are both odd functions, to
simplify the calculation of the total area.
A P ~ = 2 ( .gamma. a tan - 1 ( .pi. .gamma. a ) ) 2 sin ( .gamma.
b tan - 1 ( .pi. 2 .gamma. b ) ) = 4 .gamma. a tan - 1 ( .pi.
.gamma. a ) sin ( .gamma. b tan - 1 ( .pi. 2 .gamma. b ) ) ( 105 )
##EQU00037##
[0135] Just like the uniform resolution projection previously, all
the terms except k and |A.sub.P| may cancel in the equation for the
number of pixels N.sub.{tilde over (p)}.
Np = k A P = k 4 .gamma. a tan - 1 ( .pi. .gamma. A ) sin ( .gamma.
b tan - 1 ( .pi. 2 .gamma. b ) ) ( 106 ) ##EQU00038##
[0136] In equation 89, the resolution at (a, b)=(0, 0) (forward,
center), can again be equal to k. Therefore, scaling of our desired
forward resolution by |A.sub.{tilde over (P)}| to calculate the
required number of pixels may again be desired.
[0137] Referring now to FIGS. 6 and 7, a flowchart 400 and
corresponding schematic representation 500 illustrating method
steps that may be implemented to transmit virtual reality content
according to some aspects of the disclosure are shown. Both FIGS. 6
and 7 will reference common numeral identifiers as appropriate to
help illustrate the relationship between method steps for
transmitting virtual reality content process. Beginning at 405, any
virtual reality content may be received, uploaded and/or recorded,
(i.e., transmitted) to the system so long as the projection is
known or can be determined from the content. In some embodiments,
the content may be uploaded in two dimensional equirectangular
format and come from one or multiple virtual reality recording
devices. Alternatively, the content may be in 360.degree. spherical
three dimensional format. Where content is in 360.degree. spherical
three dimensional format, for every projection mapping from a three
dimensional sphere content to a flat two dimensional projections
there may be non-uniform distortions. By rotating this projection
and applying it to the same content different renditions of the
same content can be generated with more or less distortion on the
sphere of possible viewing directions. Using the viewing direction
(either intended or determined by the virtual reality projection
device) and correlating it to the rendition delivered can improve
the visible resolution for the user, therefore improving the
immersion of same virtual reality content, even with the same
hardware and bandwidth constraints.
[0138] According to additional aspects of the disclosure, custom
projections can be implemented to magnify the range of distortions
across a projection (as explained above). This can exaggerate the
differences in resolution depending on viewing angle, thus enabling
the delivery of higher resolution projections for the most relevant
view direction (as compared to those portions of the virtual
reality content that are not immediately visible). All of this can
be accomplished while also reducing the bitrate of the transmitted
video, as the content in its new projection format can be encoded
in such a way that its bitrate is lower than the original content,
but the resolution in the view direction is the same as the
original content.
[0139] At 410, the content can be mapped using a resolution-defined
projection. The resolution-defined projection may be generated, for
example, according to one or more of the developed projections with
Gaussian, Cosine, Uniform, and Cauchy resolution distributions or
combinations thereof. The mapping of the content may be repeated as
to generate at least one rendition using variously-rotated and/or
variously-shaped projections that can be encoded at 415 for
transmission at 420 and decoding at 425 at the VR device. Storing
(not shown) may optionally occur at one or more stages of the
process. In some fully integrated embodiments, such as PCs,
wireless devices, and those with immersive displays, the content is
encoded at 416, stored at 421 and accessed accordingly at 426,
without a need of either or both transmission or decoding.
[0140] The rendition(s) can use the same or different projections
rotation in azimuth and elevation as the base projections, or could
even implement combinations thereof. However, unlike in the
currently implemented video streaming processes in which multiple
renditions at different sizes and bitrates are used, the renditions
generated herein are fundamentally different because of the
different projections used to provide optimized resolutions and
encoding. In accordance with some aspects of the disclosure, the
different projections can generally be pre-determined defined, for
example, by a desired resolution distribution according to the view
direction and/or, on the fly, for example, by a desired resolution
distribution that adapts to parameters of the device, bandwidth
limitations, and the such.
[0141] In some embodiments, data definitive of the mapping and/or
the desired resolution may be included along the virtual reality
content for organized transmission at 420 of dynamically selected
projection versions according to the viewing direction and decoding
425 at the virtual reality projection device. According to aspects
of the disclosure, the dynamically selected projection versions may
include only a subset of a generated rendition. The virtual reality
projection device may include an app installed that is configured
to transmit virtual reality content requests (including, for
example, viewing direction defined requests) and receive the
corresponding data for decoding accordingly. Depending on the
desired resolution and bandwidth available for the transmission of
the renditions, one or more renditions may be transmitted to the
user's virtual reality device. Once the appropriate rendition has
been received by the display device, a reverse projection is
applied to the content to map it back to the full sphere of content
it was derived from. In some embodiments, the virtual reality
device already has knowledge of which rendition it requested, so
using the correct parameters for projection may be trivial.
[0142] According to some aspects, any additional renditions
transmitted may be used as a small buffer to the user's viewing
direction, for example. Once the content is decoded at 425,
inverse-projection content may be projected to a sphere at 430.
Using the resolution-defined projection parameters, the virtual
reality content can be projected in accordance with the desired
resolution distribution at 435, so that it is of higher resolution
in the areas of interest. The virtual reality user views virtual
reality content with higher resolution in their view direction
without data unnecessarily transmitted to perfectly recreate
content in their periphery or out of their field of view. This
method can be used to reduce virtual reality content bitrate
without sacrificing visible resolution, or to improve visible
resolution without increasing bitrate, or a combination of improved
visible resolution and reduced bitrate. Since it can be possible to
deliver higher visible resolution content with fewer pixels,
constraints on resolution imposed by the processing power of the
display device can also be mitigated. For example, many virtual
reality devices are capable of decoding a maximum number of pixels
per second so by projecting the virtual reality content in a
different way, the visible resolution can be further improved
without exceeding this maximum processing speed.
[0143] Referring now to FIGS. 8 and 9, a flowchart 600 and
corresponding schematic representation 700 illustrating method
steps that may be implemented to process and deliver virtual
reality content according to some aspects of the disclosure are
shown. Both FIGS. 6 and 7 will reference common numeral identifiers
as appropriate to help illustrate the relationship between method
steps for processing and transmitting virtual reality content.
[0144] Beginning at 605, 360.degree. virtual reality content is
transmitted to a processing device. According to aspects of the
disclosure, in order to use traditional media encoding and
compression techniques, the content can be re-mapped to a
two-dimensional rectangular projection to avoid extraneous data. At
610, as previously described, a set of projections and/or rotations
can then be mapped. At 615, each of those renditions can then be
encoded at one or more bitrates. At 620, head and/or eye movement
tracking can be used to determine a viewing direction of the user.
This data can be obtained from the head mounted virtual reality
devices' gyroscopes and/or a sensor tracking the user's pupils
during viewing of content or other methods of view direction
tracking preexisting in the virtual reality device. At 620, the
viewing direction data may be used to select a subset of renditions
to be delivered to the virtual reality device at 625. In addition
to or alternatively at 620, detection of available bandwidth and/or
viewing device parameters may be used to determine the selection of
the projections. Where viewing direction is used for the selection,
for example, by detecting the user's viewing direction and, taking
into account the fact that human vision degrades away from the
focus point, the subset of virtual reality content that can be
selected is optimized for that viewing direction. Where the viewing
device parameters are used for the selection, for example, the
maximum DPI of the screen may be used for the selection, to ensure
the subset closely matches the maximum DPI of the screen and avoid
unnecessary processing. The subset of renditions selected are then
transmitted to the virtual reality device of the user for display
of the virtual reality content at 630.
[0145] According to additional aspects of the disclosure, in
addition to the use of viewing direction, the process steps can
further be used to automatically generate projections based on the
content or recorded moving patterns. For example, the regions in
virtual reality content in which greater motion is detected through
motion detection algorithms, a resolution distribution that
includes greater resolution in those corresponding regions could be
determined. By taking into account that the user may be focused on
the activity/movement, the higher definition in those regions then
provide a more immersive experience to the user.
[0146] According to yet additional aspects, this method could also
be used in conjunction with the implementation of crowd-sourced
data to define, at least in part, which areas of the virtual
reality content should include the higher distributions and/or
establish moving patterns that can be used for predictive
generation of the renditions. More specifically, by aggregating
data from many viewers of the same or other virtual reality content
(viewing direction, moving patterns, focus, emotional reactions
and/or physiological changes), the data may be used to improve the
virtual reality experience, including, for example, to create
desired resolution distribution profiles specific to the virtual
reality content or viewer or to select one or more existing
renditions.
[0147] The many features and advantages of the invention are
apparent from the detailed specification, and thus, it is intended
by the appended claims to cover all such features and advantages of
the invention which fall within the true spirit and scope of the
invention. Further, because numerous modifications and variations
will readily occur to those skilled in the art, it is not desired
to limit the invention to the exact construction and operation
illustrated and described, and accordingly, all suitable
modifications and equivalents may be resorted to, falling within
the scope of the invention.
* * * * *
References