U.S. patent application number 15/860392 was filed with the patent office on 2018-07-05 for method and system for providing a viewport division scheme for virtual reality (vr) video streaming.
This patent application is currently assigned to Black Sails Technology Inc.. The applicant listed for this patent is Black Sails Technology Inc.. Invention is credited to Chia-Chi Chang, Yongtao Tang, Zhuo Wang, Ruoxi Zhao, Haoyan Zu.
Application Number | 20180192044 15/860392 |
Document ID | / |
Family ID | 62711388 |
Filed Date | 2018-07-05 |
United States Patent
Application |
20180192044 |
Kind Code |
A1 |
Wang; Zhuo ; et al. |
July 5, 2018 |
Method and System for Providing A Viewport Division Scheme for
Virtual Reality (VR) Video Streaming
Abstract
A method and a system for providing a viewport division scheme
for VR video streaming are disclosed. A projection area of VR video
is obtained. The projection area is divided into a plurality of
grid blocks. A core area is defined in the projection area and
divided into a plurality of core blocks. A plurality of viewport
blocks are obtained by combining the core blocks. A mapping
relationship between the plurality of viewport blocks and the grid
blocks is established. The plurality of viewport blocks are related
to a user's field of view. The plurality of viewport blocks are
obtained in accordance with a mapping table. The video data is
processed by different methods for a viewport area and a
non-viewport area to have a better viewing experience.
Inventors: |
Wang; Zhuo; (Sunnyvale,
CA) ; Tang; Yongtao; (San Leandro, CA) ; Zhao;
Ruoxi; (San Jose, CA) ; Zu; Haoyan; (Newark,
CA) ; Chang; Chia-Chi; (San Jose, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Black Sails Technology Inc. |
Sunnyvale |
CA |
US |
|
|
Assignee: |
Black Sails Technology Inc.
Sunnyvale
CA
|
Family ID: |
62711388 |
Appl. No.: |
15/860392 |
Filed: |
January 2, 2018 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62441936 |
Jan 3, 2017 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06T 3/40 20130101; H04N
21/816 20130101; H04L 43/0888 20130101; H04N 13/378 20180501; H04N
19/70 20141101; H04N 13/117 20180501; G06T 9/001 20130101; G06T
15/205 20130101; H04N 21/234345 20130101; H04N 13/383 20180501;
H04N 13/332 20180501; H04N 21/2335 20130101; H04N 13/275 20180501;
H04N 9/8715 20130101; H04L 67/38 20130101; H04N 13/344 20180501;
H04N 5/76 20130101; H04N 13/398 20180501; H04N 13/139 20180501;
H04N 21/231 20130101; H04N 21/234363 20130101; H04N 21/234381
20130101; G06F 3/011 20130101; H04N 19/44 20141101; G06F 3/012
20130101; H04N 13/161 20180501; H04L 65/601 20130101; H04N 13/189
20180501; H04N 19/40 20141101; H04N 13/122 20180501; G06T 2215/16
20130101; G06T 15/04 20130101 |
International
Class: |
H04N 13/106 20180101
H04N013/106; H04L 29/06 20060101 H04L029/06 |
Claims
1. A method for providing a viewport division scheme for VR video
streaming, comprising: obtaining a projection area of VR video;
dividing the projection area into a plurality of grid blocks;
defining a core area in the projection area; dividing the core area
into a plurality of core blocks; obtaining a plurality of viewport
blocks by combining the plurality of core blocks; and establishing
a mapping relation between the plurality of viewport blocks and the
plurality of grid blocks, wherein the plurality of viewport blocks
is related to a user's field of view.
2. The method according to claim 1, wherein the step of obtaining
the projection area of VR video comprises: projecting the VR video
into a rectangle with the aspect ratio of 2:1 by equivalent
rectangle matrix to obtain the projection area.
3. The method according to claim 1, wherein the step of dividing
the projection area into the plurality of grid blocks comprises:
dividing the projection area into a plurality of equal
portions.
4. The method according to claim 1, further comprising: creating
identifications for the plurality of grid blocks, the plurality of
core blocks and the plurality of viewport blocks, and establishing
a mapping relationship between the plurality of viewport blocks and
the plurality of grid blocks by the identifications.
5. The method according to claim 4, wherein the step of creating
identifications for the plurality of grid blocks, the plurality of
core blocks and the viewport, and establishing the mapping
relationship between the plurality of viewport blocks and the
plurality of grid blocks by the identifications comprises: setting
the identification id.sub.block of a grid block as a positive
integer started from zero, so as to satisfy Equation (1)
id.sub.block [0,n-1],id.sub.block N.sup.+ (1); setting an edge
length of the grid block to satisfy Equation (2), width block =
height video n 2 ( 2 ) ##EQU00013## where height.sub.video
represents an image height, and N.sup.+ represents a positive
integer; setting the core area to be a set of grid blocks
Set.sub.core which satisfy Equation (3), Set core = { id block id
block = { n 4 , n 4 + 1 , , 3 n 4 - 2 , 3 n 4 - 1 } } ; ( 3 )
##EQU00014## defining a core block to have a width equal to that of
the grid blocks and a height equal to that of the core area, and
setting the identification id.sub.core.sub._.sub.block of the core
block, so as to satisfy Equation (4), id core_block = { id block %
2 n 2 , id block .di-elect cons. Set core null , others ; ( 4 )
##EQU00015## defining a viewport block in the core area to have a
width determined by a user's field of view and a height equal to
that of the core area, where a width
width.sub.viewport.sub._.sub.block of the viewport block satisfies
Equation (5), width viewport_block = fov * height video .pi. , ( 5
) ##EQU00016## where fov represents the user's field of view;
defining the number of the core blocks in the viewport block to be
i, which satisfies equation (6), i = width viewport_block width
block ; ( 6 ) ##EQU00017## for any identification of the viewport
block, the identifications of the core blocks in the viewport block
can be obtained according to Equation (7), f ( id viewport_block )
= { { id viewport_block , id viewport_block + 1 , , id
viewport_block + i - 2 , id viewport_block + i - 1 } ; id
viewport_block .di-elect cons. [ 0 , 2 n 2 - i ] { id
viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i
- ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block
.di-elect cons. [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
##EQU00018## for any identification of the core block, the
identification of the viewport block can be obtained uniquely
according to Equation (8),
id.sub.viewport.sub._.sub.block=f(id.sub.core.sub._.sub.block)=id.sub.cor-
e.sub._.sub.block.sub._.sub.1st (8), where
id.sub.core.sub._.sub.block.sub._.sub.1st represents the
identification of the first core block that is included in the
viewport block.
6. The method according to claim 1, wherein the step of dividing
the core area into the plurality of core blocks comprises: dividing
the core area into the plurality of core blocks with equal
areas.
7. The method according to claim 1, wherein the plurality of core
blocks each have a height equal to that of the core area, and the
core area has an area equal to an integral times that of the area
of a grid block.
8. The method according to claim 1, further comprising: dividing
the VR video data into a plurality of sub-data according to the
plurality of grid blocks, wherein each sub-data corresponds to one
of the grid blocks, and determining a mapping relationship between
the plurality of sub-data and the viewport according to the mapping
relationship between the plurality of grid blocks and the
viewport.
9. The method according to claim 1, further comprising: creating a
plurality of viewport blocks according to a user's field of
view.
10. A system for providing a viewport division scheme for VR video
streaming, comprising: a configuration module, configured to obtain
a projection area of VR video; a first division module, configured
to divide the projection area into a plurality of grid blocks; a
second division module, configured to define a core area in the
projection area; a third division module, configured to divide the
core area into a plurality of core blocks; a combination module,
configured to combine the core blocks to provide a plurality of
viewport blocks; and a mapping module, configured to establish a
mapping relationship between the plurality of viewport blocks and
the grid blocks, wherein the plurality of viewport blocks are
related to a user's field of view.
11. The system according to claim 1, wherein the configuration
module projects the VR video into a rectangle with the aspect ratio
of 2:1 by equivalent rectangle matrix to obtain the projection
area.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims the priority and benefit of U.S.
provisional application 62/441,936, filed on Jan. 3, 2017, which is
incorporated herein by reference in its entirety.
BACKGROUND OF THE DISCLOSURE
Field of the Disclosure
[0002] The present disclosure relates to video processing
technology, and more particularly, to a method and a system for
providing a viewport division scheme for virtual reality (VR) video
streaming BACKGROUND OF THE DISCLOSURE
[0003] Virtual Reality (VR) is a computer simulation technology for
creating and experiencing a virtual world. For example, a
three-dimensional real-time image can be presented based on a
technology which tracks a user's head, eyes or hands. In the
network-based virtual reality technology, full-view video data can
be pre-stored on a server, and then transmitted to a display
device. A display device can be glasses, a head-mounted display,
etc. A video is displayed on the display device according to a
user's field of view.
[0004] However, a high-resolution video data occupies large
transmission of bandwidth and requires high computing power from
the display devices. Presenting high-resolution VR video on
internet is difficult. Precisely, the existing video streaming
technology can not fulfill the virtual reality. Therefore, in order
to present VR video smoothly in real-time, it is desirable to
further improve the existing video streaming technology to save
bandwidth and reduce performance requirements for display devices,
by a new way to encode and store the VR video data on the
server.
[0005] It is a problem in the art to play back video data on a
player end with a better viewing experience through less video data
transmission.
SUMMARY OF THE DISCLOSURE
[0006] In view of this, the present disclosure provides a method
and a system for providing a viewport division scheme for virtual
reality (VR) video streaming, which determine a viewport block of
the VR video data and process the VR video data by different
methods for a viewport area and a non-viewport area so as to have a
better viewing experience at a player end through less data
transmission.
[0007] According to one aspect of the present disclosure, there is
provided a method for providing a viewport division scheme for VR
video streaming, comprising:
[0008] obtaining a projection area of VR video;
[0009] dividing the projection area into a plurality of grid
blocks;
[0010] defining a core area in the projection area;
[0011] dividing the core area into a plurality of core blocks;
[0012] combining the core blocks to provide a plurality of viewport
blocks; and
[0013] establishing a mapping relation between the plurality of
viewport blocks and the plurality of grid blocks, wherein the
plurality of viewport blocks are related to a user's field of
view.
[0014] Preferably, the step of obtaining a projection area of VR
video comprises:
[0015] projecting the VR video into a rectangle with the aspect
ratio of 2:1 by equivalent rectangle matrix to obtain the
projection area.
[0016] Preferably, the step of dividing the projection area into a
plurality of grid blocks comprises:
[0017] dividing the projection area into a plurality of equal
portions; and
[0018] dividing each of the plurality of equal portions into the
same number of grid blocks.
[0019] Preferably, the method further comprises creating
identifications for the plurality of grid blocks, the plurality of
core blocks and the viewport, and establishing a mapping
relationship between the viewport and the plurality of grid blocks
by the identifications.
[0020] Preferably, the step of creating identifications for the
plurality of grid blocks, the plurality of core blocks and the
viewport, and establishing a mapping relationship between the
viewport and the plurality of grid blocks by the identifications
comprises:
[0021] setting the identificationid.sub.block of a grid block as a
positive integer started from zero, so as to satisfy Equation
(1)
id.sub.block [0,n-1],id.sub.block N.sup.+ (1);
[0022] setting an edge length of the grid block to satisfy Equation
(2),
width block = height video n 2 ( 2 ) ##EQU00001##
[0023] where height.sub.video represents an image height, and
N.sup.+ represents a positive integer;
[0024] setting the core area to be a set of grid blocks
Set.sub.core which satisfy Equation (3),
Set core = { id block | id block = { n 4 , n 4 + 1 , , 3 n 4 - 2 ,
3 n 4 - 1 } } ; ( 3 ) ##EQU00002##
[0025] defining a core block to have a width equal to that of the
plurality of grid blocks and a height equal to that of the core
area, and setting the identification id.sub.core.sub._.sub.block of
the core block, so as to satisfy Equation (4),
id core_block = { id block % 2 n 2 , id block .di-elect cons. Set
core null , others ; ( 4 ) ##EQU00003##
[0026] defining a viewport block in the core area to have a width
determined by a user's field of view and a height equal to that of
the core area, where a width width.sub.viewport.sub._.sub.block of
the viewport block satisfies Equation (5),
width viewport_block = fov * height video .pi. , ( 5 )
##EQU00004##
[0027] where fov represents the user's field of view;
[0028] defining the number of the core blocks in the viewport block
to be i, which satisfies equation (6),
i = width viewport_block width block ; ( 6 ) ##EQU00005##
[0029] for any identification of the viewport block, the
identifications of the core blocks in the viewport block can be
obtained according to Equation (7),
f ( id viewport_block ) = { { id viewport_block , id viewport_block
+ 1 , , id viewport_block + i - 2 , id viewport_block + i - 1 } ;
id viewport_block .di-elect cons. [ 0 , 2 n 2 - i ] { id
viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i
- ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block
.di-elect cons. [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
##EQU00006##
[0030] for any identification of the core block, the identification
of the viewport block including the core block can be obtained
uniquely according to Equation (8),
id.sub.viewport.sub._.sub.block=f(id.sub.core.sub._.sub.block)=id.sub.co-
re.sub._.sub.block.sub._.sub.1st (8),
[0031] where id.sub.core.sub._.sub.block.sub._.sub.1st represents
the identification of the first core block that is included in the
viewport block.
[0032] Preferably, the step of dividing the core area into the
plurality of core blocks comprises:
[0033] dividing the core area into the plurality of core blocks
with equal areas.
[0034] Preferably, the plurality core blocks each have a height
equal to of the central and the core area has an area equal to
integral times of the area of a grid block.
[0035] Preferably, the method further comprises dividing the VR
video data into a plurality of sub-data according to the plurality
of grid blocks, wherein each sub-data corresponds to one of the
grid blocks, and determining a mapping relationship between the
plurality of sub-data and the viewport according to the mapping
relationship between the plurality of grid blocks and the
viewport.
[0036] Preferably, the method further comprises creating a
plurality of viewport blocks according to a user's field of
view.
[0037] According to another aspect of the present disclosure, there
is provided a system for providing a viewport for VR video
streaming, comprising:
[0038] a configuration module, configured to obtain a projection
area of VR video;
[0039] a first division module, configured to divide the projection
area into a plurality of grid blocks;
[0040] a second division module, configured to define a core area
in the projection area;
[0041] a third division module, configured to divide the core area
into a plurality of core blocks;
[0042] a combination module, configured to combine the core blocks
to provide a plurality of viewport blocks;
[0043] a mapping module, configured to establish a mapping
relationship between the viewport and the plurality of grid blocks,
wherein the plurality of viewport blocks are related to a user's
field of view.
[0044] Preferably, the configuration module
[0045] projects the VR video into a rectangle with the aspect ratio
of 2:1 by equivalent rectangle matrix to obtain the projection
area.
[0046] In this embodiment, the VR source video data is divided into
a plurality of grid blocks, and a mapping table between the
viewport blocks and the grid blocks is established according to a
specific field of view. When the user's field of view is changed,
the viewport block is retrieved according to the mapping table, and
the video data of a viewport area and a non-viewport area are
processed by different methods to have a better view
experience.
BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The above and other objects, advantages and features of the
present disclosure will become more fully understood from the
detailed description given hereinbelow in connection reference with
the appended drawings, and wherein:
[0048] FIG. 1 is a diagram illustrating an example the
network-based virtual reality playback system;
[0049] FIG. 2 is a flowchart presents the process of the playback
system in FIG. 1;
[0050] FIG. 3 is a flowchart presents the method for providing a
viewport division scheme for VR video streaming;
[0051] FIG. 4A-4C schematically show a core area, grid blocks and
core blocks of the video data; and
[0052] FIG. 5 is a block diagram illustrating a system for
providing a viewport for VR video streaming.
DETAILED DESCRIPTION OF THE DISCLOSURE
[0053] Exemplary embodiments of the present disclosure will be
described in more details below with reference to the accompanying
drawings. In the drawings, like reference numerals denote like
members. The figures are not drawn to scale, for the sake of
clarity. Moreover, some well known parts may not be shown.
[0054] FIG. 1 is a diagram illustrating an example network of a VR
playback system. The VR playback system 10 includes a server 100
and a display device 120 which are coupled with each other through
a network 110, and a VR device. For example, the server 100 may be
a stand-alone computer server or a server cluster. The server 100
is used to store various video data and to store various
applications that process these video data. For example, various
daemons run on the server 100 in real time, so as to process
various video data in the server 100 and to respond various
requests from VR devices and the display device 120. The network
110 may be a selected one or selected ones from the group
consisting of an internet, a local area network, an internet of
things, and the like. For example, the display device 120 may be
any of the computing devices, including a computer device having an
independent display screen and a processing capability. The display
device 120 may be a personal computer, a laptop computer, a
computer workstation, a server, a mainframe computer, a palmtop
computer, a personal digital assistant, a smart phone, an
intelligent electrical apparatus, a game console, an iPAD/iPhone, a
video player, a DVD recorder/player, a television, or a home
entertainment system. The display device 120 may store VR player
software as a VR player. When the VR player is started, it requests
and downloads various video data from the server 100, and renders
and plays the video data in the display device. In this example,
the VR device 130 is a stand-alone head-mounted device that can
interact with the display device 120 and the server 100, and
transmit information to the display device 120 and/or the server
100 by signal, including change of the position of a user's helmet,
change of the eye sight, and the like. The display device 120 can
process flexibly the video data which is played according to the
information. In some embodiments, when a user's position is
changed, the display device 120 determines that a central viewport
of the user has been changed, and starts to play video data with a
high resolution in the central viewport.
[0055] In the above embodiment, the VR device 130 is a stand-alone
head-mounted device. However, those skilled in the art should
understand that the VR device 130 is not limited thereto, and the
VR device 130 may also be an all-in-one head-mounted device. The
all-in-one head-mounted device itself has a display screen, so that
it is not necessary to connect the all-in-one head-mounted device
with the external display device. For example, in this example, if
the all-in-one head-mounted device is used as the VR device, the
display device 120 may be omitted. At this point, the all-in-one
head-mounted device is configured to obtain video data from the
server 100 and to perform playback operation, and the all-in-one
head-mounted device is also configured to detect user's field of
view and to adjust the playback operation according to the user's
field of view.
[0056] FIG. 2 is a flowchart showing a method used in the VR
playback system of FIG. 1. The method includes the following
steps.
[0057] In step S10, the server performs source video data
processing.
[0058] In step S20, display device side obtains some related
information by interacting with the VR device.
[0059] In step S30, according to the related information, the
display device side requests the server side to provide the video
data and receives the video data.
[0060] In step S40, the display device side renders the received
video data.
[0061] Wherein, the step S10 is used to process the video data
stored on the server side. In this step, the video data can be
divided into those in a viewport are and those in a non-viewport
area, and then can be processed by different methods to reduce data
transmission for the non-viewport. Therefore, the present
disclosure further provides a method for providing a viewport
division scheme to achieve the above functions.
[0062] FIG. 3 is a flowchart presents the method for providing a
viewport division scheme for VR video streaming. Specifically, the
method comprises following steps.
[0063] In step S100, a projection area of VR video is obtained.
[0064] Although the VR video data is captured in a 360-degree
panoramic view of the real-world environment, a projection process
may be included in a VR video production process and a rendering
process to change the VR video data to two-dimensional spatial
data. The original video data may be subjected to the projection
process, for example, an equivalent rectangle matrix process with
the aspect ratio of 2:1, to obtain a projection area. In a case
that the VR video data has been subjected to the projection
process, the projection area can be obtained by decoding the VR
video data. Two main VR video formats include the video data with
the aspect ratio of 1:1 and the aspect ratio of 2:1, which are
well-known in the industry as film sources. In some cases, the
server needs to perform a normalization process in view of the
user's requirements. For example, the video data with the aspect
ratio of 1:1 is processed into the video data with the aspect ratio
of 2:1 which is suitable for the user's VR device. In the
equivalent rectangle matrix projection, for example, in an
equally-spaced cylindrical projection, the meridians can be
rendered to be vertical straight lines having a constant pitch, or
the latitude circles can be rendered to be horizontal straight
lines having a constant pitch. It should be noted that the VR video
data here refers to original data and/or intermediate data in any
VR processing.
[0065] In step S200, a core area of the projection area is
divided.
[0066] As described above, the projection area of the VR video data
is obtained in the previous step S100. In this step, the core area
of the projection area is divided. The data after being projected
is generally distorted. For example, a full-view image may be
rendered by an equivalent rectangle matrix projection process to
obtain an area with the aspect ratio of 2:1. According to
stretching characteristics of the equivalent rectangle matrix
projection, when an image is close to a pole, the image will be
stretched and distorted. The degree of pixel distortion will be
higher. Therefore, according to stretching characteristics of the
equivalent rectangle matrix projection, a central portion will be
defined at the pole.
[0067] In step S300, a block size is defined according to the
parameter N.
[0068] The parameter N is a given parameter. In this step, an area
of each grid block is obtained according to the N value. For
example, assuming that the normalized video data has an area S and
N is 32, each grid block has a block size equal to S/32.
[0069] In step S400, the core area is divided into grid blocks.
[0070] After the core area is determined, the core area can be
divided into M core blocks based on a given value M. The core
blocks may be numbered for identifying each core block. Generally,
the number starts from 0.
[0071] In step S500, the projection area is divided into grid
blocks.
[0072] In this step, the projection area is divided into a
plurality of equal portions, each of which corresponds to a grid
block. It should be understood that the grid block here is a
portion of the VR video data. After that, each grid block is
numbered. Generally, the number starts from 0.
[0073] In step S600, the core blocks are combined into a viewport
block.
[0074] The viewport block contains one or more core blocks. When a
field of view of the VR device is changed, the viewport block is
also changed. Accordingly, the core blocks in the viewport block
are also changed. Therefore, the core blocks may be combined into
the viewport block according to the user's field of view.
[0075] In step S700, a mapping table between the viewport blocks
and the grid blocks is established.
[0076] Based on the viewport block of each VR video data, which is
obtained in step S600, the mapping table between the viewport
blocks and the grid blocks is established. In this embodiment, the
VR source video data is divided into a plurality of grid blocks,
and a viewport block is defined according to a specific field of
field of view (fov) to establish the mapping table between the
viewport blocks and the grid blocks. When the user's field of view
is changed, the viewport block is changed and obtained according to
the mapping table. The video data is processed by different methods
for the viewport area and for the non-viewport area, so as to have
a better viewing experience. Further, the user's field of view is
proportional to the area of the viewport block. Many viewport
blocks can be defined for many fields of views (fov). In an
alternative embodiment, the grid blocks, the core blocks and the
viewport blocks have identifications, and a mapping relationship
between the viewport blocks and the grid blocks is established with
the identifications.
[0077] It should be noted that in the above embodiments, the video
data is not actually segmented. Therefore, in an optional
embodiment of the present disclosure, a segmentation step may also
be included in which the video data is divided into a plurality of
sub-data according to a plurality of grid blocks which are obtained
according to the above embodiment. Each sub-data corresponds to a
grid block. According to the above mapping relationship between the
viewport blocks and the grid blocks, one can determine the mapping
relationship between the plurality of sub-data and the plurality of
viewport blocks.
[0078] Under some limitations, mathematical equations can be
established based on the identifications of the grid blocks, the
core blocks and the viewport blocks, so that the identification of
the viewport block can be determined according to the
identifications of the grid blocks and/or the core blocks, or the
identifications of the core blocks can be determined according to
the identification of the viewport block. Details will be described
in the following paragraphs.
[0079] Assuming that an image of the video data has the aspect
ratio of 2:1 and is divided into n grid units with the aspect ratio
of 1:1, and each grid unit is a grid block and has the
identification id.sub.block as follows,
id.sub.block [0,n-1],id.sub.block N.sup.+ (1);
[0080] An edge length of the grid block satisfies Equation (1).
width block = height video n 2 ( 2 ) ##EQU00007##
[0081] where height.sub.video represents an image height, and
N.sup.+ represents a positive integer;
[0082] According characteristics of the equivalent rectangle matrix
projection, the core area is defined as a set of grid blocks
Set.sub.core which satisfy Equation (3),
Set core = { id block id block = { n 4 , n 4 + 1 , , 3 n 4 - 2 , 3
n 4 - 1 } } ; ( 3 ) ##EQU00008##
[0083] A core block is defined to have a width equal to that of the
grid blocks and a height equal to that of the core are. The
identification id.sub.core.sub._.sub.block of the core block
satisfies Equation (4),
id core_block = { id block % 2 n 2 , id block .di-elect cons. Set
core null , others ; ( 4 ) ##EQU00009##
[0084] A viewport block is defined as having blocks in the core
area, which are determined by a user's field of view fov and have a
width equal to that of the core area. A width
width.sub.viewport.sub._.sub.block of the viewport block satisfies
Equation (5),
width viewport_block = fov * height video .pi. , ( 5 )
##EQU00010##
[0085] The number of the core blocks in the viewport block is
defined as i, which satisfies equation (6),
i = width viewport_block width block ; ( 6 ) ##EQU00011##
[0086] For any identification of the viewport block, the
identifications of the core blocks in the viewport block can be
obtained according to Equation (7),
f ( id viewport_block ) = { { id viewport_block , id viewport_block
+ 1 , , id viewport_block + i - 2 , id viewport_block + i - 1 } ;
id viewport_block .di-elect cons. [ 0 , 2 n 2 - i ] { id
viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i
- ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block
.di-elect cons. [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
##EQU00012##
[0087] For any identification of the core block, the identification
of the viewport block including the core block, which represents
the identification of the first core block, can be obtained
uniquely according to Equation (8), The identification satisfies
Equation (8),
id.sub.viewport.sub._.sub.block=f(id.sub.core.sub._.sub.block)=id.sub.co-
re.sub._.sub.block.sub._.sub.1st (8),
[0088] where id.sub.core.sub._.sub.block.sub._.sub.1st represents
the identification of the first core block that is included in the
viewport block.
[0089] FIG. 4A-4C schematically show a core area, grid blocks and
core blocks of the video data.
[0090] In this example, 400 represents a projection area, 401
represents a core area, and N is equal to 32. That is to say, the
projection area is divided into 32 grid blocks, which are numbered
as 0-31. The core blocks are numbered as 0-7. Assuming that one
viewport block (not shown) contains two core blocks which are
numbered as 1 and 2, the viewport block contains those grid blocks
being numbered as 9-10 and 17-18. Thus, it is possible to establish
a mapping relationship between the viewport blocks and the grid
blocks. In this case, the identifications of the grid blocks, the
core blocks and the viewport blocks may be numbered and be verified
by the above equations.
[0091] It should be understood that this example is given only for
the mapping relationship among the grid blocks, the core blocks and
the viewport blocks under ideal conditions. That is to say, in this
example, the core blocks and the grid blocks and the viewport
blocks overlap each other. However, the present disclosure is not
limited to this and can be applied in a case that the grid blocks,
the core blocks and the viewport blocks partially overlap each
other.
[0092] FIG. 5 is a block diagram illustrating a system for
providing a viewport for VR video streaming.
[0093] The system includes a configuration module 501, a first
division module 503, a second division module 502, a third division
module 504, a combination module 505, and a mapping module 506.
[0094] The configuration module 501 is configured to obtain a
projection area of VR video. In a case that the VR video data has
been processed into two-dimensional data, the VR video data can
decoded to obtain the projection area of the video data from a
parameter. In another case that the video data is not processed
into two-dimensional data, the projection area can be obtained by
projecting the VR video into a rectangle by an equivalent rectangle
matrix projection process. Two main VR video formats include the
video data with the aspect ratio of 1:1 and the aspect ratio of
2:1, which are well-known in the industry as film sources. The
original VR video format may be normalized to have the aspect ratio
of 2:1, and then further processed.
[0095] The first division module 503 is configured to divide the
projection area into N grid blocks. For example, the projection
area is divided into 32 equal portions. As a result, the 32 grid
blocks each are a portion of the projection area with equal
size.
[0096] The second division module 502 is configured to divide a
core area of the projection area according to the user's field of
view. The core area is defined in the projection area. For example,
for a rectangle, a central portion in the X-axis direction or the
Y-axis direction is defined as the core area.
[0097] The third division module 504 is configured to divide the
core area into a plurality of core blocks. The core area is divided
into a plurality of grid units, generally having equal area, but is
not limited thereto. For example, if an area of an edge unit is
insufficient, the area of the edge unit will be smaller.
[0098] The combination module 505 is configured to combine the core
blocks into a plurality of viewport blocks. The core blocks are
combined according to a predetermined rule. For example, two
adjacent core blocks are combined into one viewport block. In
another example, a core block at a central point and those core
blocks surrounding the central point are combined into one viewport
block.
[0099] The mapping module 506 is configured to establish a mapping
relationship between the viewport blocks and the grid blocks. Each
viewport block contains a plurality of core blocks. Each core block
may are mixed with one or more grid blocks. Correspondingly, one or
more grid blocks may be retrieved according to the core blocks.
[0100] By establishing the mapping relationship between the
viewport block and the grid blocks, the system can locate the
viewport block and improve viewing experience effect by optimizing
the video data of the viewport block.
[0101] Preferably, the above system for providing a viewport for VR
video streaming may further comprise a segmentation module (not
shown). The segmentation module divides the video data into a
plurality of sub-data according to the plurality of grid blocks.
Each sub-data corresponds to a grid. The mapping relationship
between the plurality of sub-data and the plurality of viewport
blocks can be determined by the mapping relationship between the
plurality of viewport blocks and the plurality of grid blocks.
[0102] The configuration module projects the VR video into a
rectangle with the aspect ratio of 2:1 by equivalent rectangle
matrix to obtain the projection area.
[0103] The embodiments were chosen and described in order to best
explain the principles of the disclosure and its practical
applications, to thereby enable others skilled in the art to best
utilize the disclosure. The disclosure is intended to cover
alternatives, modifications and equivalents that may be included
within the spirit and scope of the disclosure as defined by the
appended claims.
[0104] The foregoing descriptions of specific embodiments of the
present disclosure have been presented, but are not intended to
limit the disclosure to the precise forms disclosed. It will be
readily apparent to one skilled in the art that many modifications
and changes may be made in the present disclosure. Any
modifications, equivalence, variations of the preferred embodiments
can be made without departing from the doctrine and spirit of the
present disclosure.
* * * * *