U.S. patent application number 12/588461 was filed with the patent office on 2011-03-31 for method and apparatus for controlling display operations.
This patent application is currently assigned to ARM LIMITED. Invention is credited to Daren Croxford, Elvind Liland, Ashley Stevens, Joe Tapply.
Application Number | 20110074800 12/588461 |
Document ID | / |
Family ID | 41350432 |
Filed Date | 2011-03-31 |
United States Patent
Application |
20110074800 |
Kind Code |
A1 |
Stevens; Ashley ; et
al. |
March 31, 2011 |
Method and apparatus for controlling display operations
Abstract
A graphics processing system includes a graphics processor 1
that renders output frames that are written to a frame buffer in a
memory 2 for display on a display 7. Comparison and control
hardware 5 of the graphics processing system operates to compare
successive output frames that are being generated for display, and
then controls one or more aspects of the way in which the display
of the output frames generated by the graphics processor 1 is
carried out. In one preferred embodiment, the rate at which the
display device 7 is updated (refreshed) from the frame buffer is
controlled on the basis of the output frame comparisons. In another
preferred embodiment, the format used when storing the output
frames in the frame buffer is selected on the basis of the output
frame comparisons.
Inventors: |
Stevens; Ashley; (Cambridge,
GB) ; Liland; Elvind; (Trondheim, NO) ;
Croxford; Daren; (Cambridge, GB) ; Tapply; Joe;
(Cambridge, GB) |
Assignee: |
ARM LIMITED
Cambridge
GB
|
Family ID: |
41350432 |
Appl. No.: |
12/588461 |
Filed: |
October 15, 2009 |
Current U.S.
Class: |
345/545 |
Current CPC
Class: |
G06T 11/40 20130101 |
Class at
Publication: |
345/545 |
International
Class: |
G09G 5/36 20060101
G09G005/36 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 25, 2009 |
GB |
0916924.4 |
Claims
1. A method of operating a data processing system in which a stream
of output frames to be displayed is generated by the data
processing system and written to a frame buffer for display on a
display device, the method comprising: comparing output frames to
be displayed, and controlling at least one aspect of the way in
which the output frames are provided for display on the display
device on the basis of the comparison.
2. The method of claim 1, wherein the step of controlling at least
one aspect of the way in which the output frames are provided for
display on the display device on the basis of the comparison of the
output frames comprises controlling the rate at which the display
device is updated from the frame buffer on the basis of the
comparison.
3. The method of claim 2, further comprising: always updating the
display device at a minimum rate irrespective of the result of the
frame comparisons.
4. The method of claim 2, comprising updating only part of the
frame displayed on the display device from the frame buffer on the
basis of the comparison of the output frames.
5. The method of claim 1, wherein the step of controlling at least
one aspect of the way in which the output frames are provided for
display on the display device on the basis of the comparison of the
output frames comprises controlling the way that the output frame
is stored in the frame buffer on the basis of the output frame
comparisons.
6. The method of claim 5, wherein the step of controlling the way
that the output frame is stored in the frame buffer on the basis of
the output frame comparisons comprises selecting the format that
the frame is stored in the frame buffer in on the basis of the
comparisons.
7. The method of claim 6, further comprising associating with the
output frame stored in the frame buffer control information
indicative of the format that the frame is stored in in the frame
buffer.
8. The method of claim 6, comprising using different frame buffer
formats for different portions of a frame.
9. The method of claim 1, wherein the step of comparing the output
frames to be displayed comprises comparing blocks of data
representing particular regions of the respective output frames
with each other.
10. The method of claim 9, wherein the data processing system is a
tile-based graphics processing system and each data block that is
compared corresponds to a rendered tile that the graphics
processing system produces.
11. A data processing system comprising: a data processor for
generating a stream of output frames to be displayed; a frame
buffer for storing an output frame to be displayed; a write
controller for writing an output frame generated by the data
processor to the frame buffer; a display device for displaying an
output frame; a display controller for reading an output frame from
the frame buffer and for providing it to the display device for
display; and processing circuitry for comparing output frames to be
displayed and for controlling at least one aspect of the way in
which the output frames are provided for display on the display
device on the basis of the comparison.
12. The system of claim 11, wherein the processing circuitry
controls at least one aspect of the way in which the output frames
are provided for display on the display device on the basis of the
comparison of the output frames by controlling the rate at which
the display device is updated from the frame buffer on the basis of
the comparison.
13. The system of claim 12, wherein the display controller
comprises processing circuitry arranged to always update the
display device at a minimum update rate.
14. The system of claim 12, wherein the processing circuitry can
update only part of the frame displayed on the display device from
the frame buffer on the basis of the comparison of the output
frames.
15. The system of claim 11, wherein the processing circuitry
controls at least one aspect of the way in which the output frames
are provided for display on the display device on the basis of the
comparison of the output frames by controlling the way that the
output frame is stored in the frame buffer on the basis of the
output frame comparisons.
16. The system of claim 15, wherein the processing circuitry is
arranged to select the format that the frame is stored in the frame
buffer in on the basis of the output frame comparisons.
17. The system of claim 16, further comprising processing circuitry
for associating with the output frame stored in the frame buffer
control information indicative of the format that the frame is
stored in in the frame buffer.
18. The system of claim 16, wherein the processing circuitry can
use different frame buffer formats for different portions of a
frame.
19. The system of claim 11, wherein the processing circuitry is
arranged to compare the output frames to be displayed by comparing
blocks of data representing particular regions of the respective
output frames with each other.
20. The system of claim 19, wherein the data processing system is a
tile-based graphics processing system and each data block that is
compared corresponds to a rendered tile that the graphics
processing system produces.
21. An apparatus for use in a data processing system in which
successive output frames to be displayed are generated by the data
processing system and written to a frame buffer for display on a
display device, the apparatus comprising: processing circuitry
arranged to compare output frames to be displayed, and to control
at least one aspect of the way in which the output frames are
provided for display on the display device on the basis of the
comparison.
22. The apparatus of claim 21, wherein the processing circuitry
controls at least one aspect of the way in which the output frames
are provided for display on the display device on the basis of the
comparison of the output frames by controlling the rate at which
the display device is updated from the frame buffer on the basis of
the comparison.
23. The apparatus of claim 22, wherein the processing circuitry can
update only part of the frame displayed on the display device from
the frame buffer on the basis of the comparison of the output
frames.
24. The apparatus of claim 21, wherein the processing circuitry
controls at least one aspect of the way in which the output frames
are provided for display on the display device on the basis of the
comparison of the output frames by controlling the way that the
output frame is stored in the frame buffer on the basis of the
output frame comparisons.
25. The apparatus of claim 24, wherein the processing circuitry is
arranged to select the format that the frame is stored in the frame
buffer in on the basis of the output frame comparisons.
26. The apparatus of claim 25, further comprising processing
circuitry for associating with the output frame stored in the frame
buffer control information indicative of the format that the frame
is stored in in the frame buffer.
27. The apparatus of claim 25, wherein the processing circuitry can
use different frame buffer formats for different portions of a
frame.
28. The apparatus of claim 21, wherein the processing circuitry is
arranged to compare the output frames to be displayed by comparing
blocks of data representing particular regions of the respective
output frames with each other.
29. The apparatus of claim 28, wherein the apparatus is for use in
a tile-based graphics processing system and each data block that
the apparatus compares corresponds to a rendered tile that the
graphics processing system produces.
30. A computer readable storage medium storing computer software
code for performing the method of any one of claims 1 when the
program element is run on a data processor.
Description
[0001] The present invention relates to the controlling of display
operations where an output is to be displayed on a display device
such as an LCD display, and in particular to such arrangements
where a data processing system, such as a graphics processing
system, is providing an output to be displayed on a display
device.
[0002] As is known in the art, the output of an, e.g., graphics
processing system that is to be displayed is usually written to a
so-called "frame buffer" in memory when it is ready for display.
The frame buffer is then read by a display controller and output to
a display device (which may, e.g., be a screen or printer) for
display.
[0003] The reading of the frame buffer and its provision to the
display device is a relatively expensive operation. For example,
LCD displays are typically refreshed at a constant high rate
typically between 60-70 Hz. Each such "refresh" involves reading
the complete frame buffer from memory and writing the contents to
the display. This involves, inter alia, a lot of power hungry
memory and display accesses.
[0004] The bandwidth of reading from and writing to the frame
buffer can also be a significant system bandwidth and power cost,
particularly in the case of high definition (HD) displays.
[0005] One known technique for trying to reduce the power
consumption of display operations is to reduce the display refresh
rate under the control of the application that is generating the
output to be displayed. For example, if the application knows that
the output frame has not changed or will not change for a while, it
can signal the display controller to defer, or reduce the rate of,
refreshing the display. However, these arrangements rely on the
application and/or operating system itself being able to perform
the necessary display control.
[0006] The Applicants believe therefore that there remains scope
for improvements to display operations in data processing
systems.
[0007] According to a first aspect of the present invention, there
is provided a method of operating a data processing system in which
a stream of output frames to be displayed is generated by the data
processing system and written to a frame buffer for display on a
display device, the method comprising:
[0008] the data processing system comparing output frames to be
displayed, and controlling at least one aspect of the way in which
the output frames are provided for display on the display device on
the basis of the comparison.
[0009] According to a second aspect of the present invention, there
is provided a data processing system comprising:
[0010] a data processor for generating a stream of output frames to
be displayed;
[0011] a frame buffer for storing an output frame to be
displayed;
[0012] a write controller for writing an output frame generated by
the data processor to the frame buffer;
[0013] a display device for displaying an output frame;
[0014] a display controller for reading an output frame from the
frame buffer and for providing it to the display device for
display; and
[0015] processing circuitry for comparing output frames to be
displayed and for controlling at least one aspect of the way in
which the output frames are provided for display on the display
device on the basis of the comparison.
[0016] According to a third aspect of the present invention, there
is provided a display control apparatus for use in a data
processing system in which successive output frames to be displayed
are generated by the data processing system and written to a frame
buffer for display on a display device, the apparatus
comprising:
[0017] processing circuitry arranged to compare output frames to be
displayed, and to control at least one aspect of the way in which
the output frames are provided for display on the display device on
the basis of the comparison.
[0018] The present invention relates to arrangements in which a
data processing system, such as a graphics processing system,
produces a stream of output frames to be displayed on a display
device, such as a screen. However, in the present invention, one or
more aspects of the way in which the display of output frames on
the display device is carried out is controlled on the basis of a
comparison of output frames to be displayed. As will be discussed
further below, the Applicants have recognised that by comparing
output frames, a measure, for example, of the extent to which
different output frames differ from each other can be derived, and,
moreover, that this information can then be used to advantageously
control the operation for displaying the output frames in use.
[0019] For example (and again as will be discussed further below),
if it is found that the output frames are not changing, the update
or refresh rate of the display device can be reduced without
significantly degrading the image as seen by the user, thereby
reducing the power, etc., that will be consumed by the display
operation.
[0020] Similarly, and again as will be discussed further below, it
would be possible to alter the format of the stored output frame
data (e.g. as between a lower quality format requiring fewer bits
of data and a higher quality format requiring more bits of data)
depending upon the rate at which the display is found to be
changing. This could then be used, e.g., to control the format that
the output frames are stored in in the frame buffer (and hence the
bandwidth needed for the display operations) dynamically in
use.
[0021] Thus, the present invention can be used to reduce
significantly the power consumed and memory bandwidth used for
display operations, in effect by facilitating the identification of
opportunities to perform the display operations in a more efficient
manner.
[0022] Moreover, in contrast to the prior art schemes discussed
above, the present invention does not depend upon any application
or operating system operation, will work for a variety of sources
(and not, e.g., only for specific applications), can provide a more
immediate and dynamic response and control, and can provide greater
power and bandwidth savings than known prior art systems.
[0023] The aspect or aspects of the way in which the output frames
are provided for display that is or are controlled in response to
the output frame comparisons in the present invention can relate to
any suitable and desired aspect of the process for displaying the
output frames. For example, as discussed above, it could (and,
indeed, preferably does) relate to the update or refresh rate of
the display and/or to the format that the output frames are stored
in the frame buffer.
[0024] In a particularly preferred embodiment, the way that output
frames are written to, and/or are read from, the frame buffer is
controlled on the basis of the output frame comparisons. Most
preferably the rate of writing frames to and/or of reading frames
from the frame buffer, and/or the format in which the frames are
written to and/or read from the frame buffer, is controlled on the
basis of the frame comparisons.
[0025] In one particularly preferred embodiment, the update or
refresh rate of the display device (i.e. the rate at which frames
are read from the frame buffer and provided to the display device)
is controlled on the basis of the comparisons of the output frames
that are generated by the data processing system.
[0026] Most preferably, if it is determined by the output frame
comparisons that the output image is not changing or is not
changing very rapidly (e.g., and preferably, at less than a
threshold rate), the display device update (refresh) rate is
reduced, and vice-versa. In this case, when it is determined that
the output frames being generated for display are only changing at
a low rate, the image that is displayed on the display device is
updated at a lower rate, thereby reducing power consumption and
bandwidth for the display operation.
[0027] Most preferably, if it is determined by the output frame
comparison that a new output frame should be considered to be the
same as the current output frame being displayed, then the output
display is not updated, but if it is determined by the output frame
comparison that a new output frame should be considered to be
different to the output frame currently being displayed, then the
output display is updated.
[0028] By controlling the rate that the display device is updated
in this manner, unnecessary updates to the display device can be
avoided or eliminated, thereby saving power and bandwidth in the
display device, the frame buffer memory, the interconnect and the
display controller, etc.
[0029] For example, if one considers first order effects of frame
buffer accesses only, and ignores on-chip interconnect, display
controller, and video output power consumption, reducing the
display refresh rate for an LCD display from 60 fps (frames per
second) to 20 fps using the present invention may save 200 mW and
316 MB/s for HD and 75 mW and 120 MB/s for 1024.times.768
resolution displays.
[0030] It is believed that such arrangements may be new and
advantageous in their own right. Thus, according to a fourth aspect
of the present invention, there is provided a method of operating a
data processing system in which a stream of output frames to be
displayed is generated by the data processing system and written to
a frame buffer for display on a display device, the method
comprising:
[0031] the data processing system comparing output frames to be
displayed, and controlling the rate at which the display device is
updated from the frame buffer on the basis of the comparison.
[0032] According to a fifth aspect of the present invention, there
is provided a data processing system comprising:
[0033] a data processor for generating a stream output frames to be
displayed;
[0034] a frame buffer for storing an output frame to be
displayed;
[0035] a write controller for writing an output frame generated by
the data processor to the frame buffer;
[0036] a display device for displaying an output frame;
[0037] a display controller for reading an output frame from the
frame buffer and for providing it to the display device for
display; and
[0038] processing circuitry for comparing output frames to be
displayed and for controlling the rate at which the display device
is updated from the frame buffer on the basis of the
comparison.
[0039] According to a sixth aspect of the present invention, there
is provided a display control apparatus for use in a data
processing system in which successive output frames to be displayed
are generated by the data processing system and written to a frame
buffer for display on a display device, the apparatus
comprising:
[0040] processing circuitry arranged to compare output frames to be
displayed, and to control the rate at which the display device is
updated from the frame buffer on the basis of the comparison.
[0041] As will be appreciated by those skilled in the art, these
aspects and embodiments of the invention may and preferably do
include any one or more or all of the preferred and optional
features of the invention described herein, as appropriate. Thus,
for example, preferably successive generated output frames are
compared and if they are determined to be the same, the display
device is not updated, and vice-versa.
[0042] In a preferred arrangement of these aspects and embodiments
of the invention, the display operation is configured to always
update the display device at a minimum rate (e.g. to always update
the display after a predetermined time period since the last
update), irrespective of the result of the frame comparisons. This
will help to avoid the displayed image fading away or degrading too
much as a consequence of the operation of the present invention
(and the minimum rate is preferably selected accordingly so as to
achieve this).
[0043] Thus, in a particularly preferred embodiment, the display
device is updated (the frame buffer is read out to the display
device) only when the output frame has been determined to change,
subject to an overall minimum update rate (that is chosen, e.g., to
avoid flicker). This will keep the number of display device updates
to a minimum whilst still ensuring a satisfactory displayed
image.
[0044] A down counter may, for example, be used to trigger the
"minimum rate" display updates.
[0045] It would also be possible to set a maximum update rate, if
desired. This may be useful where, for example, the data processing
system can produce output frames at a rate that is much higher than
is needed for the display device. Again, a "maximum update rate"
down counter could be used to prevent display updates at too high a
rate.
[0046] In a preferred embodiment of these arrangements of the
present invention, the display device's backlight is also
controlled to reduce the perception of flickering at lower update
rates (by adjusting the backlight to keep the display brightness
constant). For example where, as discussed below, only a portion of
the displayed image is updated, the older portions of the image may
have faded, whilst the recently updated portion of the image will
be brighter. The backlight can be controlled to minimise the
disparity between these different portions of the display. Where
the backlight can be controlled independently for different
portions of the display, the backlight may be controlled with finer
granularity to achieve this, if desired.
[0047] Although in one preferred embodiment it is the entire output
frame that is updated (or not) on the display device on the basis
of the output frame comparisons, if the display device supports the
possibility of updating only part (but not all) of the displayed
frame, then the present embodiment can equally be applied in
respect of particular regions or parts of the output display only.
Thus, in a preferred embodiment, the frame comparisons assess
whether a particular part or parts of the output frame have
changed, and then those particular parts of the frame on the
display device are updated (or not) accordingly. This would then
facilitate only updating on the display device those parts or
regions of the displayed frame that have changed, rather than
having to update the whole displayed frame.
[0048] In another particularly preferred embodiment, the way that
the output frame is stored in in the frame buffer (such as, and
preferably, the frame buffer format that is used) is controlled
(selected) on the basis of the output frame comparisons. This may
be instead of or in addition to varying the display device update
rate on the basis of the output frame comparisons.
[0049] In a particularly preferred such embodiment, the output
frame is stored in a relatively higher or a relatively lower
quality format on the basis of the output frame comparisons. Most
preferably a lower quality format is used where the output frame is
determined to be changing a lot and/or relatively rapidly (as in
this case a lower quality format may be acceptable), but a higher
quality format is used when the output frames are not changing so
much and/or are determined to be static (as this will provide a
higher quality display whose quality may be visible on a relatively
static or unchanging display, but not really appreciable where the
display is changing more rapidly).
[0050] In these arrangements, the lower quality format may, e.g.,
be a lossy or more lossy format (e.g. have a higher compression
ratio), whereas the higher quality format may be, e.g., a less
lossy format (have a lower compression ratio) or lossless.
[0051] In one particularly preferred arrangement of these
embodiments of the invention, the lower quality format that is used
is a YUV format or similar (which, as is known in the art, is a
format that is often used for video where the image is constantly
moving), such as YUV 4:2:0, and the higher quality format is an RGB
format or similar, such as RGB 8:8:8 or RGB 10:10:10 (which is a
much higher quality format than YUV 4:2:0 for example, but whose
additional quality tends only really to be visible on static images
such as when viewing a static computing desktop or user interface
(UI)).
[0052] Other possible arrangements would be to use higher and lower
frame buffer resolutions, higher and lower dynamic ranges (colour
depth) for the frame buffer, and other more or less lossy
compression schemes. In the case of rendering the image at a lower
resolution (e.g. when the image is changing rapidly) the display
controller or another system component may then scale the image to
full size for display.
[0053] Another arrangement would be to perform partial frame
updates of the frame in the frame buffer when a lower quality frame
buffer is desired (acceptable). For example, the data processor
could be configured to generate only a part of the new frame each
time, such that, for example, for a first new frame only the top
left portion of the frame is generated, followed a frame later by
the bottom right portion of the frame and so on until the whole
frame has been generated anew. Other suitable such schemes could
generate appropriately only every other line (or column) of the
frame when a new frame is generated. This will reduce the workload
for generating the output frame in the first place, thereby
reducing power consumption and data processor bandwidth.
[0054] For displays that support partial frame updates, portions of
the frame only could be sent to the display, again reducing power
consumption and bandwidth.
[0055] These arrangements of the present invention therefore
effectively facilitate using a lower quality frame buffer format,
thereby reducing power consumption and bandwidth, when there would
be less or no perceived benefit to using a higher quality frame
buffer format (e.g. because the output display is changing rapidly
such that the user will be unable to perceive a great deal of
detail in the displayed image in any event), but reverting to a
higher quality image format (such as RGB) when the additional
quality of the image will actually be appreciated by the viewer
(e.g. because the image is static).
[0056] Thus, by selectively and dynamically switching between
different frame buffer formats based on the comparisons of the
output frames, a system that has, for example, improved quality as
compared to a system that always uses a YUV format frame buffer,
but that reduces memory bandwidth and power consumption as compared
to a system that always uses an RGB format frame buffer, can be
provided.
[0057] It is again believed that these arrangements may be new and
advantageous in their own right. Thus, according to a seventh
aspect of the present invention, there is provided a method of
operating a data processing system in which a stream of output
frames to be displayed is generated by the data processing system
and written to a frame buffer for display on a display device, the
method comprising:
[0058] the data processing system comparing output frames to be
displayed, and selecting the way in which the output frames are
stored in the frame buffer on the basis of the comparison.
[0059] According to an eighth aspect of the present invention,
there is provided a data processing system comprising:
[0060] a data processor for generating a stream of output frames to
be displayed;
[0061] a frame buffer for storing an output frame to be
displayed;
[0062] a write controller for writing an output frame generated by
the data processor to the frame buffer;
[0063] a display device for displaying an output frame;
[0064] a display controller for reading an output frame from the
frame buffer and for providing it to the display device for
display; and
[0065] processing circuitry for comparing output frames to be
displayed and for selecting the way in which the output frames are
stored in the frame buffer on the basis of the comparison.
[0066] According to a ninth aspect of the present invention, there
is provided a display control apparatus for use in a data
processing system in which successive output frames to be displayed
are generated by the data processing system and written to a frame
buffer for display on a display device, the apparatus
comprising:
[0067] processing circuitry arranged to compare output frames to be
displayed, and to select the way in which the output frames are
stored in the frame buffer on the basis of the comparison.
[0068] As will be appreciated by those skilled in the art, these
aspects and embodiments of the invention may and preferably do
include any one or more or all of the preferred and optional
features of the invention described herein, as appropriate. Thus,
for example, the format to be used when storing the output frames
in the frame buffer is preferably selected on the basis of the
frame comparisons, and the system preferably selects between a
lower quality frame buffer format, such as YUV, and a higher
quality format such as RGB.
[0069] In a particularly preferred arrangement of these aspects and
embodiments of the invention, when the output frame comparisons
indicate that the output frame is changing at greater than a
particular, preferably predetermined rate, the output frames are
preferably stored in the frame buffer in a lower quality fashion,
e.g., in a lower quality format, but when the rate of changing of
the output frames drops (is lower) (e.g. and preferably changes at
less than a particular, preferably predetermined rate (which may be
same or a different rate to threshold rate for using a lower
quality frame buffer format)) and/or the image becomes static, the
output frames are stored in the frame buffer in a higher quality
fashion, e.g., format (and vice-versa).
[0070] In these aspects and embodiments of the present invention,
the data processing system and/or data processor could be
controlled to generate its output frames in the desired frame
buffer format (or other fashion) in the first place (for storage
then in the frame buffer in that desired (appropriate) format or
fashion). Alternatively, there could be some form of format
conversion or processing stage or element (circuitry) that converts
the output frames, if necessary, to the desired frame buffer
format, etc., before they are stored in the frame buffer. (For
example, the output frame could be generated in one format by the
data processor but then converted to the desired frame buffer
format before it is stored in the frame buffer).
[0071] In these arrangements, the stored output frame preferably
has associated with it (e.g. stored with it or in association with
it) control information to allow the stored frame in the frame
buffer to be provided in the appropriate format to the display
device (where that is necessary). This control information may,
e.g., indicate what format the frame has been stored in in the
frame buffer, and/or may indicate, or comprise, an algorithm for
converting the data in the frame buffer to the desired final
display format.
[0072] Then, when the frame is read from the frame buffer for
display, this control information is preferably also read (e.g. by
the display controller) to allow the frame buffer format and/or
necessary conversion process to be determined so that the stored
frame can be converted appropriately (and if necessary) to the
desired format for the display device.
[0073] Again, although in one preferred arrangement of these
embodiments of the invention it is the entire output frame that is
stored in the desired frame buffer format, etc., it would equally
be possible, as for the display updating, to apply these techniques
in respect of particular portions or regions of the (stored) output
frame only. Thus, for example, different frame buffer formats could
be used for different portions of a frame (and in one preferred
embodiment this is done), e.g., depending upon the determined rate
of change of the respective portions of the frame.
[0074] Thus, in a preferred embodiment, the frame comparisons
assess whether a particular part or parts of the output frame have
changed, and then the way in which those particular parts of the
frame are stored in the frame buffer (such as the frame buffer
format to be used) is selected accordingly. This would then
facilitate varying the frame buffer format, for example, for those
parts or regions of the displayed frame that have changed, without
having to change the format for the whole frame buffer.
[0075] Other aspects of the writing and reading of the output
frames to and from the frame buffer could also or instead be
controlled on the basis of the comparison of the output frames
generated by the data processing system, if and as desired.
[0076] For example, the level of anti-aliasing (e.g. of
multisampling or supersampling) being used could be varied based on
the output frame comparisons, e.g. to use better anti-aliasing
(thereby giving higher (perceived) image quality) when it is
determined that the image is only changing slowly and/or is static
and vice-versa.
[0077] It would also or instead be possible to vary the rate at
which the output frames are produced by the data processing system
based on the output frame comparisons, for example, to reduce the
rate of output frame generation (thereby reducing power
consumption) when it is determined that the image is only changing
slowly and/or is static and vice-versa.
[0078] The output frames that are generated by the data processing
system can be compared in any suitable and desired manner. As
discussed above, the comparison is preferably so as to determine or
at least estimate whether the frames are changing or have changed,
and/or the rate at which the frames are changing. Any comparison
process that is able to assess this may be used.
[0079] Thus, in a particularly preferred embodiment, the comparison
process is so as to assess or estimate (and is to be used to assess
or estimate) the correlation between successive, and/or sequences
of, output frames generated by the data processing system (i.e. the
extent to which the output frames are similar to each other). Thus,
the comparison is preferably so as to determine whether one frame
is the same as (or at least sufficiently similar to) the other (or
another) frame or not.
[0080] Then, if the comparison process indicates that successive
frames are the same (the correlation is high) that would suggest,
e.g., that the image is static for a period of time (in which case,
as discussed above, it may be possible to reduce the display update
rate, for example), and vice-versa.
[0081] Preferably the comparison process uses some form of
threshold or tide value or values (parameter or parameters) to
determine whether the output frames should be considered to be
changing or not (to be different or to be static) for the purposes
of the control operation of the present invention. Then, if the
comparison indicates a value to one or other side of the
appropriate threshold, the display operation will be controlled
accordingly.
[0082] In a particularly preferred embodiment, the comparison
process compares one output frame with its immediately preceding
output frame (i.e. such that adjacent frames in the stream of
output frames generated by the data processing system are compared
with each other). This is preferably done by comparing a newly
generated output frame with the (appropriate) frame that is
currently stored in the frame buffer.
[0083] Thus, in a preferred embodiment a newly generated output
frame is compared with an output frame already stored in the frame
buffer, most preferably so as to determine whether the new output
frame should be considered to be the same as (or at least
sufficiently similar to) the already stored output frame or
not.
[0084] The comparison is preferably repeated for each new output
frame that is generated, i.e. such that there will be successive
comparisons of pairs of output frames as the output frames are
generated by the data processing system.
[0085] In this case, the system could, e.g., count how many
compared frames are considered to be the same as each other (or to
differ from each other), e.g. in a given period and then compare
that count to a threshold value to determine whether the image
should be considered to be static (unchanging) or not.
[0086] Most preferably, the comparison process determines whether a
respective pair of frames being compared should be considered to be
the same as each other or not, and the display control is then
performed on the basis of that assessment, e.g., to update display
device if the frames are considered to be different but not
otherwise, and so on. This will then provide a dynamic and
"instant" response to the output frames that are being
produced.
[0087] Other arrangements would, of course, be possible. For
example, a series of more than two output frames could be compared
with each other if desired.
[0088] The comparison process preferably compares some or all of
the content of each output frame that is being compared.
[0089] In a particularly preferred embodiment, the comparison is
performed by comparing information representative of and/or derived
from the content of one output frame with information
representative of and/or derived from the other output frame, e.g.,
and preferably, to assess the similarity or otherwise of the output
frames.
[0090] The Applicants have recognised that in many data processing
system arrangements, an output frame for display will be generated
and stored in the frame buffer on a data-block-by-data-block basis,
rather than directly as a single, overall, output "frame".
[0091] This will be the case, for example, and as will be
appreciated by those skilled in the art, in a tile-based graphics
processing system, in which case each block of data that is
generated and stored in the frame buffer may correspond to a "tile"
that the rendering process of the graphics processor produces.
[0092] (As is known in the art, in tile-based rendering, the two
dimensional output array or frame of the rendering process (the
"render target") (e.g., and typically, that will be displayed to
display the scene being rendered) is sub-divided or partitioned
into a plurality of smaller regions, usually referred to as
"tiles", for the rendering process. The tiles (sub-regions) are
each rendered separately (typically one after another). The
rendered tiles (sub-regions) are then recombined to provide the
complete output array (frame) (render target), e.g. for
display.
[0093] Other terms that are commonly used for "tiling" and "tile
based" rendering include "chunking" (the sub-regions are referred
to as "chunks") and "bucket" rendering. The terms "tile" and
"tiling" will be used herein for convenience, but it should be
understood that these terms are intended to encompass all
alternative and equivalent terms and techniques.)
[0094] Where the present invention is implemented in a data
processing system in which the overall, "final" output frame of the
data processing system is stored in the frame buffer on a
block-by-block basis, then in a particularly preferred embodiment
the output frames are compared by comparing respective data blocks
of the output frames. In other words, the comparison process
preferably comprises comparing blocks of data representing
particular regions of the respective output frames with each
other.
[0095] In a particularly preferred embodiment of these arrangements
of the present invention, the data blocks are compared by comparing
a data block that is newly generated by the data processing system
(for a new output frame) with at least one data block of a previous
output frame, e.g., and preferably, that is already stored in the
frame buffer.
[0096] Thus, in a particularly preferred embodiment of the present
invention, the data processing system comprises a data processing
system in which the output frame is stored in the frame buffer by
writing blocks of data representing particular regions of the
output frame to the frame buffer, and the comparison process
comprises comparing a block of data that is to be written to the
frame buffer to at least one block of data already stored in the
frame buffer. This will then preferably be repeated for plural data
blocks making up the respective output frame being generated.
[0097] The blocks of data that are considered and compared in these
embodiments of the present invention can each represent any
suitable and desired region (area) of the overall output frame that
is to be stored in the frame buffer. So long as the overall output
frame is divided or partitioned into a plurality of identifiable
smaller regions each representing a part of the overall output
frame, and that can accordingly be represented as blocks of data
that can be identified and compared, then the sub-division of the
output frame into blocks of data can be done as desired.
[0098] Each block of data preferably represents a different part
(sub-region) of the overall output frame (although the blocks could
overlap if desired). Each block should represent an appropriate
portion (area) of the output frame, such as a plurality of data
positions within the frame. Suitable data block sizes would be,
e.g., 8.times.8, 16.times.16 or 32.times.32 data positions in the
output frame.
[0099] In one particularly preferred embodiment, the output frame
is divided into regularly sized and shaped regions (blocks of
data), preferably in the form of squares or rectangles. However,
this is not essential and other arrangements could be used if
desired.
[0100] Where the data processor is a tile-based graphics processor,
then in a particularly preferred embodiment, each data block that
is compared corresponds to a rendered tile that the graphics
processor produces as its rendering output. This is a particularly
straightforward way of implementing these arrangements of the
present invention, as the graphics processor will generate the
rendering tiles directly, and so there will be no need for any
further processing to "produce" the data blocks that will be
considered and compared. In this case therefore, as each rendered
tile generated by the graphics processor is to be written to the
frame buffer, it will preferably be compared with a rendered tile
or tiles already stored in the frame buffer.
[0101] In these arrangements of the present invention, the
(rendering) tiles that the output frame is divided into for
rendering purposes can be any desired and suitable size or shape.
The rendered tiles are preferably all the same size and shape, as
is known in the art, although this is not essential. In a preferred
embodiment, each rendered tile is rectangular, and preferably
16.times.16, 32.times.32 or 8.times.8 sampling positions in
size.
[0102] In a particularly preferred embodiment of these arrangements
of the present invention, the comparison may be, and preferably is,
also or instead performed using data blocks of a different size
and/or shape to the tiles that the rendering process operates on
(produces).
[0103] For example, in a preferred embodiment, a or each data block
that is considered and compared may be made up of a set of plural
"rendered" tiles, and/or may comprise only a sub-portion of a
rendered tile. In these cases there may be an intermediate stage
that, in effect, "generates" the desired data block from the
rendered tile or tiles that the graphics processor generates.
[0104] In one preferred embodiment, the same block (region)
configuration (size and shape) is used across the entire output
frame. However, in another preferred embodiment, different block
configurations (e.g. in terms of their size and/or shape) are used
for different regions of a given output frame. Thus, in one
preferred embodiment, different data block sizes may be used for
different regions of the same output frame.
[0105] Where the comparison process involves comparing respective
blocks of data, then the comparison is, similarly, preferably so as
to determine whether the new data block is the same as (or at least
sufficiently similar to) the already stored data block or not, and
preferably comprises comparing some or all of the content of the
new data block with some or all of the content of the already
stored data block.
[0106] In a particularly preferred embodiment, the comparison is
performed by comparing information representative of and/or derived
from the content of the new output data block with information
representative of and/or derived from the content of the stored
data block. In such a case, the information representative of the
content of each data block is preferably in the form of a
"signature" for the data block which is generated from or based on
the content of the data block. (Such a data block content
"signature" may comprise, e.g., and preferably, any suitable set of
derived information that can be considered to be representative of
the content of the data block, such as a checksum, a CRC, or a hash
value, etc., derived from (generated for) the data block. Suitable
signatures would include standard CRCs, such as CRC32, or other
forms of signature such as MD5, SHA-1, etc . . . )
[0107] Thus, in a particularly preferred arrangement of these
embodiments of the present invention, a signature, such as a CRC
value, is generated for each data block that is to be written to
the frame buffer (e.g. and preferably, for each output rendered
tile that is generated) and the comparison process comprises
comparing the signatures of the respective data blocks.
[0108] The signatures for the data blocks (e.g. rendered tiles)
that are stored in the frame buffer should be stored appropriately.
Preferably they are stored with the frame buffer. Then, when the
signatures need to be compared, the stored signature for a data
block can be retrieved appropriately.
[0109] The current, completed (newly generated) data block (e.g.
rendered tile) (e.g., and preferably, its signature) can be
compared with one, or with more than one, data block that is
already stored in the frame buffer. It should be compared with the
equivalent data block (or blocks, if appropriate) already stored in
the frame buffer (i.e. the data block(s) in the frame buffer
occupying the same position (the same data block (e.g. tile)
position)) as the completed, new data block is to be written
to).
[0110] In one preferred embodiment, all the data blocks making up
the output frame being considered are compared. However, in other
arrangements some but not all of the blocks are compared. This
could be the case where, for example, only a portion of the output
frame is to be considered, or where, for example, it is known that
a certain portion of the frame is always going to change or be
static (such that that portion doesn't need checking). In the
latter case, information about whether portions of the image are
static or dynamic could, e.g., be provided to the comparison
processing circuitry, for example by the application in
question.
[0111] The former case could occur, for example, where, as
discussed above, the display arrangement supports updating only
part but not all of the displayed frame. In this case the
comparisons could be carried out in respect of particular regions
of the output frame only, i.e. for those parts of the frame that
are being, or that are potentially to be, updated.
[0112] In a particularly preferred embodiment, the number of data
blocks that are compared with a stored data block or blocks for
respective output frames is varied, e.g., and preferably, on a
frame-by-frame, or over sequences of frames, basis. This may be
particularly applicable where, for example, only portions of the
display are to be updated or different regions are to be stored in
different formats. It is preferably based on the expected
correlation (or not) between successive output frames.
[0113] In a preferred embodiment, it is possible for the software
application that is controlling the display to indicate and control
which regions of the frame are compared, e.g., in particular, and
preferably, to indicate which regions of the frame the data
comparison process should be performed for. This would then allow
the comparison calculation to be "turned off" by the application
for regions of the frame the application "knows" will be
static.
[0114] This may be achieved as desired. In a preferred embodiment
registers are provided that enable/disable data block (e.g.
rendered tile) signature comparison for frame regions, and the
software application then sets the registers accordingly (e.g. via
the graphics processor driver). The number of such registers may be
chosen, e.g., as a trade-off between the extra logic required for
the registers, the desired granularity of control, and the
potential savings from being able to disable the comparison
calculations.
[0115] It would also be possible to use information in registers
elsewhere in the system, for example in a graphics processor's
scissor or stencil registers, to determine where comparisons need
to be made, if desired.
[0116] The data block comparisons are preferably used, as discussed
above, to estimate or determine the similarity or not of (the
correlation between) the output frames being compared. This is
preferably estimated on the basis of the number of data blocks that
are found to match (that are considered to match) and/or that are
found to mis-match between the two frames. Most preferably the
number of mismatching (unmatching) data blocks in the frames being
compared is counted and if the number of mis-matching blocks
exceeds a particular, preferably predetermined, threshold value,
the two frames are considered not to match (i.e. the image is
considered to have changed), and vice-versa (as between those two
output frames).
[0117] Thus, in a particularly preferred embodiment, the number of
data blocks found to match or to mismatch in the two output frames
being compared is used as a measure of the correlation of the
frames and to determine how the relevant aspect of the display
operation should be controlled. For example, and preferably, if the
count of mismatching blocks exceeds a threshold value, the display
is updated, but not otherwise. Similarly, if the count of
mismatching blocks is above a threshold value (which would
preferably be a higher value than the threshold to trigger a
display update), then preferably a lower quality (less accurate)
frame buffer format is used.
[0118] The count of mismatching (or matching) data blocks may be
performed as desired, for example by using a standard counter,
using a bitmap that can be analysed, or using a hierarchical count
(e.g. a count per tile, then a level up of a count for a group of
four tiles, and so on, all the way up to a single count per frame).
A hierarchical count would allow portions of the frame that have
matches or mismatches to be searched rapidly.
[0119] In one preferred embodiment, where the number of matching
blocks is below a threshold value, the mismatching blocks are
analysed to determine the magnitude of the visual differences, and
the display operation is controlled on the basis of the determined
magnitude of the visual differences.
[0120] Where the present invention is to be used with a
double-buffered frame buffer, i.e. a frame buffer which stores two
output frames concurrently, e.g. one being displayed and one that
has been displayed and is therefore being written to as the next
output frame to display, then the comparison process of the present
invention preferably compares the newly generated output frame with
the output frame in the frame buffer that is currently being
displayed.
[0121] The comparison process and consequent display control may be
implemented in an integral part of the data processor, or there
may, e.g., be a separate hardware element or elements that perform
these functions.
[0122] In a particularly preferred embodiment, there is a "display
control" hardware element or elements that carry out the comparison
process and control the relevant display operation.
[0123] In one preferred embodiment, this hardware element or
elements is separate to the data processor, and in another
preferred embodiment is integrated in (part of) the data processor.
Thus, in one preferred embodiment, the comparison element etc., is
part of the data processor itself, but in another preferred
embodiment, the data processing system comprises a data processor,
and a separate "display control" unit or units or element or
elements that perform the frame comparison, etc.
[0124] The present invention can be used wherever a data processing
system is providing output frames intended to form an image for
display.
[0125] Similarly, although in one particularly preferred embodiment
the present invention is used in conjunction with a graphics
processing system (and the data processor is a graphics processor),
the principles of the present invention can equally be applied to
other systems that will produce output frames for display in a
similar manner to graphics processing systems. Thus the present
invention may equally be used, for example, for video processing
(and video processing analogously operates on blocks of data
analogous to the tiles in tile-based graphics processing (in which
case the data processor will be a video processor)), and for
composite image processing (again, the composition frame buffer can
be processed as distinct blocks of data).
[0126] In such arrangements, in a similar manner to that discussed
above in relation to tile-based graphics processing systems, the
data blocks that are compared (where that is done) may comprise,
e.g., video data blocks produced by the video processing system (a
video processor), and/or composite frame tiles produced by a
composition processing system, etc.
[0127] The present invention also extends to the provision of a
particular hardware element for performing the comparison and
consequent display control determination of the present invention.
As discussed above, this hardware element (logic) may, for example,
be provided as an integral part of a, e.g., graphics or other
processor, or may be a standalone element. It may be a programmable
or dedicated hardware element.
[0128] Similarly, the frame buffer that the output frames are to be
written to may comprise any suitable such buffer and may be
configured in any suitable and desired manner in memory. For
example, it may be an on-chip buffer or it may be an external
buffer (and, indeed, may be more likely to be an external buffer
(memory), as will be discussed below). Similarly, it may be
dedicated memory for this purpose or it may be part of a memory
that is used for other data as well.
[0129] The display may similarly be any suitable form of display.
In one preferred embodiment it is an LCD display.
[0130] The various functions of the present invention can be
carried out in any desired and suitable manner. For example, the
functions of the present invention can be implemented in hardware
or software, as desired. Thus, for example, the various units and
elements, etc., of the invention may comprise a suitable processor
or processors, functional units, circuitry, processing circuitry,
logic, processing logic, microprocessor arrangements, etc., that
are operable to perform the various functions, etc., such as
appropriately dedicated hardware elements and/or programmable
hardware elements that can be programmed to operate in the desired
manner.
[0131] In a preferred embodiment the data graphics, e.g., processor
and/or display control unit or units is implemented as a hardware
element or elements (e.g. ASIC). Thus, in another aspect the
present invention comprises a hardware element including the
apparatus of, or operated in accordance with the method of, any one
or more of the aspects of the invention described herein.
[0132] It should also be noted here that, as will be appreciated by
those skilled in the art, the various functions, etc., of the
present invention may be duplicated and/or carried out in parallel
on a given processor.
[0133] The present invention is applicable to any suitable form or
configuration of graphics processor and renderer, such as
processors having a "pipelined" rendering arrangement (in which
case the renderer will be in the form of a rendering pipeline). It
is particularly applicable to tile-based graphics processors and
graphics processing systems.
[0134] As will be appreciated from the above, the present invention
is particularly, although not exclusively, applicable to 2D and 3D
graphics processors and processing devices, and accordingly extends
to a 2D and/or 3D graphics processor and a 2D and/or 3D graphics
processing platform including the apparatus of, or operated in
accordance with the method of, any one or more of the aspects of
the invention described herein. Subject to any hardware necessary
to carry out the specific functions discussed above, such a 2D
and/or 3D graphics processor can otherwise include any one or more
or all of the usual functional units, etc., that 2D and/or 3D
graphics processors include.
[0135] It will also be appreciated by those skilled in the art that
all of the described aspects and embodiments of the present
invention can include, as appropriate, any one or more or all of
the preferred and optional features described herein.
[0136] The methods in accordance with the present invention may be
implemented at least partially using software e.g. computer
programs. It will thus be seen that when viewed from further
aspects the present invention provides computer software
specifically adapted to carry out the methods herein described when
installed on data processing means, a computer program element
comprising computer software code portions for performing the
methods herein described when the program element is run on data
processing means, and a computer program comprising code means
adapted to perform all the steps of a method or of the methods
herein described when the program is run on a data processing
system. The data processing system may be a microprocessor, a
programmable FPGA (Field Programmable Gate Array), etc.
[0137] The invention also extends to a computer software carrier
comprising such software which when used to operate a graphics
processor, renderer or microprocessor system comprising data
processing means causes in conjunction with said data processing
means said processor, renderer or system to carry out the steps of
the methods of the present invention. Such a computer software
carrier could be a physical storage medium such as a ROM chip, CD
ROM or disk, or could be a signal such as an electronic signal over
wires, an optical signal or a radio signal such as to a satellite
or the like.
[0138] It will further be appreciated that not all steps of the
methods of the invention need be carried out by computer software
and thus from a further broad aspect the present invention provides
computer software and such software installed on a computer
software carrier for carrying out at least one of the steps of the
methods set out herein.
[0139] The present invention may accordingly suitably be embodied
as a computer program product for use with a computer system. Such
an implementation may comprise a series of computer readable
instructions either fixed on a tangible medium, such as a computer
readable medium, for example, diskette, CD ROM, ROM, or hard disk,
or transmittable to a computer system, via a modem or other
interface device, over either a tangible medium, including but not
limited to optical or analogue communications lines, or intangibly
using wireless techniques, including but not limited to microwave,
infrared or other transmission techniques. The series of computer
readable instructions embodies all or part of the functionality
previously described herein.
[0140] Those skilled in the art will appreciate that such computer
readable instructions can be written in a number of programming
languages for use with many computer architectures or operating
systems. Further, such instructions may be stored using any memory
technology, present or future, including but not limited to,
semiconductor, magnetic, or optical, or transmitted using any
communications technology, present or future, including but not
limited to optical, infrared, or microwave. It is contemplated that
such a computer program product may be distributed as a removable
medium with accompanying printed or electronic documentation, for
example, shrink wrapped software, pre loaded with a computer
system, for example, on a system ROM or fixed disk, or distributed
from a server or electronic bulletin board over a network, for
example, the Internet or World Wide Web.
[0141] A number of preferred embodiments of the present invention
will now be described by way of example only and with reference to
the accompanying drawings, in which:
[0142] FIG. 1 shows schematically a graphics processing system that
can be operated in accordance with the present invention;
[0143] FIG. 2 shows schematically and in more detail a first
embodiment of the comparison and control hardware unit and display
controller shown in FIG. 1;
[0144] FIG. 3 shows schematically how relevant data may be stored
in memory in an embodiment of the present invention;
[0145] FIGS. 4 and 5 show schematically and in more detail a second
embodiment of the comparison and control hardware unit and display
controller shown in FIG. 1.
[0146] Like reference numerals are used for like components
throughout the figures, unless otherwise indicated.
[0147] A number of preferred embodiments of the present invention
will now be described. These embodiments will be described
primarily with reference to the use of the present invention in a
graphics processing system, although, as noted above, the present
invention is applicable to other data processing systems which
produce output frames for display, such as video processing
systems, composite image processing systems, etc.
[0148] Similarly, the following embodiments will be described
primarily with reference to the comparison of output frames by
comparing rendered tiles generated by a tile-based graphics
processor, although again and as noted above, the present invention
is not limited to such arrangements.
[0149] FIG. 1 shows schematically an arrangement of a graphics
processing system that is in accordance with the present
invention.
[0150] The graphics processing system includes, as shown in FIG. 1,
a tile-based graphics processor or graphics processing unit (GPU)
1, which, as is known in the art, will, inter alia, produce tiles
of an output frame intended to be displayed on a display device,
such as a screen or printer.
[0151] As is known in the art, in such an arrangement, once a tile
has been generated by the graphics processor 1, it would then
normally be written to a frame buffer in a memory 2 (which memory
may be DDR-SDRAM) via an interconnect 3 which is connected to a
memory controller 4. Sometime later the frame buffer will, e.g., be
read by a display controller 6 and output to a display 7. In the
present embodiment, the display device 7 is an LCD screen (other
arrangements would, of course, be possible). FIG. 1 also shows for
completeness a host CPU 8.
[0152] In the present embodiment, and in accordance with the
present invention, the above process is modified by the use of a
comparison and control unit 5, which controls one or more aspects
of the way in which the display of the output frames formed by the
tiles generated by the graphics processor 1 is carried out.
[0153] In essence, and as will be discussed further below, the
comparison and control hardware 5 operates to compare successive
output frames that are being generated for display, and then
controls, in the present embodiment, the rate at which the display
device 7 is updated (refreshed) from the frame buffer on the basis
of those comparisons. In effect, the comparison and control
hardware assesses whether successive output frames differ from each
other, and if they do not, the display 7 is not updated (subject to
a predetermined minimum update rate).
[0154] In this way, the present embodiment can avoid relatively
costly updates to the display device 7 when the output frame is not
actually changing from one frame to the next. This can save a
significant amount of bandwidth and power consumption in relation
to the display update operation.
[0155] FIG. 2 shows the comparison and control hardware unit 5 and
the display controller 6 that are used in the present embodiment in
more detail. For clarity purposes, FIG. 2 does not show the
interconnect 3 or the memory controller 4, but it will be
appreciated that the necessary communication between the comparison
and control hardware unit 5, memory 2 and display controller 6
shown in FIG. 2 is performed appropriately via the interconnect 3
and memory controller 4.
[0156] In this embodiment, the comparison and control hardware 5
operates to generate for each tile a signature representative of
the content of the tile and then compares that signature with the
signature of the corresponding tile already stored in the frame
buffer to see if the signatures match. Thus, the output frames are
compared by comparing rendered tiles generated by the graphics
processor 1.
[0157] FIG. 3 shows an exemplary memory layout for storing the
frame buffer and corresponding tile signatures in the memory 2. The
tiles making up the frame are stored in one portion 10 of the
memory 2 (thus forming the "frame buffer") and the associated
signatures for the tiles making up the frame are stored in another
portion 11 of the memory. (Other arrangements would, of course, be
possible.)
[0158] As shown in FIG. 2, tile data 25 received by the comparison
and control hardware unit 5 from the graphics processor 1 is passed
both to a buffer 21 which temporarily stores the tile data while
the signature generation and comparison process takes place, and a
signature generator 20.
[0159] The signature generator 20 operates to generate the
necessary signature for the tile. In the present embodiment the
signature is in the form of a 64-bit CRC for the tile. (Other
signature generation functions and other forms of signature such as
hash functions, etc., could also or instead be used, if desired. It
would also, for example, be possible to generate a single signature
for an RGBA tile, or a separate signature for each colour plane.
Similarly, colour conversion could be performed and a separate
signature generated for each of Y, U and V.)
[0160] Once the signature for the new tile has been generated, it
is passed to a signature comparator 23, which operates to compare
the signature of the new tile with the signature of the tile
already in the frame buffer at the tile position for the tile in
question.
[0161] The signatures for plural tiles from the previous frame are
cached in a signature buffer 22 (this buffer may be implemented in
a number of ways, e.g. buffer or cache) of the comparison and
control hardware unit 5 to facilitate their retrieval in operation
of the system, and so the signature comparator 23 fetches the
relevant signature from the signature buffer 22 if it is present
there (or triggers a fetch of the signature from the main memory 2,
as is known in the art), and compares the signature of the previous
frame's tile with the signature received from the signature
generator to see if there is a match.
[0162] If the signatures are found not to match, then a counter 26
is updated. The counter 26 thus keeps a count of how many tiles
have been found not to match for a respective pair of output frames
being compared. The counter 26 is reset to zero each time a new
output frame is started.
[0163] The count of mis-matching tiles for the current frame that
is being maintained by the counter 26 is compared to a
predetermined threshold count value or tidemark in a comparator 27.
If the count of mis-matching tiles for the output frames being
compared is found to exceed the threshold value, then it is assumed
that the two output frames being compared are different, i.e. the
image has been modified as between the two output frames being
compared, and so an appropriate "frame-modified" indicator (flag)
is set in a register 28. The "frame-modified" indicator is provided
as an input to a control unit 29 of the display controller 6.
[0164] The control unit 29 operates to trigger an update of the
display 7 from the frame buffer in the memory 2 if the
"frame-modified" indicator is in its "set" (i.e. "do an update")
state when the new output frame has been completed in the frame
buffer.
[0165] To facilitate this operation, the frame-modified indicator
is reset whenever the frame buffer is output to the display device
7, and then, as discussed above, changed to its set state when a
sufficient number of tiles have been found to differ between the
two output frames being compared.
[0166] Then, once a new output frame has been finished and is
stored in the frame buffer, the controller 29 checks the state of
the frame-modified indicator, and triggers the updating of the new
output frame from the frame buffer to the display 7 to update the
display if the frame-modified indicator is set to show that the new
frame differs from the previous frame.
[0167] The number of the tiles that must differ to trigger a
determination that the output frames differ (and thus that the
display should be updated) can be set as desired, but should, for
example, comprise a suitably small percentage of the tiles that
make up an output frame.
[0168] In one preferred arrangement it is assumed that the output
frames differ (i.e. a display update is triggered) if one pair of
tiles being compared is found to differ (i.e. such that, in effect,
the threshold number of tiles that must differ to trigger a frame
update is one). In this case, the counter 26 and threshold
comparator 27 could be eliminated, and the comparator 23 could be
configured simply to set the frame-modified indicator to trigger a
display update as soon as it is found that one pair of the tiles
making up the output frames being compared differ from each
other.
[0169] The control unit 29 of the display controller 6 triggers a
display update by providing a "fetch" control signal to a further
control unit 30 that controls the fetching of new frames from the
frame buffer for provision to the display device 7 in dependence
upon the control inputs it receives.
[0170] In this way, the updating of output frames to the display
device 7 is controlled in dependence upon the result of the
comparison of the output frames by the comparison and control
hardware 5 and in particular on the basis of whether the two output
frames are to be considered to be the same or not.
[0171] As shown in FIG. 2, the control unit 30 of the display
controller 6 controls a fetch element 31 which will read an output
frame from the frame buffer in the memory 2 and provide that output
frame via the appropriate display controller logic to the display
device 7 for display. In this way, the control unit 30 controls the
updating (or not) of a new frame to the display 7.
[0172] The control unit 30 receives two inputs that can trigger it
to trigger a frame update to the display 7. The first such input is
the input from the control unit 29 based on the frame-modified
indicator from the register 28. The other input to the controller
30 that can trigger it to cause a display update is from a minimum
update rate down counter 32.
[0173] The minimum update rate down counter 32 is set to a
predetermined value specified in a minimum update rate register 33
each time a frame is output to the display device 7 for display,
and thereafter counts down from that value until a new output frame
is provided to the display device 7 (at which point the minimum
update rate down counter is reset to its initial value). If the
minimum update rate down counter 32 ever reaches zero, it signals
the controller 30 to trigger the writing of an output frame to the
display device 7. In this way, the minimum update rate down counter
32 ensures that the display device 7 is updated at a minimum
refresh rate, irrespective of the results of the output frame
comparisons.
[0174] The display controller may also have a maximum refresh rate
down counter (not shown), which is similarly reset to a
predetermined value whenever an output frame is written to the
display device 7 and then counts down to zero thereafter, with the
display controller being configured to prevent any further updates
to the display device 7 until the maximum refresh rate down counter
has reached zero. This will stop the display device 7 being updated
at too high a rate in a situation where, for example, the graphics
processor 1 is able to generate output frames at much higher rates
than in practice are needed for the display device 7.
[0175] The initial values that minimum update rate down counter 32
and the maximum update rate down counter (where present) are set to
can preferably be set and altered automatically, e.g. and
preferably depending upon whether the system is tethered or battery
operated and upon the condition of the battery. This is preferably
done by providing signalling between the display controller 6 and a
power management unit of the system.
[0176] The display controller signalling logic necessary to
implement the present embodiment may be integrated in the display
controller or external to the display controller, as desired.
[0177] Instead of, as shown in FIG. 2, using counters and logic,
etc., integrated into the comparison and control hardware 5 and
display controller 6, it would be possible to configure the
comparison and control hardware 5 to, for example, generate an
interrupt, and then have the display update rate control and
process performed in software, with the processor (e.g. the CPU 8)
then signalling to the display controller 6 when to update the
display, if desired.
[0178] The backlight of the LCD display device 7 may be controlled
to reduce the effect of flickering (which may occur when low update
rates are being used), e.g. by adjusting the backlight to keep the
display brightness constant as the screen fades away. For devices
with an integrated display, for example, the characteristics of the
display (such as how the colour intensity changes if it isn't
refreshed) can be used to determine how the image will fade and
thus how the backlight should be controlled to reduce flicker.
[0179] As can be seen from the above, in the present embodiment,
the display controller 6 does simply not update the LCD display
device 7 at the usual constant rate of 60-70 Hz. Instead,
differences are detected between the current and subsequent output
frames, and if the frames are determined to be the same, the
display isn't refreshed, subject to still refreshing the display at
a minimum rate to avoid the image fading away or degrading.
[0180] In this way, unnecessary refreshes of the display device 7
are avoided, and when the image is static or only changing at a
relatively slow rate, the image on the display is updated at a
lower rate. This saves power and bandwidth in the display device 7,
the frame buffer memory, the interconnect, and the display
controller 6, etc.
[0181] Furthermore, this is achieved without any modification or
particular operation of the application or operating system, can
provide an immediate response whenever the output frame changes,
and can be used to ensure as lower an update rate as possible
without flickering or slow response.
[0182] As well as operating in the above manner to reduce the
refresh (update) rate of the display device 7, the present
embodiment also operates to reduce the number of frame buffer write
transactions whenever possible. This is achieved by using the tile
signature comparison process to also control whether a new tile is
written to the frame buffer or not.
[0183] In particular, if the signature comparison process
determines that the signatures of the two tiles being compared
match (i.e. the new tile should be considered to be the same as the
corresponding tile (the tile in the same tile position) that is
already stored in the frame buffer), then the new tile is not
written to the frame buffer, but instead the existing tile is
retained in the frame buffer for that tile position.
[0184] In this way, the present embodiment can avoid write traffic
for sections (tiles) of the frame buffer that don't actually change
from one frame to the next. This can save a significant amount of
bandwidth and power consumption in relation to the frame buffer
operation.
[0185] On the other hand, if the signatures do not match, then the
new tile is written to the frame buffer and the generated signature
for the tile is also written to memory.
[0186] This operation is achieved by the signature comparator 23,
if it finds that the tiles` signatures do not match, controlling a
write controller 24 to write the new tile and its signature to the
frame buffer and associated signature data store in the memory 2.
On the other hand, if the signature comparator 23 finds that the
signature of the new tile matches the signature of the tile already
stored in the frame buffer (i.e. that the tiles should be
considered to be the same), then the write controller 24
invalidates the tile and no data is written to the frame buffer
(i.e. the existing tile is allowed to remain in the frame buffer
and its signature is retained).
[0187] In this way, a tile is only written to the frame buffer in
the memory 2 if it is found that by the signature comparison to
differ from a tile that is already stored in the memory 2. This
helps to reduce the number of write transactions to the memory 2 as
a frame is being generated.
[0188] In the present embodiment, to stop incorrectly matched tiles
from existing for too long a long period of time in the frame
buffer, the ability of the signature comparison to prevent a tile
being written to the frame buffer is periodically disabled
(preferably once a second for each stored tile in the frame
buffer). This then means that when a tile for which the ability of
the signature comparison to prevent a tile being written to the
frame buffer has been disabled is newly generated, the newly
generated tile will inevitably be written to the frame buffer in
the memory 2. In this way, it can be ensured that incorrectly
matched tiles will over time always be replaced with completely new
(and therefore correct) tiles in the frame buffer. In the present
embodiment, the ability of the signature comparison to prevent a
tile being written to the frame buffer for the stored tiles is
disabled in a predetermined, cyclic, sequence, so that each second
(and/or over a set of say, 25 or 30 frames), each individual tile
position will always have a new tile written for it once.
[0189] In a particularly preferred embodiment, the tile signatures
that are generated for use in the present invention are "salted"
(i.e. have another number (a salt value) added to the generated
signature value) when they are created. The salt value may
conveniently be, e.g., the output frame number since boot, or a
random value. This will, as is known in the art, help to make any
error caused by any inaccuracies in the comparison process of the
present invention non-deterministic.
[0190] Although in the above embodiment it is the display of the
entire output frame that is updated (or not), the present invention
may also be used to control the updating of only parts of the
output display where the display process supports such operation.
This will be the case, for example, for display interconnects that
support partial frame updates. Equally, in some displays, the LCD
controller can skip lines that have not changed since the last
frame by toggling a horizontal blank signal quickly without sending
the data for the scan line. It would also be possible to modify the
interface between the display and the display controller so as to
be able to update only parts of the output frame that have been
changed, if desired.
[0191] This could be used, for example, for displays with
integrated memory, as these displays will allow modifying parts of
the display image at a time through the bus.
[0192] In this case, the tile comparison process could be used to
assess whether particular regions of the output frame have changed
or not, and then to control the updating of those particular
regions only in the manner of the present embodiment.
[0193] For example, the comparison and control hardware could keep
track of which regions of the display of the output frame have been
modified as between the output frames being compared, and force a
display refresh for those regions that have been modified, but
otherwise refresh the displayed image at a low rate for regions of
the image that remain unmodified. This would be particularly useful
for displays that support partial updates.
[0194] In this case, the tile comparison process need not be
carried out for each and every tile that is generated, but may be
done for the desired regions of the display only, if desired.
[0195] Where the present invention is being used to update only
portions of the display in this fashion, then preferably the
brightness of the portion of the image that is to be updated is
modified (for example by changing the gamma of the image portion in
the display controller, or by controlling the backlight, and/or by
manipulating the colours) so that it has the same brightness as the
rest of the image. This will help to prevent distortions to the
image caused by updating only some but not all of the displayed
image.
[0196] In the above embodiment, the rate at which the display 7 is
updated from the frame buffer is controlled on the basis of the
output frame comparisons. In another preferred embodiment of the
present invention, the comparison and control hardware 5 is
operable to vary the format in which the output frame is stored in
the frame buffer on the basis of the output frame comparisons.
[0197] The Applicants have recognised that if the output frame is
changing rapidly or significantly from frame to frame, the user
viewing the frames will be unable to perceive much detail, whereas
if the image is static, a higher quality image may be desirable.
This second embodiment of the present invention exploits this by
varying the frame buffer format that is used in dependence upon
whether the output frame comparisons indicate that the output
frames are changing significantly or are static.
[0198] FIGS. 4 and 5 show the configurations of the comparison and
control hardware 5 and the display controller 6, respectively, in
this embodiment. (The system of this embodiment is otherwise
configured as shown in FIG. 1.)
[0199] As shown in FIG. 4, in this embodiment, the comparison and
control hardware 5 receives as its input tiles 25 from the graphics
processor 1. Each tile is again then temporarily stored in a buffer
21 and passed to a signature generator 20. The buffer 21 buffers a
complete frame, which frame is then written out as a complete frame
to the frame buffer in the memory 2 when the complete frame is
stored in the buffer 21.
[0200] The signature generated for a new tile in the signature
generator 20 is again passed to a comparator 23, where it is
compared to the signature of the corresponding tile in the frame
buffer (which may be retrieved from a signature buffer 22 or, if
necessary, from the memory 2, as discussed above).
[0201] As in the preceding embodiment, if the comparator 23 finds
that the signatures do not match, it increments a counter 26 which
keeps a count of the number of tiles that have been found not to
match for the output frame in question. This counter 26 is again
reset to zero each time a new output frame is started, and then
counts the number of unmatching tiles for the frame being
processed.
[0202] The count of mis-matching tiles for the current frame is
again compared to a predetermined threshold count value or tidemark
in a comparator 27. If the count of mis-matching tiles for the
output frame being compared is found to exceed the threshold value,
then it is assumed that the two output frames being compared are
significantly different (i.e. the image is changing significantly
as between the two frames). This then triggers the storing of the
output frame in the frame buffer in a lower quality format. This
process will now be described.
[0203] As shown in FIG. 4, the frame that is stored in the buffer
21 may either be passed directly from the buffer 21 to the write
controller 24 that writes the frame to the frame buffer in the
memory 2, or it may be passed to the write controller 24 via an
encoder 40 that is operable to encode the frame from the buffer 21
into a different format.
[0204] In the present embodiment, it is assumed that the tiles are
produced by the graphics processor in an RGB 8:8:8 or RGB 10:10:10
format. This is a higher quality format, but the additional quality
can only really be perceived if the image is static. The encoder 40
therefore operates to convert the frame (tiles) from this format
into a lower quality, lossy compressed, format, in this case YUV
4:2:0 (which is often used for video where the image is constantly
moving).
[0205] The write controller 24 is then controlled to store either
the original, high quality format frame (tiles) directly from the
buffer 21, or the lower quality format frame (tiles) from the
encoder 40 by the threshold comparator 27. In particular, if the
number of mis-matching tiles exceeds the threshold value, the write
controller is triggered to store the output from the encoder 40 in
the frame buffer (i.e. such that the lower quality format is used)
and vice-versa.
[0206] The threshold mis-matching tile value to trigger the use of
the lower quality format can be selected as desired. It should be a
higher value than the value required to trigger a display update in
the previous embodiment.
[0207] As shown in FIG. 4, as well as the output frame itself being
written to the frame buffer (in the selected format), a frame
descriptor unit 41 generates a description or identifier
(meta-data) 42 that indicates the format that the frame is being
stored in. The frame format descriptor 42 is then stored in
association with the frame 43 in the memory 2 to allow the display
controller to identify the format that the frame has been stored in
in the frame buffer (as will be discussed further below). The frame
format descriptor information 42 may be an identifier indicating
the format used, and/or an algorithm indicating how the data should
be converted to the desired display format, etc. Alternatively, the
frame format could be signalled to the display controller 6, if
desired.
[0208] In an alternative arrangement, the comparator 27 could
control the graphics processor 1 to produce its tiles in the lower
quality format directly, where that is required.
[0209] FIG. 5 shows the corresponding display controller operation.
As shown in FIG. 5, when the display controller receives a frame 51
from the frame buffer for display, it first checks the format
descriptor 42 associated with the frame in question in a frame
descriptor decoder 52. As a result of that decoding, it then either
passes the frame in the form it is received via a pass-through path
53 to the display controller logic for display, or, if the input
frame is in its encoded format, passes it to a frame decoder 54 for
converting to the appropriate format for display before again
passing the frame to the display controller logic and the
display.
[0210] The display controller may conveniently, for example, read
the frame format descriptor during the VSYNC period immediately
after the frame buffer swop before then reading and decoding the
actual frame data. This makes it possible to straightforwardly
switch frame buffer formats on-the-fly from frame to frame.
[0211] It can be seen therefore that in this embodiment the
comparison and control hardware 5 is operable to store the frame in
the frame buffer in a desired format together with control
information to indicate which format has been written. The display
controller 6 then reads the information describing the frame buffer
format, reads out the frame buffer data, and, if necessary,
converts the frame buffer format to the desired output format. The
different formats to use are based on the comparison of the output
frames to see whether successive output frames differ significantly
from each other or not.
[0212] This embodiment accordingly allows a lower image quality
frame buffer format to be used when the image is determined to be
changing rapidly (such that the user will be unable to receive a
great deal of detail and therefore a lower image quality will be
acceptable), thereby reducing power consumption and bandwidth, but
while still being able to revert to a higher quality image format
when the image becomes static (and thus the higher quality of the
image can be perceived by the user).
[0213] By being able to dynamically alter the frame buffer format
used based on the rate of change of the image in this way, an
arrangement which improves image quality compared with frame
buffers that always use lower quality format, but which also
reduces memory bandwidth compared to systems which would always use
a higher quality frame buffer format, is provided.
[0214] Although in the above embodiment, the higher quality format
is described as being RGB, and the lower quality format as being
YUV, other arrangements to provide higher and lower quality formats
could be used if desired. For example, other compressed, lossy
formats could be used for the lower quality formats. It would also,
for example, be possible to use different frame buffer resolutions,
and/or dynamic ranges (colour depths), etc., to provide the higher
and lower quality formats, if desired. Where the image is rendered
at a lower resolution, then the display controller or another
system component could scale the image to full size for
display.
[0215] It would also be possible to provide only partial frame
updates, e.g. such that only a region of each new frame is
generated, rather than generating the entire new frame, where a
lower quality format is desired, with the higher quality format
then being the generation of the entire new frame each time.
[0216] Equally, although the above embodiments are described with
reference to selecting a particular format for the entire frame
that is stored in the frame buffer, it would be possible to vary
the format for particular regions of the frame only (i.e. to have
different regions of the frame stored in different formats). This
could be done by using the tile comparisons to identify those
regions of the frame that are changing more significantly and then
using a different format for those regions of the frame only, in a
similar manner to the process discussed above for updating part of
the display only.
[0217] In this case, the buffer 21 could, for example, simply store
part of the frame, or part of the frame that is stored in the frame
buffer 21 could be passed through the encoder 40 for writing to the
frame buffer, with the remainder of the frame not being so-encoded
before it is written to the frame buffer. Similarly, the frame
descriptor 42 could be configured to appropriately indicate the
region or regions of the frame and what format they are encoded
in.
[0218] Although the above embodiment has been described as being an
alternative arrangement to the control of the display update rate
based on the frame comparisons, as will be appreciated by those
skilled in the art, it would be possible to perform both the format
selection and display update rate control based on the frame
comparisons if desired. Thus in a preferred embodiment, the frame
comparisons are used to both control the rate at which the display
is updated and to select the format that the frame is stored in in
the frame buffer. In this case two different mis-matching tile
count thresholds are preferably used, a first, lower threshold to
trigger frame updates, and a second, higher threshold to trigger
the use of a lower quality frame buffer format.
[0219] Although the present invention has been described above with
particular reference to the controlling of the display update rate
and/or the frame buffer format based on the comparisons of the
output frames, the comparisons of the output frames could be used
to control other aspects of the system operation as well or
instead, if desired. For example, if the output frame comparisons
show that the image is static for a period of time, the graphics
processor could be controlled to reduce the rate at which it
produces the output frames, thereby saving power. Similarly, if it
is found by the output frame comparisons that the output image is
static for a period of time, then the system could be controlled to
re-render the frame using better anti-aliasing, thereby increasing
the perceived image quality.
[0220] A number of other alternatives and arrangements of the above
embodiments and of the present invention could be used if
desired.
[0221] For example, it would be possible to provide hardware
registers that enable/disable the tile signature comparisons for
particular frame regions, such that the signature generation and
comparison is only performed for a tile if the register for the
frame region in which the tile resides is set.
[0222] The driver for the graphics processor (for example) could
then be configured to allow software applications to access and set
these tile signature enable/disable registers, thereby giving the
software application the opportunity to control directly whether or
not and where (for which frame regions) the signature generation
and comparisons take place. This would allow a software application
to, for example, control how and whether the signature comparison
is performed. This could then be used, e.g., to eliminate the power
consumed by the signature comparison for a region of the output
frame the application "knows" will be static.
[0223] The number of such registers may chosen, for example, as a
trade-off between the extra logic required implementing and using
them and the desired granularity of control.
[0224] Although the present embodiments have been described above
with particular reference to the comparison of rendered tiles to be
written to the frame buffer, as discussed herein it is not
necessary that the data blocks forming regions of the output data
array that are compared (and e.g. have signatures generated for
them) in the manner of the present invention correspond exactly to
rendered tiles generated by the graphics processor.
[0225] For example, the data blocks that are considered and
compared in the manner of the present invention could be made up of
plural rendered tiles and/or could comprise sub-portions of a
rendered tile. Indeed, different data block sizes may be used for
different regions of the same output array (e.g. output frame)
and/or the data block size and shape could be adaptively changed,
if desired.
[0226] Where a data block size that does not correspond exactly to
the size of a rendered tile is being used, then the comparison and
control hardware unit 5 may conveniently be configured to, in
effect, assemble or generate the appropriate data blocks (and,
e.g., signatures for those data blocks) from the data, such as the
rendered tiles, that it receives from the graphics processor (or
other processor providing it data for an output array).
[0227] Although the present embodiments have been described above
with particular reference to a graphics processing system and the
generation of output frames by a graphics processor, the Applicants
have recognised that the frame comparison and control processes of
the present invention could equally be used for other systems that
generate output frames for display, particularly where the output
frame data is processed in blocks in a manner similar to the tiles
of a tile-based graphics processor, such as a video processor
(video codec) producing video blocks for a video frame buffer, and
for graphics processor image composition. Thus the processes of the
present invention may be applied equally to the image that is
being, for example, generated by a video processor.
[0228] It can be seen from the above that the present invention, in
its preferred embodiments at least, can help to reduce, for
example, display operation power consumption and memory
bandwidth.
[0229] This is achieved, in the preferred embodiments of the
present invention at least, by identifying opportunities to use
lower display update rates and/or less expensive frame buffer
formats, etc., by comparing output frames to determine whether the
output frames are changing or not. Then, for example, if the output
frame is only changing slowly or is static, the display is updated
at a lower rate, thereby reducing power consumption and bandwidth,
and if the output frame is changing rapidly, the image quality is
reduced, thereby again reducing power consumption and
bandwidth.
[0230] The present invention, in its preferred embodiments at
least, can significantly reduce the frame buffer and display
controller power consumption and frame buffer bandwidth
requirements. For example, when the display update rate is being
controlled, the frame buffer access power saving is expected to be
around 50% when the image is static.
[0231] For example, a 32-bit mobile DDR-SDRAM transfer consumes
about 2.4 nJ per 32-bit transfer. Thus, assuming a display update
rate of 60 Hz, display refresh transactions will consume about
(1920.times.1080.times.4).times.2.4 nJ/4.times.60=300 mW and 474
MB/s for HD graphics at 60 fps and
(1024.times.768.times.4).times.(2.4 nJ/4).times.60=112 mW and 180
MB/s for 1024.times.768 graphics at 60 fps.
[0232] Considering just the first order effect of frame buffer
accesses and ignoring on-chip interconnect, display control and
video output power consumption, etc., if the display refresh rate
is reduced from 60 fps to 20 fps, that will save of the order of
200 mW and 316 MB/s for HD graphics and 75 mW and 120 MB/s for
1024.times.768 graphics.
[0233] Thus the present invention is particularly applicable for
use with lower powered devices and devices where power consumption
is important. It is therefore particularly applicable for use in or
with mobile and portable devices.
* * * * *