U.S. patent application number 17/349807 was filed with the patent office on 2021-12-16 for systems, methods and computer readable medium for visual software development quality assurance.
The applicant listed for this patent is Engineer.ai Naya Limited. Invention is credited to Sachin Dev Duggal, Siddhartha Ghosh, Rohan Patel, Marco Quaglio, Joseph Rifkin.
Application Number | 20210390032 17/349807 |
Document ID | / |
Family ID | 1000005838938 |
Filed Date | 2021-12-16 |
United States Patent
Application |
20210390032 |
Kind Code |
A1 |
Quaglio; Marco ; et
al. |
December 16, 2021 |
SYSTEMS, METHODS AND COMPUTER READABLE MEDIUM FOR VISUAL SOFTWARE
DEVELOPMENT QUALITY ASSURANCE
Abstract
A computer-implemented method for identifying discrepancies
between a design image of a user interface for an application and a
screenshot of the user interface as displayed by the application
includes performing a first comparison between the design image and
the screenshot to identify one or more discrepancies between the
images, excluding from the discrepancies those corresponding to
visual elements on the screenshot that include dynamic content, and
generating an image of the screenshot, wherein the image includes a
visual indication of every discrepancy detected by the second
comparison as the identified discrepancies between the design image
and the screenshot.
Inventors: |
Quaglio; Marco; (London,
GB) ; Ghosh; Siddhartha; (London, GB) ;
Duggal; Sachin Dev; (London, GB) ; Patel; Rohan;
(London, GB) ; Rifkin; Joseph; (Los Angeles,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Engineer.ai Naya Limited |
Los Angeles |
CA |
US |
|
|
Family ID: |
1000005838938 |
Appl. No.: |
17/349807 |
Filed: |
June 16, 2021 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
63039064 |
Jun 15, 2020 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 11/366 20130101;
G06T 2200/24 20130101; G06K 9/6215 20130101; G06T 2207/30168
20130101; G06T 7/001 20130101 |
International
Class: |
G06F 11/36 20060101
G06F011/36; G06T 7/00 20060101 G06T007/00; G06K 9/62 20060101
G06K009/62 |
Claims
1. A quality assurance system for visual software development,
comprising: a quality assurance application implemented on a
computer using computer readable software instructions stored in
non-transient memory, and configured for identifying discrepancies
between a design image of a user interface for an application and a
screenshot of the user interface as displayed by the application,
wherein the quality assurance application is further configured to
perform computer-implemented steps comprising: performing a first
comparison between the design image and the screenshot to identify
one or more discrepancies between the images; excluding from the
discrepancies those corresponding to visual elements on the
screenshot that include dynamic content, the excluding comprising:
identifying which of the discrepancies are structural
discrepancies; applying a mask to every visual element
corresponding to a structural discrepancy on both the design image
and the screenshot, wherein the mask is shaped like the visual
element; performing a second comparison between the masked design
image and masked screenshot, wherein a lack of discrepancies
detected by the second comparison between a masked visual element
on the design image and the corresponding masked visual element on
the screenshot indicates that the visual element on the screenshot
includes dynamic content; and generating an image of the
screenshot, wherein the image includes a visual indication of every
discrepancy detected by the second comparison as the identified
discrepancies between the design image and the screenshot.
2. The system of claim 1, wherein the steps further comprise
generating and displaying a discrepancy map showing areas of
discrepancy between the design image and the screenshot as shaded
areas.
3. The system of claim 1, wherein performing the first or the
second comparison comprises traversing the design image and the
screenshot using an SSIM analysis on every pixel.
4. The system of claim 1, wherein the one or more discrepancies
comprise color patches, missing elements, and structural
discrepancies.
5. The system of claim 1, wherein each of color patches, missing
elements, and structural discrepancies is identified based on
different combinations of local luminance similarity, local
contrast similarity, and local structure similarity.
6. The system of claim 1, wherein applying the mask comprises
applying a contrast-based mask than applies a shadow to regions of
the visual element corresponding to the structural discrepancy
where contrast is higher than a small value.
7. The system of claim 1, further comprising generating a bug
report that includes an inventory of the identified discrepancies,
their location, and measures of the divergence of their
corresponding visual elements from the design image.
8. A computer-implemented method for identifying discrepancies
between a design image of a user interface for an application and a
screenshot of the user interface as displayed by the application,
the method comprising: performing a first comparison between the
design image and the screenshot to identify one or more
discrepancies between the images; excluding from the discrepancies
those corresponding to visual elements on the screenshot that
include dynamic content, the excluding comprising: identifying
which of the discrepancies are structural discrepancies; applying a
mask to every visual element corresponding to a structural
discrepancy on both the design image and the screenshot, wherein
the mask is shaped like the visual element; performing a second
comparison between the masked design image and masked screenshot,
wherein a lack of discrepancies detected by the second comparison
between a masked visual element on the design image and the
corresponding masked visual element on the screenshot indicates
that the visual element on the screenshot includes dynamic content;
and generating an image of the screenshot, wherein the image
includes a visual indication of every discrepancy detected by the
second comparison as the identified discrepancies between the
design image and the screenshot.
9. The method of claim 8, further comprising generating and
displaying a discrepancy map showing areas of discrepancy between
the design image and the screenshot as shaded areas.
10. The method of claim 8, wherein performing the first or the
second comparison comprises traversing the design image and the
screenshot using an SSIM analysis on every pixel.
11. The method of claim 8, wherein the one or more discrepancies
comprise color patches, missing elements, and structural
discrepancies.
12. The method of claim 11, wherein each of color patches, missing
elements, and structural discrepancies is identified based on
different combinations of local luminance similarity, local
contrast similarity, and local structure similarity.
13. The method of claim 8, wherein applying the mask comprises
applying a contrast-based mask than applies a shadow to regions of
the visual element corresponding to the structural discrepancy
where contrast is higher than a small value.
14. The method of claim 8, further comprising generating a bug
report that includes an inventory of the identified discrepancies,
their location, and measures of the divergence of their
corresponding visual elements from the design image.
Description
[0001] This application claims the benefit of U.S. Provisional
Patent Application No. 63/039,964, filed on Jun. 16, 2020, which is
incorporated herein in its entirety.
FIELD OF THE INVENTION
[0002] Embodiments of the present invention are directed to
software development.
[0003] Specifically, embodiments of the present invention are
directed to systems and methods for providing visual quality
assurance of software applications under development.
BACKGROUND
[0004] User Interface (UI) layout and functionality is an important
aspect of software development. The software development process
can include a step in which one or more display screens are
generated on a device to provide a graphical visual interface.
These are generated using executable code to provide the front end
(whether static or dynamic). The visual elements of the display
screen generated by the software executable can be graphical
elements generated for example by a software module. The visual
elements can be images that are retrieved for an image file. Other
visual elements can be fonts, colors, icons, menu structures,
buttons, etc. The software development process is, in a standard
way for development, structured where a designer generates visual
images (e.g., static images) such as in an image file as the visual
design that the executable software is to deliver when the software
is developed and running. A good UI facilitates the use and
enjoyment of an application can be dispositive of its success.
Significant resources are poured into optimal UI design that
effectively combine aesthetics and functionality. To ensure that
the design is rendered correctly by the application, quality
assurance processes are used to compare the design with the UI as
rendered by the software.
[0005] Quality Assurance (QA) for User Interfaces represents an
important step in the software development cycle. The process
involves the comparison of original designs (i.e. what the designer
and the customer agreed on), with the respective UIs (i.e. what the
developer implemented in the front-end and ultimately appears on
the screen of a computing device). The QA process is primarily
carried out by patient and meticulous QA testers. Needless to say,
it can be tedious. Indeed, visual quality check of the output of
the software development process can be time consuming, difficult,
and imprecise. In addition, the QA process for designs is sensitive
to human judgement and the outcomes can vary from individual to
another. Further, the software application often times is developed
for multiple platforms, devices, screen sizes (or resolution), or
operating environments. For example, one starting display screen
(or portion thereof) may have many corresponding application
implementations (e.g., IOS, Android, Windows, etc.). Known QA
processes are not capable of assisting in software development with
tools that address such complex situations or provide related
features.
[0006] Systems and methods have been proposed to automate or
otherwise facilitate the QA process for UIs. In particular, systems
exist that implements the ability to automatically compare the
original design of a UI with the actual UI as displayed by an
application in order to identify and correct deficiencies in the
application's rendering. However, such systems present several
deficiencies. For example, existing systems are unable to detect
dynamic content in their analyses of UIs. This inability to account
for parts of the screen that change (e.g., animation, widgets, etc)
generates false positives during automatic comparisons between
designs and displayed UIs.
[0007] What is desired is a system and method for providing quality
assurance of UIs that is more accurate than a human tester and can
operate at a pixel level. It is also desired to provide a system
and method that can detect and account for dynamic content on the
screen and minimize the generation of false positives.
SUMMARY OF THE INVENTION
[0008] Embodiment of the present invention disclose a quality
assurance system for visual software development. The quality
assurance system includes a quality assurance application
implemented on a computer using computer readable software
instructions stored in non-transient memory, and configured for
identifying discrepancies between a design image of a user
interface for an application and a screenshot of the user interface
as displayed by the application, wherein the quality assurance
application is further configured to perform computer-implemented
steps including: performing a first comparison between the design
image and the screenshot to identify one or more discrepancies
between the images; excluding from the discrepancies those
corresponding to visual elements on the screenshot that include
dynamic content, the excluding including: identifying which of the
discrepancies are structural discrepancies; applying a mask to
every visual element corresponding to a structural discrepancy on
both the design image and the screenshot, wherein the mask is
shaped like the visual element; performing a second comparison
between the masked design image and masked screenshot, wherein a
lack of discrepancies detected by the second comparison between a
masked visual element on the design image and the corresponding
masked visual element on the screenshot indicates that the visual
element on the screenshot includes dynamic content; and generating
an image of the screenshot, wherein the image includes a visual
indication of every discrepancy detected by the second comparison
as the identified discrepancies between the design image and the
screenshot.
[0009] In some embodiments, the steps further include generating
and displaying a discrepancy map showing areas of discrepancy
between the design image and the screenshot as shaded areas.
[0010] In some embodiments, performing the first or the second
comparison includes traversing the design image and the screenshot
using an SSIM analysis on every pixel.
[0011] In some embodiments, the one or more discrepancies include
color patches, missing elements, and structural discrepancies. In
some of these embodiments, each of color patches, missing elements,
and structural discrepancies is identified based on different
combinations of local luminance similarity, local contrast
similarity, and local structure similarity.
[0012] In some embodiments, applying the mask includes applying a
contrast-based mask than applies a shadow to regions of the visual
element corresponding to the structural discrepancy where contrast
is higher than a small value.
[0013] In some embodiments, the system further includes generating
a bug report that includes an inventory of the identified
discrepancies, their location, and measures of the divergence of
their corresponding visual elements from the design image.
[0014] Embodiments of the present invention also disclose a
computer-implemented method for identifying discrepancies between a
design image of a user interface for an application and a
screenshot of the user interface as displayed by the application.
The method includes performing a first comparison between the
design image and the screenshot to identify one or more
discrepancies between the images; excluding from the discrepancies
those corresponding to visual elements on the screenshot that
include dynamic content, the excluding including: identifying which
of the discrepancies are structural discrepancies; applying a mask
to every visual element corresponding to a structural discrepancy
on both the design image and the screenshot, wherein the mask is
shaped like the visual element; performing a second comparison
between the masked design image and masked screenshot, wherein a
lack of discrepancies detected by the second comparison between a
masked visual element on the design image and the corresponding
masked visual element on the screenshot indicates that the visual
element on the screenshot includes dynamic content; and generating
an image of the screenshot, wherein the image includes a visual
indication of every discrepancy detected by the second comparison
as the identified discrepancies between the design image and the
screenshot.
[0015] In some embodiments, the method further includes generating
and displaying a discrepancy map showing areas of discrepancy
between the design image and the screenshot as shaded areas.
[0016] In some embodiments, performing the first or the second
comparison includes traversing the design image and the screenshot
using an SSIM analysis on every pixel.
[0017] In some embodiments, the one or more discrepancies include
color patches, missing elements, and structural discrepancies. In
some of these embodiments, each of color patches, missing elements,
and structural discrepancies is identified based on different
combinations of local luminance similarity, local contrast
similarity, and local structure similarity.
[0018] In some embodiments, applying the mask includes applying a
contrast-based mask than applies a shadow to regions of the visual
element corresponding to the structural discrepancy where contrast
is higher than a small value.
[0019] In some embodiments, the method further includes generating
a bug report that includes an inventory of the identified
discrepancies, their location, and measures of the divergence of
their corresponding visual elements from the design image.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Various features of examples and embodiments in accordance
with the principles described herein may be more readily understood
with reference to the following detailed description taken in
conjunction with the accompanying drawings, where like reference
numerals designate like structural elements, and in which:
[0021] FIG. 1 illustrates a system for providing visual quality
control of user interfaces according to an embodiment of the
present invention.
[0022] FIG. 2 illustrates a block diagram of an exemplary system
for automatically detecting and identifying discrepancies in a user
interface of a software application, according to an embodiment of
the present invention.
[0023] FIG. 3 illustrates a visual interface of the system of an
embodiment of the present invention.
[0024] FIG. 4 illustrates a discrepancy map as a combination of a
structure map, contrast map, and luminance map according to an
embodiment of the present invention.
[0025] FIG. 5A illustrates a design-UI pair of images featuring
dynamic content 580 according to an embodiment of the present
invention.
[0026] FIG. 5B illustrates the masked out design-UI pair according
to an embodiment of the present invention.
[0027] FIG. 5C illustrates close ups of visual element on each of
the design and screenshot images, before and after masking,
according to an embodiment of the present invention.
[0028] FIG. 5D illustrates before and after masking details of
element on each of the design and UI images, according to an
embodiment of the present invention.
[0029] FIG. 6 illustrates a graph visualization of a test or an
assessment of UI rendering.
[0030] Certain examples and embodiments have other features that
are one of in addition to and in lieu of the features illustrated
in the above-referenced figures. These and other features are
detailed below with reference to the above-referenced figures.
DETAILED DESCRIPTION OF THE INVENTION
[0031] Embodiments of the present invention, provide for a software
development application that receives and loads an image that is a
designer's illustration of a display screen (or a portion thereof),
which may have been entirely or partially manually prepared by an
illustrator on an illustration software application, and receives
and loads a corresponding display screen that is generated by the
software application under development (in which the designer
illustrated images are a specified component of the design for the
software application under development).
[0032] The display screen can be a screen shot (screen capture) or
image generated (e.g., automatically generated) from the software
generated display screen. The software development application
applies an algorithm in which the software development application
traverses the contents of the image of the designer illustration
image and the contents of the display screen (e.g., in image
capture form). The traversal can sequential, simultaneously, or
some combination thereof. One or more algorithms can be applied as
part of the traversal that identifies the discrepancies between the
illustrated image and the software generated display screen. One or
more algorithms can be applied to identify discrepancies as
qualifying as bugs for reporting to the software development team.
One or more algorithms can be applied that automatically based on
the received and identified discrepancies provide an adjustment to
the software implementation of the display screen to remove or
reduce the discrepancy.
[0033] A visual interface can be generated that provides a visual
illustration of the discrepancies which can be generated using
different parameters from the algorithm. The visual interface of
generated image can provide intuitive tools for a software
developer, design illustrator, or a client to understand and in
response, decide to take action (or automated action based on the
output can be applied to the code).
[0034] FIG. 1 illustrates a system 100 for providing visual quality
control of user interfaces according to an embodiment of the
present invention. The system includes a software application 120
installed on an electronic device 130 and quality control
application 140 implemented on a server 150. The electronic device
130 is preferably a desktop computer that can communicate with
server via mobile networks or other wireless networks. Each of the
electronic device and servers is a computer system that includes a
microprocessor and volatile and non-volatile memory to configure
the computer system. The computer system 130 also includes a
network connection interface that allows the computer system 130 to
communicate with another computer system over a network. Each
software application may also be used independently or in
conjunction with another software application to strengthen or
supplement functionalities of the other software application.
[0035] The quality control application 140 is configured to provide
a visual interface that enables a customer to interact with the
system. In some embodiments, the quality control application is a
web application provided by the server and configured to run in a
conventional browser 120. The visual interface may thus be
displayed in a conventional browser 120 on the client computing
device 130.
[0036] FIG. 2 illustrates an example block diagram of a system 200
for automatically detecting and identifying discrepancies in a user
interface of a software application, according to an embodiment of
the present invention. The system 200 receives as inputs a pair of
images, a first image 210 representing the original design of a
user interface, and a second image 220 being an image of the actual
user interface, that is the application's render of the original
design 210 of the user interface. The original design image 210 or
reference image 210 is the image of the user interface as conceived
before or during the development of the application by graphic
artists or developers, and is the model for the user interface
rendered by the application. The design image 210 includes one or
more visual elements 215. The reference image 210 may be generated
through a variety of means, such as a graphic design application, a
CAD application, or any other means of generating a picture of the
intended design. The user interface image 220 or application render
220 is preferably a screenshot of the application. The user
interface 220 may also otherwise have been generated or rendered by
the application, provided that it is an exact representation of the
user interface 220 as displayed on the screen. Each of original
design image 210 or reference image 210 and the actual
application's image or screenshot 220 can be in a variety of image
file formats. These include Windows Bitmap (bmp), Portable Image
Formats (pbm, pgm, ppm), Sun Raster (sr, ras), JPEG (jpeg, jpg,
jpe), JPEG 2000 (jp2), TIFF files (tiff, tif), Portable Network
Graphics (png), among others.
[0037] The pair of images 210, 220 is first received and processed
by a normalization engine 230. The normalization engine 230 is
configured to format or convert the images 210, 220 into a format
that enables comparison. For example, the normalization engine 230
compares the pixel dimensions of the original design image 210 and
the screenshot 220. If the dimensions are not identical, the
screenshot 220 is resized to match the dimensions of the original
design image 210. The normalization engine 230 may also change the
format of an image to match the other or of both images to match to
match the preferred image format of the normalization engine 230.
In some embodiments, images are converted into PNG format. The
converted design image 210 and the screenshot 220 are then sent to
the comparison engine 240.
[0038] The comparison engine 240 is first configured to locate and
detect discrepancies between the received image and the application
screenshot 220. To that end, the comparison engine 240 compares the
received design image 210 and screenshot 220 to generate a
discrepancy map 250. FIG. 3 illustrates a visual interface 300 of
the system of the present invention. The visual interface 300
provided by the application enables the user to interact with the
system. In some embodiments, the application my run on a server as
a web app, and the visual interface of the application may be
displayed in a conventional browser on a client computing device
(see FIG. 1). The visual interface 300 displays the intended design
image 310 and the application screenshot 320 both previously
uploaded to the system as described above. Each of the images 310
and 320 exhibits visual elements 305. The visual interface further
displays a discrepancy map 330.
[0039] The discrepancy map 330 is an image that shows the result
comparison between the two images 310, 320. In particular, the
discrepancy map 320 shows locations 335 where the comparison engine
has detected some difference between the design image 310 and the
corresponding screenshot 320 of the application in development.
Thus, the parts of the images that are identical are blank on the
map, and the parts where differences exist between the reference
image 310 and the actual user interface 320 are shaded 335. The
darkness of the shading 335 in any one location reflects the
severity of the discrepancy at that location.
[0040] The comparison engine may generate the discrepancy map 330
using a variety of image comparison techniques. In one embodiment,
the discrepancy map is generated by performing a pixel-wise
comparison between original image and the UI image. Specifically,
the comparison engine traverse the content of the images 310, 320
using a Structural Similarity Index Method (SSIM) algorithm on
every pixel of the images to obtain the discrepancy map 330. SSIM a
statistical method to comprehensively evaluate differences between
images and is described in Wang, Zhou & Bovik, Alan &
Sheikh, Hamid & Simoncelli, Eero. (2004). Image Quality
Assessment: From Error Visibility to Structural Similarity. Image
Processing, IEEE Transactions on. 13. 600-612.
10.1109/TIP.2003.819861, which is incorporated herein by reference
in its entirety. In this embodiment, the method as implemented
employs a sliding window over the entire frame to calculate a local
SSIM for each pixel. The SSIM analysis yields three measures of
local similarity for each pixel: luminance similarity, contrast
similarity, and structure similarity. Each pixel similarity measure
can serve as a basis for a map, and the luminance map, contrast
map, and structure map can be generated. In some embodiments, the
user may select to display each of the maps. The combination or
superimposition of all three maps yields the discrepancy map. In
other words, the discrepancy is made of pixels each having three
dimensions or similarity measure: luminance, contrast, and
structure. FIG. 4 illustrates a discrepancy map 400 as a
combination of a structure map 410, contrast map 420, and luminance
map 430.
[0041] Referring back to FIG. 3, in some embodiments,
user-controlled parameters are provided that affect the detection
and mapping of discrepancies 335. For example, the scope parameter
336 determines the resolution of the detector: a sharp scope should
be used to highlight discrepancies in small elements in the
design-UI pair, such as characters, small icons. A blurred scope
may be employed to highlight discrepancies in big elements in the
design-UI pair, such as entire words and paragraphs, logos,
buttons, images, etc. Threshold value 337 for the map is another
parameter that may be provided to the user. The threshold value
determines the level of similarity between the design image and the
screenshot that will be flagged by the comparison engine as a
discrepancy. Threshold thus varies between lenient and strict.
Slides or other control means are provided on the user interface to
enable the user to change each parameter.
[0042] The system uses combinations of the measures of local
similarity to identify three different types of discrepancies from
the discrepancy map. A first type of discrepancy is color patches,
which are identified by areas of extreme structure similarity but
low luminance or contrast similarity. Missing or spurious elements
constitute a second type of discrepancy, which is indicated by high
contrast. Any other discrepancies that do not fall satisfy the
color patch or missing/spurious element criteria tend to indicate
generic structural or geometrical discrepancies, which indicate
discrepancies in the location and size of the visual elements of
user interfaces.
[0043] The comparison engine is configured to compile the location
and type of discrepancies between the original image and the
application image into a bug report 260 (see FIG. 2). The bug
report 260 provides an inventory of every visual element from the
original design that has changed or differs from the intended
design. The location of each visual element may be recorded using
any of a number of location or coordinate systems that are known in
the art of image processing and analysis. Further, the bugs or
visual elements that differ from the design image are visually
identified on the application screenshot. In the embodiment
illustrated in FIG. 3, visual elements that differ from the those
of the design image are surrounded with boxes 340.
[0044] Referring back to FIG. 2, a dynamic content detector 270
next processes the images in order to account for the dynamic
elements in the user interface. Dynamic content may be a
combination of text/images that are dynamically generated by some
backend process and cannot be expected to match pixel by pixel the
text and artworks used by the designer. Unlike static visual
element, dynamic elements or dynamic content can generate false
positive in the comparison engine because the dynamic visual
element on a screenshot of the actual UI is likely to differ from
the static picture of the original design. For example, a dynamic
visual element such as an animation becomes a static picture on a
screenshot having a different visual configuration than the
original visual element on the design image. This discrepancy would
generate an erroneous bug report about the visual element by the
comparison engine without the specialized dynamic content process
of the present invention. The dynamic content detector 270 serves
to identify the dynamic visual elements of the user interface.
[0045] Dynamic content detection is only applied to the visual
elements that have been identified as structural or geometrical
discrepancies. The dynamic content detector 270 is applied to such
elements to determine whether the dynamic content has caused the
mismatch. In particular, the dynamic content detector 270 applies a
mask or filter to cover all structural discrepancy elements in the
area under diagnosis (i.e., where the suspected mismatch between
the original image and the screenshot occurred). Effectively, this
filter replaces the suspect elements in both the original design
image and the screenshot with dark patches broadly having the
shapes of the elements under review. In particular, the mask
applies a shadow to regions of the bug or discrepancy where
contrast is higher than a small positive value, which are detected
as elements. Areas of where no contrast is detected are considered
background. The dark patches enable the comparison engine 240 to
focus on location and shape/size of the structural discrepancy
elements rather than the specific text/image content associated
with the element. If the comparison engine 240 does not detect any
discrepancy in the area under review after the elements under
review are masked, the comparison engine 240 will assume the
design-screenshot difference is associated with dynamic content and
will not be raised as bug to the user.
[0046] FIG. 5A illustrates a design-UI pair of images 510, 520
featuring dynamic content 580. Visual element 530, 540, and 550
differ between the images. Static element 530 is smaller in the
screenshot 510 than in the design image 520 and therefore will
trigger a bug report when processed by the comparison engine
according to the steps described above. Similarly, elements 540 are
flagged by the comparison engine because they differ between the
images as they represent different numbers. In additional, elements
550 also trigger the comparison engine because the content of the
image windows 550 has varied between the design image 510 and the
actual user interface 520.
[0047] Next, the dynamic content detector applies a mask to each of
the elements flagged by the comparison engine as discrepancies on
both images. FIG. 5B illustrates the masked out design-UI pair 510,
520 according to an embodiment of the present invention. Each mask
560 is shaped like the element and comprises a shadow or black
pixels. FIG. 5C illustrates close ups of visual element 530 on each
of the design and screenshot images, before and after masking
(i.e., covered with mask 560 in the bottom images). FIG. 5D
similarly illustrates before and after masking details of element
540 on each of the design and UI images. As illustrated, the masks
560 approximate the shapes of the elements and cover them
completely.
[0048] After masks 560 are applied to both images, the comparison
engine processes the masked-out images anew to detect
discrepancies. The masked out images are traversed pixel-wise
according to the process described above and the visual element
exhibiting discrepancies are recorded. If any visual element
previously detected on the initial pair of design-UI images (before
the masking) is still flagged as a discrepancy by the comparison
engine after being masked, that element is identified as a real bug
in the actual UI. This is because any element flagged after masking
exhibit geometric or structural discrepancies that are reflected in
the masks. In other words, the size, shape, orientation, or
location of the mask for that element differ between the design
image and the rendered image, indicating a structural discrepancy
independent of any dynamic content. This is the case for element
530 illustrated in FIG. 5C, where the masked element 530, 560 is
smaller in the masked screenshot than in the masked design image.
In contrast, elements that were previously detected as
discrepancies on the first pass through the comparison engine but
that the second round after masking did not flag are identified as
dynamic content. This is the case for elements 540 illustrated in
FIG. 5D, where the masked element 540, 560 retains the same
structure (size, shape, etc.) on both the design image and the
screenshot image. Dynamic content such as visual element 540 as
confirmed by the comparison engine is excluded from further
analysis. Though not described in reference to element 550, the
same process described above would yield the exclusion of that
element as dynamic content as well due to the changing photos
exhibited by the screenshot compared to the design image.
[0049] After the identifying and excluding dynamic interface
elements (e.g., 540), the system can proceed to the full analysis
or diagnostic of the remaining discrepancies. The full analysis
stage provides additional information to the user regarding visual
elements that exhibit differences from the design image. For
example, the full analysis can reveal three types of geometrical
discrepancies or measures of divergence between design and UI can
be identified from the discrepancy map using the combination of
similarity measures described above: element shift (visual elements
misplaced by a few pixels), element dimensions (when the size of an
element is wrong), and element mass (an expression of discrepancy
in the element's pixel count). The discrepancy detection, location,
and analysis provided by the discrepancy comparison engine of the
present invention provides an unprecedented level of precision in
evaluating UI interface errors. In particular, the comparison
engine measures and record divergences of each element that differs
between the original and image and the application UI image down to
the pixel. The detailed discrepancy information can be compiled in
the bug report, which can be provided to the application
development team responsible for the application to enable
adjustments thereof.
[0050] In some embodiments, the user may select an element
identified as exhibiting a discrepancy and in response the visual
interface of the application may display measures of the
divergences of the element from the source image (e.g., shift,
dimensions, pixel count). For example, the application may indicate
that an icon or a bloc of text is shifted 150 pixels to the left of
its position on the design image and that is 27% smaller than the
equivalent element on the design image. The application may enable
the user to manually correct the divergence or discrepancy by
selecting and manipulating the element (moving, enlargement, etc.)
on the screenshot.
[0051] In some embodiments, one or more algorithms can be applied
that automatically based on the received and identified
discrepancies provide an adjustment to the software implementation
of the display screen to remove or reduce the discrepancy. The
visual interface of generated image can provide intuitive tools for
a software developer, design illustrator, or a client to understand
and in response, decide to take action (or automated action based
on the output can be applied to the code).
[0052] Embodiments of the present invention further disclose
methods of measuring aspect of visual performance of UI (such as
page load and UI rendering times) as perceived by an end user. FIG.
6 illustrates a graph visualization 600 of a test or an assessment
of UI rendering. The test includes providing a visual element 610
or an object containing text on the UI (such as an information
bubble), and causing the object to move across the screen and
assessing generally whether the object behaves as intended, and in
particular whether the motion of the element is too fast to be read
by a user. For example, as illustrated in FIG. 6, the quality
control software causes the object to travel from the top of the
screen and decelerate until stop in the middle of the screen, where
the application awaits user input to continue the test. Upon
indication from the user (such as clicking a button provided for
this purpose by the application), the object 610 accelerates thru
the bottom of the screen (the X axis). The quality control
application of the present invention is configured to assess
whether the object has behaved as intended.
[0053] To assess the behavior of the object 610, first an edge
detection algorithm is applied record the relative position of the
object through its journey down the screen. Corresponding
datapoints for the object are recorded as it moves, which include
the coordinates on the graph (pixel points x.sub.1,y.sub.1 to
x.sub.n,y.sub.n), associated time stamps for each coordinate, and
other information such as expected pixel density, screen
resolution, and physical screen size. Using this object datapoints
and screen information, the application can calculate the speed and
acceleration perceived by an end user viewing a screen of specific
size and resolution (for example, the application may calculate
that a 55 in, 4K screen yields an acceleration of 2 cm/s.sup.2 for
a certain object made to move at a certain speed). In some
embodiments, motions of multiple objects can be measured. In some
embodiments, the testing application may be configured to use edge
detection and pattern recognition to identify the performance of
transient objects as they appear and leave the screen.
[0054] In general, it should be understood that embodiments apply
to systems, methods, and computer readable medium.
[0055] Inventions are described herein and are claimed by way of
explicit and implicit disclosure as would be understood by those of
ordinary skill in the art.
[0056] The application may be implemented on a system, server,
computing device, or computer. A system, server, computing device,
and computer can be implemented on one or more computer systems and
be configured to communicate over a network.
[0057] The computer system also includes a main memory, such as a
random access memory (RAM) or other dynamic storage device, coupled
to bus for storing information and instructions to be executed by a
processor of one or more computers of the computer system. Main
memory also may be used for storing temporary variables or other
intermediate information during execution of instructions to be
executed by a processor. Such instructions, when stored in
non-transitory storage media accessible to processor, configure the
computer system into a special-purpose machine that is customized
to perform the operations specified in the instructions and provide
or be capable of features and functionality described herein.
[0058] The computer system further includes a read only memory
(ROM) or other static storage device coupled to bus for storing
static information and instructions for processor. A storage
device, such as a magnetic disk or optical disk, is provided and
coupled to bus for storing information and instructions.
[0059] The computer system may be coupled via bus to a display,
such as an LCD, for displaying information to a computer user. An
input device, including alphanumeric and other keys, may be coupled
to bus for communicating information and command selections to
processor. Another type of user input device is cursor control,
such as a mouse, a trackball, touchscreen (e.g., on mobile phones)
or cursor direction keys for communicating direction information
and command selections to processor and for controlling cursor
movement on display. This input device typically has two degrees of
freedom in two axes, a first axis (e.g., x) and a second axis
(e.g., y), that allows the device to specify positions in a
plane.
[0060] The computer system may implement the techniques described
herein using customized hard-wired logic, one or more ASICs or
FPGAs, firmware and/or program logic which in combination with the
computer system causes or programs computer system to provide
specialized features. According to one embodiment, the techniques
herein are performed by the computer system in response to the
processor executing one or more sequences of one or more
instructions contained in main memory. Such instructions may be
read into main memory from another storage medium, such as storage
device. Execution of the sequences of instructions contained in
main memory causes the processor to perform the process steps
described herein. In alternative embodiments, hard-wired circuitry
may be used in place of or in combination with software
instructions.
[0061] The term storage media as used herein refers to any
non-transitory media that stores data and/or instructions that
cause a machine to operation in a specific fashion. Such storage
media may comprise non-volatile media and/or volatile media.
Non-volatile media includes, for example, optical or magnetic
disks, such as storage device. Volatile media includes dynamic
memory, such as main memory. Common forms of storage media include,
for example, hard disk, solid state drive, or any other magnetic
data storage medium, a CD-ROM, any other optical data storage
medium, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other
memory chip or cartridge.
[0062] Storage media is distinct from but may be used in
conjunction with transmission media. Transmission media
participates in transferring information between storage media. For
example, transmission media includes coaxial cables, copper wire
and fiber optics, including the wires that comprise bus.
Transmission media can also take the form of acoustic or light
waves, such as those generated during radio-wave and infra-red data
communications.
[0063] Various forms of media may be involved in carrying one or
more sequences of one or more instructions to the processor for
execution. For example, the instructions may initially be carried
on a magnetic disk or solid state drive of a remote computer. A
hardware bus carries the data to main memory, from which processor
retrieves and executes the instructions. The instructions received
by main memory may optionally be stored on storage device either
before or after execution by the processor.
[0064] The computer system also includes a communication interface
coupled to bus. The communication interface provides a two-way data
communication coupling to a network link that is connected to a
local network. For example, the communication interface may be an
integrated services digital network (ISDN) card, cable modem,
satellite modem, or a modem to provide a data communication
connection to a corresponding type of telephone line. As another
example, the communication interface may be a local area network
(LAN) card to provide a data communication connection to a
compatible LAN. Wireless links may also be implemented. In any such
implementation, the communication interface sends and receives
electrical, electromagnetic or optical signals that carry digital
data streams representing various types of information.
[0065] Network link typically provides data communication through
one or more networks to other data devices. For instance, network
link may provide a connection through local network to a host
computer or to data equipment operated by an Internet Service
Provider (ISP). ISP in turn provides data communication services
through the worldwide packet data communication network now
commonly referred to as the "Internet." Local network and Internet
both use electrical, electromagnetic or optical signals that carry
digital data streams. The signals through the various networks and
the signals on network link and through the communication
interface, which carry the digital data to and from the computer
system, are example forms of transmission media.
[0066] Hardware and software implementation is also illustratively
described or understood from the attached Visual QA document and
the incorporated patent application applications mentioned
above.
[0067] The computer system can send messages and receive data,
including program code, through the network(s), network link and
the communication interface. In the Internet example, a server
might transmit a requested code for an application program through
Internet, ISP, local network and the communication interface.
[0068] The received code may be executed by the processor as it is
received, and/or stored in storage device, or other non-volatile
storage for later execution.
[0069] It should be understood that variations, clarifications, or
modifications are contemplated. Applications of the technology to
other fields are also contemplated.
[0070] Exemplary systems, devices, components, and methods are
described for illustrative purposes. Further, since numerous
modifications and changes will readily be apparent to those having
ordinary skill in the art, it is not desired to limit the invention
to the exact constructions as demonstrated in this disclosure.
Accordingly, all suitable modifications and equivalents may be
resorted to falling within the scope of the invention.
[0071] Thus, for example, any sequence(s) and/or temporal order of
steps of various processes or methods that are described herein are
illustrative and should not be interpreted as being restrictive.
Accordingly, it should be understood that although steps of various
processes or methods or connections or sequence of operations may
be shown and described as being in a sequence or temporal order,
but they are not necessarily limited to being carried out in any
particular sequence or order. For example, the steps in such
processes or methods generally may be carried out in various
different sequences and orders, while still falling within the
scope of the present invention. Moreover, in some discussions, it
would be evident to those of ordinary skill in the art that a
subsequent action, process, or feature is in response to an earlier
action, process, or feature.
[0072] It is also implicit and understood that the applications or
systems illustratively described herein provide
computer-implemented functionality that automatically performs a
process or process steps. This can involve human-software
interaction such as if desired by selecting images or selecting
controls for the process that automatically carried out.
[0073] It is understood from the above description that the
functionality and features of the systems, devices, components, or
methods of embodiments of the present invention include generating
and sending signals to accomplish the actions.
[0074] It should be understood that claims that include fewer
limitations, broader claims, such as claims without requiring a
certain feature or process step in the appended claim or in the
specification, clarifications to the claim elements, different
combinations, and alternative implementations based on the
specification, or different uses, are also contemplated by the
embodiments of the present invention.
[0075] It should be understood that combinations of described
features or steps are contemplated even if they are not described
directly together or not in the same context.
[0076] The terms or words that are used herein are directed to
those of ordinary skill in the art in this field of technology and
the meaning of those terms or words will be understood from
terminology used in that field or can be reasonably interpreted
based on the plain English meaning of the words in conjunction with
knowledge in this field of technology. This includes an
understanding of implicit features that for example may involve
multiple possibilities, but to a person of ordinary skill in the
art a reasonable or primary understanding or meaning is
understood.
[0077] It should be understood that the above-described examples
are merely illustrative of some of the many specific examples that
represent the principles described herein. Clearly, those skilled
in the art can readily devise numerous other arrangements without
departing from the scope of the present invention.
* * * * *