U.S. patent application number 15/671392 was filed with the patent office on 2018-03-22 for code coverage through overlay hooks.
The applicant listed for this patent is International Business Machines Corporation. Invention is credited to Michael S. Fulton, Kevin A. Stoodley.
Application Number | 20180081792 15/671392 |
Document ID | / |
Family ID | 61620397 |
Filed Date | 2018-03-22 |
United States Patent
Application |
20180081792 |
Kind Code |
A1 |
Fulton; Michael S. ; et
al. |
March 22, 2018 |
CODE COVERAGE THROUGH OVERLAY HOOKS
Abstract
In an approach for utilizing overlay hooks to increase code
coverage, a processor inserts an overlay hook in program code at a
location within the program code corresponding to a condition
statement. A processor executes the program code. Upon reaching the
overlay hook, a processor branches to a set of instructions
operative to document an outcome of the condition statement. A
processor executes the condition statement. A processor records the
outcome of the condition statement.
Inventors: |
Fulton; Michael S.;
(Squamish, CA) ; Stoodley; Kevin A.; (Richmond
Hill, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
International Business Machines Corporation |
Armonk |
NY |
US |
|
|
Family ID: |
61620397 |
Appl. No.: |
15/671392 |
Filed: |
August 8, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
15268670 |
Sep 19, 2016 |
|
|
|
15671392 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 11/3676 20130101;
G06F 11/3624 20130101; G06F 11/362 20130101; G06F 11/302 20130101;
G06F 11/36 20130101; G06F 11/3644 20130101; G06F 11/3688
20130101 |
International
Class: |
G06F 11/36 20060101
G06F011/36 |
Claims
1. A method for utilizing overlay hooks to increase code coverage,
the method comprising: compiling, by one or more processors, a
program code with branches broken apart to test only one possible
condition state at each condition statement; analyzing, by one or
more processors, debug information to identify a location within
the program code corresponding to a condition statement;
exchanging, by one or more processors, the condition statement for
an overlay hook at the location within the program code; executing,
by one or more processors, the program code; upon reaching the
overlay hook, branching, by one or more processors, to a set of
instructions operative to document an outcome state of the
condition statement, wherein the outcome state of the condition
statement is selected from the group consisting of true, false,
less than, equal to, and greater than; executing, by one or more
processors, the condition statement; recording, by one or more
processors, the outcome state of the condition statement and the
location of the condition statement within the program code in a
table; accessing, by one or more processors, the table;
determining, by one or more processors, whether each possible
outcome state of the condition statement has been recorded in the
table; responsive to determining that each possible outcome state
of the condition statement has not been recorded in the table,
continuing to execute the program code; responsive to determining
that each possible outcome state of the condition statement has
been recorded in the table, removing, by one or more processors,
the overlay hook; and generating, by one or more processors, a
report, wherein the report includes the recorded outcome state of
the condition statement and the location within the program code
corresponding to the condition statement.
Description
BACKGROUND
[0001] The present invention relates generally to the field of code
coverage, and more particularly to runtime-directed overlay hooks
being used to track code coverage.
[0002] Code coverage is a measure used to describe the degree to
which the program code of a program is tested by a particular test
suite. A program with high code coverage has been more thoroughly
tested and has a lower chance of containing software bugs than a
program with low code coverage. To measure what percentage of code
has been exercised by a test suite, one or more coverage criteria
are used: function coverage, statement coverage, branch coverage,
and condition coverage. Function coverage requires that each
function in the program be called. Statement coverage requires each
statement in the program to be executed. Branch coverage requires
each branch of each control structure to be executed. Condition
coverage requires each Boolean sub-expression to be evaluated both
to true and false.
[0003] A combination of function coverage and branch coverage is
sometimes also called decision coverage. This criterion requires
that every point of entry and exit in the program have been invoked
at least once, and every decision in the program have taken on all
possible outcomes at least once. Modified condition/decision
coverage (MC/DC) requires that both decision and condition coverage
be satisfied and that each condition should affect the decision
outcome independently.
SUMMARY
[0004] Aspects of an embodiment of the present invention disclose a
method for utilizing overlay hooks to increase code coverage. A
processor inserts an overlay hook in program code at a location
within the program code corresponding to a condition statement. A
processor executes the program code. Upon reaching the overlay
hook, a processor branches to a set of instructions operative to
document an outcome of the condition statement. A processor
executes the condition statement. A processor records the outcome
of the condition statement.
BRIEF DESCRIPTION OF THE DRAWINGS
[0005] FIG. 1 is a functional block diagram illustrating a code
testing environment, in accordance with an embodiment of the
present invention;
[0006] FIG. 2 is a flowchart depicting operational steps of an
overlay hook program, on a computing device within the environment
of FIG. 1, in accordance with an embodiment of the present
invention; and
[0007] FIG. 3 depicts a block diagram of components of the
computing device executing the overlay hook program, in accordance
with an embodiment of the present invention.
DETAILED DESCRIPTION
[0008] Embodiments of the present invention recognize the
importance of achieving high code coverage when testing the program
code of a program. Using modified condition/decision coverage
(MC/DC), it is impossible to tell if a test is covering all the
possible conditions for a section of code. For example, if a line
of code reads--if (a<=b) then {b=5} else {b=1}, MC/DC code
coverage will only show that a<=b and a>b were tested.
However, it cannot be determined whether a<b or a=b caused the
`if` to execute for the a<=b instance. Thus, there is a need for
a code coverage method that covers these edge conditions in
testing. Embodiments of the present invention provide solutions for
this inability to determine whether each condition has actually
been tested. In this manner, as discussed in greater detail herein,
embodiments of the present invention can provide finer-grained
coverage of conditional paths through code by tracking all possible
condition codes through the use of overlay hooks.
[0009] The present invention will now be described in detail with
reference to the Figures.
[0010] FIG. 1 depicts a diagram of code testing environment 10, in
accordance with an embodiment of the present invention. FIG. 1
provides only an illustration of one embodiment and does not imply
any limitations with regard to the environments in which different
embodiments may be implemented.
[0011] In the depicted embodiment, code testing environment 10
includes computing device 100. Code testing environment 10 may
include additional computing devices, servers, computers, mobile
devices, or other devices not shown. Computing device 100 may be a
management server, a web server, laptop computer, tablet computer,
netbook computer, personal computer (PC), a desktop computer, a
smart phone, or any programmable electronic device or computing
system capable of running a program and receiving and sending data.
In other embodiments, computing device 100 may represent a server
computing system utilizing multiple computers as a server system,
such as in a cloud computing environment. In the depicted
embodiment, computing device 100 contains overlay hook program 110,
condition table 120, and debug information 130. Computing device
100 may include components, as depicted and described in further
detail with respect to FIG. 3.
[0012] Overlay hook program 110 operates as a runtime debugger that
executes overlay hooks inserted in the program code to track code
coverage by recording each condition outcome as the condition
outcome is set. An overlay hook is a segment of code that places a
hook into a program to transfer control to a debugger and overlays
the current instruction with a new instruction. When inserting an
overlay hook, overlay hook program 110 creates a location that can
be branched to, and at that location, overlay hook program 110
saves the condition statement that is overlaid by the overlay hook.
The overlay hook lines of code are inserted directly on top of
where the condition statement was in the program code. For example,
overlay hook program 110 would replace a condition statement
instruction "Compare A & B" with "Branch to comparison test,"
which is the overlay hook. The "comparison test" has instructions
to execute the condition, record the condition outcome, check if
all condition outcomes have been recorded, and return to the
program code to continue executing. It should be appreciated that
the actual overlay hook sequence will vary by operating system,
compiled language, hardware instruction set, and assembler.
[0013] Generally, overlay hook program 110 runs alongside a
debugger executing the program code. When the debugger begins
executing the program code, overlay hook program 110 initiates
running alongside the debugger. When an overlay hook is reached in
the program code, overlay hook program 110 executes the overlay
hook in the program code. Overlay hook program 110 determines
whether to remove the overlay hook by checking condition table 120
to see if each possible condition outcome has executed. If each
possible condition outcome has executed, overlay hook program 110
removes the overlay hook, reinserts the original condition
statement, and continues executing the program code. If each
possible condition outcome has not executed, overlay hook program
110 leaves the overlay hook within the program code and continues
executing the program code. When the program code has been executed
completely, overlay hook program 110 generates a report from
condition table 120. Overlay hook program 110 has access to
condition table 120 and updates condition table 120 with table
entries of the location and outcome of a condition statement. In
the depicted embodiment, overlay hook program 110 resides on
computing device 100. In another embodiment, overlay hook program
110 may reside elsewhere within code testing environment 10
provided overlay hook program 110 has access to debug information
130 and condition table 120. Overlay hook program 110 is described
in further detail with respect to FIG. 2.
[0014] Condition table 120 is updated by overlay hook program 110
with table entries of each possible condition outcome from a
condition statement in program code as each possible condition
outcome is set. Condition table 120 is a mapping table in which
each entry includes a map to the location of the condition
statement in the program code and the state of the condition once
set. The state of the condition is the condition code register, or
its equivalent depending on the instruction set architecture (ISA),
such as high, low, less than or equal to, less than, equal to,
greater than, greater than or equal to, etc. In the depicted
embodiment, condition table 120 resides on computing device 100. In
another embodiment, condition table 120 may reside elsewhere within
code testing environment 10 provided overlay hook program 110 has
access to condition table 120.
[0015] Debug information 130 is used by overlay hook program 110 to
locate where conditions are in the program code allowing overlay
hook program 110 to insert an overlay hook at each condition. Debug
information 130 is generated as an object file when compiling code
with a static compiler. Debug information 130 includes side
information associated with the program code that gives mapping
information from actual individual instructions back to the program
code. For example, "instruction A" was generated to support the
original source level statement on line 57 of "source file X." In
the depicted embodiment, debug information 130 resides on computing
device 100. In another embodiment, debug information 130 may reside
elsewhere within code testing environment 10 provided overlay hook
program 110 has access to debug information 130.
[0016] FIG. 2 depicts flowchart 200 of the steps of overlay hook
program 110, executing within code testing environment 10 of FIG.
1, in accordance with an embodiment of the present invention.
Overlay hook program 110 prepares a program code and then executes
inserted overlay hooks until each possible condition outcome has
executed.
[0017] In step 210, overlay hook program 110 prepares the program
code. Before the program code is executed, overlay hook program 110
prepares the program code as described below according to different
embodiments.
[0018] In a first embodiment, overlay hook program 110 analyzes
debug information 130 to identify where condition statements are
located in the program code and inserts an overlay hook at each
condition statement.
[0019] In a second embodiment, overlay hook program 110 compiles
the program code for use in MC/DC code coverage analysis. Overlay
hook program 110 breaks apart branches to test only one possible
condition outcome at a time. For example, the condition statement
"if (a<=b) then {b=5} else {b=1}" would be broken apart to "if
(a<b) then {b=5} else {b=1}" and "if (a=b) then {b=5} else
{b=1}." These broken apart branches create simple condition
statements in which each condition statement in the program code
only has one possible condition outcome, i.e., one condition in a
condition code register. Overlay hook program 110 analyzes debug
information 130 for where condition statements are in the program
code. Overlay hook program 110 inserts a simple overlay hook at
each simple condition statement that includes instructions to
update condition table 120 with the condition statement's location
when the condition statement executes. When an overlay hook is
reached in the program code, overlay hook program 110 executes the
overlay hook in the program code, which involves overlay hook
program 110 recording the location of the condition statement in
condition table 120. Overlay hook program 110 removes the overlay
hook and continues executing the program code. When the program
code has been executed completely, overlay hook program 110
generates a report from condition table 120.
[0020] In a third embodiment, overlay hook program 110 is a
combination of the first and second embodiments. Overlay hook
program 110 analyzes debug information 130 for where condition
statements are in the program code. Overlay hook program 110
inserts a simple overlay hook for a simple condition statement or
inserts a complex overlay hook for a complex condition statement. A
simple condition statement only has one possible condition outcome,
i.e. one condition in a condition code register. A complex
condition statement has multiple possible outcomes, i.e. multiple
possible conditions in a condition code register and perhaps
multiple condition code registers, if the instruction set
architecture supports multiple condition code registers. A simple
overlay hook records its location in condition table 120 and then
the simple overlay hook is eliminated. A complex overlay hook
operates the same as an overlay hook described in paragraphs
[0011-0012].
[0021] In step 220, overlay hook program 110 reaches an overlay
hook. In an embodiment, overlay hook program 110 runs alongside a
debugger executing the program code. When the debugger begins
executing the program code, overlay hook program 110 initiates
running alongside the debugger. In an embodiment, overlay hook
program 110 runs alongside the debugger and reaches an overlay hook
in the program code and executes the overlay hook.
[0022] In step 230, overlay hook program 110 records a condition
outcome. In the first embodiment, once the condition statement has
executed, overlay hook program 110 records the outcome of the
condition and the location of the condition as table entries in
condition table 120. In the second embodiment, once the simple
condition statement has executed, overlay hook program 110 records
the location of the condition as a table entry in condition table
120. In the third embodiment, for a complex condition statement
that is executed, overlay hook program 110 records the outcome of
the condition and the location of the condition as table entries in
condition table 120. In the third embodiment, for a simple
condition statement that is executed, overlay hook program 110
records the location of the condition as a table entry in condition
table 120.
[0023] In decision 240, overlay hook program 110 determines whether
each possible condition outcome has been recorded. For example, if
a condition statement is "less than or equal to," the condition
statement has multiple possible outcomes. Overlay hook program 110
cross-references with condition table 120 to determine if all the
possible condition outcomes have been recorded. If overlay hook
program 110 determines that all the possible condition outcomes
have been recorded (decision 240, yes branch), then overlay hook
program 110 eliminates the overlay hook and continues executing the
program code (see step 260). If overlay hook program 110 determines
that not all the possible condition outcomes have been recorded
(decision 240, no branch), then overlay hook program 110 returns to
the program code (see step 250).
[0024] In step 250, overlay hook program 110 continues executing
the program code. In an embodiment, overlay hook program 110
determines that not all the possible condition outcomes have been
recorded, therefore overlay hook program 110 keeps the overlay hook
at the condition statement and continues executing the program code
after the overlay hook.
[0025] In step 260, overlay hook program 110 removes the overlay
hook and continues executing the program code. In an embodiment,
overlay hook program 110 determines that all the possible condition
outcomes have been recorded, therefore, overlay hook program 110
removes the overlay hook from the condition statement and continues
executing the program code after the overlay hook.
[0026] In step 270, overlay hook program 110 generates a report. In
an embodiment, overlay hook program 110 generates a report of the
condition states and condition statement locations recorded in
condition table 120.
[0027] FIG. 3 is a block diagram depicting components of a computer
300 suitable for executing the overlay hook program 110. FIG. 3
displays the computer 300, the one or more processor(s) 304
(including one or more computer processors), the communications
fabric 302, the memory 306, the cache 316, the persistent storage
308, the communications unit 310, the I/O interfaces 312, the
display 320, and the external devices 318. It should be appreciated
that FIG. 3 provides only an illustration of one embodiment and
does not imply any limitations with regard to the environments in
which different embodiments may be implemented. Many modifications
to the depicted environment may be made.
[0028] As depicted, the computer 300 operates over a communications
fabric 302, which provides communications between the cache 316,
the computer processor(s) 304, the memory 306, the persistent
storage 308, the communications unit 310, and the input/output
(I/O) interface(s) 312. The communications fabric 302 may be
implemented with any architecture suitable for passing data and/or
control information between the processors 304 (e.g.
microprocessors, communications processors, and network processors,
etc.), the memory 306, the external devices 318, and any other
hardware components within a system. For example, the
communications fabric 302 may be implemented with one or more buses
or a crossbar switch.
[0029] The memory 306 and persistent storage 308 are computer
readable storage media. In the depicted embodiment, the memory 306
includes a random access memory (RAM). In general, the memory 306
may include any suitable volatile or non-volatile implementations
of one or more computer readable storage media. The cache 316 is a
fast memory that enhances the performance of computer processor(s)
304 by holding recently accessed data, and data near accessed data,
from memory 306.
[0030] Program instructions for overlay hook program 110 may be
stored in the persistent storage 308 or in memory 306, or more
generally, any computer readable storage media, for execution by
one or more of the respective computer processors 304 via the cache
316. The persistent storage 308 may include a magnetic hard disk
drive. Alternatively, or in addition to a magnetic hard disk drive,
the persistent storage 308 may include, a solid state hard disk
drive, a semiconductor storage device, read-only memory (ROM),
electronically erasable programmable read-only memory (EEPROM),
flash memory, or any other computer readable storage media that is
capable of storing program instructions or digital information.
[0031] The media used by the persistent storage 308 may also be
removable. For example, a removable hard drive may be used for
persistent storage 308. Other examples include optical and magnetic
disks, thumb drives, and smart cards that are inserted into a drive
for transfer onto another computer readable storage medium that is
also part of the persistent storage 308.
[0032] The communications unit 310, in these examples, provides for
communications with other data processing systems or devices. In
these examples, the communications unit 310 may include one or more
network interface cards. The communications unit 310 may provide
communications through the use of either or both physical and
wireless communications links. Overlay hook program 110 may be
downloaded to the persistent storage 308 through the communications
unit 310. In the context of some embodiments of the present
invention, the source of the various input data may be physically
remote to the computer 300 such that the input data may be received
and the output similarly transmitted via the communications unit
310.
[0033] The I/O interface(s) 312 allows for input and output of data
with other devices that may operate in conjunction with the
computer 300. For example, the I/O interface 312 may provide a
connection to the external devices 318, which may include a
keyboard, keypad, a touch screen, and/or some other suitable input
devices. External devices 318 may also include portable computer
readable storage media, for example, thumb drives, portable optical
or magnetic disks, and memory cards. Software and data used to
practice embodiments of the present invention may be stored on such
portable computer readable storage media and may be loaded onto the
persistent storage 308 via the I/O interface(s) 312. The I/O
interface(s) 312 may similarly connect to a display 320. The
display 320 provides a mechanism to display data to a user and may
be, for example, a computer monitor.
[0034] The programs described herein are identified based upon the
application for which they are implemented in a specific embodiment
of the invention. However, it should be appreciated that any
particular program nomenclature herein is used merely for
convenience, and thus the invention should not be limited to use
solely in any specific application identified and/or implied by
such nomenclature.
[0035] The present invention may be a system, a method, and/or a
computer program product at any possible technical detail level of
integration. The computer program product may include a computer
readable storage medium (or media) having computer readable program
instructions thereon for causing a processor to carry out aspects
of the present invention.
[0036] The computer readable storage medium can be a tangible
device that can retain and store instructions for use by an
instruction execution device. The computer readable storage medium
may be, for example, but is not limited to, an electronic storage
device, a magnetic storage device, an optical storage device, an
electromagnetic storage device, a semiconductor storage device, or
any suitable combination of the foregoing. A non-exhaustive list of
more specific examples of the computer readable storage medium
includes the following: a portable computer diskette, a hard disk,
a random access memory (RAM), a read-only memory (ROM), an erasable
programmable read-only memory (EPROM or Flash memory), a static
random access memory (SRAM), a portable compact disc read-only
memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a
floppy disk, a mechanically encoded device such as punch-cards or
raised structures in a groove having instructions recorded thereon,
and any suitable combination of the foregoing. A computer readable
storage medium, as used herein, is not to be construed as being
transitory signals per se, such as radio waves or other freely
propagating electromagnetic waves, electromagnetic waves
propagating through a waveguide or other transmission media (e.g.,
light pulses passing through a fiber-optic cable), or electrical
signals transmitted through a wire.
[0037] Computer readable program instructions described herein can
be downloaded to respective computing/processing devices from a
computer readable storage medium or to an external computer or
external storage device via a network, for example, the Internet, a
local area network, a wide area network and/or a wireless network.
The network may comprise copper transmission cables, optical
transmission fibers, wireless transmission, routers, firewalls,
switches, gateway computers and/or edge servers. A network adapter
card or network interface in each computing/processing device
receives computer readable program instructions from the network
and forwards the computer readable program instructions for storage
in a computer readable storage medium within the respective
computing/processing device.
[0038] Computer readable program instructions for carrying out
operations of the present invention may be assembler instructions,
instruction-set-architecture (ISA) instructions, machine
instructions, machine dependent instructions, microcode, firmware
instructions, state-setting data, configuration data for integrated
circuitry, or either source code or object code written in any
combination of one or more programming languages, including an
object oriented programming language such as Smalltalk, C++, or the
like, and procedural programming languages, such as the "C"
programming language or similar programming languages. The computer
readable program instructions may execute entirely on the user's
computer, partly on the user's computer, as a stand-alone software
package, partly on the user's computer and partly on a remote
computer or entirely on the remote computer or server. In the
latter scenario, the remote computer may be connected to the user's
computer through any type of network, including a local area
network (LAN) or a wide area network (WAN), or the connection may
be made to an external computer (for example, through the Internet
using an Internet Service Provider). In some embodiments,
electronic circuitry including, for example, programmable logic
circuitry, field-programmable gate arrays (FPGA), or programmable
logic arrays (PLA) may execute the computer readable program
instructions by utilizing state information of the computer
readable program instructions to personalize the electronic
circuitry, in order to perform aspects of the present
invention.
[0039] Aspects of the present invention are described herein with
reference to flowchart illustrations and/or block diagrams of
methods, apparatus (systems), and computer program products
according to embodiments of the invention. It will be understood
that each block of the flowchart illustrations and/or block
diagrams, and combinations of blocks in the flowchart illustrations
and/or block diagrams, can be implemented by computer readable
program instructions.
[0040] These computer readable program instructions may be provided
to a processor of a general purpose computer, special purpose
computer, or other programmable data processing apparatus to
produce a machine, such that the instructions, which execute via
the processor of the computer or other programmable data processing
apparatus, create means for implementing the functions/acts
specified in the flowchart and/or block diagram block or blocks.
These computer readable program instructions may also be stored in
a computer readable storage medium that can direct a computer, a
programmable data processing apparatus, and/or other devices to
function in a particular manner, such that the computer readable
storage medium having instructions stored therein comprises an
article of manufacture including instructions which implement
aspects of the function/act specified in the flowchart and/or block
diagram block or blocks.
[0041] The computer readable program instructions may also be
loaded onto a computer, other programmable data processing
apparatus, or other device to cause a series of operational steps
to be performed on the computer, other programmable apparatus or
other device to produce a computer implemented process, such that
the instructions which execute on the computer, other programmable
apparatus, or other device implement the functions/acts specified
in the flowchart and/or block diagram block or blocks.
[0042] The flowchart and block diagrams in the Figures illustrate
the architecture, functionality, and operation of possible
implementations of systems, methods, and computer program products
according to various embodiments of the present invention. In this
regard, each block in the flowchart or block diagrams may represent
a module, segment, or portion of instructions, which comprises one
or more executable instructions for implementing the specified
logical function(s). In some alternative implementations, the
functions noted in the blocks may occur out of the order noted in
the Figures. For example, two blocks shown in succession may, in
fact, be executed substantially concurrently, or the blocks may
sometimes be executed in the reverse order, depending upon the
functionality involved. It will also be noted that each block of
the block diagrams and/or flowchart illustration, and combinations
of blocks in the block diagrams and/or flowchart illustration, can
be implemented by special purpose hardware-based systems that
perform the specified functions or acts or carry out combinations
of special purpose hardware and computer instructions.
* * * * *