U.S. patent application number 14/929961 was filed with the patent office on 2017-03-09 for system and method for optimizing testing of software production incidents.
This patent application is currently assigned to Wipro Limited. The applicant listed for this patent is Wipro Limited. Invention is credited to Rajiv Kumar AGRAWAL, Bharath Kumar HEMACHANDRAN, Venkata Subramanian JAYARAMAN, Ganesh NARAYAN.
Application Number | 20170068614 14/929961 |
Document ID | / |
Family ID | 54398384 |
Filed Date | 2017-03-09 |
United States Patent
Application |
20170068614 |
Kind Code |
A1 |
JAYARAMAN; Venkata Subramanian ;
et al. |
March 9, 2017 |
System and Method for Optimizing Testing of Software Production
Incidents
Abstract
This disclosure relates generally to software testing, and more
particularly to a system and method for optimizing testing of
software production incidents. In one embodiment, the method
comprises analyzing an incident ticket using a machine learning
algorithm to identify one or more keywords in the incident ticket,
and identifying a location of the incident ticket based on the one
or more keywords, a test workspace corresponding to the incident
ticket based on the location, and a plurality of specific test
cases corresponding to the incident ticket based on the test
workspace. The identification leads to a first scenario and a
second scenario. In the first scenario, the method further
comprises initiating a learning process based on intelligence
gathered from a manual processing of the incident ticket. In the
second scenario, the method further comprises executing the
plurality of specific test cases in a test environment.
Inventors: |
JAYARAMAN; Venkata Subramanian;
(Chennai, IN) ; AGRAWAL; Rajiv Kumar; (Bangalore,
IN) ; NARAYAN; Ganesh; (Bangalore, IN) ;
HEMACHANDRAN; Bharath Kumar; (Bangalore, IN) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Wipro Limited |
Bangalore |
|
IN |
|
|
Assignee: |
Wipro Limited
|
Family ID: |
54398384 |
Appl. No.: |
14/929961 |
Filed: |
November 2, 2015 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06N 20/00 20190101;
G06F 16/285 20190101; G06F 11/3672 20130101; G06Q 10/06
20130101 |
International
Class: |
G06F 11/36 20060101
G06F011/36; G06N 99/00 20060101 G06N099/00; G06F 17/30 20060101
G06F017/30 |
Foreign Application Data
Date |
Code |
Application Number |
Sep 3, 2015 |
IN |
4673/CHE/2015 |
Claims
1. A method for optimizing testing of software production
incidents, the method comprising: categorizing, via a processor, an
incident ticket received from one or more sources based on one or
more pre-defined parameters, the incident ticket corresponding to
an obstruction in a software production; in response to
categorization, analyzing, via the processor, the incident ticket
using a machine learning algorithm to identify one or more keywords
in the incident ticket; identifying, via the processor, a location
of the incident ticket based on the one or more keywords, a test
workspace corresponding to the incident ticket based on the
location, and a plurality of specific test cases corresponding to
the incident ticket based on the test workspace, the identification
leading to a first scenario and a second scenario; in the first
scenario, initiating, via the processor, a learning process based
on intelligence gathered from a manual processing of the incident
ticket; and in the second scenario, identifying, via the processor,
a test environment for the plurality of specific test cases; and
executing, via the processor, the plurality of specific test cases
in the test environment, wherein the first scenario corresponds to
a negative identification of at least one of the location, the test
workspace, and the plurality of specific test cases, wherein the
second scenario corresponds to a positive identification of the
location, the test workspace, and the plurality of specific test
cases, wherein the incident ticket resulting in the first scenario
comprises a new incident ticket unrelated to a plurality of cast
incident tickets and not having at least one of a corresponding
location, a corresponding test workspace, and a corresponding
specific test case, and wherein the manual processing of the new
incident ticket comprises generating at least one of a solution, a
location, a test workspace, a test case, and a test
environment.
2. The method of claim 1, further comprising logging the incident
ticket in an incident repository.
3. The method of claim 1, wherein the one or more predefined
parameters comprise at least one of a person, a system, and a
location related to the incident ticket.
4. The method of claim 1, wherein identifying the location of the
incident ticket comprises referring to a keyword-location
mapping.
5. The method of claim 1, further comprising routing the incident
ticket to the identified location based on at least one of a
priority and a severity rating of the one or more keywords and the
identified location.
6. The method of claim 1, wherein identifying the test workspace
corresponding to the incident ticket comprises referring to a
location-workspace mapping.
7. The method of claim 1, wherein identifying the plurality of
specific test cases comprises: identifying a test case location
from a plurality of test workspaces by referring to a
location-workspace mapping and a search keyword mapping; and
identifying the plurality of specific test cases from the test case
location.
8. The method of claim 1, wherein identifying the plurality of
specific test cases comprises analyzing a plurality of test cases
in the test workspace based on the one or more keywords using the
machine learning algorithm.
9. The method of claim 1, further comprising verifying the
plurality of specific test cases for suitability to testing of the
incident ticket.
10. (canceled)
11. The method of claim 1, wherein identifying the test environment
comprises preparing the test environment corresponding to an
environment of the software production and based on the plurality
of specific test cases.
12. (canceled)
13. The method of claim 1, further comprising updating at least one
of a location repository, a test workspace repository, a test case
repository, a keyword-location mapping, a location-workspace
mapping, and a search keyword mapping based on the learning
process.
14. The method of claim 1, further comprising generating a report
indicating at least one of a result of the execution, a cause
leading to failure of the execution, an effectiveness of
implementation of the optimized testing, an area with high number
of incident tickets that require implementation of the optimized
testing, and a return on investment analysis related to
implementation of the optimized testing.
15. A system for optimizing testing of software production
incidents, the system comprising: at least one processor; and a
computer-readable medium storing instructions that, when executed
by the at least one processor, cause the at least one processor to
perform operations comprising: categorizing an incident ticket
received from one or more sources based on one or more pre-defined
parameters, the incident ticket corresponding to an obstruction in
a software production; in response to categorization, analyzing the
incident ticket using a machine learning algorithm to identify one
or more keywords in the incident ticket; identifying a location of
the incident ticket based on the one or more keywords, a test
workspace corresponding to the incident ticket based on the
location, and a plurality of specific test cases corresponding to
the incident ticket based on the test workspace, the identification
leading to a first scenario and a second scenario; in the first
scenario, initiating a learning process based on intelligence
gathered from a manual processing of the incident ticket; and in
the second scenario, identifying a test environment for the
plurality of specific test cases, and executing the plurality of
specific test cases in the test environment, wherein the first
scenario corresponds to a negative identification of at least one
of the location, the test workspace, and the plurality of specific
test cases, wherein the second scenario corresponds to a positive
identification of the location, the test workspace, and the
plurality of specific test cases, wherein the incident ticket
resulting in the first scenario comprises a new incident ticket
unrelated to a plurality of past incident tickets and not having at
least one of a corresponding location, a corresponding test
workspace, and a corresponding specific test case, and wherein the
manual processing of the new incident ticket comprises generating
at least one of a solution, a location, a test workspace, a test
case, and a test environment.
16. The system of claim 15, wherein identifying the location of the
incident ticket comprises referring to a keyword-location mapping,
and wherein identifying the test workspace corresponding to the
incident ticket comprises referring to a location-workspace
mapping, and wherein identifying the plurality of specific test
cases comprises identifying a test case location from a plurality
of test workspaces by referring to a location-workspace mapping and
a search keyword mapping and identifying the plurality of specific
test cases from the test case location.
17. (canceled)
18. The system of claim 15, wherein the manual processing of the
incident ticket comprises generating at least one of a solution, a
location, a test workspace, a test case, and a test environment,
and wherein the operations further comprise updating at least one
of a location repository, a test workspace repository, a test case
repository, a keyword-location mapping, a location-workspace
mapping, and a search keyword mapping based on the learning
process.
19. A non-transitory computer-readable medium storing
computer-executable instructions for: categorizing an incident
ticket received from one or more sources based on one or more
pre-defined parameters, the incident ticket corresponding to an
obstruction in a software production; in response to
categorization, analyzing the incident ticket using a machine
learning algorithm to identify one or more keywords in the incident
ticket; identifying a location of the incident ticket based on the
one or more keywords, a test workspace corresponding to the
incident ticket based on the location, and a plurality of specific
test cases corresponding to the incident ticket based on the test
workspace, the identification leading to a first scenario and a
second scenario; in the first scenario, initiating a learning
process based on Intelligence gathered from a manual processing of
the incident ticket; and in the second scenario, identifying a test
environment for the plurality of specific test cases, and executing
the plurality of specific test cases in the test environment,
wherein the first scenario corresponds to a negative identification
of at least one of the location, the test workspace, and the
plurality of specific test cases, wherein the second scenario
corresponds to a positive identification of the location, the test
workspace, and the plurality of specific test cases, wherein the
incident ticket resulting in the first scenario comprises a new
incident ticket unrelated to a plurality of past incident tickets
and not having at least one of a corresponding location, a
corresponding test workspace, and a corresponding specific test
case, and wherein the manual processing of the new incident ticket
comprises generating at least one of a solution, a location, a test
workspace, a test case, and a test environment.
Description
TECHNICAL FIELD
[0001] This disclosure relates generally to software testing, and
more particularly to a system and method for optimizing testing of
software production incidents.
BACKGROUND
[0002] In the current business environment, need for quality
assurance of a software product is paramount to the success of
Information Technology (IT) organizations. Quality assurance
primarily involves testing of a software product at various stages
of the software lifecycle to minimize defects. However, testing is
a major factor driving overall cost of the project/program and
anything done to reduce the cost of testing directly or indirectly
results in cost saving to the project/program. In this context,
automation in the area of software testing has grown and a lot of
automation techniques are now available to increase efficiency and
reduce cost. For example, these techniques include test cases
automation to reduce cycle time, process automation to reduce
overall schedule, automation of test cases when the requirements
are being developed, parallel execution along with development when
coding is being done, adding services that control the updating of
test cases on change, change management and their impacts on
testing and tools to fix them, and so forth. All these techniques
involve reducing the time taken to overall testing, thereby
reducing cost.
[0003] Additionally, the overall quality of any software product is
determined based on the total production or post release defects
that have leaked into the product. It is preferable to identify and
correct defects as soon as possible in a software production so as
not to adversely impact the customer experience or the
organization's reputation and competitiveness. However, if the
total number of production defects is very high, the time taken and
cost involved for re-rollout of a product release or for fixing and
testing the defects in the release is a challenge as testing
imposes a big bottleneck. The main reason for above is time and
cost incurred to manually identify right test cases to verify the
fix, to manually execute the identified test cases for the given
fix, and to manually identify regression test suite to make sure
that the fix has not broken any other major functionalities.
[0004] A cost analysis shows that when a defect is leaked into
production the cost of fixing that defect and testing the solution
is the costliest. For example, if a defect that could have been
detected in requirement phase is detected post release, then it
would cost about 10-100 times more to fix than if the defect had
been detected in the requirement phase itself. Further, if a defect
is leaked to release after construction, then it would be 25 times
costly to fix it. As stated above, this is mostly because of the
retesting effort that is needed.
[0005] Existing software testing techniques do not completely
address the issues stated above particularly with respect to
production or post release defects. Existing techniques to test the
production defects and/or to arrest production defects in customer
environment involve manual processing and operations and are
therefore time consuming and cost intensive. Moreoever, there is no
automated way to connect the various systems so as to optimize
fixing and testing of software production defects. All these
results in increased business spend for a particular release.
SUMMARY
[0006] In one embodiment, a method for optimizing testing of
software production incidents is disclosed. In one example, the
method comprises categorizing an incident ticket received from one
or more sources based on one or more pre-defined parameters. The
incident ticket corresponds to an obstruction in a software
production. The method further comprises analyzing the incident
ticket using a machine learning algorithm to identify one or more
keywords in the incident ticket in response to categorization. The
method further comprises identifying a location of the incident
ticket based on the one or more keywords, a test workspace
corresponding to the incident ticket based on the location, and a
plurality of specific test cases corresponding to the incident
ticket based on the test workspace. The identification leads to a
first scenario and a second scenario. In the first scenario, the
method further comprises initiating a learning process based on
intelligence gathered from a manual processing of the incident
ticket. In the second scenario, the method further comprises
identifying a test environment for the plurality of specific test
cases, and executing the plurality of specific test cases in the
test environment.
[0007] In one embodiment, a system for optimizing testing of
software production incidents is disclosed. In one example, the
system comprises at least one processor and a memory
communicatively coupled to the at least one processor. The memory
stores processor-executable instructions, which, on execution,
cause the processor to categorize an incident ticket received from
one or more sources based on one or more pre-defined parameters.
The incident ticket corresponds to an obstruction in a software
production. The processor-executable instructions, on execution,
further cause the processor to analyze the incident ticket using a
machine learning algorithm to identify one or more keywords in the
incident ticket in response to categorization. The
processor-executable instructions, on execution, further cause the
processor to identify a location of the incident ticket based on
the one or more keywords, a test workspace corresponding to the
incident ticket based on the location, and a plurality of specific
test cases corresponding to the incident ticket based on the test
workspace. The identification leads to a first scenario and a
second scenario. In the first scenario, the processor-executable
instructions, on execution, further cause the processor to initiate
a learning process based on intelligence gathered from a manual
processing of the incident ticket. In the second scenario, the
processor-executable instructions, on execution, further cause the
processor to identify a test environment for the plurality of
specific test cases, and to execute the plurality of specific test
cases in the test environment.
[0008] In one embodiment, a non-transitory computer-readable medium
storing computer-executable instructions for optimizing testing of
software production incidents is disclosed. In one example, the
stored instructions, when executed by a processor, cause the
processor to categorize an incident ticket received from one or
more sources based on one or more pre-defined parameters. The
incident ticket corresponds to an obstruction in a software
production. The stored instructions, when executed by a processor,
further cause the processor to analyze the incident ticket using a
machine learning algorithm to identify one or more keywords in the
incident ticket in response to categorization. The stored
instructions, when executed by a processor, further cause the
processor to identify a location of the incident ticket based on
the one or more keywords, a test workspace corresponding to the
incident ticket based on the location, and a plurality of specific
test cases corresponding to the incident ticket based on the test
workspace. The identification leads to a first scenario and a
second scenario. In the first scenario, the stored instructions,
when executed by a processor, further cause the processor to
initiate a learning process based on intelligence gathered from a
manual processing of the incident ticket. In the second scenario,
the stored instructions, when executed by a processor, further
cause the processor to identify a test environment for the
plurality of specific test cases, and to execute the plurality of
specific test cases in the test environment.
[0009] It is to be understood that both the foregoing general
description and the following detailed description are exemplary
and explanatory only and are not restrictive of the invention, as
claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings, which are incorporated in and
constitute a part of this disclosure, illustrate exemplary
embodiments and, together with the description, serve to explain
the disclosed principles.
[0011] FIG. 1 is a block diagram of an exemplary system for
optimizing testing of software production incidents in accordance
with some embodiments of the present disclosure;
[0012] FIG. 2 is a functional block diagram of an incident
management engine in accordance with some embodiments of the
present disclosure;
[0013] FIG. 3 is a flow diagram of an exemplary process for
optimizing testing of software production incidents in accordance
with some embodiments of the present disclosure;
[0014] FIG. 4 is a flow diagram of a detailed exemplary process for
optimizing testing of software production incidents in accordance
with some embodiments of the present disclosure; and
[0015] FIG. 5 is a block diagram of an exemplary computer system
for implementing embodiments consistent with the present
disclosure.
DETAILED DESCRIPTION
[0016] Exemplary embodiments are described with reference to the
accompanying drawings. Wherever convenient, the same reference
numbers are used throughout the drawings to refer to the same or
like parts. While examples and features of disclosed principles are
described herein, modifications, adaptations, and other
implementations are possible without departing from the spirit and
scope of the disclosed embodiments. It is intended that the
following detailed description be considered as exemplary only,
with the true scope and spirit being indicated by the following
claims.
[0017] Referring now to FIG. 1, an exemplary system 100 for
optimizing testing of software production incidents is illustrated
in accordance with some embodiments of the present disclosure. In
particular, the system 100 implements an incident management engine
for providing optimized testing of incidents corresponding to
defects or obstructions in a software production. As will be
described in greater detail in conjunction with FIG. 2, the
incident management engine interacts with multiple users (e.g.,
quality assurance team, development team, business analyst team,
and so forth) and multiple software development, testing, and
management systems or platforms (e.g., test management systems,
test environment systems, and so forth), identifies a test
workspace by analyzing an incident ticket, determines required test
cases for the incident ticket in the test workspace, and executes
the required test cases in a requisite test environment. The system
100 comprises one or more processors 101, a computer-readable
medium (e.g., a memory) 102, and a display 103. The
computer-readable medium 102 stores instructions that, when
executed by the one or more processors 101, cause the one or more
processors 101 to perform optimized testing of software production
incidents in accordance with aspects of the present disclosure. The
system 100 interacts with users via a user interface 104 accessible
to the users via the display 103.
[0018] Referring now to FIG. 2, a functional block diagram of an
incident management engine 200 implemented by the system 100 of
FIG. 1 is illustrated in accordance with some embodiments of the
present disclosure. In some embodiments, the incident management
engine 200 comprises an incident logging module 201, an incident
analysis module 202, a test suite generation module 203, and an
execution module 204. A user may interact with the incident
management engine 200 from the web browser or other interfaces. The
incident management engine 200 may further interact with one or
more continuous integration systems 205, one or more test
management systems 206, and one or more test environment systems
207.
[0019] The incident logging module 201 receives and logs all the
incidents either via a manual or an automated process whenever
there is a production failure that has been encountered in any of
the software product. Incidents may be raised by customer on phone,
via Fax, or via email. An incident ticket is automatically
generated and logged in an incident repository by the incident
logging module 201 whenever an incident is reported. The incident
ticket captures the information from a customer site. It should be
noted that incident typically provide only cursory information that
may be captured.
[0020] The incident analysis module 202 analyzes the incident
tickets so as to determine the categories of the incident tickets
and to identify a test workspace. For example, in some embodiments,
the incident tickets may be categorized based on departments they
belong to. Once categorized, the incident tickets are sorted.
Incident tickets that only require a L1 support (e.g., telephone
helpdesk, customer center support, and so forth) are moved out from
here directly. The remaining incident tickets are analyzed against
the main areas/locations which are identified for the first time
with the involvement of the business analysts, quality analysts,
and application development team. The teams provide the priority
and severity of each area. The severity and priority of the area
are the average of the all the teams. The business holds the final
veto to either accept or overwrite the whole value provided by
different teams. In some embodiments, the priority and severity
rating is done in three levels for which the threshold values may
be a configurable value. By way of example, the incident logging
module 201 may have default threshold values for each rating which
may be updated by a user. For example, in certain embodiments,
sample illustration of the rating threshold values is shown in
Table A below while sample illustration of the area/location
priority values provided by different teams is shown in Table
B:
TABLE-US-00001 TABLE A Rating Threshold Value High 1 Medium 2 Low
3
TABLE-US-00002 TABLE B BA QA Development Business Area/Location
Team Team Team Average Team Pharmacy 3 1 1 1.67 1 queue Stores 2 3
3 2.67 2 Web store 3 3 3 3 3 Billing 3 3 3 3 1
[0021] The test suite generation module 203 identifies the right
test cases needed for execution based on the analytical result. The
right test cases are identified based on their suitability for a
given incident ticket belonging to an identified test workspace.
Further, the execution module 204 executes the identified test
cases in a right test environment. The test environment may be
tools, batch jobs, or locations where test cases may be executed
either manually or automatically. The incident management engine
200 may integrate with the various systems to ensure that the right
environment is provided to the right test cases so that the result
could be appropriate. The contiguous integration systems 205 enable
the incident management engine 200 to select an appropriate or a
requisite environment that may be required for a specific test
case.
[0022] Additionally, the incident management engine 200 may
interact with the test management systems 206 that store and manage
various test cases, defects, fixes results of execution of test
cases, and so forth. The test management systems 206 may include
one or more repositories for storing various data. For example, it
may include a test cases repository to store various test cases, a
test workspace repository to store various test workspaces, and a
location repository to store various locations to which incident
tickets may belong to. In some embodiments, the test management
systems 206 may be an application lifecycle management (ALM)
platform such as IBM.TM. Rational.RTM. Jazz.TM. platform,
Microsoft.TM. Team Foundation Server (TFS), HP.RTM. ALM, and so
forth. Alternatively, the test management systems 206 may be any
test management tool such as JIRA.TM. that may be employed for
testing or project management of the software product. Further, the
incident management engine 200 may interact with the test
environment systems 207 that provide for the location from where
the test cases are operated. The execution module 204 connects with
the test environment systems 207 to access the right environment to
run the test cases.
[0023] In some embodiments, the incident management engine 200
provides for optimized testing of software production incidents by
performing following primary functions: (i) identification of test
workspaces by analyzing the production incidents/defect data logged
using keyword search analytics, (ii) determination of the required
test cases by analyzing the existing test cases in identified work
spaces using keyword search analytics, and (iii) execution of the
identified test cases in the requisite test environments and
generating analysis reports. Each of these functions will be
described in greater detail below.
[0024] The incident logging module 201 receives details of the
product incidents or the defect data from one or more sources
(e.g., such as incident reported by user on phone, via Fax, or via
email). An incident ticket is created and logged in an incident
repository corresponding to the incident is reported. The incident
tickets so created are then classified based on one or more
predefined parameters such as who logged the incident, what system
is the incident belonging to, and which location did the incident
come from. Thus, in some embodiments, the tickets are categorized
based on the person, system and location. For example, in certain
embodiments, sample illustration of categorized incident tickets is
provided in following table:
TABLE-US-00003 TABLE C Incident No. Person System Country
Country_Area Incident Description 1 XXXXX Pharma USA Pharma_USA
Pharmacy application not launching 2 YYYYY Pharma INDIA
Pharma_INDIA Optical system does not allow payments 3 ZZZZZ Ecom UK
Ecom_UK XXX home page is not opening
[0025] The categorized tickets are then analyzed based on the
keywords in the incident description to separate out the L1
tickets. In some embodiments, the condition that needs to be
satisfied for L1 incident ticket may be that the time taken to fix
the issues is less than 10 hours and the fix is purely hardware
related. The identified L1 incidents are separated out and given to
the incident management team for manual fixing. The remaining
categorized incident details are then provided to the incident
analysis module 202 where the incident description provided with
the incident is retrieved and analyzed. The incident description is
analyzed using the artificial intelligence technique for
identifying the different keywords in the incident description
text. The artificial intelligence technique may include any machine
learning algorithms such as case-based reasoning, decision tree
learning, association rule learning, cluster analysis, rule-based
learning, artificial neural networks, Bayesian networks, genetic
algorithms, fuzzy logic, inductive logic, multi-agent models,
reinforcement learning, hybrid learning, and so forth. It should be
noted that the keywords are specific details provided in the
incident description text by the customer or the operational
support team who perform the incident reporting. The keywords is
then used for keyword-location mapping, i.e., mapping the incident
tickets to the respective area/location to which the ticket has to
be routed based on the priority and severity of that keyword and
location. It should be noted that the priority and severity is not
only based on incident ticket but also based on its impact on
business. For example, in some embodiments, a sample illustration
of the different locations' priority and severity rating is
provided in following table:
TABLE-US-00004 TABLE D Location Priority Severity Pharma Queue 1 1
Stores 2 2 Web Store 3 3
[0026] The incident description is compared using the artificial
intelligence technique with the location details available in
incident analysis module 202 such as those listed in Table D. It is
compared to see if any of the identified keywords in the incident
description matches with the location names. If the identified
keywords matches then the location of where the incident ticket has
to go, severity and priority are taken in from the keyword-location
mapping information available. It should be noted that the initial
keyword of the mapping is based on the inputs provided in the
incident management engine 200 and the information that is read by
the machine learning algorithms. As will be appreciated, the
keyword-location mapping gets updated from time to time based on
the learning by the machine learning algorithms. For example, in
some embodiments, a sample illustration of the keyword-location
mapping is provided in following table:
TABLE-US-00005 TABLE E Keywords Location Priority Severity Pharma,
Health Pharma Queue 1 1 Associate, Incentive, Plan Stores 2 2 Ecom,
Homepage Web Store 3 3
[0027] If the identified keywords do not match then the incident
analysis module 202 returns the notification information to the
user stating no match could be found. The user may then be provided
with the option to enter the new location, severity, priority into
the system configuration mapping details. If the user selects an
existing location, then the incident analysis module 202 takes
keyword as inputs and updates the keyword-location mapping details
with the corresponding keywords using the machine learning
techniques automatically. As will be appreciated, with time and
usage the module keeps learning and the keyword-location mapping is
updated with more and more keywords which may be used to assign
future incident tickets.
[0028] Thus, in short, the incident analysis module 202 reads text
from the incident description, searches for keywords of the text
from the keyword-location mapping, and provides location, priority,
and severity if the text matches with those in mapping. However, if
the text does not match with those in mapping, the incident
analysis module 202 returns notification to the user, provides
option to enter the new location, and modifies the mapping details
with new location and keywords.
[0029] The identified location/area details using the keywords are
then provided to the test management system 206 to identify the
test workspaces corresponding to the location/area. The incident
tickets corresponding with the specific location are added to the
test management system. The mapping of the location and the work
spaces are maintained in the incident analysis module 202. It
should be noted that the mapping is setup using manual mechanism
for the first time. However, as new workspaces are created in the
test management system 205, the incident management engine is fed
with data of which location the workspace is going to handle. The
location-workspace mapping is updated based on this information.
Further, it should be noted that as the mapping data does not
change frequently, this data is typically kept constant. For
example, in some embodiments, a sample illustration of mapping of
the location to different workspaces identified is provided in
following table:
TABLE-US-00006 TABLE F Location Workspace Pharma Queue WS1, WS2
Stores WS4, WS3
[0030] The details of identified test workspaces are then used by
the incident analysis module 202 and the test management system 206
to identify the required or specific test cases for the incident
tickets. In some embodiments, all the test cases identified may be
further refined using a two level keyword search analytics
implemented using artificial intelligence technique. For example,
for each of the location-workspace mapping the initial search
keywords may be updated based on the expertise of the quality
assurance (QA) team for the initial setup. QA provides the search
keywords based on the earlier experience and builds the initial
mapping. QA team may provide their inputs in to the remedy/incident
mapping system. These keywords are the initial keywords that
facilitate the location-workspace-keyword mapping or the search
keyword mapping. For example, in some embodiments, a sample
illustration of the initial search keyword mapping provided by QA
team is shown in following table:
TABLE-US-00007 TABLE G Location Workspace Initial Search Keywords
Pharma Queue WS1, WS2 Pricing, Cost Stores WS4, WS3 Costing,
Financials
[0031] After the initial setup, the search keyword mapping is
enhanced using self-learning techniques implemented using
artificial intelligence technique. The incidents that have been
identified earlier may be passed through the defect triage process.
As will be appreciated, defect triage process is a process in which
project stakeholders go through new or existing defects and decide
on the action items. For example, during the triage process the key
secondary words may be added to the incident tickets. As the
additional keywords are provided, the machine learning algorithm of
the artificial intelligence technique learns the keywords and keeps
updating them to the existing table automatically. The learning
process involves understanding the input supplied by the user via
machine learning algorithm, starting the artificial intelligence
analytics to learn the keywords, and updating the keywords in the
second search keyword mapping based on the entry by the user. Thus,
the search keyword mapping gets enhanced with new keywords every
time it learns new information. For example, in some embodiments, a
sample illustration of the enhanced search keywords mapping is
provided in following table:
TABLE-US-00008 TABLE H Location Workspace Enhanced Search Keywords
Pharma Queue WS1, WS2 Pricing, Cost, Optical Stores WS4, WS3
Costing, Financials, Workers
[0032] A test case location is then identified from the different
test workspaces identified earlier based on the location-workspace
mapping and corresponding search keyword mapping. The test case
location tells the exact location where the test cases are present.
For example, in some embodiments, a sample illustration of the test
case location identification is provided in following table:
TABLE-US-00009 TABLE I Test Case Location Workspace Enhanced Search
Keywords Location Pharma Queue WS1, WS2 Pricing, Cost, Optical WS1
Stores WS4, WS3 Costing, Financials, Workers WS3
[0033] The test management system 206 provides the identified test
case location to the test suite generation module 203. The test
suite generation module 203 then identifies the specific test cases
from the test case location using the keyword search technique as
explained above. In some embodiments, the specific test cases so
identified may be optimized using optimization tools or all the
specific test cases may be selected based on an option selected by
the user. The identified specific test cases are accordingly
provided to the test suite generation module 203. Once the test
suite generation module 203 is loaded with all the test cases
resulting from the search, the tester/user may be prompted for
verification. An input from the tester is taken to verify and
confirm whether the identified test cases are sufficient to ensure
that the test cases are good enough for the validation process. If
the test cases are insufficient the user/tester is provided with
the option to update/create/delete/modify new tests into the
system. The engine 200 would learn the information keyed in and
save them to the test management system 206 along with
location-workspace mapping from where the test cases were picked up
for automatic handling of future incident tickets.
[0034] The specific test cases identified in the test suite
generation module 203 are provided to the execution module 204 for
execution. The execution module 204 establishes connection with the
test environment system 207 and identifies the right environment to
run the specific test cases. It should be noted that a production
like environment is preferable. Typically, the environment used may
be a user acceptance testing (UAT) environment or a certification
(CERT) environment. These environments are production like
environments and ensure that the defect is first reproducible, and
then apply the fix or solution, and then test if the fix is working
along with the regression around the fix.
[0035] The continuous integration system (CIS) 205 facilitates the
integration with the various systems to ensure that the right
environment is provided to the right test cases so that the result
could be appropriate. In some embodiments, the CIS 205 also works
on the result based approach. Thus, in case of failure the engine
alerts back the development team to inform that a code fix is not
working, and if found working ensures that the loop is closed and
the production ready code is built and ready for deployment. The
CIS 205 gives an option to program a set of pilot stores, that is,
if the data is programmed the solution is piloted to the specific
store(s).
[0036] As will be appreciated, the automatic execution of specific
test cases may be considered as an ideal situation. However, in
some embodiments, the execution of test cases has to be done
manually because of certain constraints in the environment or
certain limitations in automating the execution of test cases. The
execution module 204 may notify a user if such instances occur. The
execution module 204 may further provide a report that shows up the
result of execution in a graphical format indicating repeated
occurrences of such instances. For example, in some embodiments,
the published report may include a graph that indicates to a
business user how many test cases are run manually and how many are
run automatically and how many are partially run on a quarterly
basis. The report therefore provides an insight to the organization
and enables business users to infer how effective is the
automation, is the automation happening at the right place or not,
are there environment failures which hamper the automation, does
the area which gets highest ticket need to be automated, what test
cases may or should be automated, how the errors are being treated,
and so forth. The frequency of such reports may be decided by the
team.
[0037] Further, in some embodiments, the report may include a
return on investment (ROI) analysis graph. It helps in providing to
the business a view by ticket of what has happened in each and
every incident so that business may ensure that the right incident
gets the right attention. Based on the ROI analysis the funding
every quarter may be adjusted by the business that could result in
better ROI for automation. Additionally, the report may include a
comparative analysis of the result which would enable the business
user to determine for which solution there was more manual
intervention needed and how further learning may be done to improve
such manual intervention.
[0038] It should be noted that the incident management engine 200
may be implemented in programmable hardware devices such as
programmable gate arrays, programmable array logic, programmable
logic devices, and so forth. Alternatively, the incident management
engine 200 may be implemented in software for execution by various
types of processors. An identified engine of executable code may,
for instance, comprise one or more physical or logical blocks of
computer instructions which may, for instance, be organized as an
object, procedure, function, module, or other construct.
Nevertheless, the executables of an identified engine need not be
physically located together, but may comprise disparate
instructions stored in different locations which, when joined
logically together, comprise the engine and achieve the stated
purpose of the engine. Indeed, an engine of executable code could
be a single instruction, or many instructions, and may even be
distributed over several different code segments, among different
applications, and across several memory devices.
[0039] As will be appreciated by one skilled in the art, a variety
of processes may be employed for optimizing testing of software
production incidents. For example, the exemplary system 100 and the
associated incident management engine 200 may optimize testing of
software production incidents by the processes discussed herein. In
particular, as will be appreciated by those of ordinary skill in
the art, control logic and/or automated routines for performing the
techniques and steps described herein may be implemented by the
system 100 and the associated incident management engine 200,
either by hardware, software, or combinations of hardware and
software. For example, suitable code may be accessed and executed
by the one or more processors on the system 100 to perform some or
all of the techniques described herein. Similarly, application
specific integrated circuits (ASICs) configured to perform some or
all of the processes described herein may be included in the one or
more processors on the system 100.
[0040] For example, referring now to FIG. 3, exemplary control
logic 300 for optimizing testing of software production incidents
via a system, such as system 100, is depicted via a flowchart in
accordance with some embodiments of the present disclosure. As
illustrated in the flowchart, the control logic 300 includes the
step of categorizing an incident ticket received from one or more
sources based on one or more pre-defined parameters at step 301. In
response to categorization, the control logic 300 further includes
the steps of analyzing the incident ticket using a machine learning
algorithm to identify one or more keywords in the incident ticket
at step 302, and identifying a location of the incident ticket
based on the one or more keywords, a test workspace corresponding
to the incident ticket based on the location, and a plurality of
specific test cases corresponding to the incident ticket based on
the test workspace at step 303. The identification leads to a first
scenario and a second scenario. In the first scenario, the control
logic 300 further includes the step of initiating a learning
process based on intelligence gathered from a manual processing of
the incident ticket at step 304. In the second scenario, the
control logic 300 further includes the steps of identifying a test
environment for the plurality of specific test cases at step 305,
and executing the plurality of specific test cases in the test
environment at step 306.
[0041] As noted above, the incident ticket corresponds to an
obstruction (i.e., a defect, or a failure) in a software
production. Further, the one or more predefined parameters comprise
at least one of a person, a system, and a location related to the
incident ticket. In some embodiments, the first scenario
corresponds to a negative identification of at least one of the
location, the test workspace, and the plurality of specific test
cases, while the second scenario corresponds to a positive
identification of the location, the test workspace, and the
plurality of specific test cases. Further, in some embodiments, the
incident ticket resulting in the first scenario comprises a new
incident ticket unrelated to a plurality of past incident tickets
and not having at least one of a corresponding location, a
corresponding test workspace, and a corresponding specific test
case. The manual processing of such new incident ticket comprises
generating at least one of a solution, a location, a test
workspace, a test case, and a test environment.
[0042] In some embodiments, the control logic 300 further includes
the step of logging the incident ticket in an incident repository.
Additionally, in some embodiments, the control logic 300 includes
the step of routing the incident ticket to the identified location
based on at least one of a priority and a severity rating of the
one or more keywords and the identified location. Further, in some
embodiments, the control logic 300 includes the step of verifying
the plurality of specific test cases for suitability to testing of
the incident ticket. In some embodiments, the control logic 300
further includes the step of updating at least one of a location
repository, a test workspace repository, a test case repository, a
keyword-location mapping, a location-workspace mapping, and a
search keyword mapping based on the learning process. Moreover, in
some embodiments, the control logic 300 includes the step of
generating a report indicating at least one of a result of the
execution, a cause leading to failure of the execution, an
effectiveness of implementation of the optimized testing, an area
with high number of incident tickets that require implementation of
the optimized testing, and a return on investment analysis related
to implementation of the optimized testing.
[0043] In some embodiments, identifying the location of the
incident ticket at step 303 comprises referring to a
keyword-location mapping. Similarly, in some embodiments,
identifying the test workspace corresponding to the incident ticket
at step 303 comprises referring to a location-workspace mapping.
Additionally, in some embodiments, identifying the plurality of
specific test cases at step 303 comprises identifying a test case
location from a plurality of test workspaces by referring to a
location-workspace mapping and a search keyword mapping, and
identifying the plurality of specific test cases from the test case
location. In some embodiments, identifying the plurality of
specific test cases at step 303 comprises analyzing a plurality of
test cases in the test workspace based on the one or more keywords
using the machine learning algorithm. Further, in some embodiments,
identifying the test environment at step 305 comprises preparing
the test environment corresponding to an environment of the
software production and based on the plurality of specific test
cases.
[0044] Referring now to FIG. 4, exemplary control logic 400 for
optimizing testing of software production incidents is depicted in
greater detail via a flowchart in accordance with some embodiments
of the present disclosure. As illustrated in the flowchart, the
control logic 400 includes the step of logging incident tickets and
categorizing incident tickets using artificial intelligence
technique at step 401. The control logic 400 further includes the
step of checking with triage team and determining if data with
respect to the incident ticket is available at step 402. The data
may include location details corresponding to the incident ticket
and test workspace details corresponding to the location details.
The step 402 involves referring to the keyword-location mapping and
the location-workspace mapping. If the data is available at step
402, the control logic 400 further includes the step of determining
if the content of the available data is sufficient for performing a
search at step 403. If the content is not sufficient at step 403,
the control logic 400 further includes the steps of requesting
triage team to provide more details or content with respect to the
data at step 404 and checking if more details are available with
the triage team at step 405. If the content is sufficient at step
403 or if more details are available at step 405, the control logic
400 includes the steps of searching for required or specific test
cases at step 406 and determining their applicability or
suitability for testing of the incident tickets at step 407. If the
test cases are applicable at step 407, the control logic 400
includes the step of determining if automation is available for the
test cases at step 408. If automation is available at step 408, the
control logic 400 includes the step of executing automation suite
and validating results at step 409. However, if automation is not
available at step 408, the control logic 400 includes the step of
executing manual suite and validating results at step 410.
Moreover, if the data is not available at step 402, or if more
details are not available at step 405, or if the test cases are not
applicable at step 407, the control logic 400 includes the step of
starting self-learning engine to learn from manual input process at
step 411.
[0045] As will be also appreciated, the above described techniques
may take the form of computer or controller implemented processes
and apparatuses for practicing those processes. The disclosure can
also be embodied in the form of computer program code containing
instructions embodied in tangible media, such as floppy diskettes,
CD-ROMs, hard drives, or any other computer-readable storage
medium, wherein, when the computer program code is loaded into and
executed by a computer or controller, the computer becomes an
apparatus for practicing the invention. The disclosure may also be
embodied in the form of computer program code or signal, for
example, whether stored in a storage medium, loaded into and/or
executed by a computer or controller, or transmitted over some
transmission medium, such as over electrical wiring or cabling,
through fiber optics, or via electromagnetic radiation, wherein,
when the computer program code is loaded into and executed by a
computer, the computer becomes an apparatus for practicing the
invention. When implemented on a general-purpose microprocessor,
the computer program code segments configure the microprocessor to
create specific logic circuits.
[0046] The disclosed methods and systems may be implemented on a
conventional or a general-purpose computer system, such as a
personal computer (PC) or server computer. Referring now to FIG. 5,
a block diagram of an exemplary computer system 501 for
implementing embodiments consistent with the present disclosure is
illustrated. Variations of computer system 501 may be used for
implementing system 100 and incident management engine 200 for
optimizing testing of the software production incidents. Computer
system 501 may comprise a central processing unit ("CPU" or
"processor") 502. Processor 502 may comprise at least one data
processor for executing program components for executing user- or
system-generated requests. A user may include a person, a person
using a device such as such as those included in this disclosure,
or such a device itself. The processor may include specialized
processing units such as integrated system (bus) controllers,
memory management control units, floating point units, graphics
processing units, digital signal processing units, etc. The
processor may include a microprocessor, such as AMD Athlon, Duron
or Opteron, ARM's application, embedded or secure processors, IBM
PowerPC, Intel's Core, Itanium, Xeon, Celeron or other line of
processors, etc. The processor 502 may be implemented using
mainframe, distributed processor, multi-core, parallel, grid, or
other architectures. Some embodiments may utilize embedded
technologies like application-specific integrated circuits (ASICs),
digital signal processors (DSPs), Field Programmable Gate Arrays
(FPGAs), etc.
[0047] Processor 502 may be disposed in communication with one or
more input/output (I/O) devices via I/O interface 503. The I/O
interface 503 may employ communication protocols/methods such as,
without limitation, audio, analog, digital, monoaural, RCA, stereo,
IEEE-1394, serial bus, universal serial bus (USB), infrared, PS/2,
BNC, coaxial, component, composite, digital visual interface (DVI),
high-definition multimedia interface (HDMI), RF antennas, S-Video,
VGA, IEEE 802.n/b/g/n/x, Bluetooth, cellular (e.g., code-division
multiple access (CDMA), high-speed packet access (HSPA+), global
system for mobile communications (GSM), long-term evolution (LTE),
WiMax, or the like), etc.
[0048] Using the I/O interface 503, the computer system 501 may
communicate with one or more I/O devices. For example, the input
device 504 may be an antenna, keyboard, mouse, joystick, (infrared)
remote control, camera, card reader, fax machine, dongle, biometric
reader, microphone, touch screen, touchpad, trackball, sensor
(e.g., accelerometer, light sensor, GPS, gyroscope, proximity
sensor, or the like), stylus, scanner, storage device, transceiver,
video device/source, visors, etc. Output device 505 may be a
printer, fax machine, video display (e.g., cathode ray tube (CRT),
liquid crystal display (LCD), light-emitting diode (LED), plasma,
or the like), audio speaker, etc. In some embodiments, a
transceiver 506 may be disposed in connection with the processor
502. The transceiver may facilitate various types of wireless
transmission or reception. For example, the transceiver may include
an antenna operatively connected to a transceiver chip (e.g., Texas
Instruments WiLink WL1283, Broadcom BCM4750IUB8, Infineon
Technologies X-Gold 618-PMB9800, or the like), providing IEEE
802.11a/b/g/n, Bluetooth, FM, global positioning system (GPS),
2G/3G HSDPA/HSUPA communications, etc.
[0049] In some embodiments, the processor 502 may be disposed in
communication with a communication network 508 via a network
interface 507. The network interface 507 may communicate with the
communication network 508. The network interface may employ
connection protocols including, without limitation, direct connect,
Ethernet (e.g., twisted pair 10/100/1000 Base T), transmission
control protocol/internet protocol (TCP/IP), token ring, IEEE
802.11a/b/g/n/x, etc. The communication network 508 may include,
without limitation, a direct interconnection, local area network
(LAN), wide area network (WAN), wireless network (e.g., using
Wireless Application Protocol), the Internet, etc. Using the
network interface 507 and the communication network 508, the
computer system 501 may communicate with devices 509, 510, and 511.
These devices may include, without limitation, personal
computer(s), server(s), fax machines, printers, scanners, various
mobile devices such as cellular telephones, smartphones (e.g.,
Apple iPhone, Blackberry, Android-based phones, etc.), tablet
computers, eBook readers (Amazon Kindle, Nook, etc.), laptop
computers, notebooks, gaming consoles (Microsoft Xbox, Nintendo DS,
Sony PlayStation, etc.), or the like. In some embodiments, the
computer system 501 may itself embody one or more of these
devices.
[0050] In some embodiments, the processor 502 may be disposed in
communication with one or more memory devices (e.g., RAM 413, ROM
414, etc.) via a storage interface 512. The storage interface may
connect to memory devices including, without limitation, memory
drives, removable disc drives, etc., employing connection protocols
such as serial advanced technology attachment (SATA), integrated
drive electronics (IDE), IEEE-1394, universal serial bus (USB),
fiber channel, small computer systems interface (SCSI), etc. The
memory drives may further include a drum, magnetic disc drive,
magneto-optical drive, optical drive, redundant array of
independent discs (RAID), solid-state memory devices, solid-state
drives, etc.
[0051] The memory devices may store a collection of program or
database components, including, without limitation, an operating
system 516, user interface application 517, web browser 518, mail
server 519, mail client 520, user/application data 521 (e.g., any
data variables or data records discussed in this disclosure), etc.
The operating system 516 may facilitate resource management and
operation of the computer system 501. Examples of operating systems
include, without limitation, Apple Macintosh OS X, Unix, Unix-like
system distributions (e.g., Berkeley Software Distribution (BSD),
FreeBSD, NetBSD, OpenBSD, etc.), Linux distributions (e.g., Red
Hat, Ubuntu, Kubuntu, etc.), IBM OS/2, Microsoft Windows (XP,
Vista/7/8, etc.), Apple iOS, Google Android, Blackberry OS, or the
like. User interface 517 may facilitate display, execution,
interaction, manipulation, or operation of program components
through textual or graphical facilities. For example, user
interfaces may provide computer interaction interface elements on a
display system operatively connected to the computer system 501,
such as cursors, icons, check boxes, menus, scrollers, windows,
widgets, etc. Graphical user interfaces (GUIs) may be employed,
including, without limitation, Apple Macintosh operating systems'
Aqua, IBM OS/2, Microsoft Windows (e.g., Aero, Metro, etc.), Unix
X-Windows, web interface libraries (e.g., ActiveX, Java,
Javascript, AJAX, HTML, Adobe Flash, etc.), or the like.
[0052] In some embodiments, the computer system 501 may implement a
web browser 518 stored program component. The web browser may be a
hypertext viewing application, such as Microsoft Internet Explorer,
Google Chrome, Mozilla Firefox, Apple Safari, etc. Secure web
browsing may be provided using HTTPS (secure hypertext transport
protocol), secure sockets layer (SSL), Transport Layer Security
(TLS), etc. Web browsers may utilize facilities such as AJAX,
DHTML, Adobe Flash, JavaScript, Java, application programming
interfaces (APIs), etc. In some embodiments, the computer system
501 may implement a mail server 519 stored program component. The
mail server may be an Internet mail server such as Microsoft
Exchange, or the like. The mail server may utilize facilities such
as ASP, ActiveX, ANSI C++/C#, Microsoft .NET, CGI scripts, Java,
JavaScript, PERL, PHP, Python, WebObjects, etc. The mail server may
utilize communication protocols such as internet message access
protocol (IMAP), messaging application programming interface
(MAPI), Microsoft Exchange, post office protocol (POP), simple mail
transfer protocol (SMTP), or the like. In some embodiments, the
computer system 501 may implement a mail client 520 stored program
component. The mail client may be a mail viewing application, such
as Apple Mail, Microsoft Entourage, Microsoft Outlook, Mozilla
Thunderbird, etc.
[0053] In some embodiments, computer system 501 may store
user/application data 521, such as the data, variables, records,
etc. (e.g., incident tickets, locations/areas, priority and
severity ratings, keywords, test workspaces, location-workspace
mapping, search keyword mapping, test cases, and so forth) as
described in this disclosure. Such databases may be implemented as
fault-tolerant, relational, scalable, secure databases such as
Oracle or Sybase. Alternatively, such databases may be implemented
using standardized data structures, such as an array, hash, linked
list, struct, structured text file (e.g., XML), table, or as
object-oriented databases (e.g., using ObjectStore, Poet, Zope,
etc.). Such databases may be consolidated or distributed, sometimes
among the various computer systems discussed above in this
disclosure. It is to be understood that the structure and operation
of the any computer or database component may be combined,
consolidated, or distributed in any working combination.
[0054] As will be appreciated by those skilled in the art, the
techniques described in the various embodiments discussed above
results in efficient, robust, and cost-effective management and
testing of software production incidents subsequent to release of a
software product. The techniques described in the embodiments
discussed above provide an automated process of testing a defect
that is found in production, thereby ensuring a consistent and
predictive delivery of software product quality. The self-learning
mechanism of the process ensures that the process keeps improving
its efficiency after putting to use. The feedback mechanism of the
process provides feedback to user with respect to benefits and the
areas of improvement. The feedback mechanism further receives
feedback from users for continuous improvement of the process.
Additionally, the techniques described in the embodiments discussed
above analyzes the production defects and learns from the pattern,
correlates the defects with the existing test cases, and ensures a
smooth build, run, and reinstallation into production. Further, the
techniques described in the embodiments discussed above is easy to
build and use and can be integrated with any system.
[0055] The specification has described system and method for
optimizing testing of software production incidents. The
illustrated steps are set out to explain the exemplary embodiments
shown, and it should be anticipated that ongoing technological
development will change the manner in which particular functions
are performed. These examples are presented herein for purposes of
illustration, and not limitation. Further, the boundaries of the
functional building blocks have been arbitrarily defined herein for
the convenience of the description. Alternative boundaries can be
defined so long as the specified functions and relationships
thereof are appropriately performed. Alternatives (including
equivalents, extensions, variations, deviations, etc., of those
described herein) will be apparent to persons skilled in the
relevant art(s) based on the teachings contained herein. Such
alternatives fall within the scope and spirit of the disclosed
embodiments.
[0056] Furthermore, one or more computer-readable storage media may
be utilized in implementing embodiments consistent with the present
disclosure. A computer-readable storage medium refers to any type
of physical memory on which information or data readable by a
processor may be stored. Thus, a computer-readable storage medium
may store instructions for execution by one or more processors,
including instructions for causing the processor(s) to perform
steps or stages consistent with the embodiments described herein.
The term "computer-readable medium" should be understood to include
tangible items and exclude carrier waves and transient signals,
i.e., be non-transitory. Examples include random access memory
(RAM), read-only memory (ROM), volatile memory, nonvolatile memory,
hard drives, CD ROMs, DVDs, flash drives, disks, and any other
known physical storage media.
[0057] It is intended that the disclosure and examples be
considered as exemplary only, with a true scope and spirit of
disclosed embodiments being indicated by the following claims.
* * * * *