U.S. patent application number 15/920546 was filed with the patent office on 2019-09-19 for method for generating developer performance ratings.
This patent application is currently assigned to DEEPCODING LTD.. The applicant listed for this patent is DEEPCODING LTD.. Invention is credited to Sebastien Adjiman, Ronen COHEN, Arnon Yaffe.
Application Number | 20190287040 15/920546 |
Document ID | / |
Family ID | 67905825 |
Filed Date | 2019-09-19 |
United States Patent
Application |
20190287040 |
Kind Code |
A1 |
COHEN; Ronen ; et
al. |
September 19, 2019 |
METHOD FOR GENERATING DEVELOPER PERFORMANCE RATINGS
Abstract
A computer implemented method for generating a performance
rating for a developer may include monitoring developer activities
to obtain near real-time activity data; exploring the near
real-time activity data to identify entities; structuring the near
real-time activity data into data-frame objects; performing a
feature engineering procedure to measure representative behaviors
of the developer and performing a performance analysis to produce
performance rating of the developer.
Inventors: |
COHEN; Ronen; (Gesher Haziv,
IL) ; Adjiman; Sebastien; (Tel Aviv, IL) ;
Yaffe; Arnon; (Tel Aviv, IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
DEEPCODING LTD. |
Tel Aviv |
|
IL |
|
|
Assignee: |
DEEPCODING LTD.
Tel Aviv
IL
|
Family ID: |
67905825 |
Appl. No.: |
15/920546 |
Filed: |
March 14, 2018 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06Q 10/06398
20130101 |
International
Class: |
G06Q 10/06 20060101
G06Q010/06 |
Claims
1. A computer implemented method for generating a performance
rating for a developer, the method comprising: monitoring developer
activities to obtain near real-time activity data; receiving at
least one process-flow specific to a development process; exploring
the near real-time activity data related to the developer, to
identify entities based on the at least one process flow, wherein
said entities comprise at least one of: activity identifier,
content identifier, action, content type, unique content
identifier; structuring the near real-time activity data into
data-frame objects, based on the identified entities; receiving at
least one performance metric and at least one measurement related
to the at least one performance metric, wherein said at least one
performance metric and at least one measurement metric embody the
criteria that is used to rate said developer; performing a feature
engineering procedure to measure representative behaviors of the
developer, related to the at least one measurement and the at least
one performance metric; receiving a number of performance levels
related to the at least one performance metric; receiving a
performance scoring scale related to the at least one performance
metric; and performing a performance analysis on the measured
representative behaviors to produce performance rating of the
developer.
2. The method according to claim 1, wherein the performance
analysis is further producing a score card of the developer.
3. A computer implemented method for generating a performance
rating for a developer, the method comprising: monitoring developer
activities to obtain near real-time activity data; receiving at
least one process-flow specific to a development process; exploring
the near real-time activity data related to the developer, to
identify entities based on the at least one process flow, wherein
said entities comprise at least one of: activity identifier,
content identifier, action, content type, unique content
identifier; structuring the near real-time activity data into
data-frame objects, based on identified entities; receiving at
least one performance metric and at least one measurement related
to the at least one performance metric, wherein said at least one
performance metric and at least one measurement metric embody the
criteria that is used to rate said developer; performing a feature
engineering procedure to measure representative behaviors of the
developer, related to the at least one measurement and the at least
one performance metric; receiving a set of labels, the labels
representing the developer performance rating, the set of labels
provided by a customer; and performing a performance analysis on
the measured representative behaviors to produce a performance
rating mapping function based on the set of labels and obtaining
importance of the measurements.
4. The method according to claim 2, further comprising generating
performance recommendations relevant to the developer.
5. The method according to claim 1, further comprising sending the
performance rating of the developer to a receiving device.
6. The method according to claim 2, further comprising sending the
performance rating and the score card of the developer to a
receiving device.
7. The method of claim 4, further comprising sending the scorecard,
the performance rating and the performance recommendations relevant
to the developer, to a receiving device.
8. The method of claim 3, further comprising: receiving a number of
performance levels related to the at least one performance metric;
receiving a performance scoring scale related to the at least one
performance metric; and generating performance recommendations
relevant to the developer, wherein the performance analysis is
further comprising a performance scoring.
9. The method of claim 8, further comprising sending the scorecard,
and the performance recommendations relevant to the developer to a
receiving device.
10. A non-transitory computer readable storage medium storing a set
of instructions for causing a computer to generate a performance
rating for a developer, by performing the operations of: monitoring
developer activities to obtain near real-time activity data;
receiving at least one process-flow specific to a development
process; exploring the near real-time activity data related to said
developer, to identify entities based on the at least one process
flow, wherein said at least one performance metric and at least one
measurement metric embody the criteria that is used to rate said
developer; structuring the near real-time activity data into
data-frame objects, based on the identified entities; receiving at
least one performance metric and at least one measurement related
to the at least one performance metric, wherein said at least one
performance metric and at least one measurement metric embody the
criteria that is used to rate said developer; performing a feature
engineering procedure to measure representative behaviors of the
developer, related to the at least one measurement and the at least
one performance metric; receiving a number of performance levels
related to the at least one performance metric; receiving a
performance scoring scale related to the at least one performance
metric; and performing a performance analysis on the measured
representative behaviors to produce a performance rating of the
developer.
11. The non-transitory computer readable storage medium of claim
10, wherein the performance analysis is further producing a
scorecard of the developer.
12. A non-transitory computer readable storage medium storing a set
of instructions for causing a computer to generate a performance
rating for a developer, by performing the operations of: monitoring
developer activities to obtain near real-time activity data;
receiving at least one process-flow specific to a development
process; exploring the near real-time activity data related to said
developer, to identify entities based on the at least one process
flow, wherein said at least one performance metric and at least one
measurement metric embody the criteria that is used to rate said
developer; structuring the near real-time activity data into
data-frame objects, based on the identified entities; receiving at
least one performance metric and at least one measurement related
to the at least one performance metric; performing a feature
engineering procedure to measure representative behaviors of the
developer, related to the at least one measurement and the at least
one performance metric; receiving a set of labels, the set of
labels representing the developer performance rating, the set of
labels provided by a customer; and performing a performance
analysis on the measured representative behaviors to produce a
performance rating mapping function based on the set of labels and
obtaining importance of the measurements.
13. The non-transitory computer readable storage medium of claim 10
further performing the operation of generating performance
recommendations relevant to the developer.
14. The non-transitory computer readable storage medium of claim 12
further performing the operations of: receiving a number of
performance levels related to the at least one performance metric;
receiving a performance scoring scale related to the at least one
performance metric; and generating performance recommendations
relevant to the developer, wherein the performance analysis is
further comprising a performance scoring.
Description
BACKGROUND OF THE INVENTION
Field of the Invention
[0001] The invention relates generally to data analysis, and more
specifically to a method for generating developer performance
ratings.
Description of the Background Art
[0002] Various IT data centric projects, such as Enterprise
Resource Planning (ERP), regulatory reports creating in the domain
of operational risk management, and creation of product catalogs
like Amazon and eBay, carry different level of risks, such as the
project may not be completed on time, within the budget and with
the promised quality.
[0003] The performance of a developer plays a prominent role in
making the project successful.
[0004] Performance evaluation programs are frequently used by
organizations, such as companies and government agencies, to
evaluate employee performance and determine compensation for
employees. In a typical performance evaluation program, performance
is measured based on establishing a goal for the evaluation time
period. When an employee meets the goal, an incentive is awarded to
the employee. As a result, once an employee has met the goal, he or
she has no incentive to achieve a higher performance. In a similar
fashion, an employee that is far from reaching the goal has no
incentive to work toward a higher performance.
[0005] The developer performance is one of the prominent factors to
influence the risk level of a project. The closer an employee is to
a target, the more motivated he or she will be to achieve the
target. In addition, employees who are far above or below a target
should be provided with an incentive to continue improving because
the organization will benefit from their efforts.
SUMMARY OF THE INVENTION
[0006] The invention rates the developer's performance, suggests
recommendations in the spirit of how the developer can improve his
performance, and by so, helps the manager to allocate developers to
projects, in an optimized way.
[0007] Some embodiments of the present invention provide a computer
implemented method for generating a performance rating for a
developer. The method may include the following steps: monitoring
developer activities to obtain near real-time activity data;
receiving at least one process-flow specific to a development
process; exploring the near real-time activity data, to identify
entities based on the at least one process flow; structuring the
near real-time activity data into data-frame objects, based on the
identified entities; receiving at least one performance metric and
at least one measurement related to the at least one performance
metric; performing a feature engineering procedure to measure
representative behaviors of the developer, related to the at least
one measurement and the at least one performance metric; receiving
a number of performance levels for the at least one performance
metric; receiving a performance scoring scale and performing a
performance analysis on the measured representative behaviors to
produce performance rating of the developer.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The subject matter regarded as the invention is described in
detail and distinctly claimed in the concluding portion of the
specification. The invention, however, both as to organization and
method of operation, together with objects, features, and
advantages thereof, may best be understood by reference to the
following detailed description when read with the accompanying
drawings in which:
[0009] FIG. 1 is a block diagram illustrating one embodiment of a
performance rating system, according to the present invention;
[0010] FIG. 2 is a functional block diagram of another embodiment
of a performance rating engine of FIG. 1, according to the present
invention;
[0011] FIG. 3 is a functional block diagram of one embodiment of a
performance rating engine of FIG. 1, according to the present
invention;
[0012] FIG. 4 illustrates one embodiment of an analysis base table,
according to the present invention;
[0013] FIG. 5 illustrates one embodiment of a performance scoring
table, according to the present invention;
[0014] FIG. 6 illustrates one embodiment of a performance rating
screen, according to the present invention;
[0015] FIG. 7 illustrates on embodiment of a scorecard of a
developer, for the expertise metric, according to the present
invention;
[0016] FIG. 8 illustrates on embodiment of a scorecard of a
developer, for the productivity metric, according to the present
invention.
[0017] FIG. 9 illustrates one embodiment of performance
recommendations for a number of developers, according to the
present invention.
[0018] FIG. 10 shows a flowchart of method steps for computing a
performance rating, according to one embodiment of the present
invention;
[0019] It will be appreciated that, for simplicity and clarity of
illustration, elements shown in the figures have not necessarily
been drawn to scale. For example, the dimensions of some of the
elements may be exaggerated relative to other elements for clarity.
Further, where considered appropriate, reference numerals may be
repeated among the figures to indicate corresponding or analogous
elements.
DETAILED DESCRIPTION OF THE INVENTION
[0020] In the following detailed description, numerous specific
details are set forth in order to provide a thorough understanding
of the invention. However, it will be understood by those skilled
in the art that the present invention may be practiced without
these specific details. In other instances, well-known methods,
procedures, and components have not been described in detail so as
not to obscure the present invention.
[0021] Although embodiments of the invention are not limited in
this regard, discussions utilizing terms such as, for example,
"processing," "computing," "calculating," "determining,"
"establishing", "analyzing", "checking", or the like, may refer to
operation(s) and/or process(es) of a computer, a computing
platform, a computing system, or other electronic computing device,
that manipulates and/or transforms data represented as physical
(e.g., electronic) quantities within the computer's registers
and/or memories into other data similarly represented as physical
quantities within the computer's registers and/or memories or other
information non-transitory storage medium that may store
instructions to perform operations and/or processes. Although
embodiments of the invention are not limited in this regard, the
terms "plurality" and "a plurality" as used herein may include, for
example, "multiple" or "two or more". The terms "plurality" or "a
plurality" may be used throughout the specification to describe two
or more components, devices, elements, units, parameters, or the
like. The term set when used herein may include one or more items.
Unless explicitly stated, the method embodiments described herein
are not constrained to a particular order or sequence.
Additionally, some of the described method embodiments or elements
thereof can occur or be performed simultaneously, at the same point
in time, or concurrently.
[0022] The present invention provides a method for rating developer
performance, suggesting recommendations in the spirit of how the
developer can improve his performance, and, by doing so, enabling
optimization of the allocation of developers to projects and
tasks.
[0023] FIG. 1 is a block diagram illustrating a performance rating
system 100, according to some embodiments of the present invention.
Performance rating system 100 implements a performance rating
engine that uses information from past transactions involving the
developers being rated to compute a performance scoring and rating
for each developer. The term "developer" as used herein refers to
an Information Technology (IT) professional services developer,
tester, or architect.
[0024] The term "vendor" as used herein refers to a company that
sells the software/application that the developer implements.
[0025] The term "outlier" as used herein refers to a piece of data
or observation that deviates drastically from the given norm or
average of the data set.
[0026] The term "outlier detection" as used herein refers to a
process of detecting and subsequently excluding outliers from a
given set of data.
[0027] The term "correlation analysis" as used herein refers to a
method of statistical evaluation used to study the strength of a
relationship between two or more measurements.
[0028] The term "near real-time" or "nearly real-time" (NRT), in
telecommunications and computing, refers to the time delay
introduced, by automated data processing or network transmission,
between the occurrence of an event and the use of the processed
data, such as for display or feedback and control purposes.
[0029] Performance rating system 100 may include performance rating
inputs 102 and a processing unit device 120, which further may
include a performance rating engine 132, a resource allocation
engine 150 and a receiving device 140. Performance inputs 102 may
include a set of performance metrics and measurements for each
performance metric 104, performance scoring scale 106, a set of
performance levels for each metric 108, a set of performance level
weightings 110, a set of customer labels 116, near real-time
activity data 114 and process flows 116. Each of the performance
rating inputs 102 is discussed below in further detail in
conjunction with FIGS. 2-10. Near real-time activity data 114 may
include any information from near real-time activity data derived
from the monitoring of the developer being rated, during its daily
activity, when using a dedicated application.
[0030] Processing unit device 120 may be any type of processing
unit device having a processor, a memory configured to store
software and one or more input/output devices. Some examples of
processing unit devices include personal computers, laptop
computers, network-based computers and hand-held computing
devices.
[0031] Performance rating engine 132 may include an explorer 122, a
structure generator 124, a feature engineering process 126, a
performance analysis 128 and a recommendation generator 130. Each
of these elements is discussed below in further detail in
conjunction with FIGS. 2-10.
[0032] Resource allocation engine 150 is a component designed for
resource allocation of developers when there is a need to add
resources to an existing project or to allocate resources to a new
project. Resource allocation engine 150 may be based on linear
optimization algorithm, to allocate developers based on the
performance rating outputs and a set of constraints required by the
project such as the type of expertise of a developer, the time
constraint, the complexity, among other constraints.
[0033] Receiving device 140 receives the rating outputs from the
performance rating engine 132. Receiving device 140 may be any
device capable of receiving the performance rating outputs,
including any type of computer monitor or other, printer or other
printing device or display device or storage medium or other
storage device.
[0034] FIG. 2 is a functional block diagram of performance rating
engine 200, according to some embodiments of the present invention.
According to some embodiments, a data scientist may set the set of
performance metrics and measurements for each performance metric
104, the performance scoring scale 106, a set of performance levels
for each metric 108, a set of performance level weightings 110. The
process flows 116 may be designed by a system architect. Near
real-time activity data 114 relating to the developers to be rated
may be entered into the performance rating engine 132.
[0035] Set of performance metrics and set of measurements 104
embodies the criteria that the IT manager uses to rate the
different developers and forms the frame work of the performance
rating engine 132. The designer of the performance rating engine,
therefore, structures set of performance metrics and performance
measurements 104 to reflect factors related to developer
performance that are dependent on the applications used by the
developer. For example, performance metrics may include expertise,
productivity, complexity and quality, among others.
[0036] For example, the expertise metric may include any of the
following measurements, among others:
[0037] error rate--referring to what degree the developer produces
error, such as the average number of errors per day.
[0038] automation--referring to how efficient is the developer in
creating reports, such as the number of workflows divided by the
number of reports.
[0039] confidence level--referring to how confident is the
developer in what he is developing, such as the number of data
processing activities divided by the number of data review
activities.
[0040] modeling--referring to data modeling proficiency, such as
the number of data sources divided by the number of data model.
[0041] modularity--referring to how the developer is handling
complexity.
[0042] For example, the productivity metric may include any of the
following measurements, among others:
[0043] developer speed--referring how intensively the developer is
working, such as the average elapsed time between developer
actions.
[0044] development rate--referring how productive is the developer,
such as the number of objects divided by the total development
hours per day.
[0045] processing--referring how productive is the developer, such
as the average number of hours per day spent in processing
data.
[0046] For example, the complexity metric may include any of the
following measurements, among others:
[0047] compliance--referring to how messy is the input data, such
as the number of validation exceptions and group exceptions that
were violated.
[0048] source count--referring to how complex is the input data,
such the as number of data sources.
[0049] freeform complexity--referring to how complex is the
freeform report.
[0050] tabular complexity--referring to the time spent on the
longest report.
[0051] Explorer 122 may be configured to receive the near real-time
activity data 114 and the process flows 116 to identify the main
entities that participate in the development process. The main
identities may be, for example, activity types, objects types to be
developed, and the outputs that were created, among others. The
functionality of the explorer 122 may be implemented in hardware
and/or software, using any visual tool.
[0052] Given the understanding that resulted from the exploration
step, structure generator 124 may extract the entities identified
and verified by the vendor and may transform the near real-time
data into data-frame objects based on the identified identities,
where each column may represent a measurement and each row may be
an observation
[0053] According to some embodiments, data cleansing may be
applied, to remove different discrepancies such as empty rows,
inconsistencies and the like.
[0054] Feature engineering process 126 may receive as input the
structured data from the structure generator 124 and may then
measure representative behaviors of the developer, related to each
one of the performance metrics and measurements sets 104.
[0055] According to some embodiments, the output of the feature
engineering process 126 may be represented as an analysis-based
table (ABT), such the one discussed below in further details in
conjunction with FIG. 4.
[0056] In an embodiment where the data is unlabeled, the approach
applied may be the unsupervised approach, as shown in FIG. 2,
proceeding with unsupervised machine learning. Performance analysis
128 may use the performance scoring scale 106 and the performance
levels 108 to convert the measured behaviors into unitless values.
For example, a data scientist may choose the performance scoring
scaling 106 to be 60-140, and the number of performance levels 108
to be five (5), for example as (very high, high, medium, low)
related to productivity metric, (expert, proficient, competent,
advanced beginner, novice) related to the expertise metric. The
performance analysis 128 may include two stages: a performance
scoring stage and a performance rating stage.
[0057] Each developer may then be scored by a distribution modeling
based on the performance scoring scale 106, giving the score of
developer in every measurement. According to some embodiments, the
average score over all measurements may be calculated. According to
other embodiments, the score of the developer may be determined by
taking into account a set of performance level weightings 110 in
order to single out the importance of the measurement in the eyes
of the domain expert in determining the overall developer
performance.
[0058] The performance rating may be then performed. The scores may
be fed into an unsupervised machine learning algorithm to detect
developers that are similar across all the performance metrics 104
that are part of the developer performance conceptual model, for
example using K-means algorithm. FIG. 6 illustrates one embodiment
of a performance rating screen, according to the present invention,
as discussed below in further detail. A score card may be then
obtained for each developer, based on the performance levels 108.
FIG. 7 illustrates an embodiment of score cards for the expertise
metric, and FIG. 8 illustrates an embodiment of score cards for and
the productivity metric, according to the present invention and as
discussed below in further detail.
[0059] In some embodiments, recommendations how the developer may
improve his performance for at least one metric may be generated by
the recommendation generator 130, based on the scoring performance
of the developer calculated by the performance analysis 128 and the
performance levels 108. The recommendation generator 130 may use
outlier detection and anomaly detection levering correlation
analysis. FIG. 9 illustrates one embodiment of performance
recommendation for some developers, according to the present
invention and as discussed below in further detail.
[0060] In an embodiment where the data is labeled, the approach
applied may be the supervised approach, as shown in FIG. 3. In such
a case, a customer may label the developer performance rating by
inputting a set customer labels 112. Performance analysis 128 may
be performed by using a supervised machine learning algorithm known
as a classifier, such as neural network, or boosting or bagging
among others. The machine learning may be trained on the data
results from the feature engineering process 126 to produce a
performance rating function based on the set of customer labels
112. According to some embodiments, the measurements importance may
also be determined.
[0061] In some embodiments where the data is labeled, the
performance analysis 128 may further include a performance scoring
stage, similar to as described previously.
[0062] FIG. 4 illustrates one embodiment of an analysis base table
400, for a developer named Robert, according to the present
invention. As shown, each column in the analysis table represents
an entity, such as activity ID, content ID, action, content type,
unique content ID, among others, and each row represents an
observation.
[0063] FIG. 5 illustrates one embodiment of a performance scoring
table, according to the present invention. As shown, each column
may refer to a measurement being part of the set of performance
metrics and measurements 104, such as automation (related to the
expertise metric, compliance (related to the complexity metric),
among others and each row may refer to a developer. As shown, the
scoring is based on the performance scoring scale 106, which is
designated to be 60-140 in this example.
[0064] FIG. 6 illustrates one embodiment of a performance rating
screen, according to the present invention. On the lower part, it
may show the score of each developer. The hatching scheme may
correspond to the clusters, such that the K-means algorithm
detects. In this embodiment, six distinct groups of developers were
detected. The upper part is a set of filters that may dice the data
according to any criteria, such as performance measurement, any
metric or to a specific developer.
[0065] FIG. 7 illustrates one embodiment of a developer scorecard
700, for a developer named David, for a metric named expertise, and
recommendation generated for the developer David, according to the
present invention. The upper part may represent the performance
scoring related to the expertise metric, and each row may represent
the performance scoring related to each measurement being included
in the expertise metric. The lower part may represent the
recommendation generated by the recommendation generator 130.
[0066] FIG. 8 illustrates one embodiment of a developer scorecard
800, for a developer named Barbara, for a metric named
productivity, and recommendation generated for the developer
Barbara, according to the present invention. The upper part may
represent the performance scoring related to the productivity
metric, each row may represent the performance scoring related to
each measurement being included in the productivity metric. The
lower part may represent the recommendation generated by the
recommendation generator 130.
[0067] FIG. 9 illustrates one embodiment of performance
recommendations for some developers, according to the present
invention. For each developer, per project, the problem type may be
exposed, and recommendations may be generated for each of the
problem type, by the recommendation generator 130.
[0068] FIG. 10 shows a flowchart 1000 of method steps for computing
a performance rating, according to one embodiment of the present
invention.
[0069] Although the method steps are described in the context of
the system illustrated in FIGS. 1-3, any system configured to
perform the method steps is within the scope of the present
invention.
[0070] As shown in FIG. 10, in step 1002, the domain expert may
designate process-flows specific to a development process 116.
[0071] In step 1004, developer activities may be monitored to
obtain near real-time activity data 114 related to the
developer.
[0072] In step 1006, the near real-time activity data 114 may be
explored based on the process-flows 116 to obtain identify the main
entities that participate in the development process, such as
different types of activities, objects to be developed, and the
outputs that were created.
[0073] In step 1008, near real-time activity data 114 may be
structured into data-frame objects based on the identified
identities. According to some embodiments, a cleansing routine may
be part of the structuring process.
[0074] As shown in step 1010, a data scientist may designate a set
of performance metrics and measurements 104, which forms the basis
of the performance rating engine, to reflect factors related to
developer performance.
[0075] In step 1012, feature engineering procedure 126 may be
performed to measure representative behaviors of the developer,
related to any measurement and performance metric. According to
some embodiments, the output of the feature engineering process 126
may be represented as an analysis-based table (ABT), such as the
one discussed below in further detail in conjunction with FIG.
4.
[0076] In step 1014, a data scientist may designate performance
levels 108 and performance scoring scale 106.
[0077] In step 1016, performance analysis 128 may then be applied,
to obtain performance rating and scorecard of a developer. FIG. 6
illustrates one embodiment of a performance rating screen,
according to the present invention as discussed above in further
detail. FIG. 7 and FIG. 8 illustrate an embodiment of score cards
for the expertise metric and the productivity metric, according to
the present invention and as discussed above in further
details.
[0078] In step 1018, recommendations may be generated by the
recommendation generator 130. FIG. 9 illustrates one embodiment of
performance recommendation for some developers, according to the
present invention and as discussed above in further detail.
[0079] While certain features of the invention have been
illustrated and described herein, many modifications,
substitutions, changes, and equivalents will now occur to those of
ordinary skill in the art. It is, therefore, to be understood that
the appended claims are intended to cover all such modifications
and changes as fall within the true spirit of the invention.
* * * * *