U.S. patent application number 12/030684 was filed with the patent office on 2009-08-13 for system and method for conducting competitions.
Invention is credited to Gregory Eldridge, Javier Fernandez-Ivern, Michael A. Fogleman, John M. Hughes, Anthony Jefts, David Messinger, George Tsipolitis.
Application Number | 20090203413 12/030684 |
Document ID | / |
Family ID | 40939350 |
Filed Date | 2009-08-13 |
United States Patent
Application |
20090203413 |
Kind Code |
A1 |
Jefts; Anthony ; et
al. |
August 13, 2009 |
SYSTEM AND METHOD FOR CONDUCTING COMPETITIONS
Abstract
In general, in one aspect, a method for developing an asset by
competition includes posting a list of competitions, each
competition for the development of an asset, each element of the
list including a reference to one of a number of workspaces, the
one of the workspaces allocated to a respective asset. The method
includes providing a draft competition specification in one of the
workspaces. The method includes facilitating feedback from
potential competitors on the draft competition specification in the
workspace for a period of time prior to the competition. The method
includes finalizing the draft competition specification based on
the feedback from potential competitors, and holding a competition
for the development of the asset.
Inventors: |
Jefts; Anthony; (Warwick,
RI) ; Eldridge; Gregory; (Winter Springs, FL)
; Fernandez-Ivern; Javier; (Norwalk, CT) ;
Fogleman; Michael A.; (Middletown, CT) ; Hughes; John
M.; (Hebron, CT) ; Messinger; David; (West
Hartford, CT) ; Tsipolitis; George; (Cromwell,
CT) |
Correspondence
Address: |
Goodwin Procter LLP
Patent Administrator, 53 State Street
Boston
MA
02109-2881
US
|
Family ID: |
40939350 |
Appl. No.: |
12/030684 |
Filed: |
February 13, 2008 |
Current U.S.
Class: |
463/9 |
Current CPC
Class: |
A63F 9/18 20130101 |
Class at
Publication: |
463/9 |
International
Class: |
A63F 9/24 20060101
A63F009/24 |
Claims
1. A method for developing an asset by competition, comprising:
posting a list of competitions, each competition for the
development of an asset, each element of the list including a
reference to one of a number of workspaces, the one of the
workspaces for a respective asset; providing a draft competition
specification in one of the workspaces; facilitating feedback from
potential competitors on the draft competition specification in the
workspace for a period of time prior to the competition; finalizing
the draft competition specification using the feedback; and holding
a competition for the development of the asset.
2. The method of claim 1 wherein the workspace comprises a
wiki.
3. The method of claim 2, wherein the feedback comprises changes to
the specification.
4. The method of claim 1, wherein some of the competitions in the
list are upcoming.
5. The method of claim 1, wherein some of the competitions in the
list are active competitions.
6. The method of claim 1, wherein the asset comprises a software
component design, and the specification comprises a requirements
specification.
7. The method of claim 1 wherein the asset comprises an asset
selected from the group of assets consisting of: software
component, software component code, graphic design, architectural
design, product design, product specification, article, creative
work, essay, video, audio, multimedia work, and presentation; and
wherein the specification comprises a description of the desired
asset.
8. The method of claim 1, wherein facilitating feedback comprises
facilitating feedback from potential competitors and
administrators.
9. The method of claim 1, wherein the list comprises a competition
name, a planned competition start date, and the reference to the
workspace.
10. The method of claim 1, wherein the reference to the workspace
comprises a URL.
11. A system for developing an asset by competition, comprising: a
competition list comprising a list of competitions, each
competition for the development of an asset, each element of the
list including a reference to one of a number of workspaces, the
one of the workspaces allocated to a respective asset; a workspace
comprising a draft competition specification and a facility for
receiving feedback from potential competitors on the draft
competition specification in the workspace for a period of time
prior to the competition; a competition system for holding a
competition for the development of the asset using a specification
that is based on the draft competition specification taking into
account the feedback from potential competitors.
12. The system of claim 11 wherein the workspace comprises a
wiki.
13. The system of claim 12, wherein the feedback comprises changes
to the specification.
14. The system of claim 11, wherein some of the competitions in the
list are upcoming.
15. The system of claim 11, wherein the list comprises active
competitions and upcoming competitions.
16. The system of claim 11, wherein the asset is a software
component design, and the specification comprises a requirements
specification.
17. The system of claim 11 wherein the asset comprises a graphic
design, and the specification comprises a description of the
desired asset.
18. The system of claim 11, wherein the asset comprises a
mechanical part design, and the specification comprises a
description of the requirements for the desired asset.
19. The system of claim 11, wherein the list comprises a
competition name, a planned competition start date, and the
reference to the workspace.
20. The system of claim 11, wherein the reference to the workspace
comprises a URL.
Description
TECHNICAL FIELD
[0001] This invention relates to computer-based methods and systems
for facilitating the development of an asset.
BACKGROUND INFORMATION
[0002] Competitions have been held for the development of an asset,
such as essays, music, computer software, and so on.
[0003] In some competitions, a larger work is divided up into a
number of smaller pieces, and competitions held for the development
of those pieces. For example, a software program may be divided up
into components, and competitions held for the development of each
of the components. In some cases, development of a component may be
divided into design tasks and coding tasks, and a contest held for
the design of the component, and another contest held for the
coding of the component. As another example, a competition may be
held for the development of a graphic design, such as a logo.
[0004] In such competitions, specifications typically are not made
available until the competition begins. In such cases,
specifications are developed without the benefit of input from
potential competitors. Sometimes substantial interaction with the
competitors during the competition is required to clarify parts of
the specification that are not clear. Correction to specifications
may be necessary in order to fix problems identified by the
competitors, potentially requiring increases to the competition
time and reducing the likelihood of competition success.
SUMMARY OF THE INVENTION
[0005] Various asset development competitions have been used to
develop assets, such as content, computer software, graphics, user
interfaces, and so forth. Such competitions have been held, for
example, by TOPCODER, INC. of Glastonbury, Conn., and may be seen
at http://www.topcoder.com.
[0006] As described in co-pending U.S. patent application Ser. No.
11/035,783, entitled SYSTEMS AND METHODS FOR SOFTWARE DEVELOPMENT
by John M. Hughes, filed Jan. 14, 2005, incorporated herein by
reference, a software application may be developed by contest,
where the task of developing the software application is divided up
into discrete tasks, and competitions are held for the performance
of each of the discrete tasks. For example, a software program may
be divided up into components, and competitions held for the
development of each of the components. Development of a component
may be divided into design tasks and coding tasks, and a contest
held for the design of the component, and another contest held for
the coding of the component.
[0007] As described in co-pending U.S. patent application Ser. No.
11/655,768, entitled SYSTEM AND METHOD FOR DESIGN DEVELOPMENT by
John M. Hughes, filed Jan. 19, 2007, incorporated herein by
reference, design contests may be held, for example, for graphics
design and user interface development as well as other types of
designs and work product. Submissions in such contests may be
evaluated for technical merit (i.e., meeting the described
requirements) and/or based on customer affinity and/or appeal to a
designated group of individuals.
[0008] As described in co-pending U.S. patent application Ser. No.
11/755,909, entitled PROJECT MANAGEMENT SYSTEM, by Messinger et
al., filed May 31, 2007, incorporated herein by reference, a series
of contests may be managed through use of project management
system. Portions of a contest, such as a review effort, may begin
when submissions are received, deadlines are met, and so on. Such
contest systems can provide contest-related information to contest
administrators as well as potential competitors.
[0009] In various embodiments, techniques may be used to inform
potential contestants in advance about upcoming competitions. At
the same time, specifications may be made available to potential
contestants, for review and comment. Making the specifications
available in advance enables potential competitors to make better
decisions about whether to participate in a competition, and also
allows them to comment on or clarify the specification for the
competition in advance of the competition.
[0010] For example, in some embodiments, a list of competitions is
provided. Some of the competitions in the list may be upcoming. In
addition, some of the competitions in the list may be active
competitions and/or completed competitions. The list may be sorted
and/or filtered by date, type of competition and/or other
information. The list may be communicated using any suitable
information communication technique, for example by one or more of
a web page, email, RSS feed, text message, and so on. Each
competition is for the development of one or more assets. Each
competition listing may include a reference to one of a number of
workspaces, typically a workspace allocated to the respective
asset. For example, the workspace may be or may include one or more
of a shared document, a wiki page, a web page, a blog page, a
forum, a bulletin board, and so on. The workspace includes a place
for providing a draft competition specification.
[0011] The competition specification may include a technical
description of the asset to be developed. The competition
specification may include the rules and/or parameters of the
competition (e.g., dates, prizes, and so on). The competition
specification may include other information, such as other
documents and tools that may be used, reference information,
submission details, competition logistics, and so on.
[0012] Feedback may be received from potential competitors on the
draft competition specification in the workspace. Typically,
feedback is received for a period of time prior to the competition.
The feedback may include (among other things) any one or more of
comments on the specification, text changes to the specification,
graphical or design changes, other modifications to the
specification, additions to the specification, and so on. The draft
competition specification (including without limitation information
about competition logistics) may be finalized based on the
feedback, and a competition for the development of the asset held
using the finalized specification as the competition
specification.
[0013] The asset that is to be developed by competition may be any
type of asset. Just as some non-limiting examples, the asset may be
or may include software component design, graphic design, and
mechanical part design, software component, product design, product
specification or plans, creative work, artistic work, video, audio,
multimedia presentation, music, etc. The list of competitions may
include, for example, a competition name, a planned competition
start date, other pertinent meta data describing the competition
such as an overview, technologies involved, timelines, and so on
and/or a reference to the workspace. The reference to the workspace
may include a URL.
[0014] In general, in another aspect, a system for developing an
asset by competition comprises a competition list including a list
of competitions, each competition for the development of an asset,
each element of the list including a reference to one of a number
of workspaces, the one of the workspaces allocated to a respective
asset. The system includes a workspace including a draft
competition specification and a facility for receiving feedback
from potential competitors on the draft competition specification
in the workspace for a period of time prior to the competition. The
system includes a competition system for holding a competition for
the development of the asset using a specification that is based on
the draft competition specification taking into account the
feedback from potential competitors.
[0015] The systems described can be implemented as software running
on computers, mobile devices and telephones, televisions, home
entertainment centers, gaming consoles, and other devices, for
example, as described in the above-referenced co-pending patent
applications.
[0016] Other aspects and advantages of the invention will become
apparent from the following drawings, detailed description, and
claims, all of which illustrate the principles of the invention, by
way of example only.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In the drawings, like reference characters generally refer
to the same parts throughout the different views. Also, the
drawings are not necessarily to scale, emphasis instead generally
being placed upon illustrating the principles of the invention.
[0018] FIG. 1 is a flowchart illustrating by example a method for
developing work product according to an embodiment of the
invention.
[0019] FIG. 2 is a flow chart illustrating by example a method for
determining an estimated likelihood of success according to an
embodiment of the invention.
[0020] FIG. 3 is a block diagram illustrating by example an
embodiment of a development contest.
[0021] FIG. 4 is a block diagram illustrating by example an
embodiment of a contest-based software application development
process.
[0022] FIG. 5 is a demonstrative screen display in an exemplary
embodiment of the invention.
[0023] FIG. 6 is a demonstrative screen display in an exemplary
embodiment of the invention.
DETAILED DESCRIPTION
[0024] Referring to FIG. 1, in some embodiments, a method 100 for
developing an asset by competition may be applicable to any type of
asset. Just as examples, not intended to be limiting, in various
embodiments, the asset developed by competition may be or may
include text, content, graphics, design, development, data,
architecture, and/or specifications, for software, hardware, audio,
video, music, machines, equipment, devices, and/or media. The
competition may be any suitable competition. Just as examples, not
intended to be limiting, the competition may be an on-line
competition, in-person competition, live competition, and/or timed
competition. In some embodiments, the competition is an on-line
competition for the development of a portion of a computer software
application, such as a design for computer code, computer code
based on a design, or a user interface design.
[0025] Typically, a competition involves two or more participants
competing to develop an asset. A specification describes
requirements for an asset and may also include competition rules
and logistics. Standards may be specified for the asset and may be
specific to the type of asset. Standards may be described for
example, in a checklist or scorecard, and may be general or
specific as appropriate. In some embodiments, a review of an asset
is conducted using the specification and the standards in order to
identify a competition winner, and the review criteria include the
relevant standards.
[0026] In various embodiments, techniques may be used to enable
communication between potential contestants, contest
administrators, and contest sponsors in advance of upcoming asset
development competitions. In some cases, such techniques allow
contestants to be informed about the logistics of upcoming
competitions, so as to allow them to plan their schedules to allow
participation. Such techniques allow contestants to provide
feedback about the logistics, for example, to point out potential
conflicts with other competitions or events, or to otherwise give
an early indication that the planned logistics may negatively or
positively affect the success of the competition.
[0027] In various embodiments, specifications may be made available
to potential contestants in advance of the competitions for review
and comment. Making the specifications available in advance enables
potential competitors to make better decisions about whether to
participate in a competition, and also allows them to comment on or
clarify the specification for the competition in advance of the
competition. The feedback received may be used to modify the
specification or otherwise clarify the specification, to make the
competition complete more smoothly. The feedback received also may
be used to determine interest in the competition, or otherwise
determine the success of the competition.
[0028] For example, in some embodiments, a list of competitions is
provided 101. At least some of the competitions are upcoming and
have not yet started. In some embodiments, some of the competitions
may be active competitions and/or completed competitions. The list
may be provided by any suitable means. Just as a few non-limiting
examples, the list may be communicated by one or more of a web
page, email, RSS feed, text message, and so on. The list may
include such information as the name of the competition, a planned
competition start date, and a reference to the workspace. The
reference to the workspace may include a URL and/or other location
indicia.
[0029] Each competition typically is for the development of one or
more assets. Each listing may include a reference to one of a
number of workspaces, typically a workspace allocated to that
respective asset. The workspace may be any suitable workspace. Just
as a few examples, not intended to be limiting, the workspace may
be or may include one or more of a shared document, a wiki page, a
web page, a blog page, a forum, a bulletin board, and so on. The
workspace may be specific to the asset under development. In some
cases, the asset to be developed is a part of a larger or combined
asset to be developed at least in part by a series of development
competitions, and the workspace may include documents and
information associated with the larger asset.
[0030] The workspace includes a place for providing a draft
competition specification 102. The draft specification may in some
cases be first provided in a substantially complete form that would
be used as the specification if no comments are received or as a
rough draft that is expected to be further revised by the
administrator, potential competitors and/or contest sponsors. One
benefit of providing a draft specification is that potential
competitors can review the draft specification and provide
substantive feedback on the specification prior to the competition.
The potential competitors can provide input that allows for the
development of a specification that already addresses potential
concerns of competitors. In a drafting process, for example, in
which an architect or designer provides a draft, and a reviewer
provides feedback to the architect or designer, this process of
draft and revisions may be visible to those who have access to the
workspace. Access to the specification drafting and revision
process may give competitors further insight as to the desired
asset to be developed in the competition.
[0031] In some embodiments, depending on competition and/or
administrative policies, potential competitors may need to perform
certain actions or have certain privileges in order to have access
to the workspace and/or provide feedback. For example, competitors
may be required to have participated in a number of competitions,
have a rating in a type of competition, have a rating over a
predetermined threshold, etc. in order to have access and/or the
ability to provide certain types of feedback. Potential competitors
may be required to have submitted contractual documentation, such
as a non-disclosure agreement and/or an intellectual property
assignment document or other documentation or certifications as a
prerequisite to participation.
[0032] Feedback may be received on the draft competition
specification 103 in the workspace. Feedback may be received from
potential competitors, administrators, reviewers, competitions
sponsors (e.g., customers) and/or other parties. Different parties
may provide the same types or different types of feedback. For
example, some participants, such as reviewers, administrators,
sponsors, or potential competitors with particular prerequisites
may have privileges to edit text, or edit certain sections, while
others may only be able to mark text or provide comments. For
example, if prize money and/or start date/times are provided in the
competition specification, potential competitors may be allowed to
comment on the logistics of the competition but not modify the
competition specification. Potential competitors may be able to
edit technical information in the specification or add technical
information or comments to the specification.
[0033] Typically, a period of time prior to the competition is
specified for feedback. For example, if the draft specification is
first provided 2 weeks prior to a competition, a feedback period
may be specified that is from two weeks prior to the competition to
one week prior to the competition. The feedback may be any type of
feedback. Just as a few examples not intended to be limiting,
feedback may include any one or more of comments on the
specification, suggestions for changes to the specification, text
edits to the specification, suggestions for changes to graphics or
designs, edits to graphics or designs, additions to the
specification, general or specific comments on the specification
and/or the competition logistics, and so on.
[0034] In some embodiments, the feedback may include an indicia of
interest or disinterest in participating in the competition. For
example, in some implementations there may be one or more "voting"
buttons that allow a potential competitor to indicate interest in
participating in that competition. In some implementations, indicia
of interest may include a single voting button in which a potential
competitor can indicate interest in participating in that
competition. In some implementations, indicia of interest may
include a two voting buttons in which a potential competitor can
indicate interest in participating in that competition, or indicate
that the potential competitor is not interested in participating in
the competition. In some implementations, indicia of interest
includes a degree of interest. In one such implementation, as an
example, five radio buttons, each representing a different degree
of interest are provided, in which a first button indicates that a
potential competitor will participate, a second button indicates
that the potential competitor is likely to participate, a third
button indicates that the potential competitor may or may not
participate, a forth button indicates that the potential competitor
likely will not participate, and the fifth button indicates that
the potential competitor will not participate. A comment area may
also be provided for the potential competitor to give a reason for
the indicated interest indicia.
[0035] In some implementations, indicia of interest may be recorded
without storing information about who has indicated interest. In
other implementations, the indicia of interest may be stored with
information about the identity and/or qualifications of the
potential competitor(s) who have provided a positive (e.g., will
compete or are likely to compete) indicia of interest. Such
information may be used to determine the likelihood of success of
the competition. For example, in some cases if two or more
potential competitors with ratings over a predetermined threshold
(e.g., a rating threshold of 900, 1200, 1500, or 2200) have
provided a positive indicia of interest, it is likely that the
competition will be successful. Likewise, if, after a predetermined
time period, there have been no indicia of interest by a
predetermined number of potential competitors with sufficiently
high ratings, it may be beneficial to take steps to increase
participation, such as to increase prize money, modify timelines
and/or logistics, and so on. Likewise, if there is an amount of
interest above the predetermined threshold, it may be possible to
start the competition early, or to reduce prize money or points, or
to take other steps with some confidence that the competition will
complete successfully.
[0036] The indicia of interest may in some implementations be
publicly available. In such cases, all visitors can gain some
understanding about the feedback with respect to interest
expressed. The published indicia may include all information and/or
may include a summary of the indicia. For example, the indicia may
include a table demonstrating interest for range of skill level
(e.g., totals of indications from potential competitors in the
900-1199 skill rating range, totals of indications from potential
competitors in the 1200-1499 skill rating range, totals of
indications from potential competitors in the 1500-2199 skill
rating range, totals of indications from potential competitors in
the 2200+ skill rating range, etc.)
[0037] The draft competition specification may be finalized based
on the feedback 104. In some cases, this may involve simply
"locking" the specification workspace to further edits. In some
cases, this may involve a final review and edit of the
specification by a designated administrator, review board member,
and/or contest sponsor. In some cases, this may involve changes to
competition logistics. In some cases, there may be one or more
rounds of edits and comment periods.
[0038] In some cases a specification is not considered finalized
until a number of potential competitors have provided an indication
of interest in participating in the competition. For example, the
start of the competition may be postponed to a date/time later than
a scheduled time if sufficient indicia of interest are not
received. In some cases, a competition may be scheduled to begin at
some time period after sufficient indicia of interest are received.
Techniques may be used to determine the likelihood of success based
on expected participation, such as described in co-pending U.S.
Provisional Patent Application Ser. No. 61/020,703 by Fairfax et
al, entitled SYSTEM AND METHOD FOR CONDUCTING COMPETITIONS, filed
on Jan. 11, 2008, the teachings of which are incorporated herein by
reference.
[0039] In general, in another aspect, a system 200 for developing
an asset by competition may include a competition list including a
list of competitions, where each competition is for the development
of an asset. Each element of the list may include a reference to
one of a number of workspaces, where the workspace is relevant
and/or allocated to the asset competition. As shown in the figure,
Competition 1 includes a reference to Workspace 1, Competition 2
includes a reference to Workspace 2. Competition n includes a
reference to Workspace n. In should be understood that there may be
any number or combination of competitions and workspaces, and that
the number shown in the figure is intended to be exemplary.
[0040] The system includes one or more workspaces 203a, 203b, 203n,
generally 203, each including a facility (which may include or
communicate with a data store) for a competition specification
204a, 204b, 204n, generally 204. The specification facility 204 may
or may not have a provided at the time that a workspace is first
created. At some time prior to the asset development competition, a
competition specification 204 is provided in the workspace 203. In
some embodiments, the workspace 203 includes facility for receiving
feedback on the draft competition specification. The feedback may
be received from potential competitors, administrators, competition
sponsors, and so on. In some cases, the specification is designated
as a draft for a period of time prior to the competition. At some
time prior to or and the start of the competition, the
specification 204, is designated as final. There may be an
indication in the competition list of the status of the
specification (e.g., not yet provided, available for feedback,
feedback period closed, competition underway, etc.)
[0041] Competitions 205a, 205b, 205n, generally 205, are held for
the development of the specified assets using the specification 204
taking into account the feedback from potential competitors.
[0042] Referring to FIG. 3, in one embodiment, one possible
generalized implementation of a competition for the development of
an asset is shown. It should be understood that this implementation
is but one of many possible examples of competitions for which the
method of FIG. 1 and the system FIG. 2 may be applied, and that
other types of competitions also may be used. The asset may be any
sort or type of asset that may be developed by an individual or
group. As non-limiting illustrative examples related to software,
an asset may be a software program, logo, graphic design,
specification, requirements document, wireframe, static prototype,
working prototype, architecture design, component design,
implemented component, assembled or partially-assembled
application, testing plan, test cases, test code, documentation,
and so on. As non-limiting examples unrelated to software, an asset
may be a computer hardware or electronics design, or other designs
such as architecture, construction, or landscape design. Other
non-limiting examples written documents and content such as
documentation and articles for papers or periodicals (whether
on-line or on paper), research papers, scripts, multimedia content
(including without limitation video, audio, graphics images,
cartoons, sounds, graphical presentations, business presentations,
etc.), legal documents, and more.
[0043] In some embodiments, the development process is monitored
and managed by a facilitator 1000. The facilitator 1000 can be any
individual, group, or entity capable of performing the functions
described here. In some cases, the facilitator 1000 can be selected
from the distributed community of contestants based on, for
example, achieving exemplary scores on previous submissions, or
achieving a high ranking in a competition. In other cases, the
facilitator 1000 may be appointed or supplied by an entity
requesting the development, and thus the entity requesting the
competition oversees the competition.
[0044] The facilitator 1000 has a specification 1010 for an asset
to be developed by competition. In general, a specification 1010 is
intended to have sufficient information to allow contestants to
generate the desired asset. In some embodiments, the specification
has been made available to potential competitors for review and
feedback as described with reference to FIG. 1 and FIG. 2. In some
cases, the specification 1010 may include a short list of
requirements. In some cases the specification may include the
result of a previous competition, such as a design, wireframe,
prototype, and so forth. In some cases, the specification may be
the result of a previous competition along with a description of
requested changes or additions to the asset. The facilitator 1000
may review the specification 1010, and format or otherwise modify
it to conform to standards and/or to a development methodology. The
facilitator 1000 may in some cases reject the specification for
failure to meet designated standards. The facilitator 1000 may
mandate that another competition should take place to change the
specification 1010 so that it can be used in this competition. The
facilitator 1000 may itself interact with the entity requesting the
competition for further detail or information.
[0045] The facilitator specifies rules for the competition. The
rules may include the start and end time of the competition, and
the awards(s) to be offered to the winner(s) of the competition,
and the criteria for judging the competition. There may be
prerequisites for registration for participation in the
competition. In some cases, the specification may be assigned a
difficulty level, or a similar indication of how difficult the
facilitator, entity, or other evaluator of the specification,
believes it will be to produce the asset according to the
specification. In some cases, in addition to the technical
requirements, competition rules and logistics may be included in
(or even if separate, considered part of) the competition
specification.
[0046] The specification is distributed to one or more developers
1004, 1004', 1004'' (generally, 1004), who may be members, for
example, of a distributed community of asset developers. In one
non-limiting example, the developers 1004 are unrelated to each
other. For example, the developers may have no common employer, may
be geographically dispersed throughout the world, and in some cases
have not previously interacted with each other. As members of a
community, however, the developers 1004 may have participated in
one or more competitions, and/or have had previously submitted
assets subject to reviews. This approach opens the competition to a
large pool of qualified developers.
[0047] In some embodiments, the communication of the specification
can occur over a communications network using such media as email,
instant message, text message, mobile telephone call, a posting on
a web page accessible by a web browser, through a news group,
facsimile, or any other suitable communication. In some
embodiments, the communication of the specification can be
accompanied by an indication of the rules including without
limitation the prize, payment, or other recognition that is
available to the contestants that submit specified assets. In some
cases, the amount and/or type of payment may change over time, or
as the number of participants increases or decreases, or both. In
some cases submitters may be rewarded with different amounts, for
example a larger reward for the best submission, and a smaller
reward for second place. The number of contestants receiving an
award can be based on, for example, the number of contestants
participating in the competition and/or other criteria.
[0048] The recipients 1004 of the specification can be selected in
various ways. In some embodiments, members of the community may
have expressed interest in participating in a particular type of
asset development competition, whereas in some cases individuals
are selected based on previous performances in competitions, prior
projects, and/or based on other methods of measuring programming
skill of a software developer. For example, the members of the
community may have been rated according to their performance in a
previous competition and the ratings may be used to determine which
programmers are eligible to receive notification of a new
specification or respond to a notification. The community members
may have taken other steps to qualify for particular competitions,
for example, executed a non-disclosure agreement, provided evidence
of citizenship, submitted to a background check, and so forth.
Recipients may need to register for a competition in order to gain
access to a finalized competition specification.
[0049] In one embodiment, a facilitator 1000 moderates a
collaborative discussion forum among the various participants to
answer questions and/or to facilitate development by the
contestants. The collaborative forum can include such participants
as facilitators, developers, customers, prospective customers,
and/or others interested in the development of certain assets. In
one embodiment, the collaboration forum is an online forum where
participants can post ideas, questions, suggestions, or other
information. In some embodiments, only a subset of the members can
access and/or post to the forum, for example, participants in a
particular competition or on a particular team.
[0050] Upon receipt of the specification 1010, one or more of the
developers 1004 each develop assets to submit (shown as 1012, 1012'
and 1012'') in accordance with the specification 1010. The
development of the assets can be accomplished using any suitable
development tools or system, depending, for example, on the contest
rules and requirements, the type of asset, and the facilities
provided. For example, there may be specified tools and/or formats
that should be used.
[0051] Once a developer 1004 is satisfied that her asset meets the
specified requirements, she submits her submission, for example via
a communications server, email, upload, facsimile, mail, or other
suitable method.
[0052] To determine which asset will be used as the winning asset
as a result of the contest, a review process 1014 may be used. A
review can take place in any number of ways. In some cases, the
facilitator 1000 can engage one or more members of the community
and/or the facilitator and/or the entity requesting the asset. In
some embodiments, the review process includes one or more
developers acting as a review board to review submissions from the
developers 1004. A review board preferably has a small number of
(e.g., less than ten) members, for example, three members, but can
be any number. Generally, the review board is formed for only one
or a small number of related contests, for example three contests.
Review boards, in some embodiments, could be formed for an extended
time, but changes in staffing also can help maintain quality. In
some embodiments, where unbiased peer review is useful, the review
board members are unrelated (other than their membership in the
community), and conduct their reviews independently. In some
embodiments, reviewers are allocated such that they only
infrequently work on the same contests.
[0053] In some embodiments, one member of the review board member
is selected as a primary review board member. In some cases, a
facilitator 1000 acts as the primary review board member. The
primary review board member may be responsible for coordination and
management of the activities of the board. In some embodiments, the
customer and/or facilitator 1000 serves as the only review board
member(s).
[0054] In some embodiments, a screener, who may be a primary review
board member, a facilitator, or someone else, screens 1016 the
submissions before they are reviewed by the (other) members of the
review board. In some embodiments, the screening process includes
scoring the submissions based on the degree to which they meet
formal requirements outlined in the specification (e.g., format and
elements submitted). In some embodiments, scores are documented
using a scorecard, which may be a document, spreadsheet, online
form, database, or other documentation. The screener may, for
example, verify that the identities of the developers 1004 cannot
be discerned from their submissions, to maintain the anonymity of
the developers 1004 during review. A screening review 1016 may
determine whether the required elements of the submission are
included (e.g., all required files are present, and the proper
headings in specified documents). The screening review can also
determine that these elements appear complete.
[0055] In some embodiments, the screening 1016 includes a
preliminary substantive review and selection/elimination by the
entity that requested the competition. For example, if the
competition is for a wireframe, the entity may select the
wireframes that seem to be the best. This smaller group may then go
on to the next step.
[0056] In some embodiments, the screener indicates that one or more
submissions have passed the initial screening process and the
reviewers are notified. The reviewers then evaluate the submissions
in greater detail. In preferred embodiments, the review board
scores the submissions 1018 according to the rules of the
competition, documenting the scores using a scorecard. The
scorecard can be any form, including a document, spreadsheet,
online form, database, or other electronic interface or document.
There may be any number of scorecards used by the reviewers,
depending on the asset and the manner in which it is to be
reviewed.
[0057] In some embodiments, the scores and reviews from the review
board are aggregated into a final review and score. In some
embodiments, the aggregation can include compiling information
contained in one or more documents. Such aggregation can be
performed by a review board member, or in one exemplary embodiment,
the aggregation is performed using a computer-based aggregation
system. In some embodiments, the facilitator 1000 or a designated
review board member resolves discrepancies or disagreements among
the members of the review board.
[0058] In one embodiment, the submission with the highest combined
score is selected as the winning asset 1020. The winning asset may
be used for implementation, production, or for review and input
and/or specification for another competition. A prize, payment
and/or recognition is given to the winning developer.
[0059] In some embodiments, in addition to reviewing the
submissions, the review board may identify useful modifications to
the submission that should be included in the asset prior to final
completion. The review board documents the additional changes, and
communicates this information to the developer 1004 who submitted
the asset. In one embodiment, the primary review board member
aggregates the comments from the review board. The developer 1004
can update the asset and resubmit it for review by the review
board. This process can repeat until the primary review board
member believes the submission has met all the necessary
requirements. In some embodiments, the review board may withhold
payment of the prize until all requested changes are complete.
[0060] In some embodiments, a portion of the payment to a developer
of one asset in a series of assets is withheld until the until
after other competitions that make use of the asset are complete.
If any problems with the asset are identified in the further
competitions, these are provided to the reviewer(s) and the
developer, so that appropriate changes can be made by the developer
1004.
[0061] There can also be prizes, payments, and/or recognition for
the developers of the submissions other than first place
submissions. For example, the contestants that submit the second
and/or third best submissions may also receive payment, which in
some cases may be less than that of the winning contestant.
Additional prizes may be awarded for ongoing participation and/or
reliability. Payments also may be made for creative use of
technology, submitting a unique feature, or other such submissions.
In some embodiments, the software developers can appeal the score
assigned to their design, program, or other submissions.
[0062] It should be understood that the development contest model
may be applied to different portions of work that are required for
the development of an overall asset. A series of development
contests is particularly applicable to assets in which the
development may be divided into stages or portions. It can be
beneficial in many cases to size the assets developed in a single
competition such that work may be completed in several hours or a
few days. The less work required to develop a submission, the lower
the risk for the contestants that they will not win.
[0063] Referring to FIG. 4, in one illustrative example of a series
of competitions, steps for development of a software application
are shown. While described as an illustrative example in some
places as a web application, it should be understood that any sort
of software application, with any type of architecture, including
without limitation mobile applications, client/server applications,
thin-client applications, to give a few examples, may be suitable
for this type of development process.
[0064] A series of contests may be held to develop the software
application. For example, a person, referred to as a customer, may
have an idea or concept for a software application. The idea may
be, for example, thought-out in detail or only with a high level of
description. A series of development competitions (with each such
competition, for example, operating as shown with respect to FIG.
3) may be held, starting with the specification of the concept. For
example, a contest may be held for the development of application
requirements 1101. In such a contest, the initial description and
documentation provided by the customer may be used as part of the
contest specification 1010 (FIG. 3). The asset to be developed in
the competition is the application requirements documentation. It
may that the contestants need to interact with the customer to
develop the requirements. Typically, this interaction would take
place on a forum that is open to all of the contestants. The review
of the requirements may involve one or more peer reviewers (i.e.,
members of the contestant community), as well as the customer. The
selection of a requirements document may be based on the degree to
which it represents the concept presented, the actual desires of
the customer, and technical understanding and feasibility.
[0065] The requirements documentation that is developed in the
requirements contest 1101 may then be used as part of the contest
specification 1010 (FIG. 3) for a wireframe competition 1102. The
wireframe contest 1102 may be held for the development of a
wireframe (e.g., a visual guide used to suggest layout and
placement of fundamental design elements in the interface design)
of the application. The wireframe typically lays out the interface
of the application, and presents visually the way that a user will
interact with the application software. The review of the
wireframes may involve one or more peer reviewers (i.e., members of
the contestant community), as well as the customer. The selection
of a wireframe may be based on the degree to which it implements
the requirements, the actual desires of the customer, and technical
understanding and feasibility.
[0066] When the wireframe contest 1102 is complete, a contest may
be held for the development of a static prototype (e.g., an
implementation of a web site in HTML or other markup language,
typically without data persistence or other server-based
functionality) using the wireframes as a starting point. The static
prototype shows screen displays as they would look in the
application, but does not have implemented functionality. The
review of the static prototype may involve one or more peer
reviewers (i.e., members of the contestant community), as well as
the customer. The selection of a static prototype may be based on
the degree to which it implements the requirements, the actual
desires of the customer, and technical understanding and
feasibility.
[0067] When the static prototype contest 1103 is complete, a
contest may be held for the development of working prototypes
(e.g., working implementations) based on the static prototypes. The
working prototype is code that implements the requirements,
wireframes, and static prototypes, along with any other comments or
instructions provided by the customer. This working prototype may
have certain restrictions or requirements that are described in the
contest specification 1010 (FIG. 3). The working prototype may not
need to be highly scalable, or enterprise quality, but merely good
enough to try and permit customer use.
[0068] By having a customer involved in the requirements for and
selection of the deliverables, a contest-based development process
results in the efficient creation of software that the customer is
happy with. At any stage in the process, if a customer is not happy
with the final results, the customer can hold another contest to
revise the previous asset with new or changed requirements.
[0069] The working prototype may be sufficient for some customers
1105 as a useful application. For others, the working prototype is
a first step for confirming the desired requirements for a software
application. Once they have used and tested the functionality of
the working prototype, the working prototype may be used as the
input to another series of competitions for development of an
enterprise quality software application.
[0070] Shown as "STAGE 2" in the figure, another series of
contests, beginning with the development of an application
specification 1106 based on the working prototype may be held. In
some such embodiments, a contest for development of an application
specification 1106 may be held. The contest specification 1010
(FIG. 3) may include the winning working prototype and information
about changes requested from the working prototype. Other
information that may be included in the contest specification 1010
(FIG. 3) may include the required format and scope of the
application specification. In one embodiment, the application
specification is a requirements specification, including screen
displays, functionality description, and so forth. A customer may
participate on a review board for a specification, particularly to
fill in any gaps, or to clarify any problems with the inputs to the
specification competition. Of course, the specification competition
could be held without a working prototype (if STAGE 1 is skipped)
or just using wireframes and/or static prototypes as input.
Likewise, a customer may just develop its own specification, and/or
engage a consultant to develop a specification.
[0071] Once the winning application specification is selected, it
may be used as the contest specification 1010 (FIG. 3) for an
architecture competition. The asset(s) that may be developed as
part of the architecture competition may include an overall
architectural design, as well as a description of components that
may be used to build the application. An architectural design may
include a description of new components that may be built as part
of other competitions 1108, 1109, or as existing components from a
component library 1120. When a winning architecture is selected,
the resulting component specification(s) may be used as input for
component design competition(s) 1108. A customer may participate on
a review board for an architecture competition, particularly if the
customer has architecture expertise. The customer may be
particularly interested in the interfaces and integration with its
other systems. Typically, it is useful to have skilled architects
working on the review board for an architecture competition, to
identify technical issues with the architectural design.
[0072] In some embodiments, the components specified in the winning
architecture may be developed by holding a series of component
design competitions 1108. The winning component designs are then
used as input for component development competitions 1109. As
illustrative examples, component design and development
competitions have been described, for example, in the
above-referenced U.S. patent application Ser. No. 11/035,783. When
all components have been completed, they may be assembled in
assembly competitions 1110. Finally, test scripts may be developed,
tested, and run in testing competitions 1111. At the completion of
STAGE 2, an enterprise-ready software application that meets the
requirements is completed.
[0073] In some embodiments, reusable assets may be provided to
increase the speed of development in each of the various stages.
For example, templates, graphics, tools, design patterns, and so
forth may be used to increase the productivity of the contestants,
and to give a common style to make evaluation and integration
easier.
[0074] It should be understood that one, two, or more of the steps
may be performed in a different order, or combined or omitted. For
example, in STAGE 1, there may not be a need for a requirements
competition 1101. Rather, the contestants in the wireframe may
start from the description provided by the customer, and ask
questions in a forum or otherwise to generate their wireframes,
without use of more formal requirements documentation. Likewise,
there may be at any stage multiple competitions and/or multiple
levels of competitions. For example, for a complex application,
there may be an overall architecture design, and then individual
competitions for the architecture design of subassemblies. The
architecture of the subassemblies may be designed, needed
components built, and the subassemblies assembled. The
subassemblies may then be assembled into complete application in an
additional competition or competitions. Testing competitions may be
held for various portions of an application, for example, for a
subassembly, or for a distinct portion of an application, such as a
user interface with another competition held for testing of
back-end functionality.
[0075] It also should be understood that development by a series of
competitions is flexible, in that contests can be repeated if the
results were not as expected, or if additional changes or new
functionality is desired. Likewise, a customer can undertake as
much work as it likes through development by other methods, for
example, by using internal staff or outside consultants. For
example, rather than holding a contest 1104 for a working
prototype, a customer can take the static prototype, and develop
the working prototype itself.
[0076] In some embodiments, where the assets developed outside of
the contest environment are being used as the input to another
contest, it may be useful to engage reviewers to review the asset.
For example, a static prototype review can be conducted on a static
prototype developed by a customer before that static prototype is
used as input to the working prototype contest 1104. As another
example, in STAGE 2, an entity might develop a specification
itself, engage reviewers to review the specification, make any
desired changes, then use the specification in a contest for some
or all of the architecture, hold a contest for development of some
or all of the components, and then assemble the application itself.
A review can be conducted on the assembly, and a testing
competition held to develop test scripts and other
functionality.
[0077] In some embodiments, a "project plan" competition may be
held prior to the competitions described. In the project plan
competition, contestants develop a plan for development of the
customer's project needs. For example, the plan may specify a
series of development contests and timeframes for developing the
desired software application. The plan may provide approximate
costs, based on historical or other data, for similar competitions.
The plan may include development strategies, and assumptions about
customer participation. A project plan may be specified as a series
of competitions, which may then be monitored through interaction
with a competition management server. One asset developed in such a
plan may be a configuration of contests as specified in a project
management system, such as that described in co-pending U.S. patent
application Ser. No. 11/755,909.
[0078] In one embodiment, a competition web site provides a
registration process for a development project. Following project
registration, a customizable dashboard, or "cockpit" is configured
with a variety of functional widgets with which a customer can
begin the process of having development work done by a member
community. The customer can launch competitions and track projects
through delivery and member payment. In some embodiments, the web
site project-specific public or private `group` pages, generated by
the customer from the cockpit, on which the nature of the project
can be described, relevant attachments can be viewed, and the
customer can openly communicate with members via a bulletin board
or forum to answer questions, receive suggestions, and so on. In
some embodiments, these pages allow participants to see the
evolution of the project, interact, collaborate, provide status
reports, and make delivery.
[0079] In some embodiments, representatives of a customer, and
members in different roles, may have access to different
capabilities for the project. For example, some customers may be
able to post projects, while others may only be able to review
status, and other review and approve payments.
[0080] In some embodiments, a dashboard page serves as an entry
point for customers to interact with the TopCoder Community. From
this dashboard page, a registered customer may launch and track
projects, send and receive messages and monitor a variety of
information sources. The dashboard may include a profile box with
information about the individual, and a link to other information,
or to update the information, a button that allows the launching of
a project, a button or link for help and "how-to" information,
navigation to other pages, such as contests and discussion forums,
and in some cases space for broadcast messages or advertising,
search field, and so forth. In some implementations, the page is
customizable, with a framework allowing for "widgets" to be dragged
and dropped into place into a user-configured order.
[0081] In some embodiments, the dashboard includes an interface
that provides information about the user's projects, which may be
updated dynamically, such as the project name, the status of the
project, the name of the personnel associated with the project, the
status of the project, the status of payments, the number of
contest participants, the number of submissions, the project phase,
whether there are new forum posts (and in some cases, information
or a link to the post(s)).
[0082] In some embodiments, the dashboard includes a messaging
interface that indicates whether new messages are waiting, and
allows a user to compose new messages. In some embodiments, a team
information interface shows information about individuals who are
working on a current project, or, for example, a past project, or
have been designated for another reason to be included. An RSS feed
interface provides an indication of whether content has been added
to a particular web page. A news interface provides information
about new features, or, for example, newly-available components
that may be used in a competition.
[0083] Competition List
[0084] Referring to FIG. 5, in an exemplary embodiments, an excerpt
from a competition list includes such information as the date
and/or date/time of the competition 501, the client for whom the
competition is being run 502, the name of the project that the
competition is associated with 503, type of competition and asset
that is being developed 504, the technology category of the asset
506, the name of the asset 507, the status of the competition 508,
and the personnel associated with the competition, such as the
manager of the project 510, the architect and/or author of the
competition specification 511, and a reviewer 512 tasked with
reviewing the specification prior to contest launch. It should be
understood that this excerpt is exemplary, and is intended to
illustrate the types of information that may be found in such a
list. Additional information may be included, such as the date the
competition was first posted, whether or not there are competitor
registrations associated with the competition, and so on. Likewise
some information in the list may be excluded from all or some
users. For example, some users may not have access to the name of
the client 502, or the name of the project 503, or may be provided
with a code name or number.
[0085] In some embodiments, the date 501 may include a date (as
shown) or a more specific date/time indication of when the
competition is scheduled to begin. The indication may be localized
for the time zone of an individual. The client 502 listing may be
the sponsor of the competition, and/or the ultimate recipient of
the asset to be developed. The client listing 502 may be omitted
for certain competitions, or for certain viewers who are not
authorized to have client information. The project listing 503
provides an indication of which project the competition is
associated. There may not be a particular project, or the project
may be the name of a larger asset (e.g., a software application, a
multi-media presentation) for which multiple or a series of asset
development competitions are being held. The project listing 503
may be omitted for certain competitions, or for certain viewers who
are not authorized to have project information. For example, in
some cases, only administrators have access to the client and
project information, while potential and actual competitors do
not.
[0086] The asset type 504 provides an indication of the type of
asset that is to be developed. Some of the types shown in the
example include Graphic Design, Component Design, Component
Development, and Assembly. Other asset types would be appropriate
for other competitions. In some embodiments, the assets may include
each of the assets described with reference to FIG. 4. The category
of competition 506 provides an indication of the technology of the
asset. This may be helpful for potential competitors to quickly
identify competitions that may be interesting to them.
[0087] The name of the asset 507 gives a further indication of the
technology and/or the larger asset that is being developed. For
example, the Manifest Manager is shown as a .NET Generic component,
and this competition is for the design of that component. Following
design, there may be another competition for the coding of this
component and such a competition may already be scheduled in
anticipation of successful completion of this competition. Such a
competition would have a similar listing (with a different date)
and an indication that the type 504 is "Component Development."
[0088] In this example, the name 507 of each asset is a link to a
workspace for that asset. The workspace includes further
information and details about the asset, as well as a web page
allocated for the competition specification. In this example, the
competition specification is provided in a "wiki" such that any
user with sufficient permissions can make edits to the wiki, and
add or delete text. The changes to the page made by a user can be
viewed, so that a reviewer or administrator or another competitor
can see who made what changes. In addition, there is a portion of
the page dedicated to comments, such that a user can post comments,
general or specific, about the technical description, competition
logistics, or other specification information. Links to a "forum"
is also provided, such that a conversation can take place among
administrators (e.g., managers, architects), competitions sponsors
and/or potential competitors about the competition.
[0089] The status 508 shown in this list is the status of the
specification drafting, commenting, and review. The status of the
specification can change from "Not Received" to "Comments" when a
draft is made available for comments. Following the comment period,
the status will change to "Final Review" which is when the
designated reviewer will review the specification along with the
designated architect and/or the designated manager. If review
fails, the status becomes "Failed Review--Edits Required" until the
specification passes review, and then it is "Complete." Potential
competitors thus can review the list, and review the specifications
(if the potential competitor is qualified and has fulfilled any
necessary prerequisites).
[0090] The names associated with the manager 510, architect 511 and
reviewer 512 positions may be last names (as shown with the
exemplary names listed), handles/pseudonyms, first names, and/or
other identifiers. These names may be omitted for certain
competitions, or for certain viewers who are not authorized to have
full project information. For example, in some cases, only
administrators have access to the names of the manager and
architect information, while potential and actual competitors do
not. In some cases, the names are provided, but they are "handles"
(pseudonyms) used by members of a community of developers. In some
cases, the handles displayed are designated for a role (e.g.,
Contest Administrator 231)
[0091] Workspace Implementation
[0092] Referring generally to FIG. 6, in some embodiments, a screen
display provides an entry point for users to view competition
status and to review and comment on a competition specification. It
should be understood that the exemplary screen displays are not
intended to be limiting, but rather are intended only to illustrate
by example. The screen display may be implemented with HTML pages,
with HTML/Javascript pages with AJAX display technology, with a
client/server system, desktop application, browser plug-in, applet
or in any other suitable manner. For example, in some embodiments,
a commercially available content management system is configured to
provide the described interface by accessing information from other
web pages and/or back-end systems. In some embodiments, the screen
display is integrated or presented with the contest system and
other systems described in the referenced patent applications.
[0093] In some embodiments, the workspace is implemented using a
customizable framework of widgets. In this context, the widgets are
small functional components operating within a client software
program, for example, a web browser, such that the client can
present a number of these small applications as an overall combined
application. The client may be another general-purpose or
special-purpose application, however, preferably, the client is a
web browser such as INTERNET EXPLORER from Microsoft Corporation,
or FIREFOX from the Mozilla Foundation. Widgets may be implemented
in any suitable technology, depending on purpose, context, and
implementation, but in a preferred embodiment are implemented in
HTML and Javascript. The widgets may operate as part of a client
widget framework that is implemented in JavaScript. The framework
utilizes AJAX communication with a data access layer on a server,
which data access layer resides within a JBoss Application Server.
Widgets may be implemented using the framework, in order to enable
the described features.
[0094] In this example, the exemplary widgets include a
"Competition Details" widget 602, which provides details about a
competition. This includes some of the information from the list of
FIG. 5, and may also provide additional information. A "Competition
Status" widget 606, provides further details about the status of a
competition, and may include links to further information about the
competition status. A specification editing widget 608 displays the
current draft of the specification, and allows for editing of the
specification. This may be implemented as a wiki, such that changes
to the text are tracked, and user permissions can be used to allow
access. A discussion forum widget 610 provides a place for comments
and discussion about the specification 608 and/or the competition
more generally.
[0095] It should be understood that the figure shows a simplified
screen display, and there typically would be additional features on
the screen display, including without limitation additional
information about this and other competitions, navigation aids
(e.g., links, menus, etc.) to other places on the competition site,
capability to get alerts to changes, for example by subscribing to
RSS Feeds, and so on. It also should be understood that that the
use of widgets is just one possible implementation, and any other
suitable implementation for providing screen displays, for example,
using a browser (e.g., HTML pages, Javascript, ASP, etc.) and using
other approaches such as thin client, client/server, and so on.
Display on mobile devices, television/entertainment video systems
and another other suitable systems also may be used.
GENERAL APPLICABILITY
[0096] Although described here with reference to software, and
useful when implemented with regard to software assets, the
cooperatively developed asset can be any sort of tangible or
intangible object that embodies intellectual property. As
non-limiting examples, the techniques could be used for computer
hardware and electronics designs, or other designs such as
architecture, construction, or landscape design. Other non-limiting
examples for which the techniques could be used include the
development of all kinds of written documents and content such as
documentation and articles for papers or periodicals (whether
on-line or on paper), research papers, scripts, multimedia content
(including without limitation video, audio, graphics images,
cartoons, sounds, graphical presentations, business presentations,
etc.), legal documents, and more.
* * * * *
References