U.S. patent number 7,937,507 [Application Number 12/030,920] was granted by the patent office on 2011-05-03 for extended measurement word determination at a channel subsystem of an i/o processing system.
This patent grant is currently assigned to International Business Machines Corporation. Invention is credited to Mark P. Bendyk, Daniel F. Casper, John R. Flanagan, Roger G. Hathorn, Catherine C. Huang, Matthew J. Kalos, Louis W. Ricci, Gustav E. Sittmann, Harry M. Yudenfriend.
United States Patent |
7,937,507 |
Bendyk , et al. |
May 3, 2011 |
Extended measurement word determination at a channel subsystem of
an I/O processing system
Abstract
An article of manufacture, an apparatus, and a method for
determining an extended measurement word at a channel subsystem of
an I/O processing system using data from a control unit are
provided. The article of manufacture includes at least one computer
usable medium having computer readable program code logic. The
computer readable program code logic performs a method including
sending a command message to the control unit, and receiving a
transport response information unit message at the channel
subsystem in response to sending the command message to the control
unit. The computer readable program code logic additionally
extracts a plurality of time values from the transport response
information unit message as calculated by the control unit,
calculates an extended measurement word as a function of the time
values, and writes the extended measurement word to computer
readable memory in the I/O processing system.
Inventors: |
Bendyk; Mark P. (Hyde Park,
NY), Casper; Daniel F. (Poughkeepsie, NY), Flanagan; John
R. (Poughkeepsie, NY), Hathorn; Roger G. (Tucson,
AZ), Huang; Catherine C. (Poughkeepsie, NY), Kalos;
Matthew J. (Tucson, AZ), Ricci; Louis W. (Hyde Park,
NY), Sittmann; Gustav E. (Webster Groves, MO),
Yudenfriend; Harry M. (Poughkeepsie, NY) |
Assignee: |
International Business Machines
Corporation (Armonk, NY)
|
Family
ID: |
40956144 |
Appl.
No.: |
12/030,920 |
Filed: |
February 14, 2008 |
Prior Publication Data
|
|
|
|
Document
Identifier |
Publication Date |
|
US 20090210570 A1 |
Aug 20, 2009 |
|
Current U.S.
Class: |
710/33;
702/176 |
Current CPC
Class: |
G06F
13/124 (20130101) |
Current International
Class: |
G06F
13/00 (20060101) |
Field of
Search: |
;710/18,33 ;702/106 |
References Cited
[Referenced By]
U.S. Patent Documents
Foreign Patent Documents
|
|
|
|
|
|
|
3931514 |
|
Mar 1990 |
|
DE |
|
1264096 |
|
Feb 1972 |
|
GB |
|
2291990 |
|
Sep 1995 |
|
GB |
|
2291990 |
|
Feb 1996 |
|
GB |
|
63236152 |
|
Oct 1988 |
|
JP |
|
WO2006102664 |
|
Sep 2006 |
|
WO |
|
Other References
"Information Technology--Fibre Channel Protocol for SCSI, Third
Version (FCP-3)," T10 Project 1560-D, Revision 4g, Sep. 13, 2005.
cited by other .
U.S. Appl. No. 12/031,038 Non-Final Office Action dated Nov. 16,
2009. cited by other .
U.S. Appl. No. 12/030,975 Non-Final Office Action dated Oct. 22,
2009. cited by other .
U.S. Appl. No. 12/030,932 Non-Final Office Action dated Dec. 16,
2009. cited by other .
U.S. Appl. No. 12/031,023 Non-Final Office Action dated Oct. 29,
2009. cited by other .
U.S. Appl. No. 12/030,951 Non-Final Office Action dated Nov. 23,
2009. cited by other .
U.S. Appl. No. 12/030,961 Non-Final Office Action dated Dec. 17,
2009. cited by other .
U.S. Appl. No. 12/030,939 Non-Final Office Action dated Nov. 16,
2009. cited by other .
U.S. Appl. No. 12/030,989 Non-Final Office Action dated Oct. 22,
2009. cited by other .
U.S. Appl. No. 12/030,993 Non-Final Office Action dated Oct. 28,
2009. cited by other .
U.S. Appl. No. 12/031,021 Non-Final Office Action dated Jan. 8,
2010. cited by other .
U.S. Appl. No. 12/181,662--Non-Final Office Action dated Jun. 18,
2009. cited by other .
U.S. Appl. No. 12/181,662--Final Office Action dated Jan. 4, 2010.
cited by other .
Iren, et al.; "The Transport Layer: Tutorial and Survey"; ACM
Computing Surveys; vol. 31, No. 4; Dec. 1999; pp. 360-405. cited by
other .
International Search Report and Written Opinion for
PCT/EP2009/051447 dated Jul. 1, 2009. cited by other .
International Search Report and Written Opinion for
PCT/EP2009/051483 dated Jul. 27, 2009. cited by other .
Nordstrom.; "Sequence Reception Method for a Fibre Channel Protocol
Chip"; IBM Technical Disclosure Bulletin; vol. 38, No. 12; Dec.
1995; pp. 267-269. cited by other .
U.S. Appl. No. 11/548,060 Non-Final Office Action dated Apr. 15,
2008. cited by other .
U.S. Appl. No. 11/548,093 Non-Final Office Action dated Apr. 17,
2008. cited by other .
"Protocol for Insochronous Traffic Over Fiber Channel Switching";
IBM Technical Disclosure Bulletin; vol. 37, No. 06B; Jun. 1994. pp.
377-380. cited by other .
Sachs, M.W.; "I/O Marker Changing"; IBM Technical Disclosure
Bulletin; vol. 37, No. 02A; Feb. 1994; pp. 75-76. cited by other
.
Simmons et al.; "A Performance Comparison of Three Supercomputers:
Fujitsu VP-2600, NEC SX-3, and CRAY Y-MP"; ACM, Conference on High
Performance Networking and Computing, Proceedings of the 1991
ACM/IEEE conference on Supercomputing, Albuquerque, New Mexico;
Jul. 1991; pp. 150-157. cited by other .
Srikrishnan et al.; "Sharing FCP Adapters Through Virtualization";
IBM J. Res. & Dev., vol. 51, No. 1/2; Jan./Mar. 2007; pp.
103-118. cited by other .
Stone, et al.; "When the CRC and TCP Checksum Disagree"; SIGCOMM
'00, Stockholm, Sweden; Jul. 2000; 10 pages. cited by other .
U.S. Appl. No. 12/183,315, filed Jul. 31, 2008. cited by other
.
U.S. Appl. No. 12/183,323, filed Jul. 31, 2008. cited by other
.
U.S. Appl. No. 12/183,305, filed Jul. 31, 2008. cited by other
.
"z/Architecture-Principles of Operation," IBM Publication No.
SA22-7832-04, 5th Ed., Sep. 2005. cited by other .
Brice, et al.; U.S. Appl. No. 11/464,613; "Flexibility Controlling
the Transfer of Data Between Input/Output Devices and Memory";
filed Aug. 15, 2006; Specification having 23 pages and Drawings
having 4 sheets. cited by other .
Casper, et al.; U.S. Appl. No. 11/548,060; "Facilitating
Input/Output Processing by Using Transport Control Words to Reduce
Input/Output Communications"; filed Oct. 16, 2006; Specification
having 32 pages and Drawings having 12 sheets. cited by other .
Casper, et al.; U.S. Appl. No. 11/548,093; "Facilitating Access to
Status and Measurement Data Associated With Input/Output
Processing"; filed Oct. 16, 2006; Specification having 33 pages and
Drawings having 12 sheets. cited by other .
Dauby, et al. "Contention Resolution Between Two Processors"; IBM
Technical Disclosure Bulletin; vol. 26; No. 10A; Mar. 1984; 3
pages. cited by other .
DeVeer, J.A.; "Control Frame Multiplexing on Serial I/O Channels";
IBM Technical Disclosure Bulletin; vol. 32; No. 10A; Mar. 1990; pp.
39-40. cited by other .
Golasky, Richard; "Link-Level Error Recovery With Tape Backup";
Dell Power Solutions; Aug. 2005; pp. 88-91. cited by other .
"IBM.RTM. z/Architecture Principles of Operation", Publication No.
SA22-7832-05, 6th Edition, Apr. 2007. 1,215 pages separated into 4
electronic attachments. cited by other .
Peterson, David; "Information Technology, Fibre Channel Protocol
for SCSI, Fourth Version (FCP-3)", Draft Proposed American National
Standard, Jun. 2004; pp. 1-142. cited by other .
Snively, et al.; "Fibre Channel Single Byte Command Code Sets-3
Mapping Protocol (FC-SB-3)"; T11/Project 1357-D/Rev. 1.6, INCITS;
Mar. 2003; pp. 1-206. cited by other .
Snively, et al.; "Fibre Channel, Framing and Signaling"; (FC-FS)
Rev. 1.70; NCITS Working Draft Proposed American National Standard
for Information Technology; Feb. 2002; pp. i-575. cited by other
.
Written Opinion and International Search Report for
PCT/EP2009/051445 dated Jun. 25, 2009. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051446 dated Jun. 25, 2009. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051462 dated Jul. 1, 2009. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051450 dated Jul. 7, 2009. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051459 dated Jun. 23, 2009. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051463 dated Jul. 22, 2009. cited by other .
U.S. Appl. No. 12/030,912 Non-Final Office Action dated Mar. 18,
2010. cited by other .
U.S. Appl. No. 12/030,954 Non-Final Office Action dated Jan. 21,
2010. cited by other .
Written Opinion and International Search Report for
PCT/EP2009/051461 dated Sep. 22, 2009. cited by other .
U.S. Appl. No. 12/031,182 Non-Final Office Action dated Jan. 22,
2010. cited by other .
U.S. Appl. No. 12/031,201 Non-Final Office Action dated Jan. 25,
2010. cited by other .
International Search Report; International Application No.
PCT/EP2009/059184; International Filing Date: Jul. 16, 2009; Date
of mailing: Jan. 14, 2010; 9 pages. cited by other .
Z/Architecture .cndot. Principles of Operation SA 22-7832-01
International Business Machines Corporation, 2nd Edition, Oct.
2001. cited by other .
U.S. Appl. No. 12/030,967 Restriction Requirement Mailed Dec. 29,
2009. cited by other .
ANSI INCITS 433-2007, Information Technology Fibre Channel Link
Services (FC-LS), Jul. 2007. cited by other .
Merit, A.S. and J.H. Sorg, "Correction of Channel Measurement Data
Caused by Control Unit Queuing," IBM Technical Disclosure Bulletin,
vol. 33, No. 11, Apr. 1991, pp. 100-103. cited by other .
Fibre Channel Single Byte Command Code Sets-2 Mapping Protocol
(FC-SB-3), T11/Project 1357-D/Rev 1.6, INCITS Mar. 2003. cited by
other .
U.S. Appl. No. 12/030,951 Non-Final Office Action dated May 20,
2010. cited by other .
U.S. Appl. No. 12/031,038, Non-Final Office Action Mailed Apr. 15,
2010. cited by other .
U.S. Appl. No. 12/030,975. Final Office Action Mailed May 13, 2010.
cited by other .
U.S. Appl. No. 12/030,967, Notice of Allowance mailed Apr. 23,
2010. cited by other .
U.S. Appl. No. 12/030,985, Non Final Office Action Mailed May 5,
2010. cited by other .
U.S. Appl. No. 12/031,042, Non-Final Office Action Mailed Apr. 5,
2010. cited by other .
U.S. Appl. No. 12/031,021, Notice of Allowance Mailed Apr. 30,
2010. cited by other .
U.S. Appl. No. 12/183,305, Non-Final Office Action Mailed May 11,
2010. cited by other .
U.S. Appl. No. 12/183,315 Notice of Allowance dated Jun. 15, 2010.
cited by other .
Z/Series Input/Output Configuration Program user.cndot.s Guide for
IYP IOCP, SB10-7029-03c. International Business Machines
Corporation, Jun. 2003. cited by other .
U.S. Appl. No. 10/435,906 Notice of Allowance dated Nov. 3, 2005.
cited by other .
U.S. Appl. No. 10/435,906 Non-Final Office Action dated Jul. 15,
2005. cited by other .
U.S. Appl. No. 11/126,727 Non-Final Office Action dated Jun. 8,
2006. cited by other .
U.S. Appl. No. 11/126,727 Non-Final Office Action dated Jun. 21,
2006. cited by other .
U.S. Appl. No. 11/126,728 Notice of Allowance Jun. 13, 2007. cited
by other .
U.S. Appl. No. 11/126,727 Notice of Allowance Sep. 27, 2006. cited
by other .
U.S. Appl. No. 11/126,728 Non-Final Office Action Apr. 16, 2007.
cited by other .
U.S. Appl. No. 11/282,393 Non-Final Office Action Jun. 11, 2007.
cited by other .
U.S. Appl. No. 11/282,393 Non-Final Office Action Sep. 13, 2007.
cited by other .
U.S. Appl. No. 11/282,393 Notice of Allowance dated Jan. 31, 2008.
cited by other .
U.S. Appl. No. 11/469,573 Non-Final Office Action dated Apr. 11,
2008. cited by other .
U.S. Appl. No. 11/965,833 Non-Final Office Action dated Mar. 17,
2009. cited by other .
U.S. Appl. No. 11/965,833 Notice of Allowance dated May 26, 2009.
cited by other .
U.S. Appl. No. 11/965,866 Non-Final Office Action dated Jun. 3,
2010. cited by other .
U.S. Appl. No. 11/965,916 Notice of Allowance Nov. 20, 2008. cited
by other .
U.S. Appl. No. 12/030,989 Final Office Action dated May 24, 2010.
cited by other .
SCSI Primary Commands-4 (SPC-4); Project T10/1731-D. Rev 11. INCITS
Apr. 14, 2010. cited by other .
Laudon et al. .cndot.System Overview of the SGI Origin 200/2000
Product Line, Feb. 23-26, 1997, iEEE Publication pp. 150-156. cited
by other .
U.S. Appl. No. 10/435,906 Restriction Requirements dated Apr. 19,
2005. cited by other .
Behrs, Jr, "Adaptive Prediction Minimizes Data Overrun and
Underrun", IBM, Nov. 1994, pp. 1-3. http://
priorartdatabase.com/IPCOM/000114189. cited by other .
U.S. Appl. No. 12/031,038, Notice of Allowance Mailed Oct. 6, 2010.
cited by other .
Final Office Action for U.S. Appl. No. 12/030,912 mailed Oct. 6,
2010. cited by other .
U.S. Appl. No. 12/031,182 Final Office Action Mailed Jul. 22, 2010.
cited by other .
U.S. Appl. No. 12/030,932, Final Office Action Mailed Jul. 23,
2010. cited by other .
U.S. Appl. No. 12/030,951, Final Office Action Mailed Oct. 26,
2010. cited by other .
U.S. Appl. No. 12/030,961 Notice of Allowance Mailed Nov. 5, 2010.
cited by other .
U.S. Appl. No. 12/030,967 Notice of Allowance Mailed Oct. 7, 2010.
cited by other .
U.S. Appl. No. 12/031,201, Final Office Action Mailed Aug. 3, 2010.
cited by other .
U.S. Appl. No. 12/031,042, Final Office Action Mailed Oct. 25,
2010. cited by other .
U.S. Appl. No. 12/031,021, Notice of Allowance Mailed Jul. 8, 2010.
cited by other .
U.S. Appl. No. 12/181,662, Notice of Allowance Mailed Aug. 4, 2010.
cited by other .
U.S. Appl. No. 12/364,615, Non Final Office Action Mailed Oct. 7,
2010. cited by other .
U.S. Appl. No. 12/183,305, Notice of Allowance Mailed Nov. 1, 2010.
cited by other .
IBM, "Method and Apparatus to Monitor PAV Utilization", Feb. 2005,
pp. 1-3. http://priorartdatabase.com/IPCOM/000082878. cited by
other .
Moore et al., "Reconfiguration of Storage Elements to Improve
Performance", IBM, Dec. 1983, pp. 1-3.
http://priorartdatabas.com/IPCOM/000047719. cited by other .
U.S. Appl. No. 12/183,315, Notice of Allowance Mailed on Jun. 15,
2010. cited by other .
U.S. Appl. No. 12/030,939 Notice of Alllowance mailed Jun. 29,
2010. cited by other .
U.S. Appl. No. 12/030,954 Non Final Office Action mailed Jul. 12,
2010. cited by other .
U.S. Appl. No. 12/030,961 Final Office Action mailed Jul. 12, 2010.
cited by other .
U.S. Appl. No. 12/030,993 Non Final Office Action mailed Jun. 28,
2010. cited by other .
U.S. Appl. No. 12/031,021 Notice of Allowance mailed Jul. 8, 2010.
cited by other .
U.S. Appl. No. 12/031,023 Non Final Office Action mailed Jun. 11,
2010. cited by other .
U.S. Appl. No. 12/183,323 Non Final Office Action mailed Jul. 19,
2010. cited by other .
U.S. Appl. No. 12/392,246 Notice of Allowance mailed Jul. 15, 2010.
cited by other .
U.S. Appl. No. 12/030,975, Notice of Allowance Mailed Oct. 19,
2010. cited by other .
Cakmakci, Melih, et al. "Bi-Directional Communication amoung
"Smart" Compoents in a Networked Control System", University of
Michigan: Department of Mechanical Engineering, 2005 American
control conference, Jun. 8-10, 2005, Portland, OR, pp. 627-632.
cited by other .
U.S. Appl. No. 12/030,954, Notice of Allowance mailed Dec. 28,
2010. cited by other .
Ishikawa, Hikaru, et al. "Bi-Directional OFDM Transmission using
Adaptive Modulation that spreads Data Symbols". Power Line
Communications and its Applications, 2006, IEEE International
Symposium on, vol., No., pp. 202-207. cited by other .
Tachikawa, T., et al. "ARQ protocols for bi-directional data
transmission," Information Networking, 1998 (ICOIN-12)
Proceedings., Twelfth International Conference on., vol., No., pp.
468-473, Jan. 21-23, 1998. cited by other.
|
Primary Examiner: Tsai; Henry W
Assistant Examiner: Mamo; Elias
Attorney, Agent or Firm: Cantor Colburn LLP Campbell;
John
Claims
What is claimed is:
1. An article of manufacture comprising at least one computer
usable medium having computer readable program code logic to
determine an extended measurement word at a channel subsystem of an
input/output (I/O) processing system using data from a control
unit, the computer readable program code logic for performing a
method comprising: sending a command message to the control unit;
receiving a transport response information unit message at the
channel subsystem in response to sending the command message to the
control unit; extracting a plurality of time values from the
transport response information unit message as calculated by the
control unit; calculating an extended measurement word as a
function of the extracted time values; and writing the calculated
extended measurement word to computer readable memory in the I/O
processing system.
2. The article of manufacture of claim 1 wherein the computer
readable program code logic performing the method further
comprises: storing a start subchannel time prior to sending the
command message to the control unit; storing an ending status time
upon identifying an ending status in the transport response
information unit message; and determining a total system I/O
operation time at the channel subsystem as a time difference value
between the ending status time and the start subchannel time.
3. The article of manufacture of claim 2 wherein the extracted time
values include a total device time parameter indicating elapsed
time from when the control unit received the command message until
the control unit transmitted the transport response information
unit message for a device I/O operation; and the calculated
extended measurement word includes a function pending time
calculated as a time difference value between the total system I/O
operation time and the total device time parameter.
4. The article of manufacture of claim 1 wherein the computer
readable program code logic performing the method further
comprises: storing a sending time at the channel subsystem upon
sending the command message to the control unit; storing a
receiving time upon receiving the transport response information
unit message at the channel subsystem; and determining a total
channel time at the channel subsystem as a time difference value
between the receiving time and the sending time.
5. The article of manufacture of claim 4 wherein the extracted time
values include a total device time parameter indicating elapsed
time from when the control unit received the command message until
the control unit transmitted the transport response information
unit message for a device I/O operation; and the calculated
extended measurement word includes an initial command response time
calculated as a time difference value between the total channel
time and the total device time parameter.
6. The article of manufacture of claim 1 wherein the extracted time
values include: a total device time parameter indicating elapsed
time from when the control unit received the command message until
the control unit transmitted the transport response information
unit message for a device I/O operation; a defer time parameter
indicating control unit defer time; a device busy time parameter
indicating elapsed time that the command message was queued at the
control unit waiting on a reserved device; and a device active only
time parameter indicating elapsed time between a channel end (CE)
and a device end (DE) at the control unit in response to the
control unit holding the CE until the DE is available, and further
wherein the calculated extended measurement word includes a device
connect time calculated as a time difference value between the
total device time parameter, the defer time parameter, the device
busy time parameter and the device active only time parameter.
7. The article of manufacture of claim 1 wherein the extracted time
values include a defer time parameter indicating control unit defer
time and a queue time parameter indicating elapsed time that an I/O
operation was queued at the control unit excluding queue time for a
reserved device, and further wherein the calculated extended
measurement word includes a device disconnect time set equal to the
defer time parameter and a control unit queuing time set equal to
the queue time parameter.
8. The article of manufacture of claim 1 wherein the computer
readable program code logic performing the method further
comprises: determining an elapsed time between a channel end (CE)
and a device end (DE) at the channel subsystem, wherein the
extracted time values include a device active only time parameter,
and the calculated extended measurement word includes a device
active only time calculated as a time summation value of the
elapsed time and the device active only time parameter.
9. The article of manufacture of claim 1 wherein the extracted time
values include a device busy time parameter indicating elapsed time
that the command message was queued at the control unit waiting on
a reserved device; and the calculated extended measurement word
includes a device busy time calculated as a time summation value of
the device busy time parameter and additional device busy time as
determined by the channel subsystem.
10. The article of manufacture of claim 1 wherein the extracted
time values are extracted from an extended status section of the
transport response information unit message.
11. An apparatus for determining an extended measurement word in an
input/output (I/O) processing system, the apparatus comprising: a
channel subsystem in communication with a control unit, the channel
subsystem configured to perform a method comprising: sending a
command message to the control unit; receiving a transport response
information unit message at the channel subsystem in response to
sending the command message to the control unit; extracting a
plurality of time values from the transport response information
unit message as calculated by the control unit; calculating an
extended measurement word as a function of the extracted time
values; and writing the calculated extended measurement word to
computer readable memory in the I/O processing system.
12. The apparatus of claim 11 wherein the method further comprises:
storing a start subchannel time prior to sending the command
message to the control unit; storing an ending status time upon
identifying an ending status in the transport response information
unit message; and determining a total system I/O operation time at
the channel subsystem as a time difference value between the ending
status time and the start subchannel time.
13. The apparatus of claim 12 wherein the extracted time values
include a total device time parameter indicating elapsed time from
when the control unit received the command message until the
control unit transmitted the transport response information unit
message for a device I/O operation; and the calculated extended
measurement word includes a function pending time calculated as a
time difference value between the total system I/O operation time
and the total device time parameter.
14. The apparatus of claim 11 wherein the channel subsystem further
performs: storing a sending time at the channel subsystem upon
sending the command message to the control unit; storing a
receiving time upon receiving the transport response information
unit message at the channel subsystem; and determining a total
channel time at the channel subsystem as a time difference value
between the receiving time and the sending time.
15. The apparatus of claim 14 wherein the extracted time values
include a total device time parameter indicating elapsed time from
when the control unit received the command message until the
control unit transmitted the transport response information unit
message for a device I/O operation; and the calculated extended
measurement word includes an initial command response time
calculated as a time difference value between the total channel
time and the total device time parameter.
16. The apparatus of claim 11 wherein the extracted time values
include: a total device time parameter indicating elapsed time from
when the control unit received the command message until the
control unit transmitted the transport response information unit
message for a device I/O operation; a defer time parameter
indicating control unit defer time; a device busy time parameter
indicating elapsed time that the command message was queued at the
control unit waiting on a reserved device; and a device active only
time parameter indicating elapsed time between a channel end (CE)
and a device end (DE) at the control unit in response to the
control unit holding the CE until the DE is available, and further
wherein the calculated extended measurement word includes a device
connect time calculated as a time difference value between the
total device time parameter, the defer time parameter, the device
busy time parameter and the device active only time parameter.
17. The apparatus of claim 11 wherein the extracted time values
include a defer time parameter indicating control unit defer time
and a queue time parameter indicating elapsed time that an I/O
operation was queued at the control unit excluding queue time for a
reserved device, and further wherein the calculated extended
measurement word includes a device disconnect time set equal to the
defer time parameter and a control unit queuing time set equal to
the queue time parameter.
18. The apparatus of claim 11 wherein the channel subsystem
includes a channel subsystem timer, and the channel subsystem
further performs: determining an elapsed time between a channel end
(CE) and a device end (DE) at the channel subsystem using the
channel subsystem timer, wherein the extracted time values include
a device active only time parameter, and the calculated extended
measurement word includes a device active only time calculated as a
time summation value of the elapsed time and the device active only
time parameter.
19. The apparatus of claim 11 wherein the extracted time values
include a device busy time parameter indicating elapsed time that
the command message was queued at the control unit waiting on a
reserved device; and the calculated extended measurement word
includes a device busy time calculated as a time summation value of
the device busy time parameter and additional device busy time as
determined by the channel subsystem.
20. The apparatus of claim 11 wherein the extracted time values are
extracted from an extended status section of the transport response
information unit message.
21. A method for determining an extended measurement word at a
channel subsystem of an input/output (I/O) processing system using
data from a control unit, the method comprising: sending a command
message to the control unit; receiving a transport response
information unit message at the channel subsystem in response to
sending the command message to the control unit; extracting a
plurality of time values from the transport response information
unit message as calculated by the control unit; calculating an
extended measurement word as a function of the extracted time
values; and writing the calculated extended measurement word to
computer readable memory in the I/O processing system.
22. The method of claim 21 further comprising: storing a start
subchannel time prior to sending the command message to the control
unit; storing an ending status time upon identifying an ending
status in the transport response information unit message; and
determining a total system I/O operation time at the channel
subsystem as a time difference value between the ending status time
and the start subchannel time, wherein the extracted time values
include a total device time parameter and the calculated extended
measurement word includes a function pending time calculated as a
time difference value between the total system I/O operation time
and the total device time parameter.
23. The method of claim 21 further comprising: storing a sending
time at the channel subsystem upon sending the command message to
the control unit; storing a receiving time upon receiving the
transport response information unit message at the channel
subsystem; and determining a total channel time at the channel
subsystem as a time difference value between the receiving time and
the sending time, wherein the extracted time values include a total
device time parameter, a defer time parameter, a device busy time
parameter and a device active only time parameter, and the
calculated extended measurement word includes a device connect time
calculated as a time difference value between the total device time
parameter, the defer time parameter, the device busy time parameter
and the device active only time parameter.
24. An article of manufacture comprising at least one computer
usable medium having computer readable program code logic to
determine an extended measurement word at a channel subsystem of an
input/output (I/O) processing system using data from a control
unit, the computer readable program code logic for performing a
method comprising: sending a transport command information unit
message including a transport command control block (TCCB) as part
of a transport control word (TCW) channel program to the control
unit for execution; receiving a transport response information unit
message at the channel subsystem in response to sending the
transport command information unit message to the control unit,
wherein the transport response information unit message includes a
status section and an extended status section, the extended status
section further including a transport status header (TSH) defining
characteristics of a transport status area (TSA) of the extended
status section; extracting a plurality of time values from the
extended status section of the transport response information unit
message as calculated by the control unit using one or more control
unit timers, wherein the extracted time values include at least one
of a total device time parameter, a defer time parameter, a queue
time parameter, a device busy time parameter, a device active only
time parameter, and appended device sense data; calculating an
extended measurement word as a function of the extracted time
values, wherein the calculated extended measurement word includes
at least one of a device connect time, a function pending time, a
device disconnect time, a control unit queuing time, a device
active only time, a device busy time, and an initial command
response time; and writing the calculated extended measurement word
to computer readable memory in the I/O processing system.
25. An apparatus for determining an extended measurement word in an
input/output (I/O) processing system, the apparatus comprising: a
channel subsystem in communication with a control unit, the control
unit capable of controlling a device, the channel subsystem
configured to perform a method comprising: sending a transport
command information unit message including a transport command
control block (TCCB) as part of a transport control word (TCW)
channel program to the control unit for execution; receiving a
transport response information unit message at the channel
subsystem in response to sending the transport command information
unit message to the control unit, wherein the transport response
information unit message includes a status section and an extended
status section, the extended status section further including a
transport status header (TSH) defining characteristics of a
transport status area (TSA) of the extended status section;
extracting a plurality of time values from the extended status
section of the transport response information unit message as
calculated by the control unit using one or more control unit
timers, wherein the extracted time values include at least one of a
total device time parameter, a defer time parameter, a queue time
parameter, a device busy time parameter, a device active only time
parameter, and appended device sense data; calculating an extended
measurement word as a function of the extracted time values,
wherein the calculated extended measurement word includes at least
one of a device connect time, a function pending time, a device
disconnect time, a control unit queuing time, a device active only
time, a device busy time, and an initial command response time; and
writing the calculated extended measurement word to computer
readable memory in the I/O processing system.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present disclosure relates generally to input/output
processing, and in particular, to providing feedback data
associated with input/output processing to a channel subsystem.
2. Description of Background
Input/output (I/O) operations are used to transfer data between
memory and I/O devices of an I/O processing system. Specifically,
data is written from memory to one or more I/O devices, and data is
read from one or more I/O devices to memory by executing I/O
operations.
To facilitate processing of I/O operations, an I/O subsystem of the
I/O processing system is employed. The I/O subsystem is coupled to
main memory and the I/O devices of the I/O processing system and
directs the flow of information between memory and the I/O devices.
One example of an I/O subsystem is a channel subsystem. The channel
subsystem uses channel paths as communications media. Each channel
path includes a channel coupled to a control unit, the control unit
being further coupled to one or more I/O devices.
The channel subsystem may employ channel command words (CCWs) to
transfer data between the I/O devices and memory. A CCW specifies
the command to be executed. For commands initiating certain I/O
operations, the CCW designates the memory area associated with the
operation, the action to be taken whenever a transfer to or from
the area is completed, and other options.
During I/O processing, a list of CCWs is fetched from memory by a
channel. The channel parses each command from the list of CCWs and
forwards a number of the commands, each command in its own entity,
to a control unit coupled to the channel. The control unit then
processes the commands. The channel tracks the state of each
command and controls when the next set of commands are to be sent
to the control unit for processing. The channel ensures that each
command is sent to the control unit in its own entity. Further, the
channel infers certain information associated with processing the
response from the control unit for each command.
Performing I/O processing on a per CCW basis may involve a large
amount of processing overhead for the channel subsystem, as the
channels parse CCWs, track state information, and react to
responses from the control units. Therefore, it may be beneficial
to shift much of the processing burden associated with interpreting
and managing CCW and state information from the channel subsystem
to the control units. Simplifying the role of channels in
communicating between the control units and an operating system in
the I/O processing system may increase communication throughput as
less handshaking is performed. However, altering command sequences,
as well as roles of the channel subsystem and the control units,
can cause difficulties in maintaining legacy information associated
with the I/O processing. Timer values used to verify various
portions of a successful command sequence may be unavailable at the
channel subsystem without enhanced messaging from the control units
to provide extended measurement data. Such enhanced messaging would
necessitate additional functionality in both the control units and
the channel subsystem to provide and use the extended measurement
data. Accordingly, there is a need in the art for determining an
extended measurement word at a channel subsystem of an I/O
processing system based on extended measurement word data provided
by a control unit of the I/O processing system.
BRIEF SUMMARY OF THE INVENTION
Embodiments of the invention include an article of manufacture that
includes at least one computer usable medium having computer
readable program code logic to determine an extended measurement
word at a channel subsystem of an I/O processing system using data
from a control unit. The computer readable program code logic
performs a method including sending a command message to the
control unit, and receiving a transport response information unit
message at the channel subsystem in response to sending the command
message to the control unit. The computer readable program code
logic additionally extracts a plurality of time values from the
transport response information unit message as calculated by the
control unit, calculates an extended measurement word as a function
of the time values, and writes the extended measurement word to
computer readable memory in the I/O processing system.
Additional embodiments include an apparatus for determining an
extended measurement word in an I/O processing system. The
apparatus includes a channel subsystem in communication with a
control unit, where the control unit is capable of commanding and
determining status of an I/O device. The channel subsystem sends a
command message to the control unit, and receives a transport
response information unit message at the channel subsystem in
response to sending the command message to the control unit. The
channel subsystem further extracts a plurality of time values from
the transport response information unit message as calculated by
the control unit, calculates an extended measurement word as a
function of the time values, and writes the extended measurement
word to computer readable memory in the I/O processing system.
Further embodiments include a method for determining an extended
measurement word at a channel subsystem of an I/O processing system
using data from a control unit. The method includes sending a
command message to the control unit, and receiving a transport
response information unit message at the channel subsystem in
response to sending the command message to the control unit. The
method also includes extracting a plurality of time values from the
transport response information unit message as calculated by the
control unit, calculating an extended measurement word as a
function of the time values, and writing the extended measurement
word to computer readable memory in the I/O processing system.
An additional embodiment includes an article of manufacture
including at least one computer usable medium having computer
readable program code logic to determine an extended measurement
word at a channel subsystem of an I/O processing system using data
from a control unit. The computer readable program code logic
performs a method including sending a transport command information
unit message including a transport command control block (TCCB) as
part of a transport control word (TCW) channel program to the
control unit for execution. The computer readable program code
logic receives a transport response information unit message at the
channel subsystem in response to sending the transport command
information unit message to the control unit, where the transport
response information unit message includes a status section and an
extended status section. The extended status section further
includes a transport status header (TSH) defining characteristics
of a transport status area (TSA) of the extended status section.
The computer readable program code logic also extracts a plurality
of time values from the extended status section of the transport
response information unit message as calculated by the control unit
using one or more control unit timers, where the plurality of time
values include at least one of a total device time parameter, a
defer time parameter, a queue time parameter, a device busy time
parameter, a device active only time parameter, and appended device
sense data. The computer readable program code logic additionally
calculates an extended measurement word as a function of the time
values, where the extended measurement word includes at least one
of a device connect time, a function pending time, a device
disconnect time, a control unit queuing time, a device active only
time, a device busy time, and an initial command response time. The
computer readable program code logic writes the extended
measurement word to computer readable memory in the I/O processing
system.
A further embodiment includes an apparatus for determining an
extended measurement word in an I/O processing system. The
apparatus includes a channel subsystem in communication with a
control unit. The control unit is capable of commanding and
determining status of an I/O device. The channel subsystem sends a
transport command information unit message including a transport
command control block (TCCB) as part of a transport control word
(TCW) channel program to the control unit for execution. The
channel subsystem also receives a transport response information
unit message at the channel subsystem in response to sending the
transport command information unit message to the control unit,
where the transport response information unit message includes a
status section and an extended status section. The extended status
section further including a transport status header (TSH) defining
characteristics of a transport status area (TSA) of the extended
status section. The channel subsystem additionally extracts a
plurality of time values from the extended status section of the
transport response information unit message as calculated by the
control unit using one or more control unit timers, where the
plurality of time values include at least one of a total device
time parameter, a defer time parameter, a queue time parameter, a
device busy time parameter, a device active only time parameter,
and appended device sense data. The channel subsystem further
calculates an extended measurement word as a function of the time
values, where the extended measurement word includes at least one
of a device connect time, a function pending time, a device
disconnect time, a control unit queuing time, a device active only
time, a device busy time, and an initial command response time. The
channel subsystem writes the extended measurement word to computer
readable memory in the I/O processing system.
Other articles of manufacture, apparatuses, and/or methods
according to embodiments will be or become apparent to one with
skill in the art upon review of the following drawings and detailed
description. It is intended that all such additional articles of
manufacture, apparatuses, and/or methods be included within this
description, be within the scope of the present invention, and be
protected by the accompanying claims.
BRIEF DESCRIPTION OF THE DRAWINGS
The subject matter which is regarded as the invention is
particularly pointed out and distinctly claimed in the claims at
the conclusion of the specification. The foregoing and other
objects, features, and advantages of the invention are apparent
from the following detailed description taken in conjunction with
the accompanying drawings in which:
FIG. 1 depicts one embodiment of an I/O processing system
incorporating and using one or more aspects of the present
invention;
FIG. 2a depicts one example of a prior art channel command
word;
FIG. 2b depicts one example of a prior art channel command word
channel program;
FIG. 3 depicts one embodiment of a prior art link protocol used in
communicating between a channel and control unit to execute the
channel command word channel program of FIG. 2b;
FIG. 4 depicts one embodiment of a transport control word channel
program, in accordance with an aspect of the present invention;
FIG. 5 depicts one embodiment of a link protocol used to
communicate between a channel and control unit to execute the
transport control word channel program of FIG. 4, in accordance
with an aspect of the present invention;
FIG. 6 depicts one embodiment of a prior art link protocol used to
communicate between a channel and control unit in order to execute
four read commands of a channel command word channel program;
FIG. 7 depicts one embodiment of a link protocol used to
communicate between a channel and control unit to process the four
read commands of a transport control word channel program, in
accordance with an aspect of the present invention;
FIG. 8 depicts one embodiment of a control unit and a channel, in
accordance with an aspect of the present invention;
FIG. 9 depicts one embodiment of a response message communicated
from a control unit to a channel, in accordance with an aspect of
the present invention;
FIG. 10 depicts one embodiment of a timing diagram for channel and
control unit measurements of an I/O operation;
FIG. 11 depicts one embodiment of a process for determining an
extended measurement word at a channel subsystem of an I/O
processing system using data from a control unit; and
FIG. 12 depicts one embodiment of an article of manufacture
incorporating one or more aspects of the present invention.
The detailed description explains the preferred embodiments of the
invention, together with advantages and features, by way of example
with reference to the drawings.
DETAILED DESCRIPTION OF THE INVENTION
In accordance with an aspect of the present invention, input/output
(I/O) processing is facilitated. For instance, I/O processing is
facilitated by readily enabling access to information, such as
status and measurement data, associated with I/O processing.
Further, I/O processing is facilitated, in one example, by reducing
communications between components of an I/O processing system used
to perform the I/O processing. For instance, the number of
exchanges and sequences between an I/O communications adapter, such
as a channel, and a control unit is reduced. This is accomplished
by sending a plurality of commands from the I/O communications
adapter to the control unit as a single entity for execution by the
control unit, and by the control unit sending the data resulting
from the commands, if any, as a single entity.
The plurality of commands are included in a block, referred to
herein as a transport command control block (TCCB), an address of
which is specified in a transport control word (TCW). The TCW is
sent from an operating system or other application to the I/O
communications adapter, which in turn forwards the TCCB in a
command message to the control unit for processing. The control
unit processes each of the commands absent a tracking of status
relative to those individual commands by the I/O communications
adapter. The plurality of commands is also referred to as a channel
program, which is parsed and executed on the control unit rather
than the I/O communications adapter.
In an exemplary embodiment, the control unit generates a response
message including status and extended status information in
response to executing the channel program. The control unit may
also generate a response message without executing the channel
program under a limited number of communication scenarios, e.g., to
inform the I/O communications adapter that the channel program will
not be executed. The control unit may include a number of elements
to support communication between the I/O communications adapter and
I/O devices, as well as in support of channel program execution.
For example, the control unit can include control logic to parse
and process messages, in addition to one or more queues, timers,
and registers to facilitate communication and status monitoring.
The I/O communications adapter parses the response message,
extracting the status and extended status information, and performs
further calculations using the extracted information, such as
determining an extended measurement word.
One example of an I/O processing system incorporating and using one
or more aspects of the present invention is described with
reference to FIG. 1. I/O processing system 100 includes a host
system 101, which further includes for instance, a main memory 102,
one or more central processing units (CPUs) 104, a storage control
element 106, and a channel subsystem 108. The host system 101 may
be a large scale computing system, such as a mainframe or server.
The I/O processing system 100 also includes one or more control
units 110 and one or more I/O devices 112, each of which is
described below.
Main memory 102 stores data and programs, which can be input from
I/O devices 112. For example, the main memory 102 may include one
or more operating systems (OSs) 103 that are executed by one or
more of the CPUs 104. For example, one CPU 104 can execute a
Linux.RTM. operating system 103 and a z/OS.RTM. operating system
103 as different virtual machine instances. The main memory 102 is
directly addressable and provides for high-speed processing of data
by the CPUs 104 and the channel subsystem 108.
CPU 104 is the controlling center of the I/O processing system 100.
It contains sequencing and processing facilities for instruction
execution, interruption action, timing functions, initial program
loading, and other machine-related functions. CPU 104 is coupled to
the storage control element 106 via a connection 114, such as a
bidirectional or unidirectional bus.
Storage control element 106 is coupled to the main memory 102 via a
connection 116, such as a bus; to CPUs 104 via connection 114; and
to channel subsystem 108 via a connection 118. Storage control
element 106 controls, for example, queuing and execution of
requests made by CPU 104 and channel subsystem 108.
In an exemplary embodiment, channel subsystem 108 provides a
communication interface between host system 101 and control units
110. Channel subsystem 108 is coupled to storage control element
106, as described above, and to each of the control units 110 via a
connection 120, such as a serial link. Connection 120 may be
implemented as an optical link, employing single-mode or multi-mode
waveguides in a Fibre Channel fabric. Channel subsystem 108 directs
the flow of information between I/O devices 112 and main memory
102. It relieves the CPUs 104 of the task of communicating directly
with the I/O devices 112 and permits data processing to proceed
concurrently with I/O processing. The channel subsystem 108 uses
one or more channel paths 122 as the communication links in
managing the flow of information to or from I/O devices 112. As a
part of the I/O processing, channel subsystem 108 also performs the
path-management functions of testing for channel path availability,
selecting an available channel path 122 and initiating execution of
the operation with the I/O devices 112.
Each channel path 122 includes a channel 124 (channels 124 are
located within the channel subsystem 108, in one example, as shown
in FIG. 1), one or more control units 110 and one or more
connections 120. In another example, it is also possible to have
one or more dynamic switches (not depicted) as part of the channel
path 122. A dynamic switch is coupled to a channel 124 and a
control unit 110 and provides the capability of physically
interconnecting any two links that are attached to the switch. In
another example, it is also possible to have multiple systems, and
therefore multiple channel subsystems (not depicted) attached to
control unit 110.
Also located within channel subsystem 108 are subchannels (not
shown). One subchannel is provided for and dedicated to each I/O
device 112 accessible to a program through the channel subsystem
108. A subchannel (e.g., a data structure, such as a table)
provides the logical appearance of a device to the program. Each
subchannel provides information concerning the associated I/O
device 112 and its attachment to channel subsystem 108. The
subchannel also provides information concerning I/O operations and
other functions involving the associated I/O device 112. The
subchannel is the means by which channel subsystem 108 provides
information about associated I/O devices 112 to CPUs 104, which
obtain this information by executing I/O instructions.
Channel subsystem 108 is coupled to one or more control units 110.
Each control unit 110 provides logic to operate and control one or
more I/O devices 112 and adapts, through the use of common
facilities, the characteristics of each I/O device 112 to the link
interface provided by the channel 124. The common facilities
provide for the execution of I/O operations, indications concerning
the status of the I/O device 112 and control unit 110, control of
the timing of data transfers over the channel path 122 and certain
levels of I/O device 112 control.
Each control unit 110 is attached via a connection 126 (e.g., a
bus) to one or more I/O devices 112. I/O devices 112 receive
information or store information in main memory 102 and/or other
memory. Examples of I/O devices 112 include card readers and
punches, magnetic tape units, direct access storage devices,
displays, keyboards, printers, pointing devices, teleprocessing
devices, communication controllers and sensor based equipment, to
name a few.
One or more of the above components of the I/O processing system
100 are further described in "IBM.RTM. z/Architecture Principles of
Operation," Publication No. SA22-7832-05, 6th Edition, April 2007;
U.S. Pat. No. 5,461,721 entitled "System For Transferring Data
Between I/O Devices And Main Or Expanded Storage Under Dynamic
Control Of Independent Indirect Address Words (IDAWS)," Cormier et
al., issued Oct. 24, 1995; and U.S. Pat. No. 5,526,484 entitled
"Method And System For Pipelining The Processing Of Channel Command
Words," Casper et al., issued Jun. 11, 1996, each of which is
hereby incorporated herein by reference in its entirety. IBM is a
registered trademark of International Business Machines
Corporation, Armonk, N.Y., USA. Other names used herein may be
registered trademarks, trademarks or product names of International
Business Machines Corporation or other companies.
In one embodiment, to transfer data between I/O devices 112 and
memory 102, channel command words (CCWs) are used. A CCW specifies
the command to be executed, and includes other fields to control
processing. One example of a CCW is described with reference to
FIG. 2a. A CCW 200 includes, for instance, a command code 202
specifying the command to be executed (e.g., read, read backward,
control, sense and write); a plurality of flags 204 used to control
the I/O operation; for commands that specify the transfer of data,
a count field 206 that specifies the number of bytes in the storage
area designated by the CCW to be transferred; and a data address
208 that points to a location in main memory that includes data,
when direct addressing is employed, or to a list (e.g., contiguous
list) of modified indirect data address words (MIDAWs) to be
processed, when modified indirect data addressing is employed.
Modified indirect addressing is further described in U.S.
application Ser. No. 11/464,613, entitled "Flexibly Controlling The
Transfer Of Data Between Input/Output Devices And Memory," Brice et
al., filed Aug. 15, 2006, which is hereby incorporated herein by
reference in its entirety.
One or more CCWs arranged for sequential execution form a channel
program, also referred to herein as a CCW channel program. The CCW
channel program is set up by, for instance, an operating system, or
other software. The software sets up the CCWs and obtains the
addresses of memory assigned to the channel program. An example of
a CCW channel program is described with reference to FIG. 2b. A CCW
channel program 210 includes, for instance, a define extent CCW 212
that has a pointer 214 to a location in memory of define extent
data 216 to be used with the define extent command. In this
example, a transfer in channel (TIC) 218 follows the define extent
command that refers the channel program to another area in memory
(e.g., an application area) that includes one or more other CCWs,
such as a locate record 217 that has a pointer 219 to locate record
data 220, and one or more read CCWs 221. Each read CCW 220 has a
pointer 222 to a data area 224. The data area includes an address
to directly access the data or a list of data address words (e.g.,
MIDAWs or IDAWs) to indirectly access the data. Further, CCW
channel program 210 includes a predetermined area in the channel
subsystem defined by the device address called the subchannel for
status 226 resulting from execution of the CCW channel program.
The processing of a CCW channel program is described with reference
to FIG. 3, as well as with reference to FIG. 2b. In particular,
FIG. 3 shows an example of the various exchanges and sequences that
occur between a channel and a control unit when a CCW channel
program is executing. The link protocol used for the communications
is FICON (Fibre Connectivity), in this example. Information
regarding FICON is described in "Fibre Channel Single Byte Command
Code Sets-3 Mapping Protocol (FC-SB-3), T11/Project 1357-D/Rev.
1.6, INCITS (March 2003), which is hereby incorporated herein by
reference in its entirety.
Referring to FIG. 3, a channel 300 opens an exchange with a control
unit 302 and sends a define extent command and data associated
therewith 304 to control unit 302. The command is fetched from
define extent CCW 212 (FIG. 2b) and the data is obtained from
define extent data area 216. The channel 300 uses TIC 218 to locate
the locate record CCW and the read CCW. It fetches the locate
record command 305 (FIG. 3) from the locate record CCW 217 (FIG.
2b) and obtains the data from locate record data 220. The read
command 306 (FIG. 3) is fetched from read CCW 221 (FIG. 2b). Each
is sent to the control unit 302.
The control unit 302 opens an exchange 308 with the channel 300, in
response to the open exchange of the channel 300. This can occur
before or after locate command 305 and/or read command 306. Along
with the open exchange, a response (CMR) is forwarded to the
channel 300. The CMR provides an indication to the channel 300 that
the control unit 302 is active and operating.
The control unit 302 sends the requested data 310 to the channel
300. Additionally, the control unit 302 provides the status to the
channel 300 and closes the exchange 312. In response thereto, the
channel 300 stores the data, examines the status and closes the
exchange 314, which indicates to the control unit 302 that the
status has been received.
The processing of the above CCW channel program to read 4 k of data
requires two exchanges to be opened and closed and seven sequences.
The total number of exchanges and sequences between the channel and
control unit is reduced through collapsing multiple commands of the
channel program into a TCCB. The channel, e.g., channel 124 of FIG.
1, uses a TCW to identify the location of the TCCB, as well as
locations for accessing and storing status and data associated with
executing the channel program. The TCW is interpreted by the
channel and is not sent or seen by the control unit.
One example of a channel program to read 4 k of data, as in FIG.
2b, but includes a TCCB, instead of separate individual CCWs, is
described with reference to FIG. 4. As shown, a channel program
400, referred to herein as a TCW channel program, includes a TCW
402 specifying a location in memory of a TCCB 404, as well as a
location in memory of a data area 406 or a TIDAL 410 (i.e., a list
of transfer mode indirect data address words (TIDAWs), similar to
MIDAWs) that points to data area 406, and a status area 408. TCWs,
TCCBs, and status are described in further detail below.
The processing of a TCW channel program is described with reference
to FIG. 5. The link protocol used for these communications is, for
instance, Fibre Channel Protocol (FCP). In particular, three phases
of the FCP link protocol are used, allowing host bus adapters to be
used that support FCP to perform data transfers controlled by CCWs.
FCP and its phases are described further in "Information
Technology--Fibre Channel Protocol for SCSI, Third Version
(FCP-3)," T10 Project 1560-D, Revision 4, Sep. 13, 2005, which is
hereby incorporated herein by reference in its entirety.
Referring to FIG. 5, a channel 500 opens an exchange with a control
unit 502 and sends TCCB 504 to the control unit 502. In one
example, the TCCB 504 and sequence initiative are transferred to
the control unit 502 in a FCP command, referred to as FCP_CMND
information unit (IU) or a transport command IU. The control unit
502 executes the multiple commands of the TCCB 504 (e.g., define
extent command, locate record command, read command as device
control words (DCWs)) and forwards data 506 to the channel 500 via,
for instance, a FCP_Data IU. It also provides status and closes the
exchange 508. As one example, final status is sent in a FCP status
frame that has a bit active in, for instance, byte 10 or 11 of the
payload of a FCP_RSP IU, also referred to as a transport response
IU. The FCP_RES_IU payload may be used to transport FICON ending
status along with additional status information, including
parameters that support the calculation of extended measurement
words and notify the channel 500 of the maximum number of open
exchanges supported by the control unit 502.
In a further example, to write 4 k of customer data, the channel
500 uses the FCP link protocol phases, as follows:
1. Transfer a TCCB in the FCP_CMND IU.
2. Transfer the IU of data, and sequence initiative to the control
unit 502.
3. Final status is sent in a FCP status frame that has a bit active
in, for instance, byte 10 or 11 of the FCP_RSP IU Payload. The
FCP_RES_INFO field or sense field is used to transport FICON ending
status along with additional status information, including
parameters that support the calculation of extended measurement
words and notify the channel 500 of the maximum number of open
exchanges supported by the control unit 502.
By executing the TCW channel program of FIG. 4, there is only one
exchange opened and closed (see also FIG. 5), instead of two
exchanges for the CCW channel program of FIG. 2b (see also FIG. 3).
Further, for the TCW channel program, there are three communication
sequences (see FIGS. 4-5), as compared to seven sequences for the
CCW channel program (see FIGS. 2b-3).
The number of exchanges and sequences remain the same for a TCW
channel program, even if additional commands are added to the
program. Compare, for example, the communications of the CCW
channel program of FIG. 6 with the communications of the TCW
channel program of FIG. 7. In the CCW channel program of FIG. 6,
each of the commands (e.g., define extent command 600, locate
record command 601, read command 602, read command 604, read
command 606, locate record command 607 and read command 608) are
sent in separate sequences from channel 610 to control unit 612.
Further, each 4 k block of data (e.g., data 614-620) is sent in
separate sequences from the control unit 612 to the channel 610.
This CCW channel program requires two exchanges to be opened and
closed (e.g., open exchanges 622, 624 and close exchanges 626,
628), and fourteen communications sequences. This is compared to
the three sequences and one exchange for the TCW channel program of
FIG. 7, which accomplishes the same task as the CCW channel program
of FIG. 6.
As depicted in FIG. 7, a channel 700 opens an exchange with a
control unit 702 and sends a TCCB 704 to the control unit 702. The
TCCB 704 includes the define extent command, the two locate record
commands, and the four read commands in DCWs, as described above.
In response to receiving the TCCB 704, the control unit 702
executes the commands and sends, in a single sequence, the 16k of
data 706 to the channel 700. Additionally, the control unit 702
provides status to the channel 700 and closes the exchange 708.
Thus, the TCW channel program requires much less communications to
transfer the same amount of data as the CCW channel program of FIG.
6.
Turning now to FIG. 8, one embodiment of the control unit 110 and
the channel 124 of FIG. 1 that support TCW channel program
execution are depicted in greater detail. The control unit 110
includes CU control logic 802 to parse and process command messages
containing a TCCB, such as the TCCB 704 of FIG. 7, received from
the channel 124 via the connection 120. The CU control logic 802
can extract DCWs and control data from the TCCB received at the
control unit 110 to control a device, for instance, I/O device 112
via connection 126. The CU control logic 802 sends device commands
and data to the I/O device 112, as well as receives status
information and other feedback from the I/O device 112. For
example, the I/O device 112 may be busy because of a previous
reservation request targeting I/O device 112. To manage potential
device reservation contention issues that can arise when the
control unit 110 receives multiple requests to access the same I/O
device 112, the CU control logic 802 keeps track of and stores
device busy messages and associated data in a device busy queue
804. In an exemplary embodiment, an OS 103 of FIG. 1 reserves I/O
device 112 to keep other OSs 103 from accessing the I/O device 112
while the reservation is active. Although device reservation is not
required for all I/O operations, device reservation can be used to
support operations that necessitate exclusive access for a fixed
duration of time, e.g., disk formatting.
The CU control logic 802 can access and control other elements
within the control unit 110, such as CU timers 806 and CU registers
808. The CU timers 806 may include multiple timer functions to
track how much time a sequence of I/O operations takes to complete.
The CU timers 806 may further include one or more countdown timers
to monitor and abort I/O operations and commands that do not
complete within a predetermined period. The CU registers 808 can
include fixed values that provide configuration and status
information, as well as dynamic status information that is updated
as commands are executed by the CU control logic 802. The control
unit 110 may further include other buffer or memory elements (not
depicted) to store multiple messages or status information
associated with communications between the channel 124 and the I/O
device 112. The CU registers 808 may include a maximum control unit
exchange parameter that defines the maximum number of open control
unit exchanges that the control unit 110 supports.
The channel 124 in the channel subsystem 108 includes multiple
elements to support communication with the control unit 110. For
example, the channel 124 may include CHN control logic 810 that
interfaces with CHN subsystem timers 812 and CHN subsystem
registers 814. In an exemplary embodiment, the CHN control logic
810 controls communication between the channel subsystem 108 and
the control unit 110. The CHN control logic 810 may directly
interface to the CU control logic 802 via the connection 120 to
send commands and receive responses, such as transport command and
response IUs. Alternatively, messaging interfaces and/or buffers
(not depicted) can be placed between the CHN control logic 810 and
the CU control logic 802. The CHN subsystem timers 812 may include
multiple timer functions to track how much time a sequence of I/O
operations takes to complete, in addition to the time tracked by
the control unit 110. The CHN subsystem timers 812 may further
include one or more countdown timers to monitor and abort command
sequences that do not complete within a predetermined period. The
CHN subsystem registers 814 can include fixed values that provide
configuration and status information, as well as dynamic status
information, updated as commands are transported and responses are
received.
One example of a response message 900, e.g., a transport response
IU, communicated from the control unit 110 to the channel 124 upon
completion of a TCW channel program is depicted in FIG. 9. The
response message 900 provides status information to the channel 124
and may indicate that an open exchange between the channel 124 and
the control unit 110 should be closed. The status information
provided when a TCW channel program (e.g., as depicted in FIGS. 5
and 7) is executed includes additional information beyond the
status information sent upon completion of a CCW channel program
(e.g., as depicted in FIGS. 3 and 6). The response message 900
includes a status section 902 and an extended status section 904.
When the channel 124 receives the response message 900, it stores
parts of status section 902 in the subchannel for the device the
TCW was operating with and the extended status section 904 in a
memory location defined by the TCW associated with the TCW channel
program that triggered the response message 900. For example, a TCW
can designate a section of main memory 102 of FIG. 1 for storage of
the extended status section 904.
The status section 902 of the response message 900 can include
multiple fields, such as an address header 906, status flags one
908, maximum control unit exchange parameter 910, response flags
912, response code 914, residual count 916, response length 918,
reserved location 920, SPC-4 sense type 922, status flags two 924,
status flags three 926, device status 928, and a longitudinal
redundancy check (LRC) word 930. Each field in the status section
902 is assigned to a particular byte address to support parsing of
the response message 900. Although one arrangement of fields within
the status section 902 is depicted in FIG. 9, it will be understood
that the order of fields can be rearranged to alternate ordering
within the scope of the disclosure. Moreover, fields in the
response message 900 can be omitted or combined within the scope of
the invention, e.g., combining status flags two 924 and three 926
into a single field. SPC-4 is further described in "SCSI Primary
Commands-4 (SPC-4)", Project T10/1731-D, Rev 11, INCITS (May 2007),
which is hereby incorporated herein by reference in its
entirety.
In an exemplary embodiment, the address header 906 is set to the
same value as the value received by the control unit 110 in the
TCCB that initiated the TCW channel program. Although the address
header 906 is not required, including the address header 906 may
support testing to trace command and response messages on an I/O
device 112 while multiple I/O devices 112 are being accessed.
Status flags one 908 may indicate information such as the success
status of an I/O operation. Multiple bits within the status flags
one 908 can provide additional status information.
The maximum control unit exchange parameter 910 identifies the
maximum number of exchanges that the control unit 110 allows the
channel 124 to open to it. A value of zero may inform the channel
124 that the control unit 110 is not altering the current value
that the channel 124 is using. In an exemplary embodiment, the
channel 124 establishes a default value for the maximum number of
open exchanges, e.g. 64, which the control unit 110 can modify via
the maximum control unit exchange parameter 910. The value of the
maximum control unit exchange parameter 910 sent in the response
message 900 may be the actual value desired or a seed value for an
equation. For example, the value in the maximum control unit
exchange parameter 910 can be incremented and/or multiplied by the
channel 124 to determine the actual maximum number of open
exchanges, e.g. a value of "1" interpreted as "32" by the channel
124.
Using a default value for the maximum number of open exchanges
gives each control unit 110 and channel 124 a common starting point
that can be modified as determined by the control unit 110. In one
embodiment, the channel 124 checks the maximum control unit
exchange parameter 910 received in the response message 900 from
the control unit 110 to determine if the maximum control unit
exchange parameter 910 is lower than the default value or a
previously received value. If the new number is smaller than the
current number of open exchanges, the channel 124 does not drive
new I/O commands to the control unit 110 until the current number
of exchanges used is less than the new limit.
In an exemplary embodiment, the response flags field 912 uses the
standard definition as defined in FCP and can be set to default
value, e.g., two. The response code 914 may be equivalent to a
Small Computer System Interface (SCSI) status field and can be set
to a default value, such as zero. The residual count 916 for read
or write commands indicates the difference between how many bytes
were commanded to be read or written versus the number of bytes
that actually were read or written. The response length 918 is an
additional count of bytes of information in the response message
900 after the reserved location 920. The response length 918
supports variable sized response messages 900. The SPC-4 sense type
922 can be assigned to a particular value based upon message type,
e.g., a transport response IU=7F hexadecimal. In one embodiment,
the status flags two 924 is set to a value of 80 hexadecimal to
indicate that the I/O operation completed, with a valid value of
the residual count 916. Status flags three 926 is set to a value of
one when the I/O operation completed, indicating that extended
status 904 is included as part of the response message 900. The
device status 928 relays status information generated by the I/O
device 112. The LRC word 930 is a check word that covers the other
fields in the status section 902 of the response message 900 to
verify the integrity of the status section 902. The LRC word 930
can be generated through applying an exclusive-or operation to an
initial seed value with each field included in the LRC calculation
in succession.
The extended status section 904 provides information to the channel
subsystem 108 and the OS 103 associated with operating the control
unit 110 in a transport mode capable of running a TCW channel
program. The extended status section 904 may support configurable
definitions with different type status definitions for each type.
In an exemplary embodiment, the extended status section 904
includes a transport status header (TSH) 932, a transport status
area (TSA) 934, and an LRC word 936 of the TSH 932 and the TSA 934.
The TSH 932 may include extended status length 940, extended status
flags 942, a DCW offset 944, a DCW residual count 946, and a
reserved location 948. The TSH 932 is common for the different
formats, with the each format defined by a type code in the
extended status flags 942. The TSA 934 may include a total device
time parameter 950, defer time parameter 952, queue time parameter
954, device busy time parameter 956, device active only time
parameter 958, and appended device sense data 960. Each of these
fields is described in greater detail in turn.
The extended status length 940 is the size of the extended status
section 904. In an exemplary embodiment, the extended status flags
942 has the following definition:
Bit 0--The DCW offset 944 is valid.
Bit 1--The DCW residual count 946 is valid.
Bit 2--This bit set to a one informs the OS 103 of FIG. 1 in a
definitive manner when the control unit 110 had to access slow
media for data, e.g., a cache miss.
Bit 3--Time parameters 950-958 are valid. The type code set to a
one and this bit set to a one indicates that all or the time
parameters 950-958 are valid.
Bit 4--Reserved.
Bits 5 to 7--These three bits are the type code that defines the
format of the TSA 934 of the extended status section 904. The names
of the encodes are: 0. Reserved. 1. I/O Status. The extended status
section 904 contains valid ending status for the transport-mode I/O
operation. 2. I/O Exception. The extended status section 904
contains information regarding termination of the transport-mode
I/O operation due to an exception condition. 3. Interrogate Status.
The extended status section 904 contains status for an interrogate
operation. 4. to 7. Reserved.
The DCW offset 944 indicates an offset in the TCCB of a failed DCW.
Similarly, the DCW residual count 946 indicates the residual byte
count of a failed DCW (i.e., where execution of the DCWs was
interrupted).
In an exemplary embodiment, the TSA 934 definition when the type
code of ES flags 942 indicates a type of I/O Status includes time
parameters 950-958, as well as optionally appended device sense
data 960. The time parameters 950-958 represent time values and can
be scaled to any time units, such as microseconds. The CU timers
806 of FIG. 8 are used to calculate the time parameters 950-958,
and the CU registers 808 can also be employed to capture values of
the CU timers 806 on a triggering event.
The total device time parameter 950 is the elapsed time from when
the control unit 110 received the transport command IU until it
sent the transport response IU (i.e., response message 900) for the
I/O operation. The defer time parameter 952 indicates control unit
defer time. This is the time accumulated by the control unit 110
working with the I/O device 112 when no communication with the
channel 124 is performed. On CCW channel programs, such as that
depicted in FIG. 3, the control unit 302 disconnects from the
channel 300 during this time.
The queue time parameter 954 is the time that an I/O operation is
queued at the control unit 110, but does not include queue time for
device busy time where the I/O device 112 is reserved by another
channel 124 under control of a different OS 103 on the same system
or on another system. The device busy time parameter 956 is the
time that a transport command IU is queued at the control unit 110
waiting on a device busy caused by the I/O device 112 being
reserved by another channel 124 under control of a different OS 103
on the same system or on another system.
The device active only time parameter 958 is the elapsed time
between a channel end (CE) and a device end (DE) at the control
unit 110, when the control unit 110 holds the CE until DE is
available. The CE may indicate that the portion of an I/O operation
involving a transfer of data or control information between the
channel 124 and the control unit 110 has been completed. The DE may
indicate that the device portion of an I/O operation is completed.
The appended device sense data 960 is supplemental status that the
control unit 110 provides conditionally in response to an active
unit check (UC) bit in the device status 928.
The LRC word 936 is a longitudinal redundancy check word of the TSH
932 and the TSA 934, calculated in a similar fashion as the LRC
word 930 in the status 902 section of the response message 900. The
LRC word 936 can be calculated on a variable number of words,
depending upon the number of words included in the appended device
sense data 960.
FIG. 10 depicts an exemplary timing diagram 1000 illustrating how
the time parameters 950-958 are calculated in relation to each
other, as well as other time values calculated by the channel
subsystem 108 of FIG. 8. It will be understood that the timing
diagram 1000 depicts one example of a timing sequence, and as such,
may vary depending upon response times of various I/O processing
system elements. At start subchannel A 1002, the channel subsystem
108 captures a starting time value using the CHN subsystem timers
812 and the CHN subsystem registers 814 of FIG. 8. After an initial
processing and communication propagation delay, the channel 124
sends a transport command IU to the control unit 110. The control
unit 110 receives the transport command IU and creates a time stamp
of transport command IU C 1004 using the CU timers 806 and the CU
registers 808. The control unit 110 parses the transport command IU
to extract DCWs and send commands to a targeted I/O device, such as
the I/O device 112. The I/O device 112 may be busy completing a
command from another channel 124 under control of a different OS
103 on the same system or on another system. The control unit 110
uses the CU timers 806 to track device busy time 1006 and writes
the value to the device busy time parameter 956 of the response
message 900 of FIG. 9. If another channel 124 under control of a
different OS 103 on the same system or on another system attempts
to reserve or use the same I/O device 112, multiple requests are
queued on the device busy queue 804. The waiting time associated
with the device busy queue 804 is included in the device busy time
1006.
In I/O processing systems that run CCW channel programs, the
control unit 110 provides a command response at time CMR B 1008 to
acknowledge that an initial command has been received, and the
control unit 110 is ready for additional commands. However, when a
TCW channel program is run, the control unit 110 does not respond
at time CMR B 1008; rather, the control unit 110 waits until the
TCW channel program terminates to provide a response message to the
channel 124, such as the response message 900 of FIG. 9. Thus, to
perform extended measurement calculations that utilize time CMR B
1008, I/O processing systems that run TCW channel programs must
employ an alternate approach to derive timing information.
Queue time 1010 indicates time that an I/O operation is queued at
the control unit 110, but does not include the queue time for the
device busy time 1006, where the I/O device 112 is reserved by
another channel 124 under control of a different OS 103 on the same
system or on another system. The queue time 1010 is written to the
queue time parameter 954 in the response message 900 of FIG. 9.
The time accumulated by the control unit 110 working with the I/O
device 112 is illustrated as CU defer time 1012 in FIG. 10. The CU
defer time 1012 is written to the defer time parameter 952 in the
response message 900 of FIG. 9.
Device active only time 1014 represents time between CE and DE at
the control unit 110, if the control unit 110 does not present CE
status until the DE status is available. The device active only
time 1014 is written to the device active only time parameter 958
in the response message 900 of FIG. 9.
Once the control unit 110 completes the I/O operation requested in
the transport command IU, the control unit 110 sends a transport
response IU, e.g., the response message 900 of FIG. 9, to the
channel 124 at control unit response IU D 1016 time stamp. Total
device time 1018 can be calculated as the difference between the
time stamps of control unit response IU D 1016 and transport
command IU C 1004. The total device time 1018 is written to the
total device time parameter 950 in the response message 900 of FIG.
9. In response to the channel 124 receiving the transport response
IU, total channel time 1020 can be calculated as the time from when
the channel 124 sent the transport command IU to the control unit
110, until when the channel 124 received the transport response IU.
At time stamp ending status E 1022, the channel 124 identifies that
an ending status was received in the transport response IU from the
control unit 110. The channel subsystem 108 calculates total system
I/O operation time 1024 as the time difference between time stamp
ending status E 1022 and start subchannel A 1002.
In an exemplary embodiment, an extended measurement word (EMW)
including multiple time values provides I/O measurement information
for I/O operations performed at the channel 124 or a subchannel.
The channel subsystem 108 can use the time parameters 950-958
received in the response message 900 along with time values derived
from the CHN subsystem timers 812 to calculate the EMW. The EMW may
be stored in the CHN subsystem registers 814 or written to the main
memory 102 of FIG. 1. The EMW includes a device connect time, a
function pending time, a device disconnect time, a control unit
queuing time, a device active only time, a device busy time, and an
initial command response time. The device connect time is
calculated as total device time parameter 950-defer time parameter
952-device busy time parameter 956-device active only time
parameter 958. While the function pending time is calculated in an
I/O processing system supporting CCW channel programs as CMR B
1008-start subchannel A 1002, an I/O processing system supporting
TCW channel programs can calculate the function pending time as the
total system I/O operation time 1024-the total device time
parameter 950. The device disconnect time is set equal to the defer
time parameter 952. The control unit queuing time is set equal to
the queue time parameter 954. The device active only time is
calculated as the time between CE and DE at the channel 124+the
device active only time parameter 958. The device busy time is
calculated as the device busy time parameter 956+any other device
busy time information available to the channel subsystem 108. The
initial command response time is calculated as total channel time
1020-the total device time parameter 950. The EMW time values can
provide the OS 103 of FIG. 1 with insight as to the performance,
congestion, and efficiency of I/O operations occurring at the I/O
devices 112. The OS 103 or a higher-level application program may
respond in turn by altering I/O operation requests to better
balance elements of the I/O processing system 100 and reduce
delays.
Turning now to FIG. 11, a process 1100 for determining an extended
measurement word at a channel subsystem of an I/O processing system
using data from a control unit will now be described in accordance
with exemplary embodiments, and in reference to the I/O processing
system 100 of FIG. 1. At block 1102, the channel 124 in the channel
subsystem 108 sends a command message to the control unit 110. The
command message may be a transport command IU, including a TCCB
with multiple DCWs as part of a TCW channel program. The control
unit 110 receives the command message, parses it, and initiates I/O
operations as commanded in the DCWs to the I/O device 112. Upon
termination of the TCW channel program on the control unit 110, the
control unit 110 reports status information to the channel 124 in a
transport response IU message (e.g., response message 900 of FIG.
9, including a status section 902 and an extended status section
904). The transport response IU message includes a plurality of
time values as determined by the control unit 110, such as the time
parameters 950-958 of FIG. 9 as determined using the CU timers 806
of FIG. 8.
At block 1104, the channel subsystem 108 receives the transport
response IU message in response to sending the command message to
the control unit 110. Communication between the channel subsystem
108 and the control unit 110 may be managed by the CU control logic
802 and the CHN control logic 810 of FIG. 8 for a specific channel
124 of the channel subsystem 108.
At block 1106, the channel subsystem 108 extracts a plurality of
time values from the transport response IU message as calculated by
the control unit 110. For example, the time values may be extracted
from the extended status section 904 of the response message 900 of
FIG. 9, where the response message 900 depicts one embodiment of
the transport response IU message.
At block 1108, the channel subsystem 108 calculates an extended
measurement word as a function of the time values. The extended
measurement word can include a number of time values, such as a
device connect time, a function pending time, a device disconnect
time, a control unit queuing time, a device active only time, a
device busy time, and an initial command response time. Some of the
time values in the extended measurement word may also incorporate
values calculated at the channel subsystem 108, such as total
channel time and total system I/O operation time. Time values
calculated at the channel subsystem 108 can utilize one or more CHN
subsystem timers 812 of FIG. 8 to provide a time base.
At block 1110, the channel subsystem 108 writes the extended
measurement word to computer readable memory in the I/O processing
system 100, such as the main memory 102. The channel subsystem 108
may also write the contents of the extended status section 904 of
the response message 900 to the main memory 102 for the OS 103 or
other programs to access. In an exemplary embodiment, the specific
location to write the extended status section 904 in the I/O
processing system 100 is established by a TCW that includes an
address pointer to a desired write location.
Technical effects of exemplary embodiments include determining an
enhanced measurement word using time data provided by a control
unit in an I/O processing system. The channel receiving the time
data can gain insight into the performance of the control unit and
an I/O device controlled by the control unit over a period of time
encompassing multiple I/O operations. Advantages include acquiring
timing performance data without interrupting the execution of a TCW
channel program on a control unit. Thus, programs designed to gauge
performance of CCW channel programs can gauge the performance of
TCW channel programs in a seamless or near seamless fashion, while
gaining advantages of higher communication throughput due in part
to exchanging fewer messages per channel program.
As described above, embodiments can be embodied in the form of
computer-implemented processes and apparatuses for practicing those
processes. In exemplary embodiments, the invention is embodied in
computer program code executed by one or more network elements.
Embodiments include a computer program product 1200 as depicted in
FIG. 12 on a computer usable medium 1202 with computer program code
logic 1204 containing instructions embodied in tangible media as an
article of manufacture. Exemplary articles of manufacture for
computer usable medium 1202 may include floppy diskettes, CD-ROMs,
hard drives, universal serial bus (USB) flash drives, or any other
computer-readable storage medium, wherein, when the computer
program code logic 1204 is loaded into and executed by a computer,
the computer becomes an apparatus for practicing the invention.
Embodiments include computer program code logic 1204, for example,
whether stored in a storage medium, loaded into and/or executed by
a computer, or transmitted over some transmission medium, such as
over electrical wiring or cabling, through fiber optics, or via
electromagnetic radiation, wherein, when the computer program code
logic 1204 is loaded into and executed by a computer, the computer
becomes an apparatus for practicing the invention. When implemented
on a general-purpose microprocessor, the computer program code
logic 1204 segments configure the microprocessor to create specific
logic circuits.
While the invention has been described with reference to exemplary
embodiments, it will be understood by those skilled in the art that
various changes may be made and equivalents may be substituted for
elements thereof without departing from the scope of the invention.
In addition, many modifications may be made to adapt a particular
situation or material to the teachings of the invention without
departing from the essential scope thereof. Therefore, it is
intended that the invention not be limited to the particular
embodiment disclosed as the best mode contemplated for carrying out
this invention, but that the invention will include all embodiments
falling within the scope of the appended claims. Moreover, the use
of the terms first, second, etc. do not denote any order or
importance, but rather the terms first, second, etc. are used to
distinguish one element from another. Furthermore, the use of the
terms a, an, etc. do not denote a limitation of quantity, but
rather denote the presence of at least one of the referenced
item.
* * * * *
References