U.S. patent application number 13/749962 was filed with the patent office on 2013-05-30 for sequencing packets from multiple threads.
This patent application is currently assigned to JUNIPER NETWORKS, INC.. The applicant listed for this patent is Juniper Networks, Inc.. Invention is credited to Chih-Wei CHAO, Dongyi JIANG, Rakesh Gopala Krishnan NAIR, Jiaxiang SU.
Application Number | 20130136134 13/749962 |
Document ID | / |
Family ID | 47682862 |
Filed Date | 2013-05-30 |
United States Patent
Application |
20130136134 |
Kind Code |
A1 |
CHAO; Chih-Wei ; et
al. |
May 30, 2013 |
SEQUENCING PACKETS FROM MULTIPLE THREADS
Abstract
A device may reserve a slot for a received packet in a packet
ordering queue (POQ), convey the packet to one of a plurality of
threads for processing, obtain the packet from the one of the
plurality of threads after the packet has been processed, organize
the packet in the POQ in accordance with a position of the reserved
slot, and release the packet from the POQ if the reserved slot is a
head of the POQ.
Inventors: |
CHAO; Chih-Wei; (Saratoga,
CA) ; JIANG; Dongyi; (Milpitas, CA) ; NAIR;
Rakesh Gopala Krishnan; (Sunnyvale, CA) ; SU;
Jiaxiang; (San Jose, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Juniper Networks, Inc.; |
Sunnyvale |
CA |
US |
|
|
Assignee: |
JUNIPER NETWORKS, INC.
Sunnyvale
CA
|
Family ID: |
47682862 |
Appl. No.: |
13/749962 |
Filed: |
January 25, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11877146 |
Oct 23, 2007 |
8379647 |
|
|
13749962 |
|
|
|
|
Current U.S.
Class: |
370/392 |
Current CPC
Class: |
H04L 45/74 20130101;
H04L 47/34 20130101 |
Class at
Publication: |
370/392 |
International
Class: |
H04L 12/56 20060101
H04L012/56 |
Claims
1-20. (canceled)
21. A method comprising: receiving, at a device, a packet;
identifying, by the device, an address holder for the received
packet, the address holder indicating a queue associated with the
packet; identifying, by the device and based on the address holder,
the queue associated with the packet; reserving, by the device, a
slot for the packet in the queue, the slot being selected based on
an order in which the packet was received by a component of the
device in relation to other received packets; and placing, by the
device, the packet in the reserved slot.
22. The method of claim 21, where identifying the address holder
for the received packet includes: locating a record for a flow
associated with the packet, the address holder being associated
with the flow.
23. The method of claim 22, where locating the record includes:
extracting header information from the packet; and locating the
record associated with the flow using the header information.
24. The method of claim 21, where reserving the slot for the packet
includes writing an arrival position in the packet.
25. The method of claim 24, where writing the arrival position
includes writing an index to a tail of the queue in the packet.
26. The method of claim 21, where placing the packet in the
reserved slot includes writing a reference to the packet in the
slot.
27. The method of claim 21, where identifying the queue associated
with the packet includes: locking the address holder to prevent a
different component of the device from modifying contents of the
address holder.
28. A system comprising: one or more processors to: receive a
packet; identify an address holder for the received packet, the
address holder indicating a queue associated with the packet;
identify, based on the address holder, the queue associated with
the packet; reserve a slot for the packet in the queue, the slot
being selected based on an order in which the packet was received
by a component of the system in relation to other received packets;
and place the packet in the reserved slot.
29. The system of claim 28, where, when identifying the address
holder for the received packet, the one or more processors are
further to: locate a record for a flow associated with the packet,
the address holder being associated with the flow.
30. The system of claim 29, where, when locating the record, the
one or more processors are further to: extract header information
from the packet; and locate the record associated with the flow
using the header information.
31. The system of claim 28, where, when reserving the slot for the
packet, the one or more processors are further to write an arrival
position in the packet.
32. The system of claim 31, where, when writing the arrival
position, the one or more processors are further to write an index
to a tail of the queue in the packet.
33. The system of claim 28, where, when placing the packet in the
reserved slot, the one or more processors are further to write a
reference to the packet in the slot.
34. The system of claim 28, where, when identifying the queue
associated with the packet, the one or more processors are to: lock
the address holder to prevent a different component of the system
from modifying contents of the address holder.
35. A non-transitory computer-readable medium including
instructions, the instructions comprising: one or more instructions
that, when executed by one or more processors of a device, cause
the one or more processors to: receive a packet; identify an
address holder for the received packet, the address holder
indicating a queue associated with the packet; identify, based on
the address holder, the queue associated with the packet; reserve a
slot for the packet in the queue, the slot being selected based on
an order in which the packet was received by a component,
associated with the device, in relation to other received packets;
and place the packet in the reserved slot.
36. The computer-readable medium of claim 35, where the one or more
instructions that cause the one or more processors to identify the
address holder for the received packet include: one or more
instructions that cause the one or more processors to locate a
record for a flow associated with the packet, the address holder
being associated with the flow.
37. The computer-readable medium of claim 36, where the one or more
instructions that cause the one or more processors to locate the
record include: one or more instructions that cause the one or more
processors to extract header information from the packet; and one
or more instructions to locate the record associated with the flow
using the header information.
38. The computer-readable medium of claim 35, where the one or more
instructions that cause the one or more processors to reserve the
slot for the packet include: one or more instructions that cause
the one or more processors to write an arrival position in the
packet.
39. The computer-readable medium of claim 38, where the one or more
instructions that cause the one or more processors to write the
arrival position include: one or more instructions that cause the
one or more processors to write an index to a tail of the queue in
the packet.
40. The computer-readable medium of claim 35, where the one or more
instructions that cause the one or more processors to place the
packet in the reserved slot include: one or more instructions that
cause the one or more processors to write a reference to the packet
in the slot.
Description
BACKGROUND
[0001] Today's network device, such as a router or a switch, may
employ multiple threads to handle received packets. By distributing
work among several threads, the network device may accelerate
packet switching, routing, rendering firewall services, and/or
other types of processing.
SUMMARY
[0002] According to one aspect, a device may include a plurality of
service threads for processing packets, a packet ordering queue
(POQ) for ordering the packets after the plurality of service
threads process the packets, a packet distribution thread, and a
packet ordering thread. The packet distribution thread may be
configured to receive the packets, reserve slots in the POQ for the
packets in an order of their arrival, and queue the packets for the
plurality of service threads. The packet ordering thread may be
configured to obtain the packets from the plurality of service
threads, place information relating to the packets in the reserved
slots in the POQ, and release the packets from the reserved slots
in the order of their arrival.
[0003] According to one aspect, a method may include reserving a
slot for a received packet in a packet ordering queue (POQ),
conveying the packet to one of a plurality of threads for
processing, obtaining the packet from the one of the plurality of
threads after the packet has been processed, organizing the packet
in the POQ in accordance with a position of the reserved slot, and
releasing the packet from the POQ if the reserved slot is a head of
the POQ.
[0004] According yet another aspect, a device may include means for
receiving packets in a sequence, means for reserving slots for the
received packets in a packet ordering queue (POQ), means for
transferring the packets to a plurality of flow threads, means for
obtaining the packets from the plurality of flow threads
out-of-order, means for placing the out-of-order packets in the
reserved slots, and means for sending the packets from the POQ in
the sequence.
BRIEF DESCRIPTION OF THE DRAWINGS
[0005] FIG. 1 is a diagram of an exemplary network in which
concepts described herein may be implemented;
[0006] FIG. 2 is a block diagram of an exemplary network device of
the exemplary network of FIG. 1;
[0007] FIG. 3 is a block diagram of an exemplary service module of
FIG. 2;
[0008] FIG. 4 is a functional block diagram of the exemplary
service module of FIG. 3;
[0009] FIG. 5 is a functional diagram of an exemplary packet
ordering queue (POQ) of FIG. 4;
[0010] FIG. 6 is a flow diagram of an exemplary process for placing
a packet in an exemplary ingress queue (IN_QUEUE) of FIG. 4;
[0011] FIG. 7 is a flow diagram of an exemplary process for
sequencing packets using a packet ordering queue (POQ) of FIG.
4;
[0012] FIG. 8 illustrates flow of packets through some of the
components in FIG. 4; and
[0013] FIGS. 9A-9G illustrate another exemplary POQ in different
configurations.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
[0014] The following detailed description refers to the
accompanying drawings. The same reference numbers in different
drawings may identify the same or similar elements.
[0015] In the following, a circular buffer may be used to sequence
processed packets from multiple threads. When the packets are
received at a component, the multiple threads may process the
packets in parallel. However, because processing delays for
different threads are variable, the threads may complete their
processing of the packets out-of-order. In such a case, the
circular buffer may be used to organize the processed packets in
the same order that the packets were received.
[0016] The term "thread," as used herein, may refer to a sequence
of instructions that may run independently of other threads. The
thread may share data with other threads. As used herein, the term
"thread" may also be interpreted to refer to a program, an instance
of a program, or a thread instance.
[0017] The term "packet," as used herein, may refer to an Internet
Protocol (IP) packet, datagram, cell, a fragment of an IP packet,
or other types of data that may be carried at a specified
communication layer. For example, a packet may refer to an IP
packet that has been pre-pended with additional header fields
(e.g., metadata fields).
[0018] The term "circular buffer," as used herein, may refer to a
buffer where the first block of memory follows the last block of
memory. A block of memory that follows another block of memory in a
circular buffer may not necessarily occupy contiguous regions of
memory (e.g., a circular buffer that is made of a linked list).
[0019] The term "source," as used herein, may refer to a node
(e.g., network node, etc.) from which packets originate. The term
"destination," as used herein, may refer to a node to which the
packets are sent. A source or a destination may be a logical
network source or destination that is associated with one or more
identifiers, such as an Internet Protocol (IP) address, a port
number, etc.
[0020] The term "flow," as used herein, may refer to a stream of
packets from a source to a destination. As used herein, the term
"flow statistics" may refer to information about a flow, such as a
byte count, a number of packets, etc.
[0021] The term "address," as used herein, may refer not only to a
memory address, but, to a reference to a region of memory. The term
"reference," as used herein, may include a network address, a
variable name, a memory offset, or other data or information (e.g.,
an identifier) that may be used to obtain the location of the
region of memory. The address or the reference may be used to
obtain contents of the memory region.
[0022] As used herein, the term "service module" may refer to,
within a network device, a logical or physical unit for performing
a set of tasks or a service for a packet that arrives at the
network device. For example, a service module may provide a load
balancing service, a firewall service, an intrusion detection
system (IDS) service, a hypertext transfer protocol (HTTP) service,
an encryption/decryption service, etc. A service module may be
implemented in hardware, software, or a combination of hardware and
software.
[0023] In the following, a circular buffer may be used to sequence
packets in the same order that the packets are received at a
component (e.g., a data plane, a hardware module for inspecting a
packet's compliance to a security requirement, etc.). When the
packets are received, multiple threads may process them in
parallel. However, because processing delays for different threads
are variable, the threads may finish processing the packets
out-of-order.
[0024] To ensure processed packets are in a proper sequence, slots
on the circular buffer may be reserved in the order the packets
were received. After the packets have been processed by threads,
the packets may be placed in the reserved slots. The referenced
packets may be released from the slot in the order the slots are
reserved, and, therefore, in the order that the packets were
received at the component.
[0025] Without the circular buffer, the packets may be sent from
the circular buffer out-of-order. For example, if packet A arrives
before packet B and thread X finishes processing packet B before
thread Y finishes processing packet A, packet B may be sent from
the component before packet A. With the circular buffer, packet B
may be prevented from being sent before packet A is processed and
sent.
[0026] FIG. 1 is a diagram of an exemplary network 100 in which
concepts described herein may be implemented. Network 100 may
include the Internet, an ad hoc network, a local area network
(LAN), a wide area network (WAN), a metropolitan area network
(MAN), a cellular network, a public switched telephone network
(PSTN), an intranet, or a combination of networks.
[0027] As shown, network 100 may include N network devices 102-1
through 102-N (collectively referred to herein as "network device
102"). Each of network devices 102-1 through 102-N may include a
switch, a router, a server, a firewall, and/or another type of
device. While network devices 102 may be implemented as different
types of devices, in the following paragraphs, network devices 102
will be described herein in terms of a router, a switch, or a
firewall.
[0028] FIG. 2 is a block diagram of an exemplary network device
102. As shown, network device 102 may include a controller 202, M
line interfaces 204-1 through 204-M (collectively referred to
herein as "line interface 204"), a service module 206, a switch
fabric 208, and a communication path(s) 210. Depending on
implementation, network device 102 may include additional, fewer,
or different components than those illustrated in FIG. 2. For
example, in one implementation, network device 102 may include
additional service modules for rendering network services, such as
a firewall service, a load balancing service, etc.
[0029] Controller 202 may include one or more devices for managing
routes and/or types of information that demand centralized
processing. Controller 202 may manage routes (e.g., accept or
disseminate routes) in accordance with routing/signaling protocols.
Line interfaces 204-1 through 204-M may include devices for
receiving packets from network devices in network 100 and for
transmitting the packets to other network devices in network 100.
In addition, each of line interfaces 204-1 through 204-M may
perform packet forwarding, packet classification, and/or internal
redirection of packets to other components in network device 102
(e.g., service module 206).
[0030] Service module 206 may include hardware, software, or a
combination of hardware and software for rendering services related
to a received packet. The services may be rendered via multiple
threads. After processing the packet, service module 206 may drop
the packet or direct the packet to another service module (not
shown) or to one of line interfaces 204-1 through 204-M.
[0031] Switch fabric 208 may include switches for conveying packets
to/from line interfaces 204-1 through 204-M and/or service module
206 from/to others of line interfaces 204-1 through 204-M or
service modules (not shown). Communication path(s) 210 may provide
an interface through which components of network device 102 can
communicate with one another.
[0032] FIG. 3 is a block diagram of service module 206. As shown,
service module 206 may include a processing unit 302 and a memory
304. Depending on implementation, service module 206 may include
additional, fewer, and/or different components than those
illustrated in FIG. 3.
[0033] Processing unit 302 may include one or more processors,
microprocessors, Application Specific Integrated Circuits (ASICs),
and/or Field Programmable Gate Arrays (FPGAs), and/or other
processing logic. Memory 304 may include static memory, such as
read only memory (ROM), and/or dynamic memory, such as random
access memory (RAM), or onboard cache, for storing data and
machine-readable instructions.
[0034] FIG. 4 is a functional block diagram of service module 206.
As shown, service module 206 may include packet distribution thread
(PDT) 402, P address holders 404-1 through 404-P (collectively
referred to herein as address holders 404 and individually as
address holder 404-x), T packet ordering queues (POQs) 406-1
through 406-T (collectively referred to herein as POQs 406 and
individually POQ 406-x), an ingress queue (IN_QUEUE) 408, R service
threads 410-1 through 410-R (collectively referred to herein as
service threads 410 and individually as service thread 410-x), an
egress queue (OUT_QUEUE) 412, and a packet ordering thread (POT)
414. Depending on implementation, service module 206 may include
additional, fewer, or different functional components than that
illustrated in FIG. 4. For example, service module 206 may contain
other threads for communication services.
[0035] PDT 402 may include an instance of a thread for preparing
packets to be distributed to service threads 410-1 through 410-R.
When PDT 402 receives a packet, PDT 402 may store information,
which will be described below, related to POQ 406-x in the packet,
and hand off the packet to IN_QUEUE 408. In some implementations,
the order in which PDT 402 receives packets may mirror the order in
which the packets are received by service module 206.
[0036] Address holder 404-x may store either a null value (e.g.,
value that does not indicate an address or a reference) or an
address of POQ 406-x, and may provide the address of POQ 406-x to a
thread. In some implementations, an address holder may be
stand-alone or may be part of another block of memory, such as a
link in a linked list, a record (e.g., a flow record), a table,
etc. A thread that needs to access POQ 406-x may do so by first
locating address holder 404-x that stores the address of POQ
406-x.
[0037] POQs 406 may include circular buffers to organize packets
that are placed in OUT_QUEUE 412. More than one POQ 406-x may be
provided, so that packets that are placed in one POQ 406-x may not
block packets that are placed in other POQs 406. As already
explained in part, slots (e.g., a block of memory) in POQ 406-x may
be reserved for packets in the order that the packets are received
by PDT 402. After the packets have been processed by service
threads 410, the packets may be placed in the reserved slots by
writing references to the packets in the reserved slots. Once the
references have been written in the slots in POQ 406-x, the packets
may be released from POQ 406-x. In some implementations, releasing
the packets may entail additional processing, such as sending the
packets from service module 206 to another component in network
device 102 or to another network device.
[0038] IN_QUEUE 408 may include one or more blocks of memory for
queuing packets. IN_QUEUE 408 may receive packets from PDT 402 and
hold the packets until the packets are distributed to one or more
of service threads 410.
[0039] Service thread 410-x may include an instance of a thread for
processing packets that are received, rendering a particular
service for the packet (e.g., security violation detection), and
handing off the packet to OUT_QUEUE 412. In FIG. 4, even though
service threads 410-1 through 410-R are shown as being identical,
in a different implementation, one or more of service threads 410-1
through 410-R may render a different service. OUT_QUEUE 412 may
include one or more blocks of memory for queuing packets. OUT_QUEUE
412 may receive a packet from service thread 410-x and temporarily
hold the packet until the packet is removed by POT 414.
[0040] POT 414 may include an instance of a thread/process for
sequencing packets that are in OUT_QUEUE 412. POT 414 may remove a
packet from OUT_QUEUE 412, access the information that has been
stored by PDT 402 in the packets, and use the information to order
the packets in POQ 406-x.
[0041] In some implementations, one or more of the components in
FIG. 4 may be implemented in memory 304 during runtime. In such an
implementation, address holder 404-x and/or POQ 406-x for
sequencing a packet may not yet exist when the packet is received
at PDT 402. Furthermore, if the packet is received and address
holder 404-x and/or POQ 406-x for sequencing the packet does not
exist, address holder 404-x and/or POQ 406-x may be dynamically
created in memory 304.
[0042] FIG. 5 is a functional block diagram of POQ 406-x. As shown,
POQ 406-x may include slots 502-1 through 502-S. Each of slots
502-1 through 502-S may be associated with a predetermined index
(e.g., slot 502-1 may be associated with an index of 1) that can be
used to identify a particular slot. Furthermore, slots 502-1
through 502-S may be either reserved for packets (e.g., packets
504-1 through 504-3) or un-reserved. In FIG. 5, a reserved slot and
an un-reserved slot are shown with "R" and "U," respectively, and
reserved slots 502-2 through 502-4 are illustrated as being
associated with packets 504-1 through 504-3.
[0043] In FIG. 5, reserved slots 502-2 through 502-5 may include a
head, which is the position that is occupied by slot 502-2 in POQ
406-x, and a tail, which is the position occupied by slot 502-5 in
POQ 406-x. A head may identify the first of the reserved slots, and
the tail may identify the slot after the last reserved slot.
Different slots may become the head and the tail, as PDT 402 makes
new reservations in POQ 406-x and POT 414 removes (e.g.,
"un-reserves") existing reservations from POQ 406-x.
[0044] When a PDT 402 receives a new packet, PDT 402 may reserve a
slot by writing, in the packet, the index associated with the tail
slot as the packet's arrival position, and by designating the slot
that immediately follows the last tail as the new tail. For
example, in FIG. 5, assume that PDT 402 receives a new packet. PDT
402 may reserve a slot by writing the index of the tail slot (e.g.,
"5") in the packet, and setting slot 502-6 as the tail. Thus, each
time PDT 402 makes a reservation in POQ 406-x, the tail may shift
to a different slot, unless the tail slot is the same as the head
slot. In such a case, no more slots may be available for
reservation.
[0045] A slot in POQ 406-x may become available for reservation,
when a slot becomes un-reserved. Un-reserving the slot may involve
placing the packet in the reserved slot, and releasing the packet
from POQ 406-x if the reserved slot is the head of POQ 406-x.
[0046] More specifically, when a POT 414 obtains a packet from
OUT_QUEUE 412, POT 414 may place the packet in the reserved slot
by: matching the arrival position of the packet (e.g., the index
that has been written into the packet by PDT 402) to one of indices
of reserved slots 502-1 through 502-5; and writing a reference to
the packet into the slot whose index matches the arrival position.
In FIG. 5, the marked slot is indicated by a darkened packet (e.g.,
packet 504-2). In contrast, slot 502-2 that is associated with
packet 504-1 is not dark, indicating POT 414 has not found packet
504-1 in OUT_QUEUE 412 and matched the index of "2" with the
arrival position of packet 504-1.
[0047] If the slot is also the head of POQ 406-x, POT 414 may
release the packet from POQ 406-x. To release the packet, POT 414
may designate a slot that immediately follows the head as the new
head of POQ 406-x. For example, in FIG. 5, assuming that packet
504-1 has been placed in OUT_QUEUE 412 and that the arrival
position of packet 504-1 matches index "2," POT 414 may release
packet 504-1 from POQ 406-x by designating slot 502-3 as the head
of POQ 406-x.
[0048] In the above, because packets that are associated with the
reserved slots of POQ 406-x are in the order of their arrival at
PDT 402, sweeping movement of the head of POQ 406-x in one
direction (e.g., the clockwise direction) may also release the
packets in the order of their arrival. By moving the head to a
different slot only when a processed packet is associated with the
head slot, POT 414 may ensure the release of the packet in the
proper sequence.
[0049] In brief, in POQ 406-x, reserved slots may include a head
and a tail. A head may identify the first of the reserved slots,
and the tail may identify the slot after the last reserved slot.
Different slots may become the head and the tail, as PDT 402 makes
new reservations in POQ 406-x and POT 414 removes (e.g.,
"un-reserves") existing reservations from POQ 406-x. Reserving a
slot in POQ 406-x may shift a tail to a new position and render the
slot unavailable for another reservation. Un-reserving the slot may
free the slot and release a packet that is referenced by the
slot.
Exemplary Process for Placing a Packet in an Exemplary Ingress
Queue
[0050] The above paragraphs describe system elements that are
related to devices and/or components for sequencing packets from
multiple threads. FIGS. 6 and 7 are flow diagrams of exemplary
processes 600 and 700 that are capable of being performed by one or
more of these devices and/or components.
[0051] FIG. 6 is a flow diagram of an exemplary process for placing
a packet in IN_QUEUE 408. Process 600 may start at block 602 where
a new packet may be received (block 602). Depending on
implementation, the packet may be received at PDT 402 or at another
component (e.g., a buffer within service module 206).
[0052] It may be determined if there is an address holder 404-x for
the packet (block 604). In one implementation, address holder 404-x
may be part of a flow record. In such a case, header information
from the packet may be extracted and used to locate the flow record
to which the packet is related. If address holder 404-x does not
yet exist (e.g., a flow record for the packet does not exist),
process 600 may proceed to block 612, where the packet may be
placed in IN_QUEUE 408. In one implementation, address holder 404-x
for the flow may be created by one of service threads 410.
[0053] Returning to block 604, if address holder 404-x exists,
process 600 may proceed to block 606, where address holder 404-x
may be examined to determine if POQ 406-x exists. In some
implementations, existence of address holder 404-x may imply the
existence of POQ 406-x (e.g., POQ 406-x is created at the same time
as address holder) and block 606 may be omitted. During the
examination of address holder 404-x, address holder 404-x may
temporarily be locked to prevent another thread (e.g., an instance
of POT 414) from modifying contents of address holder 404-x, delete
POQ 406-x, or modify POQ 406-x.
[0054] If POQ 406-x does not exist, POQ 406-x may be allocated
(block 608). Allocating POQ 406-x may include writing a reference
to POQ 406-x in address holder 404-x. From block 608, process 600
may proceed to block 610.
[0055] Returning to block 606, if POQ 406-x already exists, process
may proceed to block 610, where a reference to a slot for the
packet may be reserved in POQ 406-x. The reservation may include
involve writing an arrival position, which is the index of the tail
of POQ 406-x, in the packet. After the writing, the tail of the POQ
406-x may be assigned to another slot that follows the current tail
slot. Once the tail has been assigned, the lock on address holder
404-x may be released.
[0056] The packet may be placed in IN_QUEUE 408 (block 612).
Placing the packet in IN_QUEUE 408 may involve writing a reference
to the packet in a slot within IN_QUEUE 408. Depending on whether
other threads are inserting/removing packets to/from IN_QUEUE 408
or whether a single instruction can remove/insert the packet,
IN_QUEUE 408 may or may not be locked during the placement of the
packet. The inserted packet may wait in IN_QUEUE 408 until the
packet is distributed to service threads 410. If a new packet
arrives at PDT 402, process 600 may return to block 602.
Exemplary Process for Sequencing Packets Using a Packet Ordering
Queue
[0057] FIG. 7 is a flow diagram of an exemplary process for 408 for
sequencing packets using POQ 406-x. Prior to the start of process
700, packets in IN_QUEUE 408 may have been distributed to service
threads 410. Service threads 410 may process the packets and place
the packets in OUT_QUEUE 412.
[0058] Process 700 may start at block 702, where a packet may be
removed from OUT_QUEUE 412. Depending on whether service threads
410 are inserting packets in OUT_QUEUE 412, OUT_QUEUE 412 may be
locked while the packet is being removed from OUT_QUEUE 412 to
prevent a thread from reading OUT_QUEUE 412 while another thread is
writing to OUT_QUEUE 412.
[0059] It may be determined if the packet includes a reference to
POQ 406-x (block 704). Block 704 may determine if the packet may be
sequenced (e.g., ordered). If the packet does not include the
reference, process 700 may proceed to block 706.
[0060] Depending on implementation, additional processing may be
performed at block 706. For example, in one implementation, the
packet may be copied and/or dropped. In another implementation, the
packet may be transmitted from service module 206 to one of line
interfaces 204-1 through 204-M.
[0061] Returning to block 704, if the packet includes a reference
to POQ 406-x, the reference may be used to locate POQ 406-x, so
that the packet may be placed in the reserved slot. The packet may
be placed in the reserved slot by: matching an arrival position,
which may have been written in the packet by process 600, to an
index of a slot in POQ 406-x (block 708); and writing the arrival
position of the packet in the slot.
[0062] It may be determined if the slot is the head of POQ 406-x
(block 710). If the slot is not the head, process 700 may return to
block 706. Otherwise, process may proceed to block 712.
[0063] At block 712, the packet may be released from POQ 406-x. To
release the packet, POT 414 may designate a slot that follows the
head as the new head of POQ 406-x. As explained above in the
description of POQ 406-x, because packets that are referenced by
the reserved slots of POQ 406-x are in the order of their arrival,
a movement of the head of POQ 406-x in one direction (e.g., to the
clockwise direction) from one slot to another slot may release the
packets in the proper order (i.e., the order in which the packets
were received).
[0064] It may be determined if POQ 406-x is empty (block 714). If
POQ 406-x is not empty, process 700 may go to block 716. At block
716, it may be determined if a head slot of POQ 406-x contains a
reference to a packet. In such a case, process 700 may proceed to
block 712, where the packet may be released. Returning to block
716, if the head slot does not contain a reference to a packet,
process 700 may go to block 706.
[0065] Returning to block 714, if POQ 406-x is empty (e.g., all
slots in POQ 406-x are un-reserved), process 700 may proceed to
block 718, where POQ 406-x may be removed. Removing POQ 406-x may
involve locking address holder 404-x, returning a block of memory
that has been allocated for POQ 406-x to a free pool, and setting
the contents of address holder 404-x to a null value. Returning the
block of memory to the free pool may render the block available for
future use.
[0066] At block 706, process 700 may await for an arrival of a
packet in OUT_QUEUE 412. When the packet is detected, process 700
may start anew at block 702.
Example
[0067] The following example, with reference to FIGS. 8 and 9A-9G,
illustrates sequencing packets from multiple threads. The example
is consistent with exemplary processes 600 and 700 described above
with reference to FIGS. 6 and 7.
[0068] In the example, as illustrated in FIG. 8, service module 800
includes a flow table 802, a POQ 804, an IN_QUEUE 806, flow threads
808-1 and 808-2, and an OUT_QUEUE 810. Other components of service
module 800 are not illustrated in FIG. 8. As further shown, flow
table 802 includes record 802-1 and 802-2, each of which contains
address holder 802-1B, and 802-2B. The contents of address holders
802-1B and 802-2B may be obtained based on the address of flow
table 802, which is known. Records 802-1 and 802-2 also includes
other flow fields, 802-1A and 802-2A, which contain information
related to flows (e.g., flow statistics) that are associated with
incoming packets.
[0069] As illustrated in FIG. 9A, assume that POQ 804 has four
slots 902-1 through 902-4. Also assume that the tail and the head
of POQ 804 are at slot 902-1.
[0070] When packet X arrives at service module 800, a PDT (not
shown) may look up a flow record for the flow to which packet X
belongs. Upon finding flow record 802-1, the PDT determines that
there is address holder 802-1B for packet X. The PDT locks flow
record 802-1. Using flow record 802-1, the PDT obtains contents of
address holder 802-1B and examines the contents to determine if a
POQ exists. The PDT determines that POQ 804 exists based on a
non-null value in address holder 802-1B, and prepares packet X to
be placed in IN_QUEUE 806.
[0071] To prepare packet X, the PDT reserves a slot in POQ 804 by
assigning the tail of POQ 804 to slot 902-2 and writing the slot
number "0" (i.e., the arrival position of packet X) in packet X.
Furthermore, the PDT writes an identifier associated with PDQ 804
in packet X. The PDT unlocks flow record 802-1, and places packet X
in IN_QUEUE 806.
[0072] FIG. 9B shows the configuration of POQ 804 after packet X
has been placed in IN_QUEUE 806. In contrast to FIG. 9A, slot 902-1
is shown as being associated with packet X, and the tail of POQ 804
is at slot 902-2.
[0073] When packet Y arrives at service module 800, the PDT
performs similar operations for packet Y as the ones PDT has
performed for packet X. In brief, the PDT looks up a flow record
for packet Y, and finds flow record 802-1. The PDT locks flow
record 802-1, and prepares packet Y to be placed in IN_QUEUE
806.
[0074] To prepare packet Y, the PDT reserves a slot in POQ 804 by
assigning the tail of POQ 804 to slot 902-3 and writing the slot
number "1" (i.e., the arrival position of packet Y) in packet Y.
Furthermore, the PDT writes an identifier associated POQ 804 in
packet Y. The PDT unlocks flow record 802-1 and places packet Y in
IN_QUEUE 806.
[0075] FIG. 9C shows the configuration of POQ 804 after packet Y
has been placed in IN_QUEUE 806. Slot 902-2 is associated with
packet Y, and the tail of POQ 804 has shifted to slot 902-3.
[0076] Assume that flow thread 808-1 removes packet X and flow
thread 808-2 removes packet Y from IN_QUEUE 806. In addition,
assume that flow thread 808-2 finishes processing packet Y before
flow thread 808-1 finishes processing packet X. Consequently,
packet Y is placed in OUT_QUEUE 810 before packet X.
[0077] From OUT_QUEUE 810, a POT (not shown) removes packet Y. The
POT examines packet Y and determines that packet Y includes a
reference to POQ 804. Subsequently, the POT matches packet Y's
arrival position, which is "1" written by the PDT, to the index of
1 of slot 902-2. The POT writes a reference to packet Y in slot
902-2.
[0078] FIG. 9D shows the configuration of POQ 804 after the
preceding operation has been performed by the POT. The
configuration is similar to that shown in FIG. 9C, except that
packet Y is darkened to illustrate that packet Y's arrival position
has been matched to the index of slot 902-2.
[0079] The POT determines that slot 902-2, which is associated with
packet Y, is not the head of POQ 804 and proceeds to handle packet
X. The POT remove packet X from OUT_QUEUE 810 and examines packet
X. The POT determines that packet X includes a reference to POQ
804. Subsequently, the POT matches packet X's arrival position,
which is "0," to the index 0 of slot 902-1. The POT writes a
reference to packet X in slot 902-1.
[0080] FIG. 9E shows the configuration of POQ 804 after the
preceding operation has been performed by the POT. The
configuration is similar to that shown in FIG. 9D, except that
packet X is darkened to illustrate that packet X's arrival position
has been matched to the index of slot 902-1.
[0081] The POT determines that slot 902-1, which is associated with
packet X, is the head of POQ 804. The POT releases packet X from
POQ 804, by designating slot 902-2 as the new head of POQ 804. FIG.
9F shows the configuration of POQ 804 after packet X has been
released from POQ 804.
[0082] After packet X has been released, the POT realizes POQ 804
is not empty. The POT determines if the head slot (e.g., "1" in
FIG. 9F) contains a reference to packet Y. Upon finding that the
head slot contains the reference, the POT releases packet Y from
POT 804, by designating slot 902-3 as the new head of POQ 804. FIG.
9G shows the configuration of POQ after packet Y has been released
from POQ 804.
[0083] The above example demonstrates how a circular buffer may be
used to sequence processed packets in the same order that the
packets were received at PDT 402. After the packets are received,
multiple threads may process them in parallel. However, because
processing delays for different threads are variable, the threads
may finish processing the packets out-of-order.
[0084] To ensure processed packets are output in a proper sequence,
slots on the circular buffer may be reserved in the sequence that
the packets were received. After the packets are processed by the
threads, the packets may be placed in the reserved slots. The
referenced packets may be released from the slots only in the order
that the slots have been reserved, and, therefore, in the order the
packets were received at PDT 402.
CONCLUSION
[0085] The foregoing description of implementations provides
illustration, but is not intended to be exhaustive or to limit the
implementations to the precise form disclosed. Modifications and
variations are possible in light of the above teachings or may be
acquired from practice of the teachings.
[0086] For example, while series of blocks have been described with
regard to exemplary processes illustrated in FIGS. 6 and 7, the
order of the blocks may be modified in other implementations. In
addition, non-dependent blocks may represent acts that can be
performed in parallel to other blocks.
[0087] It will be apparent that aspects described herein may be
implemented in many different forms of software, firmware, and
hardware in the implementations illustrated in the figures. The
actual software code or specialized control hardware used to
implement aspects does not limit the invention. Thus, the operation
and behavior of the aspects were described without reference to the
specific software code--it being understood that software and
control hardware can be designed to implement the aspects based on
the description herein.
[0088] Further, certain portions of the implementations have been
described as "logic" that performs one or more functions. This
logic may include hardware, such as a processor, an application
specific integrated circuit, or a field programmable gate array,
software, or a combination of hardware and software.
[0089] Even though particular combinations of features are recited
in the claims and/or disclosed in the specification, these
combinations are not intended to limit the invention. In fact, many
of these features may be combined in ways not specifically recited
in the claims and/or disclosed in the specification.
[0090] No element, act, or instruction used in the present
application should be construed as critical or essential to the
implementations described herein unless explicitly described as
such. Also, as used herein, the article "a" is intended to include
one or more items. Where one item is intended, the term "one" or
similar language is used. Further, the phrase "based on" is
intended to mean "based, at least in part, on" unless explicitly
stated otherwise.
* * * * *