DCD Design Note 135.0

 

 

D0 FCC Run-II Network

 

 

by

Phil DeMar

 

5/15/99

 

 

Abstract: This Design Note describes the proposed network implementation for the FCC component of the D0 Run-II network. For Run-II, D0 will require high performance, high bandwidth network infrastructure in the FCC. The network infrastructure will need to simultaneously support data flows for streaming of on-line data to mass storage, the movement of unreconstructed data into & out the reconstruction farm, and the movement of reconstructed data between mass storage & the central analysis system for off-line analysis. Data rates for each of these data streams is in excess of 100Mb/s. A single, very high performance LAN switch is proposed to support all FCC D0 Run-II computing systems. The switch, a Catalyst 6509, has an aggregate switch fabric of 32Gb/s. Gigabit ethernet ports would be provided for the major FCC D0 Run-II computing systems, as well as the connections to D0 on-line, D0 off-line, and the general facility LAN. The reconstruction farm nodes would be directly supported on the switch.

Enter the document body after this page break. Do not remove the page break. This text will not print.

I. FCC D0 Run-II Components:

There are four major components of D0 Run-II computing in the FCC, from a networking perspective, are:

  1. The Central Analysis Server (CAS). A large SMP system dedicated to producing small datasets for off-line analysis from reconstructed event data. The CAS is expected to be equipped with multiple gigabit ethernet interfaces to facilitate high data rate transfers with mass storage (the tape robot…). It will also likely support a lower speed interface for interactive access from the off-line network

  2. The Enstore Cluster. The front end systems for the D0 tape robot facility. Six "mover" nodes are envisioned on FCC2 to move data into & out of the tape robot. Each is expected to support a gigabit ethernet interface. There will likely be one or more Enstore operator mount systems on FCC1, also with gigabit ethernet interfaces.

  3. The Reconstruction Farm. A small number (initially 1 or 2) of I/O nodes that will handle data movement out of and back into the tape robot. The I/O nodes will in turn stage the data for processing by a large number (initially 50, scaling to 200+) of worker nodes. Worker nodes will have 100Mb/s ethernet interfaces; I/O nodes will have a single gigabit ethernet interface.

  4. The Reconstruction Input Pipeline (RIP). Not an FCC computing resource, but rather a pipeline of the experiment data collected by D0 on-line network, and streamed to the tape robot facility in the FCC.

There are other, less significant (from a network perspective…) components of FCC D0 Run-II computing, including the SAM cluster, several database servers, and DMPs. In addition, there are network connections to the D0 off-line network, and to the general facility network (off-site access). However, the requirements for FCC D0 Run-II network support are driven by the four major components, and the network requirements for these components can comfortably be included in the proposed solution for the four major components.

 

II. FCC D0 Run-II Switch Port & Aggregate Thruput Requirements:

 

There are two switch capacity issues of concern for the FCC D0 Run-II network, the number of switch ports needed, and the aggregate maximum network traffic the switch fabric must be able to handle at any particular moment. The number of switch ports is further divided into switch port types, namely gigabit ethernet or fast (10/100) ethernet. Finally, it is useful to identify which components are intended to support interactive users, as these components will be logically separated from the non-interactive components of the D0 Run-II network infrastructure.. The port requirements and estimated data rates imposed on the network by the major FCC D0 Run-II components are listed below:

1000B-SX 100B-X

Component ports ports Max. Data Rates

CAS 6 - 1200 Mb/s (150 MByte/s)

Enstore (FCC2) 6 - 1200 Mb/s (150 MByte/s)

Enstore (FCC1) 2 - <minimal>

RIP (from D0 on-line) 1 - 120 Mb/s (15 MByte/s)

Reconstruction Farm 2 ~200 160 Mb/s (20 MByte/s)

SAM cluster - 8 <minimal>

CAS (interactive) - 1 <minimal>

DMPs (interactive) 3 - <minimal>

DB servers (interactive) 1 1 <minimal>

D0 Off-line (interactive) 1 - 400 Mb/s

Site LAN (interactive) 1 - <minimal>

Total 23 ~200 ~3.1 Gb/s

A number of the components listed as needing gigabit ethernet switch ports but generating modest maximum data rates will likely initially be equipped with only fast ethernet interfaces. However, it is the number of gigabit ethernet switch ports required that is at issue, and these components (the DMPs and DB servers…) may be equipped with gigabit ethernet interfaces at some future date. The gigabit uplinks into the D0 off-line LAN, and the general site LAN (for off-site access) may also be significantly underutilized, but reflect a general philosophy to over-engineer links into the core network in order to provide very high quality response & performance.

 

III. Proposed FCC D0 Run-II Network:

 

The proposed implementation for the FCC D0 Run-II network is a single, high capacity, high performance switch/router to support all FCC D0 Run-II computing systems. Attaching all FCC D0 computing systems to a common switch provides the simplest overall network architecture, and eliminates the potential bottlenecks inherent with trunk links needed to interconnect different switches. While the different components of FCC D0 Run-II computing would share the same switch fabric, they would not necessarily be part of the same logical LAN. The switch would be configured with logically separate partitions to preserve some level of router isolation between the various FCC D0 Run-II systems & clusters. The routing between most of the different switch partitions would be handled within the switch, not via multiple connections to an external router.

The Cisco Catalyst 6509 is proposed as the switch platform. It has a 32Gb/s capacity switch fabric, an order of magnitude greater than D0’s estimated aggregate maximum data rate. The 6509 has 8 slots available for switch port modules. 48-port 10/100B-TX modules, and 8-port 1000B-SX modules are available today, and a 16-port 1000B-SX module will be available shortly. The 6509 would be located in FCC2 (location yet to be determined…).

The proposed network configuration for the FCC D0 Run-II network is shown in figure 1.

figure 1.

The Catalyst 6509 would be configured into four partitions, the Tape Robot partition (Enstore movers), the Reconstruction Farm partition (farm I/O & worker nodes), the CAS partition (CAS server), and a general interactive partition for the rest of the FCC D0 Run-II systems. Traffic between the Tape Robot, Farm, and CAS partitions would be routed internally within the switch/router via layer-3 switching. Traffic external to those three partitions would be routed externally via the "core" router in FCC2E. The interactive partition would (probably…) also be routed externally thru the "core" router in the FCC2E. The off-line network at D0 would have its own gigabit connection to the FCC2E core router.

IV. Procurement Specifications & Issues:

The Catalyst 6509 would be procured with a minimal configuration, equipped with just enough LAN ports to accommodate the D0 Run-II computing tests scheduled for July & August of this year. The reason for a minimal configuration is that several 6509 modules that are well suited for FCC D0 Run-II network support will not be available until the Fall of this year. These modules include a 16-port gigabit ethernet module, and a daughter-board router module for central supervisor. Waiting these modules to become available, instead of (for example…) procuring multiple 8-port gigabit ethernet modules preserves open slots on the 6509 chassis, allowing greater flexibility in meeting D0 networking needs as Run-II progresses.

The proposed initial procurement configuration is shown below. The cost of the Catalyst 6509 equipped as described below is $ 42.7k, with supervisor (main processor…) module and redundant power supplies.

Module Type Number Ports/module Total # of ports

10/100B-TX (RJ21) 1 48 48

1000B-SX 2 8 16

The Catalyst 6509 supervisor module does have two additional 1000B-SX ports, raising the total number of gigabit ethernet ports to 18 for the initial procurement.

Additional switch port modules would be procured as needed. For example, when the D0 reconstruction farms is expanded, additional 48-port 10/100 modules would be procured to satisfy port requirements for the additional farm nodes. The 16-port gigabit ethernet module would be procured when available, and one of the original 8-port modules redeployed in another 6509. The configuration for the switch in the spring of 2000, when Run-II starts, is expected to be four 48-port 10/100 modules, one 8-port gigabit ethernet module, and one 16-port gigabit ethernet module. Total ports available would be 192 fast ethernet ports and 26 gigabit ethernet ports (including the two ports on the supervisor module). Two unused slots would be available for additional ports, if needed.

V. Implementation Issues:

The new Catalyst 6509 (FCC2E) would be installed within 30 days of delivery, and made available for the various tests scheduled as part the preparations for Run-II computing support.