DCD Design Note 152.3

DZero Run-II Network Upgrade:

Physical Layout and Address Space

Darryl Wohlt

April 6, 2000

Historically the DZero Workgroup LAN has been a collection of 100Mbps FDDI backbones dedicated to specific areas of effort (Offline/Interactive computing, Online computing and data acquisition, software development, data managment), connected together by a large-scale FDDI switch (DEC Gigaswitch, s-s-hub-2). One port of that switch connected the entire DZero Workgroup to a "hub" router which provided access to the outside world. 24-bit IP subnets were allocated to each FDDI, with some FDDIs having multiple subnets.

Run-II will impose a higher demand on computing and networking resources than a 100Mbps multi-LAN can satisfy (Demar, Design Note 135). The new architecture proposed for DZero Run-II will employ three 32Gbps large-chassis ethernet switches and seven 6Gbps "satellite" switches connected over 1Gbps ethernet fiber links, and an expanded IP address space tuned for improved routing efficiency.

A Cisco 6509 9-slot ethernet switch will be installed in Feynman Computer Center 2^nd floor (FCC2W) for Run-II Computing, and two 6509s will be installed in DZero Assembly Building 2^nd floor computer room (DAB2CR) for Online and Offline networking. The Run-II Computing and Online switches will be logically partitioned into multiple broadcast domains assigned to specific purposes.

This Cisco 6509 will be located in FCC2W near the Reconstruction Farms, and will initially have 18 gigabit ethernet interfaces (two of which will be long haul LX types) and 144 10/100BaseTX interfaces. It will be partitioned into four VLANs for Interactive, Central Analysis computing, Enstore and Reconstruction Farms, and will provide layer-3 switching (IP routing) among them. The VLANs will respectively be numbered 227, 401, 164, and 229.

This switch will provide gigabit ethernet and 100bps ethernet connectivity to the Enstore systems , Farm I/O nodes and worker nodes, Central Analysis systems, RIP nodes, SAM cluster and ODS test systems, all of which are on the second floor. Connections to 100Mbps systems on the first floor of FCC will be made to a Cisco 2948G switch called s-d0-fcc1w-1, which will be linked to s-d0-fcc2w over a gigabit link. Gigabit connections to first floor computer systems will be made over intrabuilding fiber.

A Cisco 6509 containing 10 gigabit ethernet ports (including one LX) and 240 10/100 ports will be located in the "Online" network rack in the DZero Assembly Building 2^nd floor computer room (DAB2CR). It will be partitioned into six VLANs for Event Data, Level 3 Control, Level 2, Accelerator, Online Interactive, Offline Interactive, and RIP. The VLANs will be numbered 301, 303, 302, (TBD), 231, 227, and 164, respectively.

This switch will provide connections for the three online hosts (d0ola, d0olb, d0olc), the L3 Data systems, L3 Control systems, L2 Test systems, L2 Trigger systems, various servers, the Linux Farm, and the Control Room PC's. Fiber links will run from the 6509 to three 3Com switches in the Moving Counting House for L3 Control, L2 Trigger, and VME nodes. One gigabit port will reside in VLAN 227 for a connection to s-d0-dab2cr-offline.

This 6509 will contain a Multilayer Switch Feature Card (MSFC) that will provide IP routing among the Online Interactive, Level 2, and Level 3 Control VLANs. Routing with access control (TBD) will be done between the Online Interactive and Offline Interactive VLANs. There will be no routing to/from the RIP and Event Data VLANs.

Routing for the Accelerator VLAN is still under examination.

A Cisco 6509 containing 10 gigabit ethernet ports (including one LX) and 24 100BaseFX MTRJ ports will be located in the "Offline" network rack in the DZero Assembly Building 2^nd floor computer room (DAB2CR). It will contain only one VLAN, 227, for Offline Interactive.

This switch will provide 100BaseFX connections for twenty-one 24-port 10/100 hubs distributed in the office areas in DAB. Another 100BaseFX port will temporarily connect to a FastEthernet/FDDI switch, to provide connectivity for the legacy FDDI-based networks in the DZero campus.

Each of the three trailer clusters (TCE, TCN, TCW) will have two Catalyst 2948G switches, each with 48 10/100BaseTX ports and 2 gigabit ethernet slots. The 2948G pairs will initially be connected together at their gigabit ports and a single 100BaseFX uplink from each switch pair will be connected to s-d0-dab2cr-offline. As bandwidth requirements increase, each 2948G may receive its own gigabit uplink to the 6509.

A 1000BaseSX link will run between the central core router (an 8540, r-s-hub-fcc) and s-d0-fcc2w to provide connectivity between DZero and the rest of the site. The legacy FDDI router connection for the DZero Workgroup, currently running between r-s-hub-0 and a port on s-s-hub-2, will be removed and a new 100BaseFX uplink will be established between s-s-hub-2 and s-d0-fcc2w.

A gigabit ethernet link (1000BaseLX - long haul) will be connected between the Central Computing VLAN on the Run-II Computing switch (s-d0-fcc2w) and the RIP VLAN on the Online switch (s-d0-dab2cr-online). This will be for RIP data transfers only; no interactive traffic will be carried.

Another 1000BaseLX gigabit ethernet link will join the Offline Interactive VLANs on s-d0-fcc2w and s-d0-dab2cr-offline to provide DZero users connectivity to Run-II networking and the rest of the site.

Access to the Online systems at DAB will be restricted through the use of access control lists in the routing component of s-d0-dab2cr-online. A 1000BaseSX ethernet link will connect the Offline Interactive VLANs on the Online switch (s-d0-dab2cr-online) and the Offline switch (s-d0-dab2cr-offline). This will be the only path by which interactive users can access the Online systems.

Figures 1 and 2 show the network layout with, and without, the 8540.

DZero currently uses these IP subnets (mask length in parentheses):

111 - Offline, FDDI nodes (/24)

113 - Online, FDDI nodes (/24)

221 - Offline, DAB (/24)

222 - Offline, D0FS in FCC (/24)

224 - Offline, trailers (/24)

226 - Offline, DAB (/24)

The proposed address allocation for DZero is:

164 - RIP/Enstore (/24)

165 .0 - Level3 (/25)

165.128 - reserved (/25)

222 - reserved (/24)

223.0 - reserved (/25)

223.128 - SAM (/25)

223.252, 223.248 … 223.224 - Central Analysis (/30)

224-227 - Offline Interactive, all locations (/22)

228-229 - Run-II farms (/23)

230.0 - Event Data (/25)

230.128 - Level2 (/25)

231 - Online Interactive (/24)

Subnet 164 will be assigned to Enstore systems and to the RIP interfaces of the D0Olx machines at DAB.

Subnet 165 will be split into two 25-bit subnets, and the Level 3 systems will occupy the lower subnet, using a gateway at 131.225.165.126. The upper subnet will remain reserved.

Subnet 223 will be subdivided assymetrically. The lower half (223.0/25) will be reserved. A 26-bit quarter at 223.128 (addresses 223.129-190) will be occupied by SAM, with a gateway at 131.225.223.190. The upper 26-bit quarter at 223.192 (addresses 223.193-254) will be set aside for up to xx 30-bit subnets for Central Analysis system interfaces. These will be assigned in descending order from the top (223.252). Eight 30-bit subnets will be initially assigned to CAS at 223.252, 223.248 … 223.224, using VLANs 401 through 408. In each 4-address subnet, the first and fourth addresses are the network address and broadcast address, respectively. The second address is the node address, and the third is the gateway. It is intended, if the CAS system design no longer requires separate subnets on each interface, that the single address for the system is selected with future expansion in mind, perhaps the topmost address in the subnet group.

Subnets 224-227 will be used for all Offline Interactive systems. This will be a supernet with a netmask of 255.255.252.0 and a default gateway of 131.225.227.200. Many existing offline systems (nearly 270) are already in this range, but their mask and gateway settings should be revised. The router will contain secondary addresses at 224.200, 225.200, and 226.200 until all systems comply with the 22-bit mask and gateway of 227.200.

Subnets 228-229 will be assigned to Farm worker nodes and the I/O server node d0bbin. The netmask will be 255.255.254.0 and the gateway will be 131.225.229.200. There are no legacy systems within this address range.

Subnet 230 will be divided in half. The lower half (230.0/25) will be used for Event Data with a gateway at 131.225.230.126. The upper half is for Level 2 systems, with a gateway at 131.225.230.254.

Subnet 231 will be reserved for the Online Interactive systems and the Interactive interfaces of the D0Olx machines. An access control list will be implemented to ensure that only subnets 224-227 (and perhaps some selected others) can reach 231.

As new requests arrive for IP addresses for DZero systems, the new IP address spaces should be used. As nodes in the legacy subnets are decommissioned or reassigned, the old addresses should be retired. While still occupied, subnets 111, 113, 221, and 222 should be treated as Interactive subnets, and retained in the router configuration until they are emptied. Once emptied, these subnets should be dismissed to the subnet pool. DZero system managers should be encouraged to readdress nodes into the new structure at every opportunity.