This post is my snippet of Brad Hedlund article about ToR and EoR design which is accessible via the following link:
http://bradhedlund.com/2009/04/05/top-of-rack-vs-end-of-row-data-center-designs/
Top of Rack Design
ToR is sometimes called In-Rack design.
Benefits:
- All copper cabling for servers stays within the rack as relatively
- there is no need to for a large copper cabling infrastructure
- Any network upgrades or issues with the rack switches will generally only affect the servers within that rack
- any future support of 40 or 100 Gigabit on twisted pair will likely have very short distance limitations (in-rack distances). This too is another key factor why Top of Rack would be selected over End of Row.
Moving the ToR concept inside the blade enclosure itself:
One significant drawback of the Top of Rack design is the increased management domain with each rack switch being a unique control plane instance that must be managed.
For example, in a data center with 40 racks, where each rack contained (2) “Top of Rack” switches, the result would be 80 switches on the floor just providing server access connections (not counting distribution and core switches). That is 80 copies of switch software that need to be updated, 80 configuration files that need to be created and archived, 80 different switches participating in the Layer 2 spanning tree topology, 80 different places a configuration can go wrong.
The top of rack design typically also requires higher port densities in the Aggregation switches.
Summary of Top of Rack advantages (Pro’s):
- Copper stays “In Rack”. No large copper cabling infrastructure required.
- Lower cabling costs. Less infrastructure dedicated to cabling and patching. Cleaner cable management.
- Modular and flexible “per rack” architecture. Easy “per rack” upgrades/changes.
- Future proofed fiber infrastructure, sustaining transitions to 40G and 100G.
- Short copper cabling to servers allows for low power, low cost 1oGE (10GBASE-CX1), 40G in the future.
- Ready for Unified Fabric today.
Summary of Top of Rack disadvantages (Con’s):
- More switches to manage. More ports required in the aggregation.
- Potential scalability concerns (STP Logical ports, aggregation switch density).
- More Layer 2 server-to-server traffic in the aggregation.
- Racks connected at Layer 2. More STP instances to manage.
- Unique control plane per 48-ports (per switch), higher skill set needed for switch replacement.
End of Row Design
Another variation of this design can be referred to as “Middle of Row”
Summary of End of Row advantages (Pro’s):
- Fewer switches to manage. Potentially lower switch costs, lower maintenance costs.
- Fewer ports required in the aggregation.
- Racks connected at Layer 1. Fewer STP instances to manage (per row, rather than per rack).
- Longer life, high availability, modular platform for server access.
- Unique control plane per hundreds of ports (per modular switch), lower skill set required to replace a 48-port line card, versus replacing a 48-port switch.
Summary of End of Row disadvantages (Con’s):
- Requires an expensive, bulky, rigid, copper cabling infrastructure. Fraught with cable management challenges.
- More infrastructure required for patching and cable management.
- Long twisted pair copper cabling limits the adoption of lower power higher speed server I/O.
- More future challenged than future proof.
- Less flexible “per row” architecture. Platform upgrades/changes affect entire row.
- Unified Fabric not a reality until late 2010.
ToR Fabric Extender
