Thermal regulation and air flow are two important parts of any HPC installation. Adding Infiniband increases the amount of physical material at the back of the servers which can obstruct fans, reduce airflow and lead to overheating and damaged equipment. EDR cabling is significantly thicker than FDR which adds to the difficulty.
Many thanks to Moegemat Alexander from Workplace Services for sorting out the cabling in our racks. The work is not quite finished as we still need to add four more GPU servers and do some additional cable work but it’s looking a lot neater than it did a week ago. The power cables have two colours as we run each dual power supply from a different DB phase, each one being powered from one of two UPS. The number of unused plug points is due to the fact that we keep the amp rating of each PDU below 50% of its rated capacity so that in the event we lose a UPS or DB board the remaining PDU can power the entire stack.