Mellanox NIC's on Linux

calendar Apr 19, 2022 · 1 min read · bash networking mellanox  ·
Share on: linkedin copy

A while back I had to configure a bunch of servers with Mellanox Connect-X 4 NIC's to talk to an ethernet Switch(Dell S6000-ON to be exact) and the NIC's that my customer had fitted not only had really old firmware but they were also in Infiniband(I think) mode, not Ethernet. So i had to do some legwork..

First job was to update the firmware. 

 1apt update && apt install wget unzip gcc make dkms unzip  linux-headers-$(uname -r) -y
 2mkdir /tmp/mellanox && cd /tmp/mellanox
 3wget https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p84948234/v182403/fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tgz
 4gunzip fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tgz
 5tar -xf fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tar
 6mv fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754/* .
 7rm -rf fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754
 8cd /tmp/mellanox
 9wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
10tar -xvf mft-4.10.0-104-x86_64-deb.tgz
11cd /tmp/mellanox/mft-4.10.0-104-x86_64-deb/
12bash ./install.sh
13mst start
14mst status
15export OUT1=$(mst status | grep cr0 )
16TRIMMEDOUT=(${OUT1//-/ })
17CARDID=${TRIMMEDOUT[0]}
18echo $CARDID
19flint -d $CARDID dc /tmp/mellanox/orig_firmware.ini
20flint -y -d $CARDID -i  /tmp/mellanox/fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin -allow_psid_change burn

Set both ports on the NIC to Ethernet mode

 1mst start
 2
 3export OUT1=$(mst status | grep cr0 )
 4TRIMMEDOUT=(${OUT1//-/ })
 5CARDID=${TRIMMEDOUT[0]}
 6echo $CARDID
 7mlxconfig -d $CARDID query
 8
 9#Turn both ports from VPI/Auto to Ethernet only:
10mlxconfig -y -d $CARDID set LINK_TYPE_P1=2 LINK_TYPE_P2=2
11
12#turn off bootrom
13mlxconfig -y -d $CARDID set BOOT_OPTION_ROM_EN_P1=false
14mlxconfig -y -d $CARDID set BOOT_OPTION_ROM_EN_P2=false
15mlxconfig -y -d $CARDID set LEGACY_BOOT_PROTOCOL_P1=0
16mlxconfig -y -d $CARDID set LEGACY_BOOT_PROTOCOL_P2=0