How to find which IB switch is running SM master

Run the following command on the IB switch sminfo

Kernel parameter to collect the VMcore to be added in /etc/sysctl.conf

kernel.hardlockup_panic=1 kernel.hung_task_panic=1 kernel.max_rcu_stall_to_panic=1 kernel.panic_on_rcu_stall=1 kernel.softlockup_panic=1 kernel.unknown_nmi_panic=1

How to connect to The node when System is hung, and unable to login

ssh -o ConnectTimeout=1 -o ConnectionAttempts=1 user@host "nice -n -20 bash" It gets you a high-priority shell when nothing else works.

How to drop the NVME drive from DB node.

date; dbmcli -e alter physicaldisk NVME_4_2 drop for replacement

How to Generate the NMI from the ILOM ( when system is hung)

1) log into iloms via ssh. 2) start /SP/console on the Node 3) set /HOST/generate_host_nmi=true on the ILOM 4) start /SP/console node, watch for it to start dumping core. Leave it at this point while it finishes dumping core and rebooting (this may take 30 minutes or more). NOTE: Do not wait on this step. Move on to step 5. 5) Verify that the node

ipmitool commands to list all serial numbers

ipmitool fru print | awk ' /^FRU Device Description/ {fru=$0} /[Ss]erial/ {print fru " -> " $0} ' /opt/oracle.SupportTools/CheckHWnFWProfile -action list -mode serial_numbers .

Command to find the large file in the /u01 mount point

nice find /u01 -mount -type f -size +10240k -exec ls -l {} \;|awk '{print $5, $7$6$8, $3, $9}'|awk '{printf("%9.1fMB %s %12s %s\n", $1/1024/1024, $2, $3, $4) }' | sort -nr | head -20