Troubleshooting Network Issues in Oracle RAC 19c – Step-by-Step Guide
Oracle Real Application Clusters (RAC) 19c provides high availability and scalability, but network issues can impact node communication, cluster stability, and performance. Here’s a step-by-step approach to troubleshooting network problems in an Oracle RAC environment.
🛠️ Step 1: Verify the Network Configuration
Check if the Public, Private (Interconnect), and VIP addresses are correctly configured.
🔹 List all network interfaces on each node
ifconfig -a # Linux
ip a show # Alternative Linux command
SELECT inst_id, host_name, ip_address FROM gv$instance;
🔹 Check SCAN (Single Client Access Name) settings:
srvctl config scan
srvctl config scan_listener
🔹 Ensure VIP addresses fail over properly:
srvctl status nodeapps
🛠️ Step 2: Check Cluster Interconnect Status
The private network (interconnect) should be low latency and have no packet loss.
🔹 Verify private network settings:
oifcfg getif
crsctl stat res -t | grep -i interconnect
🔹 Test connectivity between nodes:
ping
traceroute
🔹 Check for packet loss:
ifstat -i eth1 # Check interconnect interface
netstat -i # Verify dropped packets
🛠️ Step 3: Validate SCAN and Listener Configuration
A misconfigured SCAN listener can cause connection failures.
🔹 Check SCAN listener status:
srvctl status scan_listener
🔹 Validate SCAN DNS resolution:
nslookup scan-name
🔹 Manually test listener connectivity:
tnsping scan_name
lsnrctl status LISTENER_SCAN1
🔹 Restart SCAN listener (if needed):
srvctl stop scan_listener
srvctl start scan_listener
🛠️ Step 4: Check CRS and GI Logs for Errors
🔹 View CRS logs for network-related failures:
cat /u01/app/grid/diag/crs/hostname/crs/trace/crsd.log | grep -i “network”
🔹 Check Grid Infrastructure logs:
cat /u01/app/grid/diag/tnslsnr/hostname/listener/alert/log.xml
🛠️ Step 5: Resolve Network Latency Issues
🔹 Check for Jumbo Frames Support (recommended for RAC interconnects):
ifconfig eth1 mtu 9000
🔹 Ensure CPU and Memory are not overloaded:
top
vmstat 5
🔹 Test TCP performance between nodes:
iperf -c -i 1 -t 10
💡 Key Takeaways
✔ Always validate SCAN, VIP, and interconnect settings first.
✔ Use ping, traceroute, netstat, and iperf to diagnose connectivity issues.
✔ Ensure correct DNS resolution for SCAN addresses.
✔ Monitor CRS logs for network-related errors.
✔ Optimize interconnect performance using Jumbo Frames and low-latency interfaces.