Introduction

There are some tuning benefits you can utilize for your Minio deployments to make things a bit smoother. Here I’ll walk you through how to configure certain “ethtool” settings and have them persist across reboots, specifically on RX and TX buffers and coalescing values.

The problem

RX and TX buffers should match on your NICs for Minio workloads, as you often have a great deal of RX and TX traffic with Minio – just the nature of the product. Balancing these settings can help avoid any issues.
Additionally, you may find setting these values appropriately can really help if you notice you are seeing RX or TX errors.

eno49: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 94:57:a5:65:66:88  txqueuelen 1000  (Ethernet)
        RX packets 140199322548  bytes 179294234824331 (163.0 TiB)
        RX errors 251  dropped 0  overruns 242  frame 9
        TX packets 223368389303  bytes 251870769249701 (229.0 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0x93800000-93ffffff

eno50: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 94:57:a5:65:66:88  txqueuelen 1000  (Ethernet)
        RX packets 211515121085  bytes 262249370338198 (238.5 TiB)
        RX errors 12150  dropped 0  overruns 10062  frame 2088
        TX packets 13630229628  bytes 15066199525661 (13.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0x94800000-94ffffff

Yikes. We shouldn’t be seeing any RX errors. This can manifested itself in some 503 errors in your API calls. No bueno!

Investigation

Check your RX and TX settings on your NICs. Note: If you have a bonded interface, you’ll want to check these settings on the physical NICs that make up your bond.

Addtionally, we’ll want to check the coalescing values on our NICs. This refers to the aggregation of network interrupts. Setting this appropriately can reduce the number of network interrupts processed by the CPU.

$ ethtool -g eno49
Ring parameters for eno49:
Pre-set maximums:
RX:             4078
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4078
Current hardware settings:
RX:             453
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4078

$ ethtool -l eno49
Channel parameters for eno49:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       30
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       8

Yikes again. As you can see above, the RX settings don’t match the hardware maximums, nor do the coalescing values match the hardware maximum. I need to update these! In your case, you will want to check the ‘ethtool -g and ethtool -l” commands for each NIC and set them appropriately for your environment. The values I have may NOT be the same as your values.

The Fix

Now we need to set these values appropriately. Because I have two NICs, I’m setting both. NOTE: In my case, my LACP flapped when I made these settings changes. You will want to take appropriate measures.

ethtool -G eno49 rx 4078 tx 4078
ethtool -G eno50 rx 4078 tx 4078
ethtool -L eno49 combined 30
ethtool -L eno50 combined 30

Check the values after you apply these settings! Validation is your friend.

$ ethtool -l eno49
Channel parameters for eno49:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       30
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       30

$ ethtool -g eno49
Ring parameters for eno49:
Pre-set maximums:
RX:             4078
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4078
Current hardware settings:
RX:             4078
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4078

Beautiful. To keep things concise, I didn’t add the validation on my other NIC. Don’t worry, I checked it too!

Persistence

Unfortunately, those settings do *NOT* persist on reboot. Hence, lets make this persist.

Create a settings file

vi /etc/sysconfig/network-scripts/ethtool-settings.sh

Add the ethtool settings you want to configure. Remember, this is a shell script, so you need to create it as such.

#!/bin/bash
ethtool -G eno49 rx 4078 tx 4078
ethtool -G eno50 rx 4078 tx 4078
ethtool -L eno49 combined 30
ethtool -L eno50 combined 30

Save the file.

If you’re using SELinux, you may also need to add SELinux context:

chcon -t bin_t /etc/sysconfig/network-scripts/ethtool-settings.sh

Now create a systemd service to run this at start

vi /etc/systemd/system/ethtool-settings.service
[Unit]
Description=Set ethtool settings on startup

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/etc/sysconfig/network-scripts/ethtool-settings.sh

[Install]
WantedBy=multi-user.target

Bob’s your uncle! You’re good to go.