Linux Kernel Parameter In EKS NODE (with Packer)

1. Requirements

• Need to edit some kernel parameters on EKS nodes
  • ex) mqtt server requires large number of network connections → Increase nf_conntrack_max’s value
    
• Instead of using user-data or a DaemonSet, the kernel parameters will be configured during the image build process with Packer.
  • As a result, the EKS nodes will already have the necessary kernel settings applied without requiring any additional configuration

---

2. Solution

Add custom conf file containing kernel parameter setting in /etc/sysctl.d directory
→ sysctl will apply settings from custom conf file

1) Packer file

provisioner "file" {
  source      = "99-customnode.conf"
  destination = "/tmp/99-customnode.conf"
}

provisioner "shell" {
  inline = [
    "sudo mv /tmp/99-customnode.conf /etc/sysctl.d/99-customnode.conf",
    "sudo chmod 644 /etc/sysctl.d/99-customnode.conf",
  ]
}

2) Result

• Add the above script to the existing Packer file to create a new AMI, then apply it to the EKS nodegroup
• Check kernel parameter value
  • ex ) net.core.somaxconn : 4096 → 32768

---

3. Trouble Shooting

1) Issue

Some kernel parameters were not set according to the 99-customnode.conf file.
ex) net.nf_conntrack_max =262144
( in 99-customnode.conf, net.nf_conntrack_max =1000000)

2) Root cause analysis

• I confirmed that kube-proxy configures some of the kernel parameters in k8s document

  (net.nf_conntrack_max → 32768 * 8core = 262144)

  
  kube-proxy document
  

• While the systemd-sysctl.service runs before the sysinit phase as an early boot service, kube-proxy is executed later because it runs under the kubelet, which belongs to the multi-user target

  kube-proxy overwrites some kernel parameters

  • systemd-sysctl.service
    
  • kubelet.service
    

3) Solution

Make Custom Service

• Run after kubelet.service
• Execute shell script
  • Check if kube-proxy is running with crictl
  • Execute command (sysctl —load 99-customnode.conf)

Service file

[Unit]
Description=Apply custom sysctl after kube-proxy starts
After=kubelet.service
Requires=kubelet.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/wait-and-apply-sysctl.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Shell Script

#!/bin/bash

for i in {1..60}; do
  FOUND=$(sudo crictl ps -a>/dev/null | grep kube-proxy | grep Running)
  if [[ ! -z "$FOUND" ]]; then
    break
  fi
  sleep 5
done

sysctl --load /etc/sysctl.d/99-customnode.conf

Packer Script

  provisioner "file" {
    source      = "99-customnode.conf"
    destination = "/tmp/99-customnode.conf"
  }
  provisioner "file" {
    source      = "wait-and-apply-sysctl.sh"
    destination = "/tmp/wait-and-apply-sysctl.sh"
  }

  provisioner "file" {
    source      = "custom-sysctl.service"
    destination = "/tmp/custom-sysctl.service"
  }

  provisioner "shell" {
    inline = [
      "sudo mv /tmp/99-customnode.conf /etc/sysctl.d/99-customnode.conf",
      "sudo chmod 644 /etc/sysctl.d/99-customnode.conf",
      "sudo mv /tmp/wait-and-apply-sysctl.sh /usr/local/bin/wait-and-apply-sysctl.sh",
      "sudo mv /tmp/custom-sysctl.service /etc/systemd/system/custom-sysctl.service",
      "sudo chmod +x /usr/local/bin/wait-and-apply-sysctl.sh",
      "sudo systemctl enable custom-sysctl.service"
    ]
  }

4) Result

• kernel parameters have been perfectly set based on 99-customnode.conf
  

---

4. Conclusion

Ensuring consistency and reliability across nodes is critical.

While DaemonSets or user-data scripts offer a relatively easy way to apply kernel parameter configurations, they inherently rely on runtime execution, which can introduce variability and risk of human error.

By leveraging Packer to embed these settings into a custom AMI during the build image process, we treat our infrastructure as predictable (embracing the principles of Infrastructure as Code)

This method not only minimizes operational overhead but also significantly reduces the chance of configuration drift