小樱 发表于 2021/7/31 21:34

Linux centos7系统执行yum跳出Exiting on user cancel解决办法,总是被Killed,内存不足杀死进程,无法启动锐速tcp加速,kvm虚拟化动态内存超售(HugePages)

Linux centos7系统执行yum跳出Exiting on user cancel解决办法,总是被Killed,内存不足杀死进程,无法启动锐速tcp加速,kvm虚拟化动态内存超售(HugePages)

yum无法执行是内存不足导致的,定位到
@test ~]# dmesg | grep 'Out of memory'
Out of memory: Kill process 1284 (yum) score 7 or sacrifice child
Out of memory: Kill process 834 (tuned) score 4 or sacrifice child
Out of memory: Kill process 1145 (kangle) score 3 or sacrifice child
Out of memory: Kill process 833 (mysqld) score 1 or sacrifice child
Out of memory: Kill process 491 (polkitd) score 1 or sacrifice child
Out of memory: Kill process 1133 (cdnbest) score 1 or sacrifice child
Out of memory: Kill process 561 (NetworkManager) score 0 or sacrifice child
Out of memory: Kill process 562 (gmain) score 0 or sacrifice child
Out of memory: Kill process 1216 (sshd) score 0 or sacrifice child
Out of memory: Kill process 1216 (sshd) score 0 or sacrifice child
Out of memory: Kill process 1396 (lotServer.sh) score 0 or sacrifice child
Out of memory: Kill process 1112 (rsyslogd) score 0 or sacrifice child
Out of memory: Kill process 1149 (in:imjournal) score 0 or sacrifice child
Out of memory: Kill process 1432 (systemd-udevd) score 0 or sacrifice child
Out of memory: Kill process 1431 (systemd-udevd) score 0 or sacrifice child
Out of memory: Kill process 1142 (kangle) score 0 or sacrifice child
Out of memory: Kill process 1118 (crond) score 0 or sacrifice child
Out of memory: Kill process 1425 (acce-3.11.36.2-) score 0 or sacrifice child
Out of memory: Kill process 1539 (yum) score 5 or sacrifice child
Out of memory: Kill process 1540 (yum) score 7 or sacrifice child
Out of memory: Kill process 1540 (yum) score 7 or sacrifice child
Out of memory: Kill process 1540 (yum) score 7 or sacrifice child
Out of memory: Kill process 1649 (yum) score 6 or sacrifice child
Out of memory: Kill process 1656 (yum) score 6 or sacrifice child
Out of memory: Kill process 1656 (yum) score 6 or sacrifice child
Out of memory: Kill process 1656 (yum) score 6 or sacrifice child
Out of memory: Kill process 1661 (yum) score 6 or sacrifice child
Out of memory: Kill process 1667 (yum) score 6 or sacrifice child
[ 1000.197305] Out of memory: Kill process 1667 (yum) score 12 or sacrifice child

之前系统重启后锐速启动不了也是这个原因。。。


分析了下是vps服务器使用的为kvm虚拟化,查了下原来是机房kvm虚拟化做了动态内存超售
服务器总是执行不了yum跳出Exiting on user cancel,开机就自动用尽了内存,在尝试增加swap


解决办法如下,执行命令,禁止kvm虚拟化执行动态内存,优先使用这个命令尝试解决,未观察到内存下降,,基本就没戏,无法解决该问题,,,直接去服务器商提供的后台面板重装系统,安装系统完成后直接使用,不要执行reboot重启就行
rmmod virtio_balloon


或者该贴试一下,类似的问题

Linux下简单通过dmesg抓包搜索被系统内核因为物理内存不足杀死进程的日志记录
https://bbs.itzmx.com/thread-89619-1-1.html

最后发现是内核开启了HugePages 2M这个东西,导致了大量的内存占用
cat /proc/meminfo | grep Huge

输出,单个页面2M,一共198个页面,HugePages共占用内存396M
# cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
HugePages_Total:   198
HugePages_Free:      190
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:       2048 kB

解决办法,打开文件删除相关参数“vm.nr_hugepages=1172”
vi /etc/sysctl.conf

一键删除
sed -i '/^vm\.nr_hugepages=/'d /etc/sysctl.conf

查看系统当前值
cat /proc/sys/vm/nr_hugepages

该值应该始终为0,不重启修改值命令,增大该值立即生效,降低为0关闭需要重启生效
sysctl -w vm.nr_hugepages=0


某Linux系统安装脚本判断hugepages开启有BUG,导致小内存机器也强行打开2.5G hugepages,导致服务器内核崩溃无法启用开机
某些软件错误判断内存引起错误打开hugepages引起内核崩溃无法开机的BUG修复
出错相关代码,请进行修正改为-gt判断
if [[ $(grep MemTotal /proc/meminfo | awk '{print $2}') > 3500000 ]]; then
    echo "[*] Enabling huge pages"
        echo "[*] 启用 huge pages"
    echo "vm.nr_hugepages=$((1168+$(nproc)))" | sudo tee -a /etc/sysctl.conf
    sudo sysctl -w vm.nr_hugepages=$((1168+$(nproc)))
fi

修复后代码
if [[ $(grep MemTotal /proc/meminfo | awk '{print $2}') -gt 3500000 ]]; then
    echo "[*] Enabling huge pages"
        echo "[*] 启用 huge pages"
    echo "vm.nr_hugepages=$((1168+$(nproc)))" | sudo tee -a /etc/sysctl.conf
    sudo sysctl -w vm.nr_hugepages=$((1168+$(nproc)))
fi

AnonHugePages是一个必须禁用的系统内核优化
https://bbs.itzmx.com/thread-110231-1-1.html

不可名 发表于 2021/8/28 04:10

{:3030:}{:3019:}
页: [1]
查看完整版本: Linux centos7系统执行yum跳出Exiting on user cancel解决办法,总是被Killed,内存不足杀死进程,无法启动锐速tcp加速,kvm虚拟化动态内存超售(HugePages)