Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check
客户侧部署完服务后,因客户侧防火墙没做好,导致服务器被黑,重启。。。然后redis就有了下面的事故redis的pod问题如下WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.WARNING o
·
客户侧部署完服务后,因客户侧防火墙没做好,导致服务器被黑,重启。。。然后redis就有了下面的事故
redis的pod问题如下
WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
主要是4个问题,导致redis启动不了的主要是最后一行。“Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix ” ,翻译一下读取仅追加文件的错误文件格式:备份AOF文件,然后使用./redis-check-a of–fix,有一个AOF的备份文件,通过这个./redis-check-a of–fix还原,查了一下redis的备份文件为“appendonly.aof”.
问题1:
The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
解决方法:
方法1: 临时设置生效: sysctl -w net.core.somaxconn = 511
方法2: 永久生效: 修改/etc/sysctl.conf文件,增加一行
net.core.somaxconn= 511
然后执行命令
sysctl -p
问题2
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
解决方法:
解决方案
方法1: 临时设置生效: sysctl -w vm.overcommit_memory = 1
方法2: 永久生效: 修改/etc/sysctl.conf文件,增加一行
vm.overcommit_memory = 1
然后执行命令
sysctl -p
补充:
overcommit_memory参数说明:
设置内存分配策略(可选,根据服务器的实际情况进行设置)
/proc/sys/vm/overcommit_memory
可选值:0、1、2。
0, 表示内核将检查是否有足够的可用内存供应用进程使用;如果有足够的可用内存,内存申请允许;否则,内存申请失败,并把错误返回给应用进程。
1, 表示内核允许分配所有的物理内存,而不管当前的内存状态如何。
2, 表示内核允许分配超过所有物理内存和交换空间总和的内存
注意:redis在dump数据的时候,会fork出一个子进程,理论上child进程所占用的内存和parent是一样的,比如parent占用的内存为8G,这个时候也要同样分配8G的内存给child,如果内存无法负担,往往会造成redis服务器的down机或者IO负载过高,效率下降。所以这里比较优化的内存分配策略应该设置为 1(表示内核允许分配所有的物理内存,而不管当前的内存状态如何)
问题3:
WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
解决方法:
上面也提供了解决方案,'echo never > /sys/kernel/mm/transparent_hugepage/enabled' 这一行,直接执行就好了,但是这样的话,只是当前生效而已,如果电脑重启之后,又是需要重新设置的,所以把这个命令加入到启动过程中。
编辑/etc/rc.local,加入echo never > /sys/kernel/mm/transparent_hugepage/enabled。
问题4
Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
重点问题来了,docker里面的redis如何寻找到备份文件(appendonly.aof)呢?搜索?那肯定很慢,找了找,docker有提供容器挂载目录存放的位置,找到位置即可
docker ps -a |grep xxxredis #过滤其容器
docker inspect 容器id 这里打印出的详细的容器详细 找到mount部分类似下面
红框部分目录,然后直接复制,然后 cd /var/lib/docker/volumes/94dbaf4fad236d5dc94f2f825202eb279d0d3217a47cb050ec51eba1b5ff2d51/_data,进行打开,你会发现就在这里
好了,备份文件我们找到了,那么redis安装运行还有恢复工具(redis-check-aof)。同理在这里寻找,找到环境
红框部分的几个目录。一个一个找过去。功夫不顾有心人,找到了,在 /usr/local/bin目录 #实在找不到直接find / -name xxx
然后执行命令 ./redis-check-aof --fix /var/lib/docker/containers/4c38df7be60ac0517961903351ecda36566f7150b39b7d29cc1ccf2665d6eb7e/appendonly.aof 然后会出现"Continue? [y/N]:"让你是否继续,你输入Y就可以了。
xxx@iZwz9isa7dpx6izf3br71hZ:/usr/local/bin# ./redis-check-aof --fix /var/lib/docker/containers/4c38df7be60ac0517961903351ecda36566f7150b39b7d29cc1ccf2665d6eb7e/appendonly.aof
0x 366c32f: Expected prefix '*', got: '
AOF analyzed: size=57066334, ok_up_to=57066287, diff=47
This will shrink the AOF from 57066334 bytes, with 47 bytes, to 57066287 bytes
Continue? [y/N]: y
Successfully truncated AOF
然后重启pod或者容器即可发现已经起来了
kubectl exec -nxxx redis-xxx bash
cd /data
/opt/bitnami/redis-sentinel/bin/redis-check-aof --fix appendonly.aof
更多推荐
已为社区贡献6条内容
所有评论(0)