Ceph用户邮件列表Vol45-Issue1



ceph Vol 45 Issue 1

1.unfound objects blocking cluster, need help!(原文)

Hi,

I have a production cluster on which 1 OSD on a failing disk was slowing the whole cluster down. I removed the OSD (osd.87) like usual in such case but this time it resulted in 17 unfound objects. I no longer have the files from osd.87. I was able to call “ceph pg PGID mark_unfound_lost delete” on 10 of those objects.

On the remaining objects 7 the command blocks. When I try to do “ceph pg PGID query” on this PG it also blocks. I suspect this is same reason why mark_unfound blocks.

Other client IO to PGs that have unfound objects are also blocked. When trying to query the OSDs which has the PG with unfound objects, “ceph tell” blocks.

I tried to mark the PG as complete using ceph-objectstore-tool but it did not help as the PG is in fact complete but for some reason blocks.

I tried recreating an empty osd.87 and importing the PG exported from other replica but it did not help.

Can someone help me please? This is really important.

这个问题是作者一个集群中(ceph 0.94.5)出现了一个磁盘损坏以后造成了一些对象的丢失,然后在做了一定的处理以后,集群状态已经正常了,但是还是新的请求会出现block的状态,这个情况下如何处理才能让集群正常,作者贴出了pg dump,ceph -s,ceph osd dump相关信息,当出现异常的时候,需要人协助的时候,应该提供这些信息方便其他人定位问题,最后这个问题作者自己给出了自己的解决办法,出现的时候影响是当时的流量只有正常情况下的10%了,影响还是很大的

复现问题过程

[root@lab8106 ceph]# rados -p rbd put testremove testremove
[root@lab8106 ceph]# ceph osd map rbd testremove
osdmap e85 pool 'rbd' (0) object 'testremove' -> pg 0.eaf226a7 (0.27) -> up ([1,0], p1) acting

写入文件,找到文件,然后去后台删除对象
然后停止掉其中一个OSD,这里选择停掉主OSD

systemctl stop ceph-osd@1
ceph osd out 1

查看状态pg被锁住状态active+degrade,不会迁移完整,并且会检测到了有数据unfound了

然后向这个对象发起get请求

[root@lab8106 ceph]# rados -p rbd get testremove testfile

前端rados请求会卡住,后端出现 requests are blocked

看下如何处理

ceph pg 0.27 mark_unfound_lost delete

邮件列表作者的环境,这个命令也无法执行,直接卡死,后来发现有个执行窗口,就是这个对象所在的PG的OSD在启动过程中还是可以接受命令的,就在这个执行窗口执行这个命令就可以解决了

执行了以后可以执行命令

[root@lab8106 ceph]# rados  -p rbd  get testremove  a
error getting rbd/testremove: (2) No such file or directory

这个时候查询集群的状态可以看到,集群已经正常的恢复了,不会因为一个对象的丢失造成集群的PG状态卡在待迁移状态

可以看到请求是失败的但是不会像之前一样卡死的状态,卡死是比失败更严重的一种状态

如果不想看到老的 slow request ,那么就重启这个卡住的PG所在的osd,如果本来就正常了,那么这个异常状态就会消失

这个是一个需要人工干预的状态,实际上模拟的就是对象丢失的场景,什么情况下会对象丢失,一般来说,底层磁盘的故障,写下去的对象当时记录着有,正好写入完成又准备写副本的时候,磁盘坏了,这个就有比较高的概率出现,所以出现了坏盘要尽早更换

本系列是只会对列表的当天的非re进行一个汇总,这样保持了一个问题的追踪都在一篇里面,所以这一天只有这一个问题

变更记录

Why Who When
创建 武汉-运维-磨渣 2016-11-04