【障害報告】2017.3.21発生:/work1, /gscr0 ストレージ障害 (2)

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
2017年3月21日

 下記の通りストレージに障害が発生し、復旧いたしました。

 時間帯:3月21(火)18:53から20:03頃

 影響範囲:/work1, /gscr0にアクセスできない

 詳細:

(3/29追記)

Journalの問題(デットロック)でMDSの処理がスタックする問題が発生しました。これは昨年の5月11日今年の2月3日3月17日などで発生したものと同種となります。

用語説明

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


 なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

<Sキュー:170ジョブ>
2120558.t2zpbs01, 2120559.t2zpbs01, 2120560.t2zpbs01, 2120561.t2zpbs01, 2120562.t2zpbs01
2120563.t2zpbs01, 2120564.t2zpbs01, 2120565.t2zpbs01, 2120566.t2zpbs01, 2120567.t2zpbs01
2120568.t2zpbs01, 2120569.t2zpbs01, 2120570.t2zpbs01, 2120571.t2zpbs01, 2120572.t2zpbs01
2120573.t2zpbs01, 2120575.t2zpbs01, 2120576.t2zpbs01, 2120577.t2zpbs01, 2120578.t2zpbs01
2120579.t2zpbs01, 2120580.t2zpbs01, 2120581.t2zpbs01, 2120582.t2zpbs01, 2120583.t2zpbs01
2120584.t2zpbs01, 2120585.t2zpbs01, 2120586.t2zpbs01, 2120587.t2zpbs01, 2120588.t2zpbs01
2130296.t2zpbs01, 2130315.t2zpbs01, 2130319.t2zpbs01, 2130370.t2zpbs01, 2130481.t2zpbs01
2130482.t2zpbs01, 2130483.t2zpbs01, 2130484.t2zpbs01, 2130485.t2zpbs01, 2130486.t2zpbs01
2130487.t2zpbs01, 2130488.t2zpbs01, 2130489.t2zpbs01, 2130490.t2zpbs01, 2130491.t2zpbs01
2130492.t2zpbs01, 2130493.t2zpbs01, 2130494.t2zpbs01, 2130495.t2zpbs01, 2130500.t2zpbs01
2130501.t2zpbs01, 2130502.t2zpbs01, 2130503.t2zpbs01, 2130504.t2zpbs01, 2130505.t2zpbs01
2130506.t2zpbs01, 2130507.t2zpbs01, 2130508.t2zpbs01, 2130509.t2zpbs01, 2130510.t2zpbs01
2130511.t2zpbs01, 2130512.t2zpbs01, 2130513.t2zpbs01, 2130514.t2zpbs01, 2130804.t2zpbs01
2130814.t2zpbs01, 2131003.t2zpbs01, 2131005.t2zpbs01, 2131034.t2zpbs01, 2131035.t2zpbs01
2131042.t2zpbs01, 2131043.t2zpbs01, 2131045.t2zpbs01, 2131046.t2zpbs01, 2131047.t2zpbs01
2131049.t2zpbs01, 2131092.t2zpbs01, 2131111.t2zpbs01, 2131162.t2zpbs01, 2131164.t2zpbs01
2131165.t2zpbs01, 2131169.t2zpbs01, 2131173.t2zpbs01, 2131197.t2zpbs01, 2131199.t2zpbs01
2131204.t2zpbs01, 2131206.t2zpbs01, 2131207.t2zpbs01, 2131209.t2zpbs01, 2131210.t2zpbs01
2131213.t2zpbs01, 2131214.t2zpbs01, 2131215.t2zpbs01, 2131217.t2zpbs01, 2131219.t2zpbs01
2131220.t2zpbs01, 2131222.t2zpbs01, 2131223.t2zpbs01, 2131224.t2zpbs01, 2131225.t2zpbs01
2131226.t2zpbs01, 2131228.t2zpbs01, 2131230.t2zpbs01, 2131231.t2zpbs01, 2131232.t2zpbs01
2131234.t2zpbs01, 2131235.t2zpbs01, 2131237.t2zpbs01, 2131238.t2zpbs01, 2131239.t2zpbs01
2131240.t2zpbs01, 2131241.t2zpbs01, 2131242.t2zpbs01, 2131244.t2zpbs01, 2131245.t2zpbs01
2131246.t2zpbs01, 2131307.t2zpbs01, 2131310.t2zpbs01, 2131312.t2zpbs01, 2131317.t2zpbs01
2131331.t2zpbs01, 2131344.t2zpbs01, 2131345.t2zpbs01, 2131365.t2zpbs01, 2131366.t2zpbs01
2131375.t2zpbs01, 2131376.t2zpbs01, 2131380.t2zpbs01, 2131383.t2zpbs01, 2131386.t2zpbs01
2131402.t2zpbs01, 2131419.t2zpbs01, 2131430.t2zpbs01, 2131444.t2zpbs01, 2131459.t2zpbs01
2131460.t2zpbs01, 2131461.t2zpbs01, 2131462.t2zpbs01, 2131463.t2zpbs01, 2131464.t2zpbs01
2131465.t2zpbs01, 2131466.t2zpbs01, 2131467.t2zpbs01, 2131468.t2zpbs01, 2131469.t2zpbs01
2131470.t2zpbs01, 2131481.t2zpbs01, 2131483.t2zpbs01, 2131489[10].t2zpbs01, 2131489[11].t2zpbs01
2131489[12].t2zpbs01, 2131489[13].t2zpbs01, 2131489[14].t2zpbs01, 2131489[15].t2zpbs01, 2131489[16].t2zpbs01
2131489[17].t2zpbs01, 2131489[18].t2zpbs01, 2131489[19].t2zpbs01, 2131489[1].t2zpbs01, 2131489[20].t2zpbs01
2131489[2].t2zpbs01, 2131489[3].t2zpbs01, 2131489[4].t2zpbs01, 2131489[5].t2zpbs01, 2131489[6].t2zpbs01
2131489[7].t2zpbs01, 2131489[8].t2zpbs01, 2131489[9].t2zpbs01, 2131494.t2zpbs01, 2131496.t2zpbs01

<S96キュー:20ジョブ>
2130010.t2zpbs01, 2130021.t2zpbs01, 2130023.t2zpbs01, 2130024.t2zpbs01, 2130992.t2zpbs01
2131044.t2zpbs01, 2131059.t2zpbs01, 2131158.t2zpbs01, 2131333.t2zpbs01, 2131393.t2zpbs01
2131490.t2zpbs01, 2131518.t2zpbs01, 2131519.t2zpbs01, 2131520.t2zpbs01, 2131521.t2zpbs01
2131522.t2zpbs01, 2131523.t2zpbs01, 2131524.t2zpbs01, 2131525.t2zpbs01, 2131526.t2zpbs01

<Gキュー:74ジョブ>
2117869.t2zpbs03, 2117879.t2zpbs03, 2118729.t2zpbs03, 2118730.t2zpbs03, 2118757.t2zpbs03
2118758.t2zpbs03, 2118760.t2zpbs03, 2118761.t2zpbs03, 2118762.t2zpbs03, 2118764.t2zpbs03
2118765.t2zpbs03, 2118835.t2zpbs03, 2118836.t2zpbs03, 2118837.t2zpbs03, 2118838.t2zpbs03
2118839.t2zpbs03, 2118840.t2zpbs03, 2118841.t2zpbs03, 2118842.t2zpbs03, 2118843.t2zpbs03
2118844.t2zpbs03, 2118845.t2zpbs03, 2118846.t2zpbs03, 2118847.t2zpbs03, 2118848.t2zpbs03
2118849.t2zpbs03, 2118850.t2zpbs03, 2118851.t2zpbs03, 2118852.t2zpbs03, 2118861[3].t2zpbs03
2118862.t2zpbs03, 2118863.t2zpbs03, 2118867[1].t2zpbs03, 2118867[2].t2zpbs03, 2118867[3].t2zpbs03
2118867[4].t2zpbs03, 2118869[0].t2zpbs03, 2118869[1].t2zpbs03, 2118869[2].t2zpbs03, 2118869[3].t2zpbs03
2118869[4].t2zpbs03, 2118870[0].t2zpbs03, 2118870[1].t2zpbs03, 2118870[2].t2zpbs03, 2118870[3].t2zpbs03
2118870[4].t2zpbs03, 2118871[0].t2zpbs03, 2118871[1].t2zpbs03, 2118871[2].t2zpbs03, 2118871[3].t2zpbs03
2118871[4].t2zpbs03, 2118875[2].t2zpbs03, 2118876[1].t2zpbs03, 2118876[2].t2zpbs03, 2118876[3].t2zpbs03
2118876[4].t2zpbs03, 2118877[0].t2zpbs03, 2118877[1].t2zpbs03, 2118877[2].t2zpbs03, 2118877[3].t2zpbs03
2118877[4].t2zpbs03, 2118878.t2zpbs03, 2118879[0].t2zpbs03, 2118879[1].t2zpbs03, 2118879[2].t2zpbs03
2118879[3].t2zpbs03, 2118879[4].t2zpbs03, 2118880[0].t2zpbs03, 2118880[1].t2zpbs03, 2118962.t2zpbs03
2118980.t2zpbs03, 2118984.t2zpbs03, 2118985.t2zpbs03, 2118987.t2zpbs03

<Vキュー:189ジョブ>
2466527.t2zpbs-vm1, 2469137.t2zpbs-vm1, 2469165.t2zpbs-vm1, 2469429.t2zpbs-vm1, 2469430.t2zpbs-vm1
2469433.t2zpbs-vm1, 2469434.t2zpbs-vm1, 2469452.t2zpbs-vm1, 2469455.t2zpbs-vm1, 2469459.t2zpbs-vm1
2469461.t2zpbs-vm1, 2469462.t2zpbs-vm1, 2469463.t2zpbs-vm1, 2469464.t2zpbs-vm1, 2469466.t2zpbs-vm1
2469469.t2zpbs-vm1, 2469470.t2zpbs-vm1, 2469475.t2zpbs-vm1, 2469476.t2zpbs-vm1, 2469481.t2zpbs-vm1
2469482.t2zpbs-vm1, 2469487.t2zpbs-vm1, 2469494.t2zpbs-vm1, 2469499.t2zpbs-vm1, 2469500.t2zpbs-vm1
2469508.t2zpbs-vm1, 2469509.t2zpbs-vm1, 2469515.t2zpbs-vm1, 2469518.t2zpbs-vm1, 2469520.t2zpbs-vm1
2469524.t2zpbs-vm1, 2469533.t2zpbs-vm1, 2469542.t2zpbs-vm1, 2469544.t2zpbs-vm1, 2469548.t2zpbs-vm1
2469550.t2zpbs-vm1, 2469552.t2zpbs-vm1, 2469553.t2zpbs-vm1, 2469554.t2zpbs-vm1, 2469555.t2zpbs-vm1
2469556.t2zpbs-vm1, 2469557.t2zpbs-vm1, 2469559.t2zpbs-vm1, 2469560.t2zpbs-vm1, 2469561.t2zpbs-vm1
2469562.t2zpbs-vm1, 2469563.t2zpbs-vm1, 2469565.t2zpbs-vm1, 2469566.t2zpbs-vm1, 2469567.t2zpbs-vm1
2469569.t2zpbs-vm1, 2469570.t2zpbs-vm1, 2469571.t2zpbs-vm1, 2469572.t2zpbs-vm1, 2469573.t2zpbs-vm1
2469574.t2zpbs-vm1, 2469595.t2zpbs-vm1, 2469599.t2zpbs-vm1, 2469601.t2zpbs-vm1, 2469609.t2zpbs-vm1
2469610.t2zpbs-vm1, 2469615.t2zpbs-vm1, 2469616.t2zpbs-vm1, 2469617.t2zpbs-vm1, 2469623.t2zpbs-vm1
2469625.t2zpbs-vm1, 2469746.t2zpbs-vm1, 2469747.t2zpbs-vm1, 2469749.t2zpbs-vm1, 2469750.t2zpbs-vm1
2469751.t2zpbs-vm1, 2469752.t2zpbs-vm1, 2469754.t2zpbs-vm1, 2469756.t2zpbs-vm1, 2469757.t2zpbs-vm1
2469758.t2zpbs-vm1, 2469759.t2zpbs-vm1, 2469762.t2zpbs-vm1, 2469763.t2zpbs-vm1, 2469764.t2zpbs-vm1
2469765.t2zpbs-vm1, 2469766.t2zpbs-vm1, 2469767.t2zpbs-vm1, 2470450.t2zpbs-vm1, 2470451.t2zpbs-vm1
2470959.t2zpbs-vm1, 2470960.t2zpbs-vm1, 2470961.t2zpbs-vm1, 2470962.t2zpbs-vm1, 2470963.t2zpbs-vm1
2470964.t2zpbs-vm1, 2470965.t2zpbs-vm1, 2470966.t2zpbs-vm1, 2470967.t2zpbs-vm1, 2470968.t2zpbs-vm1
2470969.t2zpbs-vm1, 2470970.t2zpbs-vm1, 2470971.t2zpbs-vm1, 2470972.t2zpbs-vm1, 2470973.t2zpbs-vm1
2470974.t2zpbs-vm1, 2470975.t2zpbs-vm1, 2470976.t2zpbs-vm1, 2470977.t2zpbs-vm1, 2470978.t2zpbs-vm1
2470979.t2zpbs-vm1, 2470980.t2zpbs-vm1, 2470981.t2zpbs-vm1, 2470982.t2zpbs-vm1, 2470983.t2zpbs-vm1
2470984.t2zpbs-vm1, 2470985.t2zpbs-vm1, 2470986.t2zpbs-vm1, 2470987.t2zpbs-vm1, 2470988.t2zpbs-vm1
2470989.t2zpbs-vm1, 2470990.t2zpbs-vm1, 2470992.t2zpbs-vm1, 2470993.t2zpbs-vm1, 2470994.t2zpbs-vm1
2470995.t2zpbs-vm1, 2470996.t2zpbs-vm1, 2470997.t2zpbs-vm1, 2470998.t2zpbs-vm1, 2470999.t2zpbs-vm1
2471000.t2zpbs-vm1, 2471001.t2zpbs-vm1, 2471002.t2zpbs-vm1, 2471003.t2zpbs-vm1, 2471004.t2zpbs-vm1
2471005.t2zpbs-vm1, 2471006.t2zpbs-vm1, 2471081.t2zpbs-vm1, 2471084.t2zpbs-vm1, 2471085.t2zpbs-vm1
2471086.t2zpbs-vm1, 2471087.t2zpbs-vm1, 2471107.t2zpbs-vm1, 2471108.t2zpbs-vm1, 2471213.t2zpbs-vm1
2471224.t2zpbs-vm1, 2471225.t2zpbs-vm1, 2471227.t2zpbs-vm1, 2471228.t2zpbs-vm1, 2471229.t2zpbs-vm1
2471230.t2zpbs-vm1, 2471232.t2zpbs-vm1, 2471233.t2zpbs-vm1, 2471234.t2zpbs-vm1, 2471235.t2zpbs-vm1
2471236.t2zpbs-vm1, 2471272.t2zpbs-vm1, 2471429.t2zpbs-vm1, 2471459.t2zpbs-vm1, 2471464.t2zpbs-vm1
2471512.t2zpbs-vm1, 2471513.t2zpbs-vm1, 2471514.t2zpbs-vm1, 2471515.t2zpbs-vm1, 2471516.t2zpbs-vm1
2471517.t2zpbs-vm1, 2471518.t2zpbs-vm1, 2471519.t2zpbs-vm1, 2471520.t2zpbs-vm1, 2471521.t2zpbs-vm1
2471522.t2zpbs-vm1, 2471523.t2zpbs-vm1, 2471524.t2zpbs-vm1, 2471525.t2zpbs-vm1, 2471526.t2zpbs-vm1
2471527.t2zpbs-vm1, 2471528.t2zpbs-vm1, 2471529.t2zpbs-vm1, 2471530.t2zpbs-vm1, 2471531.t2zpbs-vm1
2471532.t2zpbs-vm1, 2471533.t2zpbs-vm1, 2471534.t2zpbs-vm1, 2471535.t2zpbs-vm1, 2471536.t2zpbs-vm1
2471538.t2zpbs-vm1, 2471612.t2zpbs-vm1, 2471614.t2zpbs-vm1, 2471615.t2zpbs-vm1, 2471616.t2zpbs-vm1
2471617.t2zpbs-vm1, 2471618.t2zpbs-vm1, 2471672.t2zpbs-vm1, 2471744.t2zpbs-vm1

<Uキュー:ジョブ>
2471268.t2zpbs-vm1, 2471270.t2zpbs-vm1

<L256キュー:ジョブ>
2131159.t2zpbs01

<L512キュー:2ジョブ>
2129087.t2zpbs01, 2131327.t2zpbs01

<Hキュー:127ジョブ>
1151725.t2zpbs05, 1151732[0].t2zpbs05, 1151732[1].t2zpbs05, 1151732[2].t2zpbs05, 1151732[3].t2zpbs05
1151732[4].t2zpbs05, 1151733[0].t2zpbs05, 1151733[1].t2zpbs05, 1151733[2].t2zpbs05, 1151733[3].t2zpbs05
1151733[4].t2zpbs05, 1151734[0].t2zpbs05, 1151734[1].t2zpbs05, 1151734[2].t2zpbs05, 1151734[3].t2zpbs05
1151734[4].t2zpbs05, 1151735[0].t2zpbs05, 1151735[1].t2zpbs05, 1151735[2].t2zpbs05, 1151735[3].t2zpbs05
1151735[4].t2zpbs05, 1151737[2].t2zpbs05, 1151738[2].t2zpbs05, 1151739[2].t2zpbs05, 1151740[2].t2zpbs05
1151741[0].t2zpbs05, 1151741[1].t2zpbs05, 1151741[2].t2zpbs05, 1151741[3].t2zpbs05, 1151741[4].t2zpbs05
1151742[2].t2zpbs05, 1151743[0].t2zpbs05, 1151743[1].t2zpbs05, 1151743[2].t2zpbs05, 1151743[3].t2zpbs05
1151743[4].t2zpbs05, 1151744[1].t2zpbs05, 1151744[2].t2zpbs05, 1151744[3].t2zpbs05, 1151744[4].t2zpbs05
1151745[0].t2zpbs05, 1151745[1].t2zpbs05, 1151745[2].t2zpbs05, 1151745[3].t2zpbs05, 1151745[4].t2zpbs05
1151746[0].t2zpbs05, 1151746[1].t2zpbs05, 1151746[2].t2zpbs05, 1151746[3].t2zpbs05, 1151746[4].t2zpbs05
1151747[0].t2zpbs05, 1151747[1].t2zpbs05, 1151747[2].t2zpbs05, 1151747[3].t2zpbs05, 1151747[4].t2zpbs05
1151748[0].t2zpbs05, 1151748[1].t2zpbs05, 1151748[2].t2zpbs05, 1151748[3].t2zpbs05, 1151748[4].t2zpbs05
1151749[0].t2zpbs05, 1151749[1].t2zpbs05, 1151749[2].t2zpbs05, 1151749[3].t2zpbs05, 1151749[4].t2zpbs05
1151750[0].t2zpbs05, 1151750[1].t2zpbs05, 1151750[2].t2zpbs05, 1151750[3].t2zpbs05, 1151750[4].t2zpbs05
1151751[0].t2zpbs05, 1151751[1].t2zpbs05, 1151751[2].t2zpbs05, 1151751[3].t2zpbs05, 1151751[4].t2zpbs05
1151752[0].t2zpbs05, 1151752[1].t2zpbs05, 1151752[2].t2zpbs05, 1151752[3].t2zpbs05, 1151752[4].t2zpbs05
1151753[0].t2zpbs05, 1151753[1].t2zpbs05, 1151753[2].t2zpbs05, 1151753[3].t2zpbs05, 1151753[4].t2zpbs05
1151754[0].t2zpbs05, 1151754[1].t2zpbs05, 1151754[2].t2zpbs05, 1151754[3].t2zpbs05, 1151754[4].t2zpbs05
1151755[0].t2zpbs05, 1151755[1].t2zpbs05, 1151755[2].t2zpbs05, 1151755[3].t2zpbs05, 1151755[4].t2zpbs05
1151756[0].t2zpbs05, 1151756[1].t2zpbs05, 1151756[2].t2zpbs05, 1151756[3].t2zpbs05, 1151756[4].t2zpbs05
1151757[0].t2zpbs05, 1151757[1].t2zpbs05, 1151757[2].t2zpbs05, 1151757[3].t2zpbs05, 1151757[4].t2zpbs05
1151758[0].t2zpbs05, 1151758[1].t2zpbs05, 1151758[2].t2zpbs05, 1151758[3].t2zpbs05, 1151758[4].t2zpbs05
1151759[0].t2zpbs05, 1151759[1].t2zpbs05, 1151759[2].t2zpbs05, 1151759[3].t2zpbs05, 1151759[4].t2zpbs05
1151760[0].t2zpbs05, 1151760[1].t2zpbs05, 1151760[2].t2zpbs05, 1151760[3].t2zpbs05, 1151760[4].t2zpbs05
1151765[0].t2zpbs05, 1151765[1].t2zpbs05, 1151765[2].t2zpbs05, 1151765[3].t2zpbs05, 1151765[4].t2zpbs05
1151766[0].t2zpbs05, 1151766[1].t2zpbs05

<Xキュー:4ジョブ>
1151888.t2zpbs05, 1151923.t2zpbs05, 1151924.t2zpbs05, 1151925.t2zpbs05