TSUBAME3.0計算サービスのWebページはこちら

TSUBAME2.5からTSUBAME3.0へのデータ移行方法の資料はこちら

【障害報告】2017.3.21発生:/work1, /gscr0 ストレージ障害 (1)

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
2017年3月21日

 下記の通りストレージに障害が発生し、復旧いたしました。

 時間帯:3月21(火)11:51から15:44頃

 影響範囲:/work1, /gscr0にアクセスできない

 詳細:

(3/29追記)

Journalの問題(デットロック)でMDSの処理がスタックする問題が発生しました。これは昨年の5月11日今年の2月3日3月17日などで発生したものと同種となります。

用語説明

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


 なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

<Sキュー:154ジョブ>
2120544.t2zpbs01, 2120546.t2zpbs01, 2120548.t2zpbs01, 2120551.t2zpbs01, 2120552.t2zpbs01
2120553.t2zpbs01, 2120554.t2zpbs01, 2120555.t2zpbs01, 2120556.t2zpbs01, 2120557.t2zpbs01
2120558.t2zpbs01, 2120559.t2zpbs01, 2120560.t2zpbs01, 2120561.t2zpbs01, 2120562.t2zpbs01
2120563.t2zpbs01, 2120564.t2zpbs01, 2120565.t2zpbs01, 2120566.t2zpbs01, 2120567.t2zpbs01
2120568.t2zpbs01, 2120569.t2zpbs01, 2120570.t2zpbs01, 2120571.t2zpbs01, 2120572.t2zpbs01
2120573.t2zpbs01, 2120575.t2zpbs01, 2120576.t2zpbs01, 2120577.t2zpbs01, 2120578.t2zpbs01
2130232.t2zpbs01, 2130238.t2zpbs01, 2130250.t2zpbs01, 2130254.t2zpbs01, 2130258.t2zpbs01
2130259.t2zpbs01, 2130268.t2zpbs01, 2130273.t2zpbs01, 2130275.t2zpbs01, 2130282.t2zpbs01
2130284.t2zpbs01, 2130290.t2zpbs01, 2130296.t2zpbs01, 2130315.t2zpbs01, 2130319.t2zpbs01
2130370.t2zpbs01, 2130474.t2zpbs01, 2130475.t2zpbs01, 2130476.t2zpbs01, 2130477.t2zpbs01
2130478.t2zpbs01, 2130479.t2zpbs01, 2130480.t2zpbs01, 2130481.t2zpbs01, 2130482.t2zpbs01
2130483.t2zpbs01, 2130484.t2zpbs01, 2130485.t2zpbs01, 2130486.t2zpbs01, 2130487.t2zpbs01
2130488.t2zpbs01, 2130489.t2zpbs01, 2130490.t2zpbs01, 2130491.t2zpbs01, 2130492.t2zpbs01
2130493.t2zpbs01, 2130494.t2zpbs01, 2130495.t2zpbs01, 2130497.t2zpbs01, 2130498.t2zpbs01
2130499.t2zpbs01, 2130500.t2zpbs01, 2130501.t2zpbs01, 2130502.t2zpbs01, 2130503.t2zpbs01
2130504.t2zpbs01, 2130804.t2zpbs01, 2130814.t2zpbs01, 2130815.t2zpbs01, 2130924.t2zpbs01
2130932.t2zpbs01, 2130933.t2zpbs01, 2130934.t2zpbs01, 2130935.t2zpbs01, 2130936.t2zpbs01
2130937.t2zpbs01, 2130938.t2zpbs01, 2130941.t2zpbs01, 2130942.t2zpbs01, 2130944.t2zpbs01
2130945.t2zpbs01, 2130946.t2zpbs01, 2130957.t2zpbs01, 2130961.t2zpbs01, 2130975.t2zpbs01
2130995.t2zpbs01, 2130997.t2zpbs01, 2130998.t2zpbs01, 2130999.t2zpbs01, 2131001.t2zpbs01
2131003.t2zpbs01, 2131004.t2zpbs01, 2131005.t2zpbs01, 2131006.t2zpbs01, 2131007.t2zpbs01
2131008.t2zpbs01, 2131009.t2zpbs01, 2131010.t2zpbs01, 2131012.t2zpbs01, 2131013.t2zpbs01
2131032.t2zpbs01, 2131033.t2zpbs01, 2131034.t2zpbs01, 2131035.t2zpbs01, 2131042.t2zpbs01
2131043.t2zpbs01, 2131045.t2zpbs01, 2131046.t2zpbs01, 2131047.t2zpbs01, 2131049.t2zpbs01
2131060.t2zpbs01, 2131091.t2zpbs01, 2131110.t2zpbs01, 2131138.t2zpbs01, 2131144.t2zpbs01
2131147.t2zpbs01, 2131149.t2zpbs01, 2131150.t2zpbs01, 2131160.t2zpbs01, 2131162.t2zpbs01
2131164.t2zpbs01, 2131165.t2zpbs01, 2131169.t2zpbs01, 2131173.t2zpbs01, 2131174.t2zpbs01
2131175.t2zpbs01, 2131176.t2zpbs01, 2131177.t2zpbs01, 2131178.t2zpbs01, 2131181.t2zpbs01
2131182.t2zpbs01, 2131184.t2zpbs01, 2131185.t2zpbs01, 2131186.t2zpbs01, 2131187.t2zpbs01
2131188.t2zpbs01, 2131189.t2zpbs01, 2131190.t2zpbs01, 2131191.t2zpbs01, 2131196.t2zpbs01
2131197.t2zpbs01, 2131198.t2zpbs01, 2131199.t2zpbs01, 2131200.t2zpbs01

<S96キュー:11ジョブ>
2130010.t2zpbs01, 2130021.t2zpbs01, 2130022.t2zpbs01, 2130023.t2zpbs01, 2130024.t2zpbs01
2130992.t2zpbs01, 2131041.t2zpbs01, 2131044.t2zpbs01, 2131059.t2zpbs01, 2131158.t2zpbs01
2131192.t2zpbs01

<Gキュー:110ジョブ>
2117869.t2zpbs03, 2117878.t2zpbs03, 2117879.t2zpbs03, 2118728.t2zpbs03, 2118747.t2zpbs03
2118748.t2zpbs03, 2118749.t2zpbs03, 2118751.t2zpbs03, 2118752.t2zpbs03, 2118753.t2zpbs03
2118754.t2zpbs03, 2118755.t2zpbs03, 2118756.t2zpbs03, 2118757.t2zpbs03, 2118760.t2zpbs03
2118761.t2zpbs03, 2118762.t2zpbs03, 2118765.t2zpbs03, 2118811.t2zpbs03, 2118812.t2zpbs03
2118813.t2zpbs03, 2118814.t2zpbs03, 2118815.t2zpbs03, 2118817.t2zpbs03, 2118827.t2zpbs03
2118835.t2zpbs03, 2118837.t2zpbs03, 2118838.t2zpbs03, 2118839.t2zpbs03, 2118840.t2zpbs03
2118841.t2zpbs03, 2118842.t2zpbs03, 2118843.t2zpbs03, 2118844.t2zpbs03, 2118845.t2zpbs03
2118846.t2zpbs03, 2118847.t2zpbs03, 2118848.t2zpbs03, 2118849.t2zpbs03, 2118850.t2zpbs03
2118851.t2zpbs03, 2118852.t2zpbs03, 2118858[0].t2zpbs03, 2118858[1].t2zpbs03, 2118858[2].t2zpbs03
2118858[3].t2zpbs03, 2118858[4].t2zpbs03, 2118859[0].t2zpbs03, 2118859[1].t2zpbs03, 2118859[2].t2zpbs03
2118859[3].t2zpbs03, 2118859[4].t2zpbs03, 2118860[0].t2zpbs03, 2118860[1].t2zpbs03, 2118860[2].t2zpbs03
2118860[3].t2zpbs03, 2118860[4].t2zpbs03, 2118861[0].t2zpbs03, 2118861[1].t2zpbs03, 2118861[2].t2zpbs03
2118861[3].t2zpbs03, 2118861[4].t2zpbs03, 2118862.t2zpbs03, 2118863.t2zpbs03, 2118865[0].t2zpbs03
2118865[1].t2zpbs03, 2118865[2].t2zpbs03, 2118865[3].t2zpbs03, 2118865[4].t2zpbs03, 2118866[0].t2zpbs03
2118866[1].t2zpbs03, 2118866[2].t2zpbs03, 2118866[3].t2zpbs03, 2118866[4].t2zpbs03, 2118867[0].t2zpbs03
2118867[2].t2zpbs03, 2118867[3].t2zpbs03, 2118869[0].t2zpbs03, 2118869[1].t2zpbs03, 2118869[2].t2zpbs03
2118869[3].t2zpbs03, 2118869[4].t2zpbs03, 2118870[0].t2zpbs03, 2118870[1].t2zpbs03, 2118870[2].t2zpbs03
2118870[3].t2zpbs03, 2118870[4].t2zpbs03, 2118871[0].t2zpbs03, 2118871[1].t2zpbs03, 2118871[2].t2zpbs03
2118871[3].t2zpbs03, 2118871[4].t2zpbs03, 2118874.t2zpbs03, 2118875[0].t2zpbs03, 2118875[1].t2zpbs03
2118878.t2zpbs03, 2118962.t2zpbs03, 2118963.t2zpbs03, 2118964.t2zpbs03, 2118965.t2zpbs03
2118966.t2zpbs03, 2118967.t2zpbs03, 2118968.t2zpbs03, 2118969.t2zpbs03, 2118970.t2zpbs03
2118971.t2zpbs03, 2118972.t2zpbs03, 2118973.t2zpbs03, 2118974.t2zpbs03, 2118975.t2zpbs03

<Vキュー:202ジョブ>
2466527.t2zpbs-vm1, 2469137.t2zpbs-vm1, 2469165.t2zpbs-vm1, 2469279.t2zpbs-vm1, 2469280.t2zpbs-vm1
2469281.t2zpbs-vm1, 2469282.t2zpbs-vm1, 2469285.t2zpbs-vm1, 2469304.t2zpbs-vm1, 2469306.t2zpbs-vm1
2469307.t2zpbs-vm1, 2469308.t2zpbs-vm1, 2469309.t2zpbs-vm1, 2469310.t2zpbs-vm1, 2469311.t2zpbs-vm1
2469312.t2zpbs-vm1, 2469313.t2zpbs-vm1, 2469314.t2zpbs-vm1, 2469315.t2zpbs-vm1, 2469316.t2zpbs-vm1
2469317.t2zpbs-vm1, 2469318.t2zpbs-vm1, 2469319.t2zpbs-vm1, 2469320.t2zpbs-vm1, 2469321.t2zpbs-vm1
2469322.t2zpbs-vm1, 2469323.t2zpbs-vm1, 2469324.t2zpbs-vm1, 2469325.t2zpbs-vm1, 2469326.t2zpbs-vm1
2469327.t2zpbs-vm1, 2469328.t2zpbs-vm1, 2469329.t2zpbs-vm1, 2469330.t2zpbs-vm1, 2469331.t2zpbs-vm1
2469332.t2zpbs-vm1, 2469333.t2zpbs-vm1, 2469334.t2zpbs-vm1, 2469335.t2zpbs-vm1, 2469336.t2zpbs-vm1
2469337.t2zpbs-vm1, 2469338.t2zpbs-vm1, 2469340.t2zpbs-vm1, 2469342.t2zpbs-vm1, 2469343.t2zpbs-vm1
2469344.t2zpbs-vm1, 2469345.t2zpbs-vm1, 2469346.t2zpbs-vm1, 2469347.t2zpbs-vm1, 2469348.t2zpbs-vm1
2469349.t2zpbs-vm1, 2469350.t2zpbs-vm1, 2469351.t2zpbs-vm1, 2469353.t2zpbs-vm1, 2469355.t2zpbs-vm1
2469356.t2zpbs-vm1, 2469595.t2zpbs-vm1, 2469599.t2zpbs-vm1, 2469601.t2zpbs-vm1, 2469609.t2zpbs-vm1
2469610.t2zpbs-vm1, 2469615.t2zpbs-vm1, 2469616.t2zpbs-vm1, 2469617.t2zpbs-vm1, 2469623.t2zpbs-vm1
2469625.t2zpbs-vm1, 2469743.t2zpbs-vm1, 2469744.t2zpbs-vm1, 2469745.t2zpbs-vm1, 2469746.t2zpbs-vm1
2469747.t2zpbs-vm1, 2469748.t2zpbs-vm1, 2469749.t2zpbs-vm1, 2469750.t2zpbs-vm1, 2469751.t2zpbs-vm1
2469752.t2zpbs-vm1, 2469754.t2zpbs-vm1, 2469756.t2zpbs-vm1, 2469757.t2zpbs-vm1, 2469758.t2zpbs-vm1
2469759.t2zpbs-vm1, 2469762.t2zpbs-vm1, 2469763.t2zpbs-vm1, 2469764.t2zpbs-vm1, 2469765.t2zpbs-vm1
2469766.t2zpbs-vm1, 2469767.t2zpbs-vm1, 2469768.t2zpbs-vm1, 2470450.t2zpbs-vm1, 2470451.t2zpbs-vm1
2470488.t2zpbs-vm1, 2470489.t2zpbs-vm1, 2470490.t2zpbs-vm1, 2470491.t2zpbs-vm1, 2470542.t2zpbs-vm1
2470543.t2zpbs-vm1, 2470544.t2zpbs-vm1, 2470545.t2zpbs-vm1, 2470546.t2zpbs-vm1, 2470547.t2zpbs-vm1
2470548.t2zpbs-vm1, 2470549.t2zpbs-vm1, 2470551.t2zpbs-vm1, 2470552.t2zpbs-vm1, 2470553.t2zpbs-vm1
2470554.t2zpbs-vm1, 2470555.t2zpbs-vm1, 2470556.t2zpbs-vm1, 2470557.t2zpbs-vm1, 2470559.t2zpbs-vm1
2470560.t2zpbs-vm1, 2470563.t2zpbs-vm1, 2470564.t2zpbs-vm1, 2470565.t2zpbs-vm1, 2470566.t2zpbs-vm1
2470568.t2zpbs-vm1, 2470569.t2zpbs-vm1, 2470570.t2zpbs-vm1, 2470571.t2zpbs-vm1, 2470572.t2zpbs-vm1
2470573.t2zpbs-vm1, 2470575.t2zpbs-vm1, 2470576.t2zpbs-vm1, 2470577.t2zpbs-vm1, 2470578.t2zpbs-vm1
2470579.t2zpbs-vm1, 2470580.t2zpbs-vm1, 2470581.t2zpbs-vm1, 2470583.t2zpbs-vm1, 2470783.t2zpbs-vm1
2470785.t2zpbs-vm1, 2470959.t2zpbs-vm1, 2470960.t2zpbs-vm1, 2470961.t2zpbs-vm1, 2470962.t2zpbs-vm1
2470963.t2zpbs-vm1, 2470964.t2zpbs-vm1, 2470965.t2zpbs-vm1, 2470966.t2zpbs-vm1, 2470967.t2zpbs-vm1
2470968.t2zpbs-vm1, 2470969.t2zpbs-vm1, 2470970.t2zpbs-vm1, 2470971.t2zpbs-vm1, 2470972.t2zpbs-vm1
2470973.t2zpbs-vm1, 2470974.t2zpbs-vm1, 2470975.t2zpbs-vm1, 2470976.t2zpbs-vm1, 2470977.t2zpbs-vm1
2470978.t2zpbs-vm1, 2470979.t2zpbs-vm1, 2470980.t2zpbs-vm1, 2470981.t2zpbs-vm1, 2470982.t2zpbs-vm1
2470983.t2zpbs-vm1, 2470984.t2zpbs-vm1, 2470985.t2zpbs-vm1, 2470986.t2zpbs-vm1, 2470987.t2zpbs-vm1
2470988.t2zpbs-vm1, 2470989.t2zpbs-vm1, 2470990.t2zpbs-vm1, 2470991.t2zpbs-vm1, 2470992.t2zpbs-vm1
2470993.t2zpbs-vm1, 2470994.t2zpbs-vm1, 2470995.t2zpbs-vm1, 2470996.t2zpbs-vm1, 2470997.t2zpbs-vm1
2470998.t2zpbs-vm1, 2470999.t2zpbs-vm1, 2471000.t2zpbs-vm1, 2471001.t2zpbs-vm1, 2471002.t2zpbs-vm1
2471003.t2zpbs-vm1, 2471004.t2zpbs-vm1, 2471005.t2zpbs-vm1, 2471006.t2zpbs-vm1, 2471081.t2zpbs-vm1
2471084.t2zpbs-vm1, 2471085.t2zpbs-vm1, 2471086.t2zpbs-vm1, 2471087.t2zpbs-vm1, 2471161.t2zpbs-vm1
2471210.t2zpbs-vm1, 2471214.t2zpbs-vm1, 2471215.t2zpbs-vm1, 2471216.t2zpbs-vm1, 2471217.t2zpbs-vm1
2471218.t2zpbs-vm1, 2471219.t2zpbs-vm1, 2471220.t2zpbs-vm1, 2471221.t2zpbs-vm1, 2471222.t2zpbs-vm1
2471223.t2zpbs-vm1, 2471237.t2zpbs-vm1, 2471271.t2zpbs-vm1, 2471272.t2zpbs-vm1, 2471273.t2zpbs-vm1
2471274.t2zpbs-vm1, 2471275.t2zpbs-vm1

<Uキュー:7ジョブ>
2471208.t2zpbs-vm1, 2471211.t2zpbs-vm1, 2471265.t2zpbs-vm1, 2471266.t2zpbs-vm1, 2471267.t2zpbs-vm1
2471268.t2zpbs-vm1, 2471270.t2zpbs-vm1

<L256キュー:1ジョブ>
2131159.t2zpbs01

<L512キュー:6ジョブ>
2129087.t2zpbs01, 2131171.t2zpbs01, 2131180.t2zpbs01, 2131183.t2zpbs01, 2131193.t2zpbs01
2131195.t2zpbs01

<Hキュー:147ジョブ>
1151675[2].t2zpbs05, 1151680[0].t2zpbs05, 1151680[1].t2zpbs05, 1151680[2].t2zpbs05, 1151680[3].t2zpbs05
1151680[4].t2zpbs05, 1151683[0].t2zpbs05, 1151683[1].t2zpbs05, 1151683[2].t2zpbs05, 1151683[3].t2zpbs05
1151683[4].t2zpbs05, 1151684[0].t2zpbs05, 1151684[1].t2zpbs05, 1151684[2].t2zpbs05, 1151684[3].t2zpbs05
1151684[4].t2zpbs05, 1151686[2].t2zpbs05, 1151695[2].t2zpbs05, 1151697[2].t2zpbs05, 1151697[3].t2zpbs05
1151697[4].t2zpbs05, 1151698[0].t2zpbs05, 1151698[1].t2zpbs05, 1151698[2].t2zpbs05, 1151698[3].t2zpbs05
1151698[4].t2zpbs05, 1151699[0].t2zpbs05, 1151699[1].t2zpbs05, 1151699[2].t2zpbs05, 1151699[3].t2zpbs05
1151699[4].t2zpbs05, 1151700[0].t2zpbs05, 1151700[1].t2zpbs05, 1151700[2].t2zpbs05, 1151700[3].t2zpbs05
1151700[4].t2zpbs05, 1151701[0].t2zpbs05, 1151701[1].t2zpbs05, 1151701[2].t2zpbs05, 1151701[3].t2zpbs05
1151701[4].t2zpbs05, 1151702[0].t2zpbs05, 1151702[2].t2zpbs05, 1151702[3].t2zpbs05, 1151702[4].t2zpbs05
1151703[0].t2zpbs05, 1151703[1].t2zpbs05, 1151703[2].t2zpbs05, 1151703[3].t2zpbs05, 1151703[4].t2zpbs05
1151704[0].t2zpbs05, 1151704[2].t2zpbs05, 1151704[3].t2zpbs05, 1151704[4].t2zpbs05, 1151705[0].t2zpbs05
1151705[1].t2zpbs05, 1151705[2].t2zpbs05, 1151705[3].t2zpbs05, 1151705[4].t2zpbs05, 1151706[0].t2zpbs05
1151706[1].t2zpbs05, 1151706[2].t2zpbs05, 1151706[3].t2zpbs05, 1151706[4].t2zpbs05, 1151707[0].t2zpbs05
1151707[1].t2zpbs05, 1151707[2].t2zpbs05, 1151707[3].t2zpbs05, 1151707[4].t2zpbs05, 1151708[0].t2zpbs05
1151708[1].t2zpbs05, 1151708[2].t2zpbs05, 1151708[3].t2zpbs05, 1151708[4].t2zpbs05, 1151709[0].t2zpbs05
1151709[1].t2zpbs05, 1151709[2].t2zpbs05, 1151709[3].t2zpbs05, 1151709[4].t2zpbs05, 1151724.t2zpbs05
1151725.t2zpbs05, 1151731[0].t2zpbs05, 1151731[1].t2zpbs05, 1151731[2].t2zpbs05, 1151731[3].t2zpbs05
1151731[4].t2zpbs05, 1151732[0].t2zpbs05, 1151732[1].t2zpbs05, 1151732[2].t2zpbs05, 1151732[3].t2zpbs05
1151732[4].t2zpbs05, 1151733[0].t2zpbs05, 1151733[1].t2zpbs05, 1151733[2].t2zpbs05, 1151733[3].t2zpbs05
1151733[4].t2zpbs05, 1151734[0].t2zpbs05, 1151734[1].t2zpbs05, 1151734[2].t2zpbs05, 1151734[3].t2zpbs05
1151734[4].t2zpbs05, 1151735[0].t2zpbs05, 1151735[1].t2zpbs05, 1151735[2].t2zpbs05, 1151735[3].t2zpbs05
1151735[4].t2zpbs05, 1151736[0].t2zpbs05, 1151736[1].t2zpbs05, 1151736[2].t2zpbs05, 1151736[3].t2zpbs05
1151736[4].t2zpbs05, 1151737[0].t2zpbs05, 1151737[1].t2zpbs05, 1151737[2].t2zpbs05, 1151737[3].t2zpbs05
1151737[4].t2zpbs05, 1151738[0].t2zpbs05, 1151738[1].t2zpbs05, 1151738[2].t2zpbs05, 1151738[3].t2zpbs05
1151738[4].t2zpbs05, 1151739[0].t2zpbs05, 1151739[1].t2zpbs05, 1151739[2].t2zpbs05, 1151739[3].t2zpbs05
1151739[4].t2zpbs05, 1151740[0].t2zpbs05, 1151740[1].t2zpbs05, 1151740[2].t2zpbs05, 1151740[3].t2zpbs05
1151740[4].t2zpbs05, 1151741[0].t2zpbs05, 1151741[1].t2zpbs05, 1151741[2].t2zpbs05, 1151741[3].t2zpbs05
1151741[4].t2zpbs05, 1151742[0].t2zpbs05, 1151742[1].t2zpbs05, 1151742[2].t2zpbs05, 1151742[3].t2zpbs05
1151742[4].t2zpbs05, 1151743[0].t2zpbs05, 1151743[1].t2zpbs05, 1151743[2].t2zpbs05, 1151743[3].t2zpbs05
1151743[4].t2zpbs05, 1151744[0].t2zpbs05

<Xキュー:4ジョブ>
1151888.t2zpbs05, 1151923.t2zpbs05, 1151924.t2zpbs05, 1151925.t2zpbs05