このページの更新は終了しました。

最新の情報はTSUBAME3.0計算サービスのWebページをご覧ください。

TSUBAME2.5からTSUBAME3.0へのデータ移行方法の資料はこちら

【障害報告】2017.5.15発生:/work1 ストレージ障害

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
2017年5月15日

 下記の通りストレージに障害が発生し、復旧いたしました。

 時間帯: 5月14(日)21:52 から 5月15(月)から10:30頃

 影響:一部の計算サーバから/work1にアクセスできない。また復旧中に一時的に/gscr0にアクセスできない。

 詳細:

/work1を構成する2台で冗長化されたMDSの1台(t2s007013)が不安定となり、一部の計算ノードとreconnecting/refused reconnectionのエラーが発生しており、work1にI/Oできなかったジョブがあると考えられます。フェイルオーバ時に資源を共有する/gscr0にもアクセス不能(09:59から10:13)となりましたが10:30に復旧しました。

 

用語説明

OST:Lustreファイルシステムにおいて、ファイルの中身を実際に保存しているディスクの集合体

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

<Sキュー:54ジョブ>
2175640.t2zpbs01, 2175642.t2zpbs01, 2176503.t2zpbs01, 2176526.t2zpbs01, 2176527.t2zpbs01
2176528.t2zpbs01, 2176529.t2zpbs01, 2176530.t2zpbs01, 2176531.t2zpbs01, 2176532.t2zpbs01
2176533.t2zpbs01, 2176534.t2zpbs01, 2176535.t2zpbs01, 2176536.t2zpbs01, 2176537.t2zpbs01
2176538.t2zpbs01, 2176539.t2zpbs01, 2176540.t2zpbs01, 2176541.t2zpbs01, 2176542.t2zpbs01
2176543.t2zpbs01, 2176556.t2zpbs01, 2176557.t2zpbs01, 2176558.t2zpbs01, 2176559.t2zpbs01
2176560.t2zpbs01, 2176561.t2zpbs01, 2176562.t2zpbs01, 2176563.t2zpbs01, 2176564.t2zpbs01
2176600.t2zpbs01, 2176601.t2zpbs01, 2176602.t2zpbs01, 2176603.t2zpbs01, 2176604.t2zpbs01
2176605.t2zpbs01, 2176606.t2zpbs01, 2176607.t2zpbs01, 2176608.t2zpbs01, 2176609.t2zpbs01
2176610.t2zpbs01, 2176611.t2zpbs01, 2176613.t2zpbs01, 2176614.t2zpbs01, 2176617.t2zpbs01
2176618.t2zpbs01, 2176620.t2zpbs01, 2176623.t2zpbs01, 2176630.t2zpbs01, 2176631.t2zpbs01
2176634.t2zpbs01, 2176635.t2zpbs01, 2176637.t2zpbs01, 2176638.t2zpbs01

<S96キュー:1ジョブ>
2176589.t2zpbs01

<Gキュー:153ジョブ>
2125261.t2zpbs03, 2125262.t2zpbs03, 2125339.t2zpbs03, 2125378[2].t2zpbs03, 2125378[3].t2zpbs03
2125378[4].t2zpbs03, 2125379[2].t2zpbs03, 2125380[0].t2zpbs03, 2125380[1].t2zpbs03, 2125380[2].t2zpbs03
2125380[3].t2zpbs03, 2125380[4].t2zpbs03, 2125381.t2zpbs03, 2125382.t2zpbs03, 2125383.t2zpbs03
2125384.t2zpbs03, 2125385.t2zpbs03, 2125386.t2zpbs03, 2125388.t2zpbs03, 2125389.t2zpbs03
2125390[0].t2zpbs03, 2125390[1].t2zpbs03, 2125390[2].t2zpbs03, 2125390[3].t2zpbs03, 2125390[4].t2zpbs03
2125391[0].t2zpbs03, 2125391[1].t2zpbs03, 2125391[2].t2zpbs03, 2125391[3].t2zpbs03, 2125391[4].t2zpbs03
2125392[0].t2zpbs03, 2125392[1].t2zpbs03, 2125392[2].t2zpbs03, 2125392[3].t2zpbs03, 2125406.t2zpbs03
2125407.t2zpbs03, 2125408.t2zpbs03, 2125409.t2zpbs03, 2125410.t2zpbs03, 2125411.t2zpbs03
2125412.t2zpbs03, 2125413.t2zpbs03, 2125414.t2zpbs03, 2125415.t2zpbs03, 2125416.t2zpbs03
2125417.t2zpbs03, 2125418.t2zpbs03, 2125419.t2zpbs03, 2125420.t2zpbs03, 2125421.t2zpbs03
2125422.t2zpbs03, 2125423.t2zpbs03, 2125424.t2zpbs03, 2125425.t2zpbs03, 2125426.t2zpbs03
2125427.t2zpbs03, 2125428.t2zpbs03, 2125429.t2zpbs03, 2125430.t2zpbs03, 2125431.t2zpbs03
2125432.t2zpbs03, 2125433.t2zpbs03, 2125434.t2zpbs03, 2125435.t2zpbs03, 2125436.t2zpbs03
2125468.t2zpbs03, 2125469.t2zpbs03, 2125472.t2zpbs03, 2125473.t2zpbs03, 2125474[0].t2zpbs03
2125474[1].t2zpbs03, 2125474[2].t2zpbs03, 2125474[3].t2zpbs03, 2125474[4].t2zpbs03, 2125475[0].t2zpbs03
2125475[2].t2zpbs03, 2125475[3].t2zpbs03, 2125475[4].t2zpbs03, 2125476[0].t2zpbs03, 2125476[1].t2zpbs03
2125476[2].t2zpbs03, 2125476[4].t2zpbs03, 2125484.t2zpbs03, 2125485.t2zpbs03, 2125486.t2zpbs03
2125487.t2zpbs03, 2125488.t2zpbs03, 2125489.t2zpbs03, 2125490.t2zpbs03, 2125491.t2zpbs03
2125492.t2zpbs03, 2125493.t2zpbs03, 2125495.t2zpbs03, 2125496.t2zpbs03, 2125497.t2zpbs03
2125498.t2zpbs03, 2125499.t2zpbs03, 2125500.t2zpbs03, 2125502.t2zpbs03, 2125503.t2zpbs03
2125504.t2zpbs03, 2125505.t2zpbs03, 2125506.t2zpbs03, 2125507.t2zpbs03, 2125508.t2zpbs03
2125509.t2zpbs03, 2125510.t2zpbs03, 2125511.t2zpbs03, 2125512.t2zpbs03, 2125513.t2zpbs03
2125514.t2zpbs03, 2125516[0].t2zpbs03, 2125516[1].t2zpbs03, 2125516[2].t2zpbs03, 2125516[3].t2zpbs03
2125516[4].t2zpbs03, 2125517[3].t2zpbs03, 2125517[4].t2zpbs03, 2125518[0].t2zpbs03, 2125518[1].t2zpbs03
2125518[3].t2zpbs03, 2125518[4].t2zpbs03, 2125519.t2zpbs03, 2125520.t2zpbs03, 2125521.t2zpbs03
2125522.t2zpbs03, 2125523.t2zpbs03, 2125524.t2zpbs03, 2125525.t2zpbs03, 2125526.t2zpbs03
2125527.t2zpbs03, 2125528.t2zpbs03, 2125529.t2zpbs03, 2125530.t2zpbs03, 2125531.t2zpbs03
2125532.t2zpbs03, 2125533.t2zpbs03, 2125534.t2zpbs03, 2125535.t2zpbs03, 2125536.t2zpbs03
2125537.t2zpbs03, 2125538.t2zpbs03, 2125539.t2zpbs03, 2125540.t2zpbs03, 2125541.t2zpbs03
2125542.t2zpbs03, 2125543.t2zpbs03, 2125544.t2zpbs03, 2125545.t2zpbs03, 2125546.t2zpbs03
2125547.t2zpbs03, 2125548.t2zpbs03, 2125549.t2zpbs03

<Vキュー:354ジョブ>
2494693.t2zpbs-vm1, 2495201.t2zpbs-vm1, 2495399.t2zpbs-vm1, 2495466.t2zpbs-vm1, 2495474.t2zpbs-vm1
2495476.t2zpbs-vm1, 2495480.t2zpbs-vm1, 2495483.t2zpbs-vm1, 2495486.t2zpbs-vm1, 2495488.t2zpbs-vm1
2495499.t2zpbs-vm1, 2495504.t2zpbs-vm1, 2495507.t2zpbs-vm1, 2495510.t2zpbs-vm1, 2495513.t2zpbs-vm1
2495571.t2zpbs-vm1, 2495572.t2zpbs-vm1, 2495573.t2zpbs-vm1, 2495574.t2zpbs-vm1, 2495575.t2zpbs-vm1
2495576.t2zpbs-vm1, 2495577.t2zpbs-vm1, 2495578.t2zpbs-vm1, 2495579.t2zpbs-vm1, 2495580.t2zpbs-vm1
2495582.t2zpbs-vm1, 2495583.t2zpbs-vm1, 2495584.t2zpbs-vm1, 2495585.t2zpbs-vm1, 2495586.t2zpbs-vm1
2495587.t2zpbs-vm1, 2495588.t2zpbs-vm1, 2495590.t2zpbs-vm1, 2495591.t2zpbs-vm1, 2495592.t2zpbs-vm1
2495593.t2zpbs-vm1, 2495594.t2zpbs-vm1, 2495636.t2zpbs-vm1, 2495637.t2zpbs-vm1, 2495638.t2zpbs-vm1
2495639.t2zpbs-vm1, 2495640.t2zpbs-vm1, 2495641.t2zpbs-vm1, 2495642.t2zpbs-vm1, 2495643.t2zpbs-vm1
2495644.t2zpbs-vm1, 2495645.t2zpbs-vm1, 2495646.t2zpbs-vm1, 2495647.t2zpbs-vm1, 2495648.t2zpbs-vm1
2495651.t2zpbs-vm1, 2495652.t2zpbs-vm1, 2495653.t2zpbs-vm1, 2495654.t2zpbs-vm1, 2495655.t2zpbs-vm1
2495656.t2zpbs-vm1, 2495658.t2zpbs-vm1, 2495659.t2zpbs-vm1, 2495660.t2zpbs-vm1, 2495661.t2zpbs-vm1
2495662.t2zpbs-vm1, 2495663.t2zpbs-vm1, 2495664.t2zpbs-vm1, 2495665.t2zpbs-vm1, 2495666.t2zpbs-vm1
2495667.t2zpbs-vm1, 2495668.t2zpbs-vm1, 2495669.t2zpbs-vm1, 2495670.t2zpbs-vm1, 2495671.t2zpbs-vm1
2495672.t2zpbs-vm1, 2495674.t2zpbs-vm1, 2495675.t2zpbs-vm1, 2495676.t2zpbs-vm1, 2495708.t2zpbs-vm1
2495765.t2zpbs-vm1, 2495769.t2zpbs-vm1, 2495772.t2zpbs-vm1, 2495855.t2zpbs-vm1, 2495888.t2zpbs-vm1
2495890.t2zpbs-vm1, 2495998.t2zpbs-vm1, 2496001.t2zpbs-vm1, 2496017.t2zpbs-vm1, 2496023.t2zpbs-vm1
2496025.t2zpbs-vm1, 2496060.t2zpbs-vm1, 2496061.t2zpbs-vm1, 2496062.t2zpbs-vm1, 2496091.t2zpbs-vm1
2496092.t2zpbs-vm1, 2496112.t2zpbs-vm1, 2496129.t2zpbs-vm1, 2496132.t2zpbs-vm1, 2496136.t2zpbs-vm1
2496226.t2zpbs-vm1, 2496237.t2zpbs-vm1, 2496297.t2zpbs-vm1, 2496298.t2zpbs-vm1, 2496299.t2zpbs-vm1
2496300.t2zpbs-vm1, 2496301.t2zpbs-vm1, 2496302.t2zpbs-vm1, 2496313.t2zpbs-vm1, 2496314.t2zpbs-vm1
2496315.t2zpbs-vm1, 2496340.t2zpbs-vm1, 2496346.t2zpbs-vm1, 2496348.t2zpbs-vm1, 2496349.t2zpbs-vm1
2496350.t2zpbs-vm1, 2496356.t2zpbs-vm1, 2496357.t2zpbs-vm1, 2496358.t2zpbs-vm1, 2496359.t2zpbs-vm1
2496360.t2zpbs-vm1, 2496361.t2zpbs-vm1, 2496362.t2zpbs-vm1, 2496363.t2zpbs-vm1, 2496432.t2zpbs-vm1
2496433.t2zpbs-vm1, 2496434.t2zpbs-vm1, 2496435.t2zpbs-vm1, 2496436.t2zpbs-vm1, 2496437.t2zpbs-vm1
2496438.t2zpbs-vm1, 2496439.t2zpbs-vm1, 2496440.t2zpbs-vm1, 2496441.t2zpbs-vm1, 2496442.t2zpbs-vm1
2496444.t2zpbs-vm1, 2496445.t2zpbs-vm1, 2496446.t2zpbs-vm1, 2496588.t2zpbs-vm1, 2496589.t2zpbs-vm1
2496590.t2zpbs-vm1, 2496591.t2zpbs-vm1, 2496594.t2zpbs-vm1, 2496622.t2zpbs-vm1, 2496628.t2zpbs-vm1
2496630.t2zpbs-vm1, 2496632.t2zpbs-vm1, 2496634.t2zpbs-vm1, 2496648.t2zpbs-vm1, 2496649.t2zpbs-vm1
2496650.t2zpbs-vm1, 2496651.t2zpbs-vm1, 2496652.t2zpbs-vm1, 2496654.t2zpbs-vm1, 2496656.t2zpbs-vm1
2496657.t2zpbs-vm1, 2496688.t2zpbs-vm1, 2496689.t2zpbs-vm1, 2496690.t2zpbs-vm1, 2496691.t2zpbs-vm1
2496692.t2zpbs-vm1, 2496693.t2zpbs-vm1, 2496694.t2zpbs-vm1, 2496695.t2zpbs-vm1, 2496696.t2zpbs-vm1
2496697.t2zpbs-vm1, 2496698.t2zpbs-vm1, 2496699.t2zpbs-vm1, 2496700.t2zpbs-vm1, 2496701.t2zpbs-vm1
2496702.t2zpbs-vm1, 2496703.t2zpbs-vm1, 2496704.t2zpbs-vm1, 2496705.t2zpbs-vm1, 2496706.t2zpbs-vm1
2496707.t2zpbs-vm1, 2496708.t2zpbs-vm1, 2496709.t2zpbs-vm1, 2496710.t2zpbs-vm1, 2496711.t2zpbs-vm1
2496712.t2zpbs-vm1, 2496713.t2zpbs-vm1, 2496714.t2zpbs-vm1, 2496715.t2zpbs-vm1, 2496716.t2zpbs-vm1
2496717.t2zpbs-vm1, 2496718.t2zpbs-vm1, 2496719.t2zpbs-vm1, 2496720.t2zpbs-vm1, 2496721.t2zpbs-vm1
2496722.t2zpbs-vm1, 2496723.t2zpbs-vm1, 2496724.t2zpbs-vm1, 2496725.t2zpbs-vm1, 2496726.t2zpbs-vm1
2496727.t2zpbs-vm1, 2496728.t2zpbs-vm1, 2496729.t2zpbs-vm1, 2496730.t2zpbs-vm1, 2496731.t2zpbs-vm1
2496732.t2zpbs-vm1, 2496733.t2zpbs-vm1, 2496734.t2zpbs-vm1, 2496735.t2zpbs-vm1, 2496736.t2zpbs-vm1
2496737.t2zpbs-vm1, 2496738.t2zpbs-vm1, 2496739.t2zpbs-vm1, 2496740.t2zpbs-vm1, 2496741.t2zpbs-vm1
2496742.t2zpbs-vm1, 2496743.t2zpbs-vm1, 2496744.t2zpbs-vm1, 2496745.t2zpbs-vm1, 2496746.t2zpbs-vm1
2496747.t2zpbs-vm1, 2496748.t2zpbs-vm1, 2496749.t2zpbs-vm1, 2496750.t2zpbs-vm1, 2496751.t2zpbs-vm1
2496752.t2zpbs-vm1, 2496753.t2zpbs-vm1, 2496754.t2zpbs-vm1, 2496755.t2zpbs-vm1, 2496756.t2zpbs-vm1
2496757.t2zpbs-vm1, 2496758.t2zpbs-vm1, 2496759.t2zpbs-vm1, 2496760.t2zpbs-vm1, 2496761.t2zpbs-vm1
2496762.t2zpbs-vm1, 2496763.t2zpbs-vm1, 2496764.t2zpbs-vm1, 2496765.t2zpbs-vm1, 2496766.t2zpbs-vm1
2496767.t2zpbs-vm1, 2496768.t2zpbs-vm1, 2496769.t2zpbs-vm1, 2496777.t2zpbs-vm1, 2496778.t2zpbs-vm1
2496779.t2zpbs-vm1, 2496780.t2zpbs-vm1, 2496781.t2zpbs-vm1, 2496782.t2zpbs-vm1, 2496783.t2zpbs-vm1
2496784.t2zpbs-vm1, 2496790.t2zpbs-vm1, 2496791.t2zpbs-vm1, 2496792.t2zpbs-vm1, 2496793.t2zpbs-vm1
2496813.t2zpbs-vm1, 2496814.t2zpbs-vm1, 2496815.t2zpbs-vm1, 2496816.t2zpbs-vm1, 2496817.t2zpbs-vm1
2496836.t2zpbs-vm1, 2496837.t2zpbs-vm1, 2496838.t2zpbs-vm1, 2496839.t2zpbs-vm1, 2496840.t2zpbs-vm1
2496841.t2zpbs-vm1, 2496850.t2zpbs-vm1, 2496860.t2zpbs-vm1, 2496861.t2zpbs-vm1, 2496863.t2zpbs-vm1
2496864.t2zpbs-vm1, 2496873.t2zpbs-vm1, 2496875.t2zpbs-vm1, 2496876.t2zpbs-vm1, 2496881.t2zpbs-vm1
2496882.t2zpbs-vm1, 2496883.t2zpbs-vm1, 2496890.t2zpbs-vm1, 2496891.t2zpbs-vm1, 2496892.t2zpbs-vm1
2496893.t2zpbs-vm1, 2496894.t2zpbs-vm1, 2496895.t2zpbs-vm1, 2496896.t2zpbs-vm1, 2496897.t2zpbs-vm1
2496898.t2zpbs-vm1, 2496899.t2zpbs-vm1, 2496900.t2zpbs-vm1, 2496901.t2zpbs-vm1, 2496902.t2zpbs-vm1
2496903.t2zpbs-vm1, 2496904.t2zpbs-vm1, 2496905.t2zpbs-vm1, 2496906.t2zpbs-vm1, 2496907.t2zpbs-vm1
2496908.t2zpbs-vm1, 2496909.t2zpbs-vm1, 2496910.t2zpbs-vm1, 2496911.t2zpbs-vm1, 2496912.t2zpbs-vm1
2496913.t2zpbs-vm1, 2496914.t2zpbs-vm1, 2496915.t2zpbs-vm1, 2496916.t2zpbs-vm1, 2496917.t2zpbs-vm1
2496918.t2zpbs-vm1, 2496919.t2zpbs-vm1, 2496920.t2zpbs-vm1, 2496921.t2zpbs-vm1, 2496922.t2zpbs-vm1
2496923.t2zpbs-vm1, 2496924.t2zpbs-vm1, 2496925.t2zpbs-vm1, 2496926.t2zpbs-vm1, 2496927.t2zpbs-vm1
2496929.t2zpbs-vm1, 2496930.t2zpbs-vm1, 2496931.t2zpbs-vm1, 2496932.t2zpbs-vm1, 2496933.t2zpbs-vm1
2496934.t2zpbs-vm1, 2496935.t2zpbs-vm1, 2496936.t2zpbs-vm1, 2496937.t2zpbs-vm1, 2496938.t2zpbs-vm1
2496939.t2zpbs-vm1, 2496940.t2zpbs-vm1, 2496941.t2zpbs-vm1, 2496942.t2zpbs-vm1, 2496943.t2zpbs-vm1
2496944.t2zpbs-vm1, 2496945.t2zpbs-vm1, 2496946.t2zpbs-vm1, 2496947.t2zpbs-vm1, 2496948.t2zpbs-vm1
2496949.t2zpbs-vm1, 2496950.t2zpbs-vm1, 2496951.t2zpbs-vm1, 2496952.t2zpbs-vm1, 2496953.t2zpbs-vm1
2496954.t2zpbs-vm1, 2496955.t2zpbs-vm1, 2496956.t2zpbs-vm1, 2496957.t2zpbs-vm1, 2496958.t2zpbs-vm1
2496959.t2zpbs-vm1, 2496960.t2zpbs-vm1, 2496961.t2zpbs-vm1, 2496962.t2zpbs-vm1, 2496963.t2zpbs-vm1
2496964.t2zpbs-vm1, 2496965.t2zpbs-vm1, 2496966.t2zpbs-vm1, 2496967.t2zpbs-vm1, 2496971.t2zpbs-vm1
2496972.t2zpbs-vm1, 2496973.t2zpbs-vm1, 2496974.t2zpbs-vm1, 2496975.t2zpbs-vm1, 2496976.t2zpbs-vm1
2496979.t2zpbs-vm1, 2496983.t2zpbs-vm1, 2496984.t2zpbs-vm1, 2496985.t2zpbs-vm1

<Uキュー:1ジョブ>
2495728.t2zpbs-vm1

<L256キュー:2ジョブ>
2176624.t2zpbs01, 2176639.t2zpbs01

<Hキュー:8ジョブ>
1188730.t2zpbs05, 1194254.t2zpbs05, 1194255.t2zpbs05, 1194256.t2zpbs05, 1194257.t2zpbs05
1194258.t2zpbs05, 1194259.t2zpbs05, 1194260.t2zpbs05

<Xキュー:5ジョブ>
1194403.t2zpbs05, 1194404.t2zpbs05, 1194405.t2zpbs05, 1194407.t2zpbs05, 1194408.t2zpbs05