【障害報告】2017.2.13発生:/work0 ストレージ障害

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
2017年2月15日

 下記の通りストレージに障害が発生し、復旧いたしました。

 時間帯:2月13(月)14:58から15:13頃

 影響:/work0へのアクセスに一時的なI/O遅延やエラーが発生した可能性があります

 詳細:

/work0 を構成する2台で冗長化されている MDS(MetaData Server) として稼働していた t2s007017 において障害が発生しました。

その結果、MDS のサービス(MDT, MetaData Target)を対となる t2s007019 にフェイルオーバーしました。

期間中、一時的なI/O遅延やエラーが発生した可能性があります。

用語説明

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


 なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

<Sキュー:160ジョブ>
2016236.t2zpbs01, 2021176.t2zpbs01, 2021194.t2zpbs01, 2021221.t2zpbs01, 2021250.t2zpbs01
2021272.t2zpbs01, 2021320.t2zpbs01, 2021392.t2zpbs01, 2021456.t2zpbs01, 2021477.t2zpbs01
2021575.t2zpbs01, 2025116.t2zpbs01, 2025117.t2zpbs01, 2025118.t2zpbs01, 2025119.t2zpbs01
2025120.t2zpbs01, 2025121.t2zpbs01, 2025122.t2zpbs01, 2025123.t2zpbs01, 2025124.t2zpbs01
2025125.t2zpbs01, 2025126.t2zpbs01, 2025127.t2zpbs01, 2025128.t2zpbs01, 2025129.t2zpbs01
2025130.t2zpbs01, 2025131.t2zpbs01, 2025132.t2zpbs01, 2025133.t2zpbs01, 2025134.t2zpbs01
2025135.t2zpbs01, 2025141.t2zpbs01, 2025142.t2zpbs01, 2025143.t2zpbs01, 2025144.t2zpbs01
2025145.t2zpbs01, 2025260.t2zpbs01, 2025395.t2zpbs01, 2025408.t2zpbs01, 2025411.t2zpbs01
2025561.t2zpbs01, 2025605.t2zpbs01, 2026375.t2zpbs01, 2026380.t2zpbs01, 2026381.t2zpbs01
2026382.t2zpbs01, 2026871.t2zpbs01, 2026876.t2zpbs01, 2027274.t2zpbs01, 2027277.t2zpbs01
2027278.t2zpbs01, 2027279.t2zpbs01, 2027280.t2zpbs01, 2027281.t2zpbs01, 2027282.t2zpbs01
2027283.t2zpbs01, 2027285.t2zpbs01, 2027286.t2zpbs01, 2027288.t2zpbs01, 2027289.t2zpbs01
2027290.t2zpbs01, 2027291.t2zpbs01, 2027292.t2zpbs01, 2027293.t2zpbs01, 2027295.t2zpbs01
2027298.t2zpbs01, 2027299.t2zpbs01, 2027300.t2zpbs01, 2027301.t2zpbs01, 2027303.t2zpbs01
2027304.t2zpbs01, 2027306.t2zpbs01, 2027308.t2zpbs01, 2027309.t2zpbs01, 2027310.t2zpbs01
2027311.t2zpbs01, 2027312.t2zpbs01, 2027313.t2zpbs01, 2027344.t2zpbs01, 2027373.t2zpbs01
2027443.t2zpbs01, 2027463.t2zpbs01, 2027664.t2zpbs01, 2027721.t2zpbs01, 2027816.t2zpbs01
2027817.t2zpbs01, 2027818.t2zpbs01, 2027819.t2zpbs01, 2027822.t2zpbs01, 2027823.t2zpbs01
2027825.t2zpbs01, 2027826.t2zpbs01, 2027827.t2zpbs01, 2027830.t2zpbs01, 2032066.t2zpbs01
2032165.t2zpbs01, 2032482.t2zpbs01, 2032790.t2zpbs01, 2034073.t2zpbs01, 2034507.t2zpbs01
2034812.t2zpbs01, 2034864.t2zpbs01, 2035833.t2zpbs01, 2036685.t2zpbs01, 2036755.t2zpbs01
2036872.t2zpbs01, 2037278.t2zpbs01, 2037467.t2zpbs01, 2037706.t2zpbs01, 2037710.t2zpbs01
2037761.t2zpbs01, 2037802.t2zpbs01, 2037820.t2zpbs01, 2038057.t2zpbs01, 2038161.t2zpbs01
2038233.t2zpbs01, 2038446.t2zpbs01, 2038850.t2zpbs01, 2039172.t2zpbs01, 2039243.t2zpbs01
2039273.t2zpbs01, 2039311.t2zpbs01, 2039314.t2zpbs01, 2039455.t2zpbs01, 2039649.t2zpbs01
2039702.t2zpbs01, 2039767[10].t2zpbs01, 2039767[11].t2zpbs01, 2039767[12].t2zpbs01, 2039767[13].t2zpbs01
2039767[1].t2zpbs01, 2039767[2].t2zpbs01, 2039767[3].t2zpbs01, 2039767[4].t2zpbs01, 2039767[5].t2zpbs01
2039767[6].t2zpbs01, 2039767[7].t2zpbs01, 2039767[8].t2zpbs01, 2039767[9].t2zpbs01, 2039777.t2zpbs01
2039778.t2zpbs01, 2039779.t2zpbs01, 2039780.t2zpbs01, 2039781.t2zpbs01, 2039782.t2zpbs01
2039783.t2zpbs01, 2039784.t2zpbs01, 2039785.t2zpbs01, 2039786.t2zpbs01, 2039787.t2zpbs01
2039788.t2zpbs01, 2039789.t2zpbs01, 2039790.t2zpbs01, 2039791.t2zpbs01, 2039792.t2zpbs01
2039793.t2zpbs01, 2039794.t2zpbs01, 2039795.t2zpbs01, 2039796.t2zpbs01, 2039797.t2zpbs01

<S96キュー:43ジョブ>
2021244.t2zpbs01, 2021261.t2zpbs01, 2023940.t2zpbs01, 2023946.t2zpbs01, 2023950.t2zpbs01
2023951.t2zpbs01, 2023952.t2zpbs01, 2023953.t2zpbs01, 2023954.t2zpbs01, 2023955.t2zpbs01
2023956.t2zpbs01, 2023957.t2zpbs01, 2023958.t2zpbs01, 2023959.t2zpbs01, 2023960.t2zpbs01
2023961.t2zpbs01, 2023962.t2zpbs01, 2023963.t2zpbs01, 2023964.t2zpbs01, 2023965.t2zpbs01
2023966.t2zpbs01, 2023967.t2zpbs01, 2023968.t2zpbs01, 2023969.t2zpbs01, 2023970.t2zpbs01
2023971.t2zpbs01, 2023972.t2zpbs01, 2023973.t2zpbs01, 2023974.t2zpbs01, 2023975.t2zpbs01
2023976.t2zpbs01, 2023977.t2zpbs01, 2023978.t2zpbs01, 2023980.t2zpbs01, 2023981.t2zpbs01
2023982.t2zpbs01, 2023983.t2zpbs01, 2023986.t2zpbs01, 2023987.t2zpbs01, 2023988.t2zpbs01
2023989.t2zpbs01, 2023990.t2zpbs01, 2023991.t2zpbs01

<Gキュー:55ジョブ>
2107441.t2zpbs03, 2108173.t2zpbs03, 2108174.t2zpbs03, 2108245.t2zpbs03, 2108253.t2zpbs03
2108260.t2zpbs03, 2108262.t2zpbs03, 2108282.t2zpbs03, 2108284.t2zpbs03, 2108299.t2zpbs03
2108306.t2zpbs03, 2108331.t2zpbs03, 2108335.t2zpbs03, 2108338.t2zpbs03, 2108339.t2zpbs03
2108340.t2zpbs03, 2108341.t2zpbs03, 2108342.t2zpbs03, 2108357.t2zpbs03, 2108360.t2zpbs03
2108361.t2zpbs03, 2108369.t2zpbs03, 2108372.t2zpbs03, 2108374.t2zpbs03, 2108377.t2zpbs03
2108382[1].t2zpbs03, 2108382[2].t2zpbs03, 2108383[2].t2zpbs03, 2108384[0].t2zpbs03, 2108384[1].t2zpbs03
2108384[2].t2zpbs03, 2108384[3].t2zpbs03, 2108384[4].t2zpbs03, 2108385.t2zpbs03, 2108387.t2zpbs03
2108391.t2zpbs03, 2108392.t2zpbs03, 2108398.t2zpbs03, 2108410.t2zpbs03, 2108417.t2zpbs03
2108421[0].t2zpbs03, 2108421[1].t2zpbs03, 2108421[2].t2zpbs03, 2108421[3].t2zpbs03, 2108421[4].t2zpbs03
2108422[0].t2zpbs03, 2108422[1].t2zpbs03, 2108422[2].t2zpbs03, 2108422[3].t2zpbs03, 2108422[4].t2zpbs03
2108423[0].t2zpbs03, 2108423[1].t2zpbs03, 2108423[2].t2zpbs03, 2108423[3].t2zpbs03, 2108423[4].t2zpbs03

<Vキュー:309ジョブ>
2402117.t2zpbs-vm1, 2403734.t2zpbs-vm1, 2403735.t2zpbs-vm1, 2403738.t2zpbs-vm1, 2403739.t2zpbs-vm1
2404674.t2zpbs-vm1, 2404678.t2zpbs-vm1, 2405307.t2zpbs-vm1, 2406153.t2zpbs-vm1, 2406163.t2zpbs-vm1
2406164.t2zpbs-vm1, 2406165.t2zpbs-vm1, 2406166.t2zpbs-vm1, 2406167.t2zpbs-vm1, 2406168.t2zpbs-vm1
2406169.t2zpbs-vm1, 2406170.t2zpbs-vm1, 2406171.t2zpbs-vm1, 2406172.t2zpbs-vm1, 2406173.t2zpbs-vm1
2406176.t2zpbs-vm1, 2406178.t2zpbs-vm1, 2406179.t2zpbs-vm1, 2406180.t2zpbs-vm1, 2406181.t2zpbs-vm1
2406182.t2zpbs-vm1, 2406191.t2zpbs-vm1, 2406192.t2zpbs-vm1, 2406193.t2zpbs-vm1, 2406194.t2zpbs-vm1
2406195.t2zpbs-vm1, 2406198.t2zpbs-vm1, 2406200.t2zpbs-vm1, 2406201.t2zpbs-vm1, 2406202.t2zpbs-vm1
2406203.t2zpbs-vm1, 2406205.t2zpbs-vm1, 2406210.t2zpbs-vm1, 2406212.t2zpbs-vm1, 2406216.t2zpbs-vm1
2406217.t2zpbs-vm1, 2406223.t2zpbs-vm1, 2406224.t2zpbs-vm1, 2406227.t2zpbs-vm1, 2406229.t2zpbs-vm1
2406230.t2zpbs-vm1, 2406231.t2zpbs-vm1, 2406232.t2zpbs-vm1, 2406236.t2zpbs-vm1, 2406238.t2zpbs-vm1
2406241.t2zpbs-vm1, 2406242.t2zpbs-vm1, 2406244.t2zpbs-vm1, 2406245.t2zpbs-vm1, 2406247.t2zpbs-vm1
2406248.t2zpbs-vm1, 2406249.t2zpbs-vm1, 2406251.t2zpbs-vm1, 2406252.t2zpbs-vm1, 2406254.t2zpbs-vm1
2406255.t2zpbs-vm1, 2406256.t2zpbs-vm1, 2406257.t2zpbs-vm1, 2406258.t2zpbs-vm1, 2406259.t2zpbs-vm1
2406260.t2zpbs-vm1, 2406261.t2zpbs-vm1, 2406262.t2zpbs-vm1, 2406263.t2zpbs-vm1, 2406264.t2zpbs-vm1
2406265.t2zpbs-vm1, 2406266.t2zpbs-vm1, 2406267.t2zpbs-vm1, 2406268.t2zpbs-vm1, 2406269.t2zpbs-vm1
2406270.t2zpbs-vm1, 2406271.t2zpbs-vm1, 2406272.t2zpbs-vm1, 2406273.t2zpbs-vm1, 2406274.t2zpbs-vm1
2406275.t2zpbs-vm1, 2406276.t2zpbs-vm1, 2406277.t2zpbs-vm1, 2406278.t2zpbs-vm1, 2406279.t2zpbs-vm1
2406280.t2zpbs-vm1, 2406281.t2zpbs-vm1, 2406282.t2zpbs-vm1, 2406283.t2zpbs-vm1, 2406284.t2zpbs-vm1
2406285.t2zpbs-vm1, 2406286.t2zpbs-vm1, 2406287.t2zpbs-vm1, 2406288.t2zpbs-vm1, 2406289.t2zpbs-vm1
2406290.t2zpbs-vm1, 2406291.t2zpbs-vm1, 2406292.t2zpbs-vm1, 2406293.t2zpbs-vm1, 2406294.t2zpbs-vm1
2406295.t2zpbs-vm1, 2406296.t2zpbs-vm1, 2406297.t2zpbs-vm1, 2406298.t2zpbs-vm1, 2406299.t2zpbs-vm1
2406300.t2zpbs-vm1, 2406301.t2zpbs-vm1, 2406302.t2zpbs-vm1, 2406303.t2zpbs-vm1, 2406304.t2zpbs-vm1
2406305.t2zpbs-vm1, 2406306.t2zpbs-vm1, 2406307.t2zpbs-vm1, 2406308.t2zpbs-vm1, 2406309.t2zpbs-vm1
2406310.t2zpbs-vm1, 2406311.t2zpbs-vm1, 2406312.t2zpbs-vm1, 2406313.t2zpbs-vm1, 2406314.t2zpbs-vm1
2406315.t2zpbs-vm1, 2406316.t2zpbs-vm1, 2406317.t2zpbs-vm1, 2406318.t2zpbs-vm1, 2406319.t2zpbs-vm1
2406320.t2zpbs-vm1, 2406321.t2zpbs-vm1, 2406322.t2zpbs-vm1, 2406323.t2zpbs-vm1, 2406324.t2zpbs-vm1
2406325.t2zpbs-vm1, 2406326.t2zpbs-vm1, 2406327.t2zpbs-vm1, 2406328.t2zpbs-vm1, 2406329.t2zpbs-vm1
2406330.t2zpbs-vm1, 2406331.t2zpbs-vm1, 2406332.t2zpbs-vm1, 2406333.t2zpbs-vm1, 2406334.t2zpbs-vm1
2406335.t2zpbs-vm1, 2406336.t2zpbs-vm1, 2406337.t2zpbs-vm1, 2406338.t2zpbs-vm1, 2406339.t2zpbs-vm1
2406376.t2zpbs-vm1, 2406377.t2zpbs-vm1, 2406378.t2zpbs-vm1, 2406379.t2zpbs-vm1, 2406380.t2zpbs-vm1
2406381.t2zpbs-vm1, 2406382.t2zpbs-vm1, 2406383.t2zpbs-vm1, 2406384.t2zpbs-vm1, 2406385.t2zpbs-vm1
2406386.t2zpbs-vm1, 2406387.t2zpbs-vm1, 2406388.t2zpbs-vm1, 2406389.t2zpbs-vm1, 2406390.t2zpbs-vm1
2406391.t2zpbs-vm1, 2406392.t2zpbs-vm1, 2406393.t2zpbs-vm1, 2406394.t2zpbs-vm1, 2406395.t2zpbs-vm1
2406396.t2zpbs-vm1, 2406397.t2zpbs-vm1, 2406398.t2zpbs-vm1, 2406399.t2zpbs-vm1, 2406400.t2zpbs-vm1
2406401.t2zpbs-vm1, 2406402.t2zpbs-vm1, 2406403.t2zpbs-vm1, 2406404.t2zpbs-vm1, 2406405.t2zpbs-vm1
2406406.t2zpbs-vm1, 2406407.t2zpbs-vm1, 2406408.t2zpbs-vm1, 2406409.t2zpbs-vm1, 2406410.t2zpbs-vm1
2406411.t2zpbs-vm1, 2406412.t2zpbs-vm1, 2406413.t2zpbs-vm1, 2406414.t2zpbs-vm1, 2406415.t2zpbs-vm1
2406416.t2zpbs-vm1, 2406417.t2zpbs-vm1, 2406418.t2zpbs-vm1, 2406420.t2zpbs-vm1, 2406421.t2zpbs-vm1
2406422.t2zpbs-vm1, 2406423.t2zpbs-vm1, 2406424.t2zpbs-vm1, 2406425.t2zpbs-vm1, 2406426.t2zpbs-vm1
2406427.t2zpbs-vm1, 2406428.t2zpbs-vm1, 2406429.t2zpbs-vm1, 2406430.t2zpbs-vm1, 2406431.t2zpbs-vm1
2406432.t2zpbs-vm1, 2406433.t2zpbs-vm1, 2406434.t2zpbs-vm1, 2406435.t2zpbs-vm1, 2406436.t2zpbs-vm1
2406437.t2zpbs-vm1, 2406439.t2zpbs-vm1, 2406440.t2zpbs-vm1, 2406441.t2zpbs-vm1, 2406442.t2zpbs-vm1
2406443.t2zpbs-vm1, 2406444.t2zpbs-vm1, 2406445.t2zpbs-vm1, 2406446.t2zpbs-vm1, 2406447.t2zpbs-vm1
2406451.t2zpbs-vm1, 2406467.t2zpbs-vm1, 2406801.t2zpbs-vm1, 2406802.t2zpbs-vm1, 2406803.t2zpbs-vm1
2406804.t2zpbs-vm1, 2406805.t2zpbs-vm1, 2406806.t2zpbs-vm1, 2406855.t2zpbs-vm1, 2406856.t2zpbs-vm1
2406857.t2zpbs-vm1, 2406898.t2zpbs-vm1, 2406899.t2zpbs-vm1, 2406900.t2zpbs-vm1, 2406902.t2zpbs-vm1
2406903.t2zpbs-vm1, 2406904.t2zpbs-vm1, 2406905.t2zpbs-vm1, 2406916.t2zpbs-vm1, 2406917.t2zpbs-vm1
2406958.t2zpbs-vm1, 2406959.t2zpbs-vm1, 2406960.t2zpbs-vm1, 2406961.t2zpbs-vm1, 2406962.t2zpbs-vm1
2406963.t2zpbs-vm1, 2406964.t2zpbs-vm1, 2406965.t2zpbs-vm1, 2406966.t2zpbs-vm1, 2406967.t2zpbs-vm1
2406968.t2zpbs-vm1, 2406969.t2zpbs-vm1, 2406970.t2zpbs-vm1, 2406971.t2zpbs-vm1, 2406987.t2zpbs-vm1
2406988.t2zpbs-vm1, 2406989.t2zpbs-vm1, 2406990.t2zpbs-vm1, 2406991.t2zpbs-vm1, 2406992.t2zpbs-vm1
2407014.t2zpbs-vm1, 2407015.t2zpbs-vm1, 2407028.t2zpbs-vm1, 2407029.t2zpbs-vm1, 2407030.t2zpbs-vm1
2407031.t2zpbs-vm1, 2407032.t2zpbs-vm1, 2407033.t2zpbs-vm1, 2407034.t2zpbs-vm1, 2407035.t2zpbs-vm1
2407046.t2zpbs-vm1, 2407047.t2zpbs-vm1, 2407048.t2zpbs-vm1, 2407049.t2zpbs-vm1, 2407050.t2zpbs-vm1
2407051.t2zpbs-vm1, 2407052.t2zpbs-vm1, 2407053.t2zpbs-vm1, 2407054.t2zpbs-vm1, 2407055.t2zpbs-vm1
2407073.t2zpbs-vm1, 2407074.t2zpbs-vm1, 2407075.t2zpbs-vm1, 2407076.t2zpbs-vm1, 2407077.t2zpbs-vm1
2407089.t2zpbs-vm1, 2407099.t2zpbs-vm1, 2407100.t2zpbs-vm1, 2407101.t2zpbs-vm1, 2407102.t2zpbs-vm1
2407108.t2zpbs-vm1, 2407117.t2zpbs-vm1, 2407120.t2zpbs-vm1, 2407121.t2zpbs-vm1, 2407122.t2zpbs-vm1
2407123.t2zpbs-vm1, 2407126.t2zpbs-vm1, 2407127.t2zpbs-vm1, 2407128.t2zpbs-vm1, 2407129.t2zpbs-vm1
2407130.t2zpbs-vm1, 2407131.t2zpbs-vm1, 2407132.t2zpbs-vm1, 2407136.t2zpbs-vm1, 2407140.t2zpbs-vm1
2407143.t2zpbs-vm1, 2407144.t2zpbs-vm1, 2407148.t2zpbs-vm1, 2407150.t2zpbs-vm1, 2407152.t2zpbs-vm1
2407153.t2zpbs-vm1, 2407154.t2zpbs-vm1, 2407155.t2zpbs-vm1, 2407156.t2zpbs-vm1

<Uキュー:3ジョブ>
2406174.t2zpbs-vm1, 2407105.t2zpbs-vm1, 2407124.t2zpbs-vm1

<L256キュー:2ジョブ>
2036056.t2zpbs01, 2036264.t2zpbs01

<L512キュー:0ジョブ>

<Hキュー:8ジョブ>
1117153.t2zpbs05, 1117124.t2zpbs05, 1117075.t2zpbs05, 1116959.t2zpbs05, 1116792.t2zpbs05
1116780.t2zpbs05, 1116873.t2zpbs05, 1117301.t2zpbs05

<Xキュー:10ジョブ>
1117365.t2zpbs05, 1117364.t2zpbs05, 1117058.t2zpbs05, 1117160.t2zpbs05, 1116786.t2zpbs05
1116784.t2zpbs05, 1116834.t2zpbs05, 1116819.t2zpbs05, 1116805.t2zpbs05, 1116867.t2zpbs05