【障害報告】2016.6.8発生:/work1 ストレージ障害 (6/30更新)

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
平成28年6月29日

 下記の通りストレージに障害が発生し、復旧いたしました。発見、報告が遅くなり申し訳ありませんでした。

 日時:6月8日(水) 11:18 から 11:28 頃

 影響: /work1 にアクセスできない

 詳細:

/work1 を構成する2台で冗長化されている MDS(MetaData Server) として稼働していた1台の t2s007015 において Lustre に障害が発生し、自動で MDS のサービスが t2s007013 にフェイルオーバーされたことで復旧しました。復旧までの間 /work1 にアクセスすることができませんでした。なお、次の障害が発生する6/24まで、/work1 の MDS は t2s007013 で稼働している状況でした。原因は現在調査中です。

用語説明

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


(6/30追記)

MDS におけるファイル open 時の処理で稀にタイミングの問題で -EREMOTE というエラーが発生することがあり、そのエラーに対して適切な処理が行なわれていなかったため バグが起こることがわかりました。Lustreサーバー側の修正を準備中です。


 なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

(Sキュー)

the number of jobs = 431
### S ##########################################
1736287.t2zpbs01,1736286.t2zpbs01,1736198.t2zpbs01,1736289.t2zpbs01,1736188.t2zpbs01
1736209.t2zpbs01,1736187.t2zpbs01,1736034.t2zpbs01,1736189.t2zpbs01,1736183.t2zpbs01
1736754.t2zpbs01,1736335.t2zpbs01,1736398.t2zpbs01,1736656.t2zpbs01,1736679.t2zpbs01
1736653.t2zpbs01,1736667.t2zpbs01,1736416.t2zpbs01,1736666.t2zpbs01,1736680.t2zpbs01
1736671.t2zpbs01,1736657.t2zpbs01,1736659.t2zpbs01,1736673.t2zpbs01,1736665.t2zpbs01
1736674.t2zpbs01,1736682.t2zpbs01,1736664.t2zpbs01,1736672.t2zpbs01,1736670.t2zpbs01
1736681.t2zpbs01,1736663.t2zpbs01,1736677.t2zpbs01,1736661.t2zpbs01,1736660.t2zpbs01
1736443.t2zpbs01,1736669.t2zpbs01,1736676.t2zpbs01,1736654.t2zpbs01,1736658.t2zpbs01
1736687.t2zpbs01,1736685.t2zpbs01,1736684.t2zpbs01,1736683.t2zpbs01,1736686.t2zpbs01
1736576.t2zpbs01,1736534.t2zpbs01,1736541.t2zpbs01,1736548.t2zpbs01,1736357.t2zpbs01
1736557.t2zpbs01,1736406.t2zpbs01,1736568.t2zpbs01,1736527.t2zpbs01,1736799.t2zpbs01
1736800.t2zpbs01,1736803.t2zpbs01,1736804.t2zpbs01,1736595.t2zpbs01,1736133.t2zpbs01
1736627.t2zpbs01,1736635.t2zpbs01,1736618.t2zpbs01,1736631.t2zpbs01,1736633.t2zpbs01
1736619.t2zpbs01,1736629.t2zpbs01,1736625.t2zpbs01,1736630.t2zpbs01,1736624.t2zpbs01
1736636.t2zpbs01,1736634.t2zpbs01,1736622.t2zpbs01,1736626.t2zpbs01,1736637.t2zpbs01
1736621.t2zpbs01,1736617.t2zpbs01,1736644.t2zpbs01,1736640.t2zpbs01,1736643.t2zpbs01
1736638.t2zpbs01,1736642.t2zpbs01,1736620.t2zpbs01,1736639.t2zpbs01,1736623.t2zpbs01
1736628.t2zpbs01,1736641.t2zpbs01,1736632.t2zpbs01,1736645.t2zpbs01,1736508.t2zpbs01
1736525.t2zpbs01,1736364.t2zpbs01,1736371.t2zpbs01,1736603.t2zpbs01,1736602.t2zpbs01
1735538.t2zpbs01,1736787.t2zpbs01,1736075.t2zpbs01

### S96 ##########################################
1736789.t2zpbs01,1736808.t2zpbs01,1736058.t2zpbs01,1736809.t2zpbs01,1736810.t2zpbs01
1736790.t2zpbs01,1736428.t2zpbs01,1736435.t2zpbs01,1736429.t2zpbs01,1736425.t2zpbs01
1736426.t2zpbs01,1736431.t2zpbs01,1736432.t2zpbs01,1736433.t2zpbs01,1736430.t2zpbs01
1736436.t2zpbs01,1736439.t2zpbs01,1736437.t2zpbs01,1736798.t2zpbs01,1736647.t2zpbs01

### G ##########################################
2035691.t2zpbs03,2035223.t2zpbs03,2035515.t2zpbs03,2035699.t2zpbs03,2035677.t2zpbs03
2035727.t2zpbs03,2035533.t2zpbs03,2035704.t2zpbs03,2035526.t2zpbs03,2035676.t2zpbs03
2035661.t2zpbs03,2035660.t2zpbs03,2035687.t2zpbs03,2035742.t2zpbs03,2035683.t2zpbs03
2035692.t2zpbs03,2035180.t2zpbs03,2035681.t2zpbs03,2035686.t2zpbs03,2035705.t2zpbs03
2035688.t2zpbs03,2035678.t2zpbs03,2035680.t2zpbs03,2035682.t2zpbs03,2035685.t2zpbs03
2035703.t2zpbs03,2035696.t2zpbs03,2035700.t2zpbs03,2035701.t2zpbs03

### X ##########################################
929312.t2zpbs05,929314.t2zpbs05,929315.t2zpbs05,929316.t2zpbs05,929297.t2zpbs05
929317.t2zpbs05,929278.t2zpbs05,929275.t2zpbs05,929288.t2zpbs05,929289.t2zpbs05
929290.t2zpbs05,929291.t2zpbs05,929298.t2zpbs05,929299.t2zpbs05,929304.t2zpbs05
929266.t2zpbs05,929267.t2zpbs05,929303.t2zpbs05,929270.t2zpbs05,929268.t2zpbs05
929271.t2zpbs05,929273.t2zpbs05,929306.t2zpbs05,929308.t2zpbs05,929313.t2zpbs05

### U ##########################################
1750121.t2zpbs-vm1,1750117.t2zpbs-vm1

### V ##########################################
1750136.t2zpbs-vm1,1750138.t2zpbs-vm1,1750139.t2zpbs-vm1,1742259.t2zpbs-vm1,1742254.t2zpbs-vm1
1748839.t2zpbs-vm1,1748886.t2zpbs-vm1,1748793.t2zpbs-vm1,1742295.t2zpbs-vm1,1742277.t2zpbs-vm1
1748812.t2zpbs-vm1,1742302.t2zpbs-vm1,1748794.t2zpbs-vm1,1748669.t2zpbs-vm1,1748795.t2zpbs-vm1
1742252.t2zpbs-vm1,1742284.t2zpbs-vm1,1748849.t2zpbs-vm1,1742273.t2zpbs-vm1,1750128.t2zpbs-vm1
1750123.t2zpbs-vm1,1748944.t2zpbs-vm1,1748887.t2zpbs-vm1,1748893.t2zpbs-vm1,1748894.t2zpbs-vm1
1750126.t2zpbs-vm1,1749052.t2zpbs-vm1,1748677.t2zpbs-vm1,1749073.t2zpbs-vm1,1750135.t2zpbs-vm1
1749095.t2zpbs-vm1,1748922.t2zpbs-vm1,1748923.t2zpbs-vm1,1748927.t2zpbs-vm1,1748928.t2zpbs-vm1
1748946.t2zpbs-vm1,1748952.t2zpbs-vm1,1748951.t2zpbs-vm1,1748953.t2zpbs-vm1,1748954.t2zpbs-vm1
1748955.t2zpbs-vm1,1748956.t2zpbs-vm1,1748957.t2zpbs-vm1,1748965.t2zpbs-vm1,1748973.t2zpbs-vm1
1748976.t2zpbs-vm1,1748988.t2zpbs-vm1,1749214.t2zpbs-vm1,1749030.t2zpbs-vm1,1749031.t2zpbs-vm1
1749226.t2zpbs-vm1,1749037.t2zpbs-vm1,1749040.t2zpbs-vm1,1749042.t2zpbs-vm1,1749248.t2zpbs-vm1
1749050.t2zpbs-vm1,1748896.t2zpbs-vm1,1749061.t2zpbs-vm1,1749062.t2zpbs-vm1,1749063.t2zpbs-vm1
1749068.t2zpbs-vm1,1749071.t2zpbs-vm1,1749081.t2zpbs-vm1,1749310.t2zpbs-vm1,1750111.t2zpbs-vm1
1749322.t2zpbs-vm1,1749697.t2zpbs-vm1,1749760.t2zpbs-vm1,1749732.t2zpbs-vm1,1749187.t2zpbs-vm1
1749363.t2zpbs-vm1,1749693.t2zpbs-vm1,1749377.t2zpbs-vm1,1749711.t2zpbs-vm1,1749728.t2zpbs-vm1
1749715.t2zpbs-vm1,1749721.t2zpbs-vm1,1749708.t2zpbs-vm1,1749770.t2zpbs-vm1,1749704.t2zpbs-vm1
1749670.t2zpbs-vm1,1749410.t2zpbs-vm1,1749767.t2zpbs-vm1,1749699.t2zpbs-vm1,1749695.t2zpbs-vm1
1749716.t2zpbs-vm1,1749651.t2zpbs-vm1,1749722.t2zpbs-vm1,1749733.t2zpbs-vm1,1749681.t2zpbs-vm1
1749749.t2zpbs-vm1,1749707.t2zpbs-vm1,1749675.t2zpbs-vm1,1749664.t2zpbs-vm1,1749742.t2zpbs-vm1
1749727.t2zpbs-vm1,1749771.t2zpbs-vm1,1749724.t2zpbs-vm1,1749705.t2zpbs-vm1,1749706.t2zpbs-vm1
1749679.t2zpbs-vm1,1749764.t2zpbs-vm1,1749758.t2zpbs-vm1,1749731.t2zpbs-vm1,1749725.t2zpbs-vm1
1749734.t2zpbs-vm1,1749468.t2zpbs-vm1,1749684.t2zpbs-vm1,1749714.t2zpbs-vm1,1749447.t2zpbs-vm1
1749745.t2zpbs-vm1,1749673.t2zpbs-vm1,1749730.t2zpbs-vm1,1749661.t2zpbs-vm1,1749718.t2zpbs-vm1
1749672.t2zpbs-vm1,1749755.t2zpbs-vm1,1749738.t2zpbs-vm1,1749729.t2zpbs-vm1,1749677.t2zpbs-vm1
1749719.t2zpbs-vm1,1749657.t2zpbs-vm1,1749726.t2zpbs-vm1,1749739.t2zpbs-vm1,1749683.t2zpbs-vm1
1749700.t2zpbs-vm1,1749717.t2zpbs-vm1,1749659.t2zpbs-vm1,1749694.t2zpbs-vm1,1749720.t2zpbs-vm1
1749748.t2zpbs-vm1,1749674.t2zpbs-vm1,1749698.t2zpbs-vm1,1749723.t2zpbs-vm1,1749669.t2zpbs-vm1
1749735.t2zpbs-vm1,1749701.t2zpbs-vm1,1749649.t2zpbs-vm1,1749743.t2zpbs-vm1,1749737.t2zpbs-vm1
1749744.t2zpbs-vm1,1749662.t2zpbs-vm1,1749666.t2zpbs-vm1,1749710.t2zpbs-vm1,1749658.t2zpbs-vm1
1749713.t2zpbs-vm1,1749709.t2zpbs-vm1,1749703.t2zpbs-vm1,1749702.t2zpbs-vm1,1749736.t2zpbs-vm1
1749763.t2zpbs-vm1,1749650.t2zpbs-vm1,1749663.t2zpbs-vm1,1749354.t2zpbs-vm1,1749686.t2zpbs-vm1
1749671.t2zpbs-vm1,1749685.t2zpbs-vm1,1749652.t2zpbs-vm1,1749688.t2zpbs-vm1,1749680.t2zpbs-vm1
1749660.t2zpbs-vm1,1749696.t2zpbs-vm1,1749712.t2zpbs-vm1,1749692.t2zpbs-vm1,1749750.t2zpbs-vm1
1749654.t2zpbs-vm1,1750010.t2zpbs-vm1,1749665.t2zpbs-vm1,1749678.t2zpbs-vm1,1749682.t2zpbs-vm1
1749667.t2zpbs-vm1,1749687.t2zpbs-vm1,1749653.t2zpbs-vm1,1749753.t2zpbs-vm1,1749754.t2zpbs-vm1
1749691.t2zpbs-vm1,1749655.t2zpbs-vm1,1749747.t2zpbs-vm1,1749589.t2zpbs-vm1,1749676.t2zpbs-vm1
1749601.t2zpbs-vm1,1749690.t2zpbs-vm1,1749656.t2zpbs-vm1,1749647.t2zpbs-vm1,1749752.t2zpbs-vm1
1749762.t2zpbs-vm1,1749645.t2zpbs-vm1,1749648.t2zpbs-vm1,1749643.t2zpbs-vm1,1749761.t2zpbs-vm1
1749646.t2zpbs-vm1,1749759.t2zpbs-vm1,1749644.t2zpbs-vm1,1749740.t2zpbs-vm1,1749766.t2zpbs-vm1
1749746.t2zpbs-vm1,1749668.t2zpbs-vm1,1749751.t2zpbs-vm1,1749756.t2zpbs-vm1,1749757.t2zpbs-vm1
1749741.t2zpbs-vm1,1749768.t2zpbs-vm1,1749769.t2zpbs-vm1,1749765.t2zpbs-vm1,1748653.t2zpbs-vm1
1748726.t2zpbs-vm1,1748772.t2zpbs-vm1,1748756.t2zpbs-vm1,1748718.t2zpbs-vm1,1748697.t2zpbs-vm1
1748774.t2zpbs-vm1,1748717.t2zpbs-vm1,1748771.t2zpbs-vm1,1748760.t2zpbs-vm1,1748708.t2zpbs-vm1
1748715.t2zpbs-vm1,1748747.t2zpbs-vm1,1748759.t2zpbs-vm1,1748767.t2zpbs-vm1,1748770.t2zpbs-vm1
1748724.t2zpbs-vm1,1748751.t2zpbs-vm1,1748723.t2zpbs-vm1,1748775.t2zpbs-vm1,1748730.t2zpbs-vm1
1748714.t2zpbs-vm1,1748732.t2zpbs-vm1,1748712.t2zpbs-vm1,1748731.t2zpbs-vm1,1748704.t2zpbs-vm1
1750017.t2zpbs-vm1,1750015.t2zpbs-vm1,1750023.t2zpbs-vm1,1750020.t2zpbs-vm1,1750019.t2zpbs-vm1
1750014.t2zpbs-vm1,1750021.t2zpbs-vm1,1750024.t2zpbs-vm1,1750022.t2zpbs-vm1,1750012.t2zpbs-vm1
1750013.t2zpbs-vm1,1748758.t2zpbs-vm1,1750011.t2zpbs-vm1,1750016.t2zpbs-vm1,1748762.t2zpbs-vm1
1750116.t2zpbs-vm1,1750122.t2zpbs-vm1,1750124.t2zpbs-vm1,1750125.t2zpbs-vm1,1750127.t2zpbs-vm1
1750132.t2zpbs-vm1,1749029.t2zpbs-vm1,1748328.t2zpbs-vm1,1748329.t2zpbs-vm1,1748327.t2zpbs-vm1
1747675.t2zpbs-vm1,1748797.t2zpbs-vm1