このページの更新は終了しました。

最新の情報はTSUBAME3.0計算サービスのWebページをご覧ください。

TSUBAME2.5からTSUBAME3.0へのデータ移行方法の資料はこちら

【障害報告】2017.2.3発生:/work1, /gscr0 ストレージ障害

(グループディスクを利用している方向けの情報です)

                             学術国際情報センター
2017年2月9日

 下記の通りストレージに障害が発生し、仮復旧いたしました。報告が遅くなり申し訳ありませんでした。

 時間帯と影響範囲:
 2月3日(金) 12:36 から 14:49 /work1利用不可
 2月3日(金) 14:31 から 14:49 /work1, /gscr0利用不可

 詳細:

/work1 を構成する2台で冗長化されている MDS(MetaData Server) として稼働していた t2s007015 において Lustre の処理がスタックする問題が発生致しました。

MDS の処理がスタックしたため、Lustre クライアントである各計算ノードからは/work1 のメタデータを取得できなくなりました。その結果、/work1 にアクセスができなくなる影響が発生しました。

復旧のため、MDS のサービス(MDT, MetaData Target)を対となる t2s007013(こちらでは/gscr0が稼働) に手動でフェイルオーバーさせたところ、t2s007013で同様の問題(Lustre処理のスタック)が発生致しました。

そのため、t2s007013 から t2s007015へ逆に再度フェイルオーバーさせたところ復旧することに成功しました。t2s007013 では/gscr0のMDTが稼働してたため、本作業時に/gscr0にアクセスできなくなる影響がでました。

現在、/work1の MDSのサービス は本来の t2s007015 で稼働しており、/gscr0はフェイルオーバー先である  同じくt2s007015 で稼働しています。

つまり、サービスが片寄状態で仮普及となっており、/gscr0を本来の t2s007013 に戻す作業は2月17日(金)のTSUBAMEの全停止時を予定しています。

なお、片寄状態となっておりますが、/gscr0は規模も小さくテンポラリな領域なのでパフォーマンスには大きな影響はないと考えています。

[備考]
2016年5月11日に発生した障害と同様のバグと思われます。
http://tsubame.gsic.titech.ac.jp/node/1461

用語説明

MDS:Lustreファイルシステムにおいて、計算ノードに対してファイルのメタ情報を実際に送受信するサーバ

MDT:Lustreファイルシステムにおいて、ファイルのメタ情報を実際に保存しているディスクの集合体


 なお、影響があったと思われるジョブ(障害発生期間中に実行中であったジョブ)の一覧は下記の通りです。

<Sキュー(297ジョブ)>
1972876.t2zpbs01, 1972877.t2zpbs01, 1972911.t2zpbs01, 1972927.t2zpbs01, 1983478.t2zpbs01
1983479.t2zpbs01, 1999697.t2zpbs01, 2000613.t2zpbs01, 2000678.t2zpbs01, 2000695.t2zpbs01
2000704.t2zpbs01, 2004337.t2zpbs01, 2004979.t2zpbs01, 2005038.t2zpbs01, 2005131.t2zpbs01
2005287.t2zpbs01, 2005311.t2zpbs01, 2005328.t2zpbs01, 2005329.t2zpbs01, 2005648.t2zpbs01
2005649.t2zpbs01, 2005656.t2zpbs01, 2005657.t2zpbs01, 2005659.t2zpbs01, 2006505.t2zpbs01
2006510.t2zpbs01, 2006512.t2zpbs01, 2006514.t2zpbs01, 2006515.t2zpbs01, 2006533.t2zpbs01
2006539.t2zpbs01, 2006549.t2zpbs01, 2006585.t2zpbs01, 2006624.t2zpbs01, 2006636.t2zpbs01
2006654.t2zpbs01, 2006711.t2zpbs01, 2006714.t2zpbs01, 2006717.t2zpbs01, 2006723.t2zpbs01
2006725.t2zpbs01, 2006726.t2zpbs01, 2006731.t2zpbs01, 2006732.t2zpbs01, 2006733.t2zpbs01
2006734.t2zpbs01, 2006735.t2zpbs01, 2006737.t2zpbs01, 2006738.t2zpbs01, 2006739.t2zpbs01
2006740.t2zpbs01, 2006741.t2zpbs01, 2006742.t2zpbs01, 2006743.t2zpbs01, 2006745.t2zpbs01
2006747.t2zpbs01, 2007333.t2zpbs01, 2007335.t2zpbs01, 2007336.t2zpbs01, 2007338.t2zpbs01
2007339.t2zpbs01, 2007340.t2zpbs01, 2007341.t2zpbs01, 2007342.t2zpbs01, 2007344.t2zpbs01
2007346.t2zpbs01, 2007347.t2zpbs01, 2007348.t2zpbs01, 2007349.t2zpbs01, 2007350.t2zpbs01
2007351.t2zpbs01, 2007352.t2zpbs01, 2007354.t2zpbs01, 2007355.t2zpbs01, 2007356.t2zpbs01
2007357.t2zpbs01, 2007775.t2zpbs01, 2007794.t2zpbs01, 2007819.t2zpbs01, 2007820.t2zpbs01
2007829.t2zpbs01, 2007830.t2zpbs01, 2007831.t2zpbs01, 2007838.t2zpbs01, 2007841.t2zpbs01
2007844.t2zpbs01, 2007848.t2zpbs01, 2007849.t2zpbs01, 2007911.t2zpbs01, 2007982.t2zpbs01
2007983.t2zpbs01, 2007984.t2zpbs01, 2007985.t2zpbs01, 2007997.t2zpbs01, 2007999.t2zpbs01
2008001.t2zpbs01, 2008003.t2zpbs01, 2008005.t2zpbs01, 2008007.t2zpbs01, 2008016.t2zpbs01
2008043.t2zpbs01, 2008055.t2zpbs01, 2008057.t2zpbs01, 2008075.t2zpbs01, 2008091.t2zpbs01
2008092.t2zpbs01, 2008093.t2zpbs01, 2008095.t2zpbs01, 2008097.t2zpbs01, 2008098.t2zpbs01
2008099.t2zpbs01, 2008100.t2zpbs01, 2008103.t2zpbs01, 2008104.t2zpbs01, 2008105.t2zpbs01
2008107.t2zpbs01, 2008114.t2zpbs01, 2008115.t2zpbs01, 2008116.t2zpbs01, 2008117.t2zpbs01
2008193.t2zpbs01, 2008201.t2zpbs01, 2008209.t2zpbs01, 2008212.t2zpbs01, 2008213.t2zpbs01
2008216.t2zpbs01, 2008218.t2zpbs01, 2008219.t2zpbs01, 2008220.t2zpbs01, 2008221.t2zpbs01
2008229.t2zpbs01, 2008230.t2zpbs01, 2008241.t2zpbs01, 2008243.t2zpbs01, 2008244.t2zpbs01
2008245.t2zpbs01, 2008252.t2zpbs01, 2008259.t2zpbs01, 2008296.t2zpbs01, 2008299.t2zpbs01
2008310.t2zpbs01, 2008323.t2zpbs01, 2008335.t2zpbs01, 2008384.t2zpbs01, 2008386.t2zpbs01
2008394.t2zpbs01, 2008406.t2zpbs01, 2008475.t2zpbs01, 2008481.t2zpbs01, 2008490.t2zpbs01
2008497.t2zpbs01, 2008564.t2zpbs01, 2008570.t2zpbs01, 2008571.t2zpbs01, 2008572.t2zpbs01
2008573.t2zpbs01, 2008595.t2zpbs01, 2008597.t2zpbs01, 2008598.t2zpbs01, 2008604.t2zpbs01
2008605.t2zpbs01, 2008610.t2zpbs01, 2008622.t2zpbs01, 2008628.t2zpbs01, 2008669.t2zpbs01
2008671.t2zpbs01, 2008672.t2zpbs01, 2008673.t2zpbs01, 2008674.t2zpbs01, 2008675.t2zpbs01
2008676.t2zpbs01, 2008677.t2zpbs01, 2008679.t2zpbs01, 2008680.t2zpbs01, 2008681.t2zpbs01
2008682.t2zpbs01, 2008683.t2zpbs01, 2008684.t2zpbs01, 2008685.t2zpbs01, 2008688.t2zpbs01
2008689.t2zpbs01, 2008695.t2zpbs01, 2008696.t2zpbs01, 2008700.t2zpbs01, 2008704.t2zpbs01
2008751.t2zpbs01, 2008769.t2zpbs01, 2008773.t2zpbs01, 2008774.t2zpbs01, 2008783.t2zpbs01
2008786.t2zpbs01, 2008787.t2zpbs01, 2008793.t2zpbs01, 2008794.t2zpbs01, 2008795.t2zpbs01
2008796.t2zpbs01, 2008797.t2zpbs01, 2008798.t2zpbs01, 2008799.t2zpbs01, 2008800.t2zpbs01
2008801.t2zpbs01, 2008802.t2zpbs01, 2008803.t2zpbs01, 2008804.t2zpbs01, 2008805.t2zpbs01
2008806.t2zpbs01, 2008807.t2zpbs01, 2008808.t2zpbs01, 2008809.t2zpbs01, 2008812.t2zpbs01
2008813.t2zpbs01, 2008814.t2zpbs01, 2008815.t2zpbs01, 2008816.t2zpbs01, 2008817.t2zpbs01
2008819.t2zpbs01, 2008820.t2zpbs01, 2008822.t2zpbs01, 2008824.t2zpbs01, 2008826.t2zpbs01
2008827.t2zpbs01, 2008828.t2zpbs01, 2008850.t2zpbs01, 2008852.t2zpbs01, 2008878.t2zpbs01
2008879.t2zpbs01, 2008885[10].t2zpbs01, 2008885[11].t2zpbs01, 2008885[12].t2zpbs01, 2008885[13].t2zpbs01
2008885[14].t2zpbs01, 2008885[15].t2zpbs01, 2008885[16].t2zpbs01, 2008885[17].t2zpbs01, 2008885[18].t2zpbs01
2008885[19].t2zpbs01, 2008885[1].t2zpbs01, 2008885[20].t2zpbs01, 2008885[2].t2zpbs01, 2008885[3].t2zpbs01
2008885[4].t2zpbs01, 2008885[5].t2zpbs01, 2008885[6].t2zpbs01, 2008885[7].t2zpbs01, 2008885[8].t2zpbs01
2008885[9].t2zpbs01, 2008886.t2zpbs01, 2008889.t2zpbs01, 2008890.t2zpbs01, 2008892.t2zpbs01
2008898.t2zpbs01, 2008906.t2zpbs01, 2008909.t2zpbs01, 2008910.t2zpbs01, 2008911.t2zpbs01
2008913.t2zpbs01, 2008914.t2zpbs01, 2008915.t2zpbs01, 2008916.t2zpbs01, 2008917.t2zpbs01
2008918.t2zpbs01, 2008919.t2zpbs01, 2008920.t2zpbs01, 2008921.t2zpbs01, 2008922.t2zpbs01
2008923.t2zpbs01, 2008924.t2zpbs01, 2008925.t2zpbs01, 2008928.t2zpbs01, 2008929.t2zpbs01
2008936.t2zpbs01, 2008937.t2zpbs01, 2008938.t2zpbs01, 2008939.t2zpbs01, 2008940.t2zpbs01
2008941.t2zpbs01, 2008942.t2zpbs01, 2008943.t2zpbs01, 2008944.t2zpbs01, 2008945.t2zpbs01
2008946.t2zpbs01, 2008947.t2zpbs01, 2008948.t2zpbs01, 2008949.t2zpbs01, 2008950.t2zpbs01
2008951.t2zpbs01, 2008952.t2zpbs01, 2008954.t2zpbs01, 2008955.t2zpbs01, 2008957.t2zpbs01
2008995.t2zpbs01, 2009003.t2zpbs01, 2009004.t2zpbs01, 2009005.t2zpbs01, 2009012.t2zpbs01
2009013.t2zpbs01, 2009017.t2zpbs01

<S96キュー(17ジョブ)>
2000574.t2zpbs01, 2005043.t2zpbs01, 2005280.t2zpbs01, 2005744.t2zpbs01, 2005745.t2zpbs01
2007917.t2zpbs01, 2007920.t2zpbs01, 2007926.t2zpbs01, 2007951.t2zpbs01, 2007964.t2zpbs01
2007987.t2zpbs01, 2007988.t2zpbs01, 2007989.t2zpbs01, 2007990.t2zpbs01, 2007991.t2zpbs01
2007992.t2zpbs01, 2008763.t2zpbs01

<Gキュー(180ジョブ)>
2103777.t2zpbs03, 2104499.t2zpbs03, 2104505.t2zpbs03, 2104506.t2zpbs03, 2104509.t2zpbs03
2104543.t2zpbs03, 2104575.t2zpbs03, 2104576.t2zpbs03, 2104577.t2zpbs03, 2104578.t2zpbs03
2104581.t2zpbs03, 2104590.t2zpbs03, 2104591.t2zpbs03, 2104605.t2zpbs03, 2104606.t2zpbs03
2104607.t2zpbs03, 2104668.t2zpbs03, 2104688.t2zpbs03, 2104728.t2zpbs03, 2104729.t2zpbs03
2104730.t2zpbs03, 2104731.t2zpbs03, 2104733.t2zpbs03, 2104734.t2zpbs03, 2104735.t2zpbs03
2104736.t2zpbs03, 2104746.t2zpbs03, 2104747.t2zpbs03, 2104749.t2zpbs03, 2104750.t2zpbs03
2104751.t2zpbs03, 2104764.t2zpbs03, 2104770.t2zpbs03, 2104771.t2zpbs03, 2104772.t2zpbs03
2104773.t2zpbs03, 2104774.t2zpbs03, 2104775.t2zpbs03, 2104776.t2zpbs03, 2104777.t2zpbs03
2104778.t2zpbs03, 2104779.t2zpbs03, 2104780.t2zpbs03, 2104781.t2zpbs03, 2104782.t2zpbs03
2104783.t2zpbs03, 2104784.t2zpbs03, 2104785.t2zpbs03, 2104786.t2zpbs03, 2104787.t2zpbs03
2104789.t2zpbs03, 2104790.t2zpbs03, 2104791.t2zpbs03, 2104792.t2zpbs03, 2104793.t2zpbs03
2104794.t2zpbs03, 2104795.t2zpbs03, 2104796.t2zpbs03, 2104797.t2zpbs03, 2104798.t2zpbs03
2104799.t2zpbs03, 2104800.t2zpbs03, 2104801.t2zpbs03, 2104803.t2zpbs03, 2104804.t2zpbs03
2104805.t2zpbs03, 2104806.t2zpbs03, 2104807.t2zpbs03, 2104808.t2zpbs03, 2104809.t2zpbs03
2104810.t2zpbs03, 2104811.t2zpbs03, 2104812.t2zpbs03, 2104813.t2zpbs03, 2104814.t2zpbs03
2104815.t2zpbs03, 2104824.t2zpbs03, 2104825.t2zpbs03, 2104826.t2zpbs03, 2104840.t2zpbs03
2104841.t2zpbs03, 2104842.t2zpbs03, 2104843.t2zpbs03, 2104844.t2zpbs03, 2104845.t2zpbs03
2104846.t2zpbs03, 2104847.t2zpbs03, 2104848.t2zpbs03, 2104849.t2zpbs03, 2104856.t2zpbs03
2104857.t2zpbs03, 2104858.t2zpbs03, 2104859.t2zpbs03, 2104860.t2zpbs03, 2104861.t2zpbs03
2104862.t2zpbs03, 2104863.t2zpbs03, 2104864.t2zpbs03, 2104865.t2zpbs03, 2104866.t2zpbs03
2104867.t2zpbs03, 2104871[102].t2zpbs03, 2104871[106].t2zpbs03, 2104871[10].t2zpbs03, 2104871[110].t2zpbs03
2104871[114].t2zpbs03, 2104871[118].t2zpbs03, 2104871[14].t2zpbs03, 2104871[18].t2zpbs03, 2104871[22].t2zpbs03
2104871[26].t2zpbs03, 2104871[2].t2zpbs03, 2104871[30].t2zpbs03, 2104871[34].t2zpbs03, 2104871[38].t2zpbs03
2104871[42].t2zpbs03, 2104871[46].t2zpbs03, 2104871[50].t2zpbs03, 2104871[54].t2zpbs03, 2104871[58].t2zpbs03
2104871[62].t2zpbs03, 2104871[66].t2zpbs03, 2104871[6].t2zpbs03, 2104871[70].t2zpbs03, 2104871[74].t2zpbs03
2104871[78].t2zpbs03, 2104871[82].t2zpbs03, 2104871[86].t2zpbs03, 2104871[90].t2zpbs03, 2104871[94].t2zpbs03
2104871[98].t2zpbs03, 2104872[0].t2zpbs03, 2104872[1].t2zpbs03, 2104872[2].t2zpbs03, 2104872[3].t2zpbs03
2104872[4].t2zpbs03, 2104873[0].t2zpbs03, 2104873[1].t2zpbs03, 2104873[2].t2zpbs03, 2104873[3].t2zpbs03
2104873[4].t2zpbs03, 2104874[0].t2zpbs03, 2104874[1].t2zpbs03, 2104874[2].t2zpbs03, 2104874[3].t2zpbs03
2104874[4].t2zpbs03, 2104877.t2zpbs03, 2104882.t2zpbs03, 2104890.t2zpbs03, 2104891.t2zpbs03
2104892.t2zpbs03, 2104901.t2zpbs03, 2104904.t2zpbs03, 2104905.t2zpbs03, 2104907.t2zpbs03
2104909.t2zpbs03, 2104910.t2zpbs03, 2104911.t2zpbs03, 2104912.t2zpbs03, 2104913.t2zpbs03
2104914.t2zpbs03, 2104915.t2zpbs03, 2104916.t2zpbs03, 2104917.t2zpbs03, 2104918.t2zpbs03
2104919.t2zpbs03, 2104920.t2zpbs03, 2104921.t2zpbs03, 2104922.t2zpbs03, 2104925.t2zpbs03
2104929.t2zpbs03, 2104930.t2zpbs03, 2104931.t2zpbs03, 2104932.t2zpbs03, 2104934.t2zpbs03
2104935.t2zpbs03, 2104936.t2zpbs03, 2104937.t2zpbs03, 2104938.t2zpbs03, 2104939.t2zpbs03

<Vキュー(319ジョブ)>
2386862.t2zpbs-vm1, 2386863.t2zpbs-vm1, 2386864.t2zpbs-vm1, 2386865.t2zpbs-vm1, 2388151.t2zpbs-vm1
2388152.t2zpbs-vm1, 2388168.t2zpbs-vm1, 2388169.t2zpbs-vm1, 2390621.t2zpbs-vm1, 2390663.t2zpbs-vm1
2390916.t2zpbs-vm1, 2390922.t2zpbs-vm1, 2390924.t2zpbs-vm1, 2390925.t2zpbs-vm1, 2390926.t2zpbs-vm1
2390927.t2zpbs-vm1, 2390928.t2zpbs-vm1, 2390929.t2zpbs-vm1, 2390930.t2zpbs-vm1, 2390931.t2zpbs-vm1
2390932.t2zpbs-vm1, 2390933.t2zpbs-vm1, 2390934.t2zpbs-vm1, 2390935.t2zpbs-vm1, 2390937.t2zpbs-vm1
2390938.t2zpbs-vm1, 2390939.t2zpbs-vm1, 2390940.t2zpbs-vm1, 2390941.t2zpbs-vm1, 2390942.t2zpbs-vm1
2390943.t2zpbs-vm1, 2390944.t2zpbs-vm1, 2390945.t2zpbs-vm1, 2390946.t2zpbs-vm1, 2390947.t2zpbs-vm1
2390948.t2zpbs-vm1, 2390949.t2zpbs-vm1, 2390950.t2zpbs-vm1, 2390951.t2zpbs-vm1, 2390952.t2zpbs-vm1
2390953.t2zpbs-vm1, 2390954.t2zpbs-vm1, 2390955.t2zpbs-vm1, 2390956.t2zpbs-vm1, 2390957.t2zpbs-vm1
2390958.t2zpbs-vm1, 2390959.t2zpbs-vm1, 2390960.t2zpbs-vm1, 2390961.t2zpbs-vm1, 2390962.t2zpbs-vm1
2390963.t2zpbs-vm1, 2390964.t2zpbs-vm1, 2390965.t2zpbs-vm1, 2390966.t2zpbs-vm1, 2390967.t2zpbs-vm1
2390968.t2zpbs-vm1, 2390969.t2zpbs-vm1, 2390970.t2zpbs-vm1, 2390971.t2zpbs-vm1, 2390972.t2zpbs-vm1
2390973.t2zpbs-vm1, 2390974.t2zpbs-vm1, 2390975.t2zpbs-vm1, 2390976.t2zpbs-vm1, 2390977.t2zpbs-vm1
2390978.t2zpbs-vm1, 2390979.t2zpbs-vm1, 2390980.t2zpbs-vm1, 2390992.t2zpbs-vm1, 2390994.t2zpbs-vm1
2390995.t2zpbs-vm1, 2390999.t2zpbs-vm1, 2391001.t2zpbs-vm1, 2391004.t2zpbs-vm1, 2391007.t2zpbs-vm1
2391008.t2zpbs-vm1, 2391058.t2zpbs-vm1, 2391065.t2zpbs-vm1, 2391070.t2zpbs-vm1, 2391071.t2zpbs-vm1
2391074.t2zpbs-vm1, 2391075.t2zpbs-vm1, 2391076.t2zpbs-vm1, 2391161.t2zpbs-vm1, 2391334.t2zpbs-vm1
2391344.t2zpbs-vm1, 2391351.t2zpbs-vm1, 2391352.t2zpbs-vm1, 2391368.t2zpbs-vm1, 2391369.t2zpbs-vm1
2391370.t2zpbs-vm1, 2391372.t2zpbs-vm1, 2391384.t2zpbs-vm1, 2391387.t2zpbs-vm1, 2391392.t2zpbs-vm1
2391409.t2zpbs-vm1, 2391418.t2zpbs-vm1, 2391420.t2zpbs-vm1, 2391421.t2zpbs-vm1, 2391425.t2zpbs-vm1
2391433.t2zpbs-vm1, 2391448.t2zpbs-vm1, 2391451.t2zpbs-vm1, 2391517.t2zpbs-vm1, 2391518.t2zpbs-vm1
2391522.t2zpbs-vm1, 2391536.t2zpbs-vm1, 2391537.t2zpbs-vm1, 2391538.t2zpbs-vm1, 2391541.t2zpbs-vm1
2391543.t2zpbs-vm1, 2391545.t2zpbs-vm1, 2391546.t2zpbs-vm1, 2391548.t2zpbs-vm1, 2391549.t2zpbs-vm1
2391550.t2zpbs-vm1, 2391551.t2zpbs-vm1, 2391552.t2zpbs-vm1, 2391553.t2zpbs-vm1, 2391554.t2zpbs-vm1
2391555.t2zpbs-vm1, 2391556.t2zpbs-vm1, 2391557.t2zpbs-vm1, 2391558.t2zpbs-vm1, 2391559.t2zpbs-vm1
2391560.t2zpbs-vm1, 2391561.t2zpbs-vm1, 2391562.t2zpbs-vm1, 2391563.t2zpbs-vm1, 2391564.t2zpbs-vm1
2391565.t2zpbs-vm1, 2391566.t2zpbs-vm1, 2391567.t2zpbs-vm1, 2391568.t2zpbs-vm1, 2391569.t2zpbs-vm1
2391570.t2zpbs-vm1, 2391571.t2zpbs-vm1, 2391572.t2zpbs-vm1, 2391573.t2zpbs-vm1, 2391574.t2zpbs-vm1
2391575.t2zpbs-vm1, 2391576.t2zpbs-vm1, 2391577.t2zpbs-vm1, 2391578.t2zpbs-vm1, 2391579.t2zpbs-vm1
2391580.t2zpbs-vm1, 2391581.t2zpbs-vm1, 2391582.t2zpbs-vm1, 2391583.t2zpbs-vm1, 2391584.t2zpbs-vm1
2391585.t2zpbs-vm1, 2391586.t2zpbs-vm1, 2391587.t2zpbs-vm1, 2391588.t2zpbs-vm1, 2391589.t2zpbs-vm1
2391590.t2zpbs-vm1, 2391591.t2zpbs-vm1, 2391592.t2zpbs-vm1, 2391593.t2zpbs-vm1, 2391594.t2zpbs-vm1
2391595.t2zpbs-vm1, 2391596.t2zpbs-vm1, 2391597.t2zpbs-vm1, 2391598.t2zpbs-vm1, 2391599.t2zpbs-vm1
2391603.t2zpbs-vm1, 2391608.t2zpbs-vm1, 2391609.t2zpbs-vm1, 2391610.t2zpbs-vm1, 2391611.t2zpbs-vm1
2391612.t2zpbs-vm1, 2391613.t2zpbs-vm1, 2391614.t2zpbs-vm1, 2391615.t2zpbs-vm1, 2391616.t2zpbs-vm1
2391617.t2zpbs-vm1, 2391618.t2zpbs-vm1, 2391619.t2zpbs-vm1, 2391620.t2zpbs-vm1, 2391621.t2zpbs-vm1
2391622.t2zpbs-vm1, 2391623.t2zpbs-vm1, 2391624.t2zpbs-vm1, 2391625.t2zpbs-vm1, 2391626.t2zpbs-vm1
2391627.t2zpbs-vm1, 2391628.t2zpbs-vm1, 2391629.t2zpbs-vm1, 2391630.t2zpbs-vm1, 2391631.t2zpbs-vm1
2391632.t2zpbs-vm1, 2391633.t2zpbs-vm1, 2391634.t2zpbs-vm1, 2391635.t2zpbs-vm1, 2391636.t2zpbs-vm1
2391637.t2zpbs-vm1, 2391638.t2zpbs-vm1, 2391639.t2zpbs-vm1, 2391640.t2zpbs-vm1, 2391641.t2zpbs-vm1
2391642.t2zpbs-vm1, 2391643.t2zpbs-vm1, 2391644.t2zpbs-vm1, 2391645.t2zpbs-vm1, 2391646.t2zpbs-vm1
2391647.t2zpbs-vm1, 2391648.t2zpbs-vm1, 2391649.t2zpbs-vm1, 2391650.t2zpbs-vm1, 2391651.t2zpbs-vm1
2391652.t2zpbs-vm1, 2391653.t2zpbs-vm1, 2391654.t2zpbs-vm1, 2391655.t2zpbs-vm1, 2391656.t2zpbs-vm1
2391657.t2zpbs-vm1, 2391658.t2zpbs-vm1, 2391659.t2zpbs-vm1, 2391660.t2zpbs-vm1, 2391661.t2zpbs-vm1
2391662.t2zpbs-vm1, 2391663.t2zpbs-vm1, 2391664.t2zpbs-vm1, 2391665.t2zpbs-vm1, 2391666.t2zpbs-vm1
2391667.t2zpbs-vm1, 2391668.t2zpbs-vm1, 2391669.t2zpbs-vm1, 2391670.t2zpbs-vm1, 2391671.t2zpbs-vm1
2391672.t2zpbs-vm1, 2391673.t2zpbs-vm1, 2391674.t2zpbs-vm1, 2391675.t2zpbs-vm1, 2391676.t2zpbs-vm1
2391677.t2zpbs-vm1, 2391678.t2zpbs-vm1, 2391679.t2zpbs-vm1, 2391680.t2zpbs-vm1, 2391681.t2zpbs-vm1
2391682.t2zpbs-vm1, 2391683.t2zpbs-vm1, 2391684.t2zpbs-vm1, 2391685.t2zpbs-vm1, 2391686.t2zpbs-vm1
2391687.t2zpbs-vm1, 2391688.t2zpbs-vm1, 2391689.t2zpbs-vm1, 2391690.t2zpbs-vm1, 2391692.t2zpbs-vm1
2391693.t2zpbs-vm1, 2391694.t2zpbs-vm1, 2391695.t2zpbs-vm1, 2391696.t2zpbs-vm1, 2391697.t2zpbs-vm1
2391698.t2zpbs-vm1, 2391699.t2zpbs-vm1, 2391700.t2zpbs-vm1, 2391701.t2zpbs-vm1, 2391702.t2zpbs-vm1
2391703.t2zpbs-vm1, 2391704.t2zpbs-vm1, 2391705.t2zpbs-vm1, 2391706.t2zpbs-vm1, 2391707.t2zpbs-vm1
2391708.t2zpbs-vm1, 2391709.t2zpbs-vm1, 2391710.t2zpbs-vm1, 2391711.t2zpbs-vm1, 2391712.t2zpbs-vm1
2391713.t2zpbs-vm1, 2391714.t2zpbs-vm1, 2391715.t2zpbs-vm1, 2391716.t2zpbs-vm1, 2391717.t2zpbs-vm1
2391718.t2zpbs-vm1, 2391719.t2zpbs-vm1, 2391720.t2zpbs-vm1, 2391721.t2zpbs-vm1, 2391722.t2zpbs-vm1
2391723.t2zpbs-vm1, 2391724.t2zpbs-vm1, 2391725.t2zpbs-vm1, 2391726.t2zpbs-vm1, 2391727.t2zpbs-vm1
2391728.t2zpbs-vm1, 2391729.t2zpbs-vm1, 2391730.t2zpbs-vm1, 2391731.t2zpbs-vm1, 2391732.t2zpbs-vm1
2391733.t2zpbs-vm1, 2391734.t2zpbs-vm1, 2391735.t2zpbs-vm1, 2391787.t2zpbs-vm1, 2391788.t2zpbs-vm1
2391789.t2zpbs-vm1, 2391790.t2zpbs-vm1, 2391791.t2zpbs-vm1, 2391792.t2zpbs-vm1, 2391793.t2zpbs-vm1
2391794.t2zpbs-vm1, 2391795.t2zpbs-vm1, 2391796.t2zpbs-vm1, 2391798.t2zpbs-vm1, 2391867.t2zpbs-vm1
2391868.t2zpbs-vm1, 2391869.t2zpbs-vm1, 2392054.t2zpbs-vm1, 2392055.t2zpbs-vm1, 2392056.t2zpbs-vm1
2392057.t2zpbs-vm1, 2392058.t2zpbs-vm1, 2392060.t2zpbs-vm1, 2392061.t2zpbs-vm1, 2392062.t2zpbs-vm1
2392063.t2zpbs-vm1, 2392064.t2zpbs-vm1, 2392065.t2zpbs-vm1, 2392066.t2zpbs-vm1

<Uキュー(1ジョブ)>
2391601.t2zpbs-vm1

<L256キュー(3ジョブ)>
2008378.t2zpbs01, 2008699.t2zpbs01, 2008997.t2zpbs01

<L512キュー(2ジョブ)>
2008694.t2zpbs01, 2008823.t2zpbs01

<Hキュー(3ジョブ)>
1115564.t2zpbs05, 1115573.t2zpbs05, 1115579.t2zpbs05

<Xキュー(2ジョブ)>
1115569.t2zpbs05, 1115571.t2zpbs05