TSUBAME2 Computing Service Classes

The 1400+ computing nodes of TSUBAME2 can be used with the following service classes:

Name Usage Application Types Charging
Interactive service This is the case when the user logs in to TSUBAME and runs normal shell sessions.  Program development, job management. Long-running programs are not allowed to run here (run-time is limited to 30 min) Free
Reservation-based service This is one of the most common cases of usage. A desired number of computing nodes can be reserved through the TSUBAME portal website. The reservation duration can be as short as one day and as long as seven days. The actual maximum capacity of machines that can be reserved at once is limited to 600 node-days for the fairness among the users. Best suited for large-scale runs Fee-based
Batch-based service In this service, each user program is executed one by one. There are two types of this service: node exclusive and node sharing. In the former, each user job exclusively uses each node, whereas in the latter multiple non-related jobs may share a single node, which might cause some performance interference. Best suited for medium-class jobs Fee-based with a partial free service
TSUBAME Grand Challenge (planned) This is still being discussed, but basically we plan to allocate a significant part (say, 1000+ nodes) of the overall TSUBAME2 to a single job to run very large parallel programs. The details of this service will be announced later. TBD TBD

Below is an overview of each service. See the user guide for more details.

Reservation-based Usage

You can reserve a desired number of nodes for desired days by selecting available slots from a web-based reservation calendar. Go to the TSUBAME portal to do reservation.

This service is based on the HPC queue in TSUBAME1, but is significantly extended for more flexible resource usage. Specifically, now you can reserve as many nodes as you want within the range of 16 to 400 nodes, and the duration can be as short as 1 day and as long as 7 days. As the previous HPC queue, this usage is best suited to very large-scale runs. Also even for small scale jobs, this would be more convenient for programs that are difficult to run as standard batch jobs running autonomously but require manual interactions, because with this service, you can be certain a priori when the service is available for use, whereas in the batch-based service, it depends on how many jobs are waiting before your job.

Batch-based Usage

This provides compute nodes to user programs in a first-come-first-served way. You specify a program you want to run and its requirements such as the number of processors and memory capacity. This is called a "job." User jobs are managed by a system called job scheduler, which runs them on compute nodes one by one.

There are multiple batch services called "queues" for differnet job requirements:

  • Node-exclusive queues (S, S96, L128, L128F, L256, L512)
  • Node-sharing queue (U, V)
  • GPU queue (G)
  • Windows queue

In the node-exclusive mode, each user program uses compute nodes exclusively. This corresponds to the SLA service in TSUBAME1. In TSUBAME2, there are 5 queues of this type with different memroy capacity: S, S96, L128, L128F, L256, and L512.

In the node-sharing mode, each user program shares a single physical node with another programs by hardware virualization. This service corresponds to the BES queue in TSUBAME1. This service mode is best suited to the case where a large number of small jobs need to be processed, such as array jobs. On the other hand, because of virtualization overheads, applications with heavy network and storage requirements may not particularly run efficiently. In addition, because of a limitation in the virualization system, this service does not support GPU programs.

The GPU queue allows user jobs to use 3 GPUs and 4 CPU cores per node. This is best suited to GPU-centric programs. Compute nodes used for this queue actually shares the same hardware as the V queue by virtualization.

Free Trial Queue

Although using the batch queues is charged, there is a small for-free queue for program development and testing under the following limitations:

  • U, S, S96, L128, L128F, L256 and L512 queue 
    • Up to 2 nodes can be used. This is exclusive usage, so all 24 CPU cores can be used.
    • Job execution time is limited to 10 minutes.
  • V queue
    • Up to 2 CPUs can be used. This is node sharing usage.
    • Job execution time is limited to 10 minutes.

To submit a job to the trial queue:

t2sub -q S -l options job-script-file

I.e., specify "S" as the queue name and do not pass group names. For example, this will run "job_script.sh" job using 2 nodes with 12 processes per node.

t2sub -q S -l select=2:mpiprocs=12 -l place=scatter job_script.sh

Grand Challenge Usage

TSUBAME共同利用 グランドチャレンジ大規模計算制度をご覧ください。