このページの更新は終了しました。

最新の情報はTSUBAME3.0計算サービスのWebページをご覧ください。

TSUBAME2.5からTSUBAME3.0へのデータ移行方法の資料はこちら

Tell me about the transfer and copy of the data.

This FAQ explain about the following things.

  • transfer files between local machine and TSUBAME.
  • file copy in the TSUBAME

transfer files between local machine and TSUBAME.

At first, this section describes the following case of file transfer from local machine to TSUBAME.

  • Case1: File transfer from Mac to TSUBAME
  • Case2: File transfer from Windows to TSUBAME

When transfer files, it is necessary to consider about the following things.

  • Remote host (io-t2 or login-t2)
  • Data storage destination (home, work or data area)
  • How to transfer files (software or command)

Case 1: File transfer from Mac to TSUBAME

Suppose transfer from the testdir directory on the Mac to TSUBAME.
It is useful to transfer files by rsync command.
$ rsync -auv testdir login_name@io-t2.g.gsic.titech.ac.jp:/work1/t2g-GROUP/testdir

Even if fail this command during file transfer, you can resume from the middle by running again. I/O node(io-t2.g.gsic.titech.ac.jp) is more suitable for data transfer than Login node(login-t2.g.gsic.titech.ac.jp). Because when using I/O node, transfer time is not limited. In this sample, specify /work1 area as data destination. work area is suitable for high speed file I/O. But when use work area, it is necessary to purchase group disk.

Case 2: File transfer from Windows to TSUBAME

There are a lot of software for file transfer. For example, WinSCP for Windows, CyberDuck for Mac.

Remote host (io-t2 or login-t2)

There are two remote hosts for data transfer. I/O node(io-t2.g.gsic.titech.ac.jp) and Login node(login-t2.g.gsic.titech.ac.jp). Each node have the following characteristics. When transfer a lot of files, it is more better to use I/O node.

item I/O Node Login node
time limitation unlimited Up to 30 minutes
Authentication Key authentication required

(Connection from off-campus) Key authentication required
(Connection from campus) Key authentication , password authentication both possible

Notes

  • When failed to transfer files, please check login to TSUBAME is success or not.
  • It is not able to login TSUBAME when a lot of connection is established in constant time.

references

Data storage destination (home, work or data area)

There are three data storage destination on TSUBAME. home(/home), work(/work0, /work1) and data(/data0). Home area can be used for each user account in non- billing , but there is a capacity limit (25GB). It is not able to increse this capacity upper limit. Therefore, if more space is needed. please consider purchase group disk.

Prohibited matter

  • The execution of the parallel program on home area is prohibited, except for debug purpose short time job. Because home area can not withstand high load by parallel job. In this case, many user are affected, for example, read and write files on home is very slow.

references

How to transfer files

File transfer by software

It is also possible to make use of free software such as CyberDuck and WinSCP.
How to use the software of these refer to the manual of each.

References

File transfer by command

rsync command

Even if advantage, that of rsync stop it on the way, it is what can reopen from the last continuance.

- transfer a file to TSUBAME from one's terminal
$ rsync -auv testdir login_name@io-t2.g.gsic.titech.ac.jp:/work1/t2g-GROUP/testdir

- transfer a file to one's terminal from TSUBAME
$ rsync -auv login_name@t2.g.gsic.titech.ac.jp:/work1/t2g-GROUP/testdir .

scp command
- ・transfer a file to TSUBAME from one's terminal
$ scp xxx.dat login_name@login-t2.g.gsic.titech.ac.jp:/work1/t2g-GROUP/testdir

- transfer a file to one's terminal from TSUBAME
$ scp login_name@login-t2.g.gsic.titech.ac.jp:/work1/t2g-GROUP/testdir .

sftp command

file copy in the TSUBAME

There are several types of the method of copying data. Any command same basic syntax:

[Command] [options] [original data] [after copying]

The operation of these, if it takes the storage limit is raised. Please run after checking these limitations.

Simple copy (move)

$ cp a.txt b.txt
$ mv a.txt b.txt

To hold the information (time stamp)

$ cp -p a.txt b.txt
$ mv a.txt b.txt

Directory for each copy

$ cp -r a b (messages not required: -r → -fr)
$ mv a b

Bulk data

$ rsync -auv /work0/soudan/UNIX ~UNIX

Meaning of options:

a: Copy, including a directory of subordinate attributes
u: In the update, I will skip the file copy of the already
v: View course

Such as, a more detailed how to use, please refer to the man.

Move Group

$ rsync -av --no-g --no-p /work1/t2g-xx2013 /work1/t2g-xx2014
$ /bin/cp -r /work1/t2g-xx2013 /work1/t2g-xx2014

Note
If the expiration date outside the disk group is locked.
In this state, you can not access. Please refer the following FAQ