View source for HTCondor에서 GPU 사용하기

== HTCondor에서 GPU 사용하기 ==

경북대학교 중점연구소 GPU 자원이 CMS Tier3 HTCondor Farm에 통합되었습니다.

=== GPU 사용대상  ===

* CMS 사용자중 신청자 

=== GPU 사용신청  ===

* 기존 CMS 사용자: [http://t2-cms.knu.ac.kr/upload/Tier3_KR_KNU_GPU.hwp 사용계획서(기존사용자용).hwp]를 간단하게 작성하여 lcg_knu@knu.ac.kr로 제출하시면 됩니다.
* 신규 CMS 사용자: [http://t2-cms.knu.ac.kr/upload/Tier3_KR_KNU.hwp 사용계획서(신규사용자용).hwp]를 작성하여 lcg_knu@knu.ac.kr로 제출하시면 됩니다.

* 별도 신청서를 받는 이유는 구체적인 수요 현황을 파악하여 향후 자원 증설시 반영하고자 합니다. 양해를 부탁드립니다.

=== GPU 사용  ===

* GPU 상태 확인

 $ condor_status -compact -constraint 'TotalGpus > 0'
 Machine        Platform    Slots Cpus Gpus  TotalGb FreCpu  FreeGb  CpuLoad ST Jobs/Min MaxSlotGb
 
 dm01.knu.ac.kr x64/CentOS7     0   40    3   188.96     40   188.96    0.00 Ui     0.00 *
 dm02.knu.ac.kr x64/CentOS7     0   48    4   377.93     48   377.93    0.00 Ui     0.00 *
 
                Machines Owner Claimed Unclaimed Matched Preempting  Drain
 
    x64/CentOS7        2     0       0         2       0          0      0
 
          Total        2     0       0         2       0          0      0

 $ condor_status -compact -constraint 'TotalGpus > 0' -af Machine TotalGpus CUDADeviceName CUDACapability
 dm01.knu.ac.kr 3 TITAN V 1024.64
 dm02.knu.ac.kr 4 TITAN Xp 1024.64

* Job submit시 다음 옵션을 추가하면 GPU를 사용할 수 있습니다.

 request_gpus = 1

=== Ex 1) Matrix ===

matrix.py

 import numpy as np
 from timeit import default_timer as timer
 from numba import vectorize
 
 @vectorize(['float32(float32, float32)'], target='cuda')
 
 def pow(a, b):
     return a ** b
 
 vec_size = 100000000
 a = b = np.array(np.random.sample(vec_size), dtype=np.float32)
 c = np.zeros(vec_size, dtype=np.float32)
 start = timer()
 c = pow(a,b)
 duration = timer() - start
 print(duration)

matrix.sh

 #!/bin/bash
 python3.6 -m virtualenv myvenv
 source myvenv/bin/activate
 pip3 install numba
 python3.6 matrix.py

matrix.sub

 executable              = matrix.sh
 arguments               = $(ClusterId)$(ProcId)
 output                  = matrix.$(ClusterId).$(ProcId).out
 error                   = matrix.$(ClusterId).$(ProcId).err
 log                     = matrix.$(ClusterId).log
 should_transfer_files   = YES
 transfer_input_files    = matrix.py 
 when_to_transfer_output = ON_EXIT
 request_GPUs = 1
 request_CPUs = 1
 queue

=== Ex 2) TensorFlow ===

tf_ex.py

 import tensorflow as tf
 
 mnist = tf.keras.datasets.mnist
 
 (x_train, y_train), (x_test, y_test) = mnist.load_data()
 x_train, x_test = x_train / 255.0, x_test / 255.0
 
 model = tf.keras.models.Sequential([
   tf.keras.layers.Flatten(input_shape=(28, 28)),
   tf.keras.layers.Dense(128, activation='relu'),
   tf.keras.layers.Dropout(0.2),
   tf.keras.layers.Dense(10, activation='softmax')
 ])
 
 model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])
 
 model.fit(x_train, y_train, epochs=5)
 model.evaluate(x_test,  y_test, verbose=2)

tf_ex.sh

 #!/bin/bash
 python3.6 -m virtualenv myvenv
 source myvenv/bin/activate
 pip3 install tensorflow-gpu==2.0.0-rc1
 python3.6 tf_ex.py

tf_ex.sub

 executable              = tf_ex.sh
 arguments               = $(ClusterId)$(ProcId)
 output                  = tf_ex.$(ClusterId).$(ProcId).out
 error                   = tf_ex.$(ClusterId).$(ProcId).err
 log                     = tf_ex.$(ClusterId).log
 transfer_input_files    = tf_ex.py
 when_to_transfer_output = ON_EXIT
 request_GPUs = 1
 request_CPUs = 1
 queue

=== Ex 3) Singularity & TensorFlow - cvmfs image  ===

sing.sh

 #!/bin/bash
 python3.6 tf_ex.py

sing.sub

 arguments               = $(ClusterId)$(ProcId)
 output                  = sing.$(ClusterId).$(ProcId).out
 error                   = sing.$(ClusterId).$(ProcId).err
 log                     = sing.$(ClusterId).log
 should_transfer_files   = YES
 when_to_transfer_output = ON_EXIT
 transfer_input_files    = tf_ex.py
 request_GPUs = 1
 request_CPUs = 1
 +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/opensciencegrid/tensorflow-gpu:latest"
 queue

=== Ex 4) Singularity & TensorFlow - local image  ===

sing-local.sub

 executable              = sing.sh
 arguments               = $(ClusterId)$(ProcId)
 output                  = sing.$(ClusterId).$(ProcId).out
 error                   = sing.$(ClusterId).$(ProcId).err
 log                     = sing.$(ClusterId).log
 should_transfer_files   = YES
 when_to_transfer_output = ON_EXIT
 transfer_input_files    = tf_ex.py
 request_GPUs = 1
 request_CPUs = 1
 +SingularityImage = "/u/user/hanbi/tensorflow-gpu.sif"
 queue

=== Ex 5) Singularity & TensorFlow - docker image  ===

sing-docker.sub

 executable              = sing.sh
 arguments               = $(ClusterId)$(ProcId)
 output                  = sing.$(ClusterId).$(ProcId).out
 error                   = sing.$(ClusterId).$(ProcId).err
 log                     = sing.$(ClusterId).log
 should_transfer_files   = YES
 when_to_transfer_output = ON_EXIT
 transfer_input_files    = tf_ex.py
 request_GPUs = 1
 request_CPUs = 1
 +SingularityImage = "docker://tensorflow/tensorflow:latest-gpu"
 queue

=== 참고자료 ===

위의 예제들은 아래 문서들에서 참고했습니다. 좋은 내용이 많으니 이용에 참고하시기 바랍니다.

* https://batchdocs.web.cern.ch/tutorial/exercise10.html
* https://www.tensorflow.org/tutorials/quickstart/beginner
* https://sylabs.io/guides/3.6/user-guide/singularity_and_docker.html
* https://sylabs.io/guides/3.6/user-guide/gpu.html