Back ups in the archive HSM#

Before data is irreversibly removed from the pool disk storage, we back up all data in the HSM (Hierachical Storage Management System). For that, we use the package packems developed in colaboration by MPI-M and DKRZ. Here is a small introduction on how we use it on Levante.

packems takes directories to be packed into tar files, and optionally pushes these directly to the tape archiving system. Features are parallel operation, batch job, and error recovery. It keeps track of the user’s archiving operations to re-use this information for later retrieval of data.

  1. In a first step, we load the necessary modules.

%bash
module load packems

  Cell In[1], line 2
    module load packems
           ^
SyntaxError: invalid syntax

  1. Login to HSM:

!slk login

/bin/bash: slk: command not found

  1. We construct a file list of all files that should be transferred to tape. That list serves as input for packems.

sourcePath="/work/ik1017/CMIP6/data/CMIP6"
MIP="DCPP"
filelist=MIP+".txt"
!find {sourcepath} -type f -exec ls -la {} \; | awk '{ print $9 " " $5 }' > {filelist} ;

find: ‘{sourcepath}’: No such file or directory

  1. If you extend the archive with new .tar files, you need to find out the last number of tars. For that, you should get the Index file from the archive and get it from there.

index="INDEX_"+MIP+".txt"
lastTar = ! $(expr $(cat {index} | grep '\.tar' | cut -d '>' -f 2 | cut -d '_' -f 2 | cut -d '.' -f 1 | sort -g | tail -n 1 ) + 0)

  1. Run packems

%bash
archive_dir="/arch/ik1017/cmip6/CMIP6_retracted"
!packems -j 12 \               #parallelisation processes
         -L    \               #follow links
         -F    \               #fail fast
         -a    \               #generate archiving commands for hsm transfer
         -x    \               #add group to index file
         -S ${archiveDir} \    #destination
         -s $(({lastTar}+1)) \ #starts with number of .tar packages
         -d ${MIP} \           #output dir for tar files
         -o ${MIP} \           #prefix for tar files
         -O by_name \
         -D by_order \         #how to add files to archive
         -i {filelist} \       #input file list
         --archive-index INDEX_${MIP}.txt

  Cell In[5], line 4
    -L    \               #follow links
    ^
IndentationError: unexpected indent