Merge pull request #36 from tabernarious/custom_pruning_interval

Custom pruning interval and additional enhancements

CHANGELOG.md

@@ -1,5 +1,11 @@
 # CHANGELOG
 
+## v3.2.4 (20210831)
+* Added an option to set a custom local pruning interval. This used to be fixed at 60 seconds.
+* Added an option to set a custom directory to save archives.
+* Fixed SSH options for SCP.
+* Limited use of filename suffix to SMB/CIFS and Local (the only modes where pruning is available).
+
 ## v3.2.3 (20210327)
 * Fixed a bug (typo) when using SCP, which resulted in 'script did not successfully complete: (can't read "::destination_parameters__sftp_sshprivatekey": no such variable' etc. This was related to adding support for keys with "BEGIN OPENSSH PRIVATE KEY" headers.
 
@@ -130,4 +136,4 @@ Developed/posted by MAG
 ## ~v2.0 (20140312)
 Developed/posted by Thomas Schockaert
 * Initially posted releases from what I gathered perusing DevCentral.
-* v11.4.0-11.6.x? compatibility.
+* v11.4.0-11.6.x? compatibility.

README.md

@@ -2,8 +2,8 @@
 F5 iApp for automated backups to the local device and to network locations.
 
 ## Usage
-* You only need the `f5.automated_backup.v3.2.3.tmpl.tcl` file (or whatever the latest version is). Download this template and import it using the F5 BIG-IP GUI.
-* The `f5.automated_backup.v3.2.3.scripts_reference.sh` file is on a reference to better understand the scripts used in the `tmpl.tcl` file. In the `tmpl.tcl` file the scripts are converted to a single line by replacing tabs with \t and newlines with \n, which makes them very difficult to understand or troubleshoot.
+* You only need the `f5.automated_backup.v3.2.4.tmpl.tcl` file (or whatever the latest version is). Download this template and import it using the F5 BIG-IP GUI.
+* The scripts folder contains pretty versions of the scripts used in the iApp template.
 
 ## Intro
 Building on the significant work of Thomas Schockaert (and several other DevCentralites) I enhanced many aspects I needed for my own purposes, updated many things I noticed requested on the forums, and added additional documentation and clarification. As you may see in several of my comments on the original posts, I iterated through several 2.2.x versions and am now releasing v3.0.0. Below is the breakdown!
@@ -69,7 +69,5 @@ Daniel Tavernier (tabernarious)
 * Using a 4096 bit private key for SFTP/SCP results in error "Unable to decrypt text of length (4338) which exceeds the max of (4048)" which may be an iApp bug/limitation of fields designated as type "password". (github Issue #12)
     * Could add a second field to accept part of the key, then combine the values.
     * Could use something like this to pull the key from the F5 filestore (though this would result in the key being accessible via the GUI). This might even work with encrypted keys: grep "sys file ssl-key /Common/KEY-NAME.key" -A1 /config/bigip.conf |tail -1 |sed 's/    cache-path //'
-* Reported issues with FTP (sending archive before finished or corrupting?) (github Issue #15)
-* Reported issues with SMB from v12.x to Windows Server 2012 (github Issue #17)
 * Add automatic pruning for FTP and SFTP/SCP.
-    * Use "dir -t" or "nlist -t" commands to pull file list...
+    * Use "dir -t" or "nlist -t" commands to pull file list...

scripts/ftp.sh

@@ -0,0 +1,47 @@
+# Currently using tabs (not 4 spaces) throughout these scripts for ease of conversion to a single-line.
+# Search for tabs and replace with string "\t"
+# Search for newlines and replace with string "\n"
+# Copy into main script between 'exec echo -e "' and '" > $scriptfile'
+
+ftp_function()
+{
+	f5masterkey=\$(f5mku -K)
+	username=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	password=\$(echo \"ENCRYPTEDPASSWORD\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	# Escape every character for safe submission of special characters in the password
+	password_escaped=\$(echo \${password} | sed \'s/./\\\\\\&/g\')
+	server=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	directory=\$(echo \"ENCRYPTEDDIRECTORY\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+
+	if \[ \"BACKUPFILENAMEEXTENSION_NODOT\" == \"scf\" \]
+	then
+		ftp_return=\$(ftp -n \${server} << END_FTP
+quote USER \${username}
+quote PASS \${password_escaped}
+binary
+put BACKUPDIRECTORY/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT \${directory}/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT
+put BACKUPDIRECTORY/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT.tar \${directory}/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT.tar
+quit
+END_FTP
+)
+	else
+		ftp_return=\$(ftp -n \${server} << END_FTP
+quote USER \${username}
+quote PASS \${password_escaped}
+binary
+put BACKUPDIRECTORY/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT \${directory}/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT
+quit
+END_FTP
+)
+	fi
+
+	if \[ \"\$ftp_return\" == \"\" \]
+	then
+		return 0
+	else
+		echo \"\$ftp_return\" >> /var/tmp/scriptd.out
+		return 1
+	fi
+}
+
+ftp_function

scripts/local_pruning.sh

@@ -0,0 +1,23 @@
+# Currently using tabs (not 4 spaces) throughout these scripts for ease of conversion to a single-line.
+# Search for tabs and replace with string "\t"
+# Search for newlines and replace with string "\n"
+# Copy into main script between 'exec echo -e "' and '" > $scriptfile'
+
+files_tokeep=\$(ls -t BACKUPDIRECTORY/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null | head -n CONSERVE\)
+for current_archive_file in `ls BACKUPDIRECTORY/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null` ; do
+	current_archive_file_basename=`basename \$current_archive_file`
+	check_file=\$(echo \$files_tokeep | grep -w \$current_archive_file_basename)
+	if \[ \"\$check_file\" == \"\" \] ; then
+		rm -f \$current_archive_file
+	fi
+done
+if \[ \"BACKUPFILENAMEEXTENSION_NODOT\" == \"scf\" \] ; then
+	tar_files_tokeep=\$(ls -t BACKUPDIRECTORY/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null | head -n CONSERVE\)
+	for current_archive_tar_file in `ls BACKUPDIRECTORY/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null` ; do
+		current_archive_tar_file_basename=`basename \$current_archive_tar_file`
+		check_file=\$(echo \$tar_files_tokeep | grep -w \$current_archive_tar_file_basename)
+		if \[ \"\$check_file\" == \"\" \] ; then
+			rm -f \$current_archive_tar_file
+		fi
+	done
+fi

scripts/scp.sh

@@ -0,0 +1,21 @@
+# Currently using tabs (not 4 spaces) throughout these scripts for ease of conversion to a single-line.
+# Search for tabs and replace with string "\t"
+# Search for newlines and replace with string "\n"
+# Copy into main script between 'exec echo -e "' and '" > $scriptfile'
+
+scp_function()
+{
+	f5masterkey=\$(f5mku -K)
+	username=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	server=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	directory=\$(echo \"ENCRYPTEDDIRECTORY\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	echo \"ENCRYPTEDPRIVATEKEY\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey} > /var/tmp/TMSHAPPNAME_scp.key
+
+	chmod 600 /var/tmp/TMSHAPPNAME_scp.key
+	scp -i /var/tmp/TMSHAPPNAME_scp.key SCPCIPHER SCPSTRICTHOSTKEYCHECKING SCPSSHOPTIONS BACKUPDIRECTORY/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT* \${username}@\${server}:\${directory}/ 2>> /var/tmp/scriptd.out
+	scp_result=\$?
+	rm -f /var/tmp/TMSHAPPNAME_scp.key
+	return \$scp_result
+}
+
+scp_function

scripts/sftp.sh

@@ -0,0 +1,21 @@
+# Currently using tabs (not 4 spaces) throughout these scripts for ease of conversion to a single-line.
+# Search for tabs and replace with string "\t"
+# Search for newlines and replace with string "\n"
+# Copy into main script between 'exec echo -e "' and '" > $scriptfile'
+
+sftp_function()
+{
+	f5masterkey=\$(f5mku -K)
+	username=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	server=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	directory=\$(echo \"ENCRYPTEDDIRECTORY\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+	echo \"ENCRYPTEDPRIVATEKEY\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey} > /var/tmp/TMSHAPPNAME_sftp.key
+
+	chmod 600 /var/tmp/TMSHAPPNAME_sftp.key
+	echo put BACKUPDIRECTORY/${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT* | sftp -b- -i /var/tmp/TMSHAPPNAME_sftp.key SFTPCIPHER SFTPSTRICTHOSTKEYCHECKING SFTPSSHOPTIONS \${username}@\${server}:\${directory}/ 2>> /var/tmp/scriptd.out
+	sftp_result=\$?
+	rm -f /var/tmp/TMSHAPPNAME_sftp.key
+	return \$sftp_result
+}
+
+sftp_function

scripts/smb_cifs.sh

@@ -0,0 +1,85 @@
+# Currently using tabs (not 4 spaces) throughout these scripts for ease of conversion to a single-line.
+# Search for tabs and replace with string "\t"
+# Search for newlines and replace with string "\n"
+# Copy into main script between 'exec echo -e "' and '" > $scriptfile'
+
+\#\!/bin/sh
+f5masterkey=\$(f5mku -K)
+username=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+password=\$(echo \"ENCRYPTEDPASSWORD\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+msdomain=\$(echo \"ENCRYPTEDMSDOMAIN\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+server=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+msshare=\$(echo \"ENCRYPTEDMSSHARE\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+mssubdir=\$(echo \"ENCRYPTEDMSSUBDIR\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+mountp=\$(echo \"ENCRYPTEDMOUNTP\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+mountvers=\$(echo \"ENCRYPTEDMOUNTVERS\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+mountsec=\$(echo \"ENCRYPTEDMOUNTSEC\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})
+cd BACKUPDIRECTORY
+if \[ \! -d \${mountp} \]
+then
+	mkdir -p \${mountp}
+	if \[ \$? -ne 0 \]
+	then
+		rm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*
+		exit 1
+	fi
+fi
+\# The password must be surrounded by two single-quotes to successfully handle special characters. Still does not support comma, single-quote, and double-quote.
+mount -t cifs //\${server}/\${msshare}\${mssubdir} \${mountp} -o user=\${username},password=\'\'\${password}\'\',domain=\${msdomain},vers=\${mountvers},sec=\${mountsec} 2>> /var/tmp/scriptd.out
+if \[ \$? -ne 0 \]
+	then
+	echo \"DEBUG: Failed to mount //\${server}/\${msshare}\${mssubdir}\ to \${mountp}\" >> /var/tmp/scriptd.out
+	rm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*
+	exit 1
+else
+	echo \"DEBUG: Successfully mounted //\${server}/\${msshare}\${mssubdir}\ to \${mountp}\" >> /var/tmp/scriptd.out
+fi
+
+latestFileOnSMB=\$(ls -t \${mountp}/\*.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null| head -n 1 2>/dev/null)
+echo \"DEBUG: Latest BACKUPFILENAMEEXTENSION_NODOT file found on SMB mount: \$latestFileOnSMB\" >> /var/tmp/scriptd.out
+
+if \[ \"X\"\${latestFileOnSMB} \!= \"X\" \]
+	then
+	sum1=\$(md5sum ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT | awk '{print \$1}')
+	sum2=\$(md5sum \${latestFileOnSMB} | awk \'{print \$1}\')
+	if \[ \${sum1} == \${sum2} \]
+	then
+		echo \"ERROR: File ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT already exists in //\${server}/\${msshare}\${mssubdir}\" >> /var/tmp/scriptd.out
+		umount \${mountp}
+		rm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*
+		exit 1
+	else
+		echo \"DEBUG: File ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT does not already exist in //\${server}/\${msshare}\${mssubdir} (continuing...)\" >> /var/tmp/scriptd.out
+	fi
+else
+	echo \"DEBUG: Destination SMB mount contains no BACKUPFILENAMEEXTENSION_NODOT files (continuing...)\" >> /var/tmp/scriptd.out
+fi
+cp ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT* \${mountp}
+rm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*
+
+if \[ \"PRUNINGMODE\" \!= \"Disabled\" \]; then
+
+	files_tokeep=\$(ls -t \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null | head -n CONSERVE\)
+	for current_archive_file in `ls \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null` ; do
+		current_archive_file_basename=`basename \$current_archive_file`
+		check_file=\$(echo \$files_tokeep | grep -w \$current_archive_file_basename)
+		if \[ \"\$check_file\" == \"\" \] ; then
+			rm -f \$current_archive_file
+		fi
+	done
+	if \[ \"BACKUPFILENAMEEXTENSION_NODOT\" == \"scf\" \] ; then
+		tar_files_tokeep=\$(ls -t \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null | head -n CONSERVE\)
+		for current_archive_tar_file in `ls \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null` ; do
+			current_archive_tar_file_basename=`basename \$current_archive_tar_file`
+			check_file=\$(echo \$tar_files_tokeep | grep -w \$current_archive_tar_file_basename)
+			if \[ \"\$check_file\" == \"\" \] ; then
+				rm -f \$current_archive_tar_file
+			fi
+		done
+	fi
+fi
+
+umount \${mountp}
+
+echo \"DEBUG: Script completed without errors\" >> /var/tmp/scriptd.out
+exit 0