In some of our earlier articles we’ve explained the setting up of chrooted sftp and chrooted ssh accounts along with the setup of an ftp server using vsftpd. In today’s automation driven enterprise infrastructure environments we may often find ourselves in situations where we need to automate the transfer of files using the ftp/sftp protocol. Setting up passwordless tranfer of files using sftp is farily stragithforward. Since sftp is based on the ssh protocol we can generate and use ssh keys to setup passwordless authentication to allow for automated file tranfers. In this article we will demonstrate three techniques using which you can automate the transfer of files using FTP protocol as well.
Method 1: Using wget
The wget command provides options to connect to the ftp server and download a file while specifying the credentials on the command line. In the below example, we connect to the ftp server with IP address 172.31.18.17 using the user name sahil and password as L#giN@123 and download the file download.txt.
[root@sahilsuri0081 ~]# wget --user=sahil --password='L#giN@123' ftp://172.31.18.17/download.txt --2018-07-27 17:39:40-- ftp://172.31.18.17/download.txt => ‘download.txt’ Connecting to 172.31.18.17:21... connected. Logging in as sahil ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD not needed. ==> SIZE download.txt ... done. ==> PASV ... done. ==> RETR download.txt ... done. [ <=> ] 0 --.-K/s in 0s 2018-07-27 17:39:40 (0.00 B/s) - ‘download.txt’ saved  [root@sahilsuri0081 ~]# ls -l download.txt -rw-r--r--. 1 root root 0 Jul 27 17:39 download.txt [root@sahilsuri0081 ~]#
The shortcoming of using this method is that we have to specify the credentials in the command itself so it’s not very secure. The wget command only allows us to download files from the ftp server. It does not allow us to upload files.
Although there is a utility named wput which works like wget but instead of downloading files it’s purpose is to upload files.
Method 2: Using curl
The curl command provides us options to download as well as upload files in an automated manner. While using the curl command we specify the username/password by using the -u option followed by the username and password delimited by a colon. We use the -O option specify the FTP server URL. Given below is an example.
[root@linuxnix ~]# curl -u sahil:L#giN@123 -O ftp://172.31.18.17/download.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 20 100 20 0 0 430 0 --:--:-- --:--:-- --:--:-- 444 [root@linuxnix ~]# ls -l download.txt -rw-r--r--. 1 root root 20 Jul 28 10:30 download.txt [root@linuxnix ~]#
Note: Curl will not download the file if the file is empty.
To upload a file using the curl command we use the -T option to specify the file name to be uploaded and we do not use the -O option to indicate the URL in this case. Given below is an example:
[root@linuxnix ~]# curl -u sahil:123 ftp://172.31.18.17/ -T file.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 69 0 0 100 69 0 1503 --:--:-- --:--:-- --:--:-- 1533
Method 3: Using here documents
In the third and final technique we’ll rely on here documents. A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command, such as ftp, cat, or the ex text editor. A limit string delineates (frames) the command list. The special symbol << precedes the limit string. This has the effect of redirecting the output of a command block into the stdin of the program or command. It is similar to interactive-program < command-file, where command-file contains To demonstrate the automation of the FTP file transfer process using here documents I’ve written a small script shown below
[root@linuxnix ~]# cat ftp.bash ftp -in 172.31.18.17 << EOF > ~/ftp-session-log.txt 2>&1 user sahil L#giN@123 get download.txt bye EOF [root@linuxnix ~]#
In the above script we’ve used the -i and -n options with the FTP command wherein -i prevents interactive login prompts and -n option prevents auto-login attempts. Therefore we can specify all the FTP commands that we intend to execute during our FTP session within the body of the here document. In the above script we are logging in to our FTP server as the user sahil by providing the user name and password and then we download a file named download.txt. After this we disconnect the session. The sessions logs are being sent to a file named ftp-session-log.txt. Let’s execute this script and then we can review the contents of this file.
[root@linuxnix ~]# bash -x ftp.bash + ftp -in 172.31.18.17 [root@linuxnix ~]# cat ftp-session-log.txt WARNING! 1 bare linefeeds received in ASCII mode File may not have transferred correctly.
In this article we demonstrated three different techniques to automate our FTP file transfers. From among the methods listed we would recommend using here documents as we can implement the setup in shell scripts and it would provide more flexibility.
Latest posts by Sahil Suri (see all)
- Containerization and orchestration concepts explained - July 17, 2019
- Using CPULimit to limit CPU utilization of a process in Linux - July 11, 2019
- 5 Different ways to display IP addresses in Centos 7 - July 8, 2019
- 3 ways to automate file transfers using ftp - July 3, 2019
- Journalctl command in Linux explained - July 1, 2019