Putty

How to Automate FTP Uploads with PSFTP

Many vendors and companies still transfer data via FTP. It could be transactional data, user data, or a myriad of other things. Hopefully they’re using SFTP or FTPS (which are different ways of achieving secure FTP transmissions) rather than FTP, for similar data transfer security reasons on HTTP vs HTTPS.

A common use case I come across, is user management. Uploading basic user data like names, email addresses and employee numbers so a vendor can update records in their cloud based product for your staff to use. If you’re using a cloud service and don’t have user automation sorted – then ask them how you can achieve it – it’s much more enjoyable to set up automation, than do repeating mindless tasks.

Assuming you have details on what to send and where, you’ll need to work out how to automate FTP uploads. Note that this is a much less secure method – you’re saving the password in plain text. Alternatives do exist such as what’s demonstrated in this YouTube video below using a Public Key and Private Key Pair, but require the ability to create a .ssh folder on the FTP server. I’d rather do it this way:

Getting the other end to do what you want isn’t always possible in the real world, so you need to consider the risks if you need to save a password in plain text somewhere (saved in a Scheduled Task as you’ll see below). They’re obtainable if someone can get onto the server (or a backup of the server, or connect to Task Scheduler remotely), which should be heavily locked down anyway, and the password for this might be saved in a password database those same staff members have access to anyway.

If those credentials were obtained by another party, what could they do? If the FTP site cleans up the data instantly that’s uploaded, then they could potentially upload whatever they wanted. Look at a worst case and decide if you’re comfortable with having the account credentials saved this way, or need to find another approach.

Again, consider these risks, try to implement the most secure method you can, and raise any concerns with management/your boss. Assuming this is a scenario where you can’t do it more securely:

Instructions

First, you’ll need software. I use PSFTP – part of PuTTY, a free and open source solution. Download the full installer, as there’s a few components of PuTTY we need.

Next, you’ll need the login details of the FTP site:
Host: e.g. sftp.contoso.com
Username
Password

Open a Command Prompt, navigate to the location that contains sftp.exe and type:

sftp username@host -pw password

You’ll probably first be prompted with a message saying ‘The server’s host key is not cached in the registry’ with some details on the fingerprint. If you’re sure you’re connecting to the right server, you can say ‘y’ to ‘Store key in cache’. Once saved, you won’t be prompted for this on the same computer/user.

At this stage, we’re just making sure you can sign in and get past the key stage. If this works, you’ll now need to work on a batch file to pass through all the commands you want to do.

In this example, I’ll be going into a folder and uploading a file. Open notepad and type your commands, which you can first test in your active connection:

cd inbound
mput filename.csv
quit

Pretty simple stuff. Save your notepad file (we’ll call it batchfile.txt), and if you haven’t already disconnected from your SFTP session, do so with the ‘quit’ command.

Connect back to the SFTP site, but this time we’ll specify the batch file to rin after connecting:

sftp username@host -pw password -b batchfile.txt -batch

I’ve also added -batch on the end to specify it’s an automated batch job – this will cause SFTP to exit on a prompt, rather than be running forever waiting for an input. You can try without -batch first if you want to test and see the prompts, but you’ll need to run this command manually rather than triggering from a Scheduled Task.

If this works as expected, great! You can automate the SFTP task – the final step is to schedule it to run, which I usually use the native Scheduled Tasks in Windows to do.

If your scheduled task is running under a different account than what you tested with, then you’ll need to do that initial host key saving – easiest way is to launch Command Prompt as that user, and connect to the FTP site.

Using Firefox with a Putty SSH tunnel as a SOCKS proxy

Hi,

The reason I wanted to do this one was viewing a site from an American IP address rather than an Australian. I tested this on my Bluehost account, and it worked perfectly. Here are the instructions:

1. Work out how to connect to your server. You’ll need Putty (available at http://www.chiark.greenend.org.uk/~sgtatham/putty/) – from the Session screen, put in your hostname, most likely port 22 for SSH, make sure SSH is selected as the connection type and click ‘Open’ at the bottom of the screen. If you can log in to that, then the SSH component works and you’re good to go for the rest of this.

2. From Putty, go down to Connection > SSH > Tunnels. From this area, add in a source port (I’ve chosen 9870 but this can be any unused port), and choose ‘Dynamic’. Then click ‘Add’ and you’ll get an entry under “Forwarded Ports”:


3. Go back to Session and connect to SSH. You can save your session to load later rather than retyping the details:

putty2
4. Click “Open” and log onto your SSH session. You only need to log on with your credentials and get to a bash prompt.

5. Open Firefox, and go to the website http://whatismyip.com – take note of your external IP address, because this should change once you’re tunnelling through SSH.

6. In Firefox, press the ‘alt’ key to bring up the top menu and go into Tools > Options. Under the “Advanced” area, go to the “Network” tab and in the “Connection” area click the “Settings” button.

firefox17. Choose the “Manual proxy configuration” radio button, and under SOCKS Host enter “127.0.0.1” and the port you entered into Putty (in my case it’s 9870). Make sure SOCKS v5 is selected.

firefox2

8. Press OK twice to get back to the main view of the Firefox Browser. Try again to get to http://whatismyip.com – if all is working, you’ll see a different IP address which is from the server you’re SSH’d to. If something isn’t configured correctly, you’ll more likely see a browser error like “The proxy server is refusing connections”.

That’s it! Handy for Australians to get access to different web content and pricing.

Update: BobGrrl on twitter has mentioned another two options, free but requiring someone you know to help you at the other end. http://openvpn.net is where you can get it, so a good free opensource solution pending you’re nice enough to someone in another country!

If you want to do this with BlueHost yourself, sign up here:

There’s a bunch of other stuff you get, not just proxy access :)

Update 19/11/2013: This has been tested under Windows 8.1 and worked perfectly. If you do any of the above settings wrong (like putting in 127.0.0.1 as your HTTP proxy rather than SOCKS Host) it probably won’t work. Also make sure your forwarded port from the very first step is still there, if you close putty and re-open it may not remember those settings.