This is one of the commonest errors that are thrown when trying to copy files over scp. The major reason for this is, the port being different from the default 22.
pradeeban@pradeeban:~$ scp -r /home/pradeeban/patches email@example.com:/mnt/patches ssh: connect to host 22.214.171.124 port 22: Connection refused lost connection
To fix this, use -P flag, and the port number. Notice the upper case. This is to maintain the consistency with the -p usage of cp command.
pradeeban@pradeeban:~$ scp -P 1984 -r /home/pradeeban/patches firstname.lastname@example.org:/mnt/patches
Find further useful tips at, http://www.linuxtutorialblog.com/post/ssh-and-scp-howto-tips-tricks