Web Mirroring
In many situations, you can pull down the contents of a website, recursively using the wget utility.
To do a straight forward mirror of a site:
wget -m http:\\some.site.com
long version:
wget –mirror http:\\some.site.com
This is actually the same as specifying:
wget -r -l inf -nr -N http:\\some.site.com
long version:
wget –recursive –levelinf –dont-remove-listing –timestamping http:\\some.site.com=
Convert for off-line reading, including giving all html files an .html extension.
wget –recursive –level1 –dont-remove-listing –timestamping –convert-links –html-extension http://some.site.com=
Other common options or variants
-l depth
--level=depth
Specifies the maximum depth level for recursion.
-L
--relative
Follows relative links only.man wget will tell you more.
Proxy servers
Set an environment label http_proxy
E.g. in bash shell…
export http_proxyhttp://proxy.mysite.com:8888=
See the manual for specifying username and password if your proxy server requires them. Options –proxy-user –proxy-passwd
Example
wget –verbose –timeout=30 –mirror –proxy-user=myuser –proxy-passwd=mypassword http://some.web.site
More info at http://www.gnu.org/manual/wget/
FTP
wget –mirror ftp://myusername:mypassword@ftp.targetsite.com/target/path
– Frank Dean 6 Dec 2002