Skip to main content

Download website with wget in linux

wget is a nice tool for downloading resources from the internet. The basic usage is wget url:

wget http://linuxreviews.org/

Therefore, wget (manual page) + less (manual page) is all you need to surf the internet. The power of wget is that you may download sites recursive, meaning you also get all pages (and images and other data) linked on the front page:

wget -r http://linuxreviews.org/

But many sites do not want you to download their entire site. To prevent this, they check how browsers identify. Many sites refuses you to connect or sends a blank page if they detect you are not using a web-browser. You might get a message like:

Sorry, but the download manager you are using to view this site is not supported. We do not support use of such download managers as flashget, go!zilla, or getright

Wget has a very handy -U option for sites like this. Use -U My-browser to tell the site you are using some commonly accepted browser:

wget -r -p -U Mozilla http://www.stupidsite.com/restricedplace.html

The most important command line options are --limit-rate= and --wait=. You should add --wait=20 to pause 20 seconds between retrievals, this makes sure you are not manually added to a blacklist. --limit-rate defaults to bytes, add K to set KB/s. Example:

wget --wait=20 --limit-rate=20K -r -p -U Mozilla http://www.stupidsite.com/restricedplace.html

A web-site owner will probably get upset if you attempt to download his entire site using a simple wget http://foo.bar command. However, the web-site owner will not even notice you if you limit the download transfer rate and pause between fetching files.

Use --no-parent


--no-parent is a very handy option that guarantees wget will not download anything from the folders beneath the folder you want to acquire. Use this to make sure wget does not fetch more than it needs to if just just want to download the files in a folder.


Source : http://linuxreviews.org/quicktips/wget/

Comments

Popular posts from this blog

Install and Uninstall VMware .bundle in Redhat - RHEL 6

Here the steps for Install and Uninstall VMware .bundle in Redhat and also Linux. I have checked with RHEL 6.

For install :
Login in as root user, Type following command

~]# sh VMware-***.bundle

Here *** will be your software and it's version.

This command will display normal GUI installation. Choose option as your wish.

For Uninstall :
Login in as root user, Type following command


~]# vmware-installer -l

This will display your product name and product version installed by VMware.

Ex:
*
*
*
Product Name         Product Version    
========== ==============
vmware-player        4.0.1.528992   

Note Name and Version, Use following command to uninstall

~]# vmware-installer -u {product-name} {product-version}

Ex:
~]# vmware-installer -u vmware-workstation 8.0.1.528992

Now you can uninstall with GUI.

* This is common for all VMware .bundle installation files. Ex: VMware-Workstation, VMware-Player etc.

Play NFS (Need For Speed) or other Games on Full Screen in Laptop with Windows 7

Now a days most of people playing NFS serious (NFS Most Wanted , NFS Carbon etc.) in Laptop.

We are facing the problem that NFS game does not play in full screen mode. It have some black bar at both right and left side.

We can fix it with simple steps, For this your video driver should be installed with it's "Graphics & Media Control Panel" application.

Here i have explained it with screen shot. It'll be mostly same for Intel boards. Settings for other video drivers may differ. You may need to search for it in their graphics control panel.


Step 1 :

Click aero button at right bottom corner for show available notifications and running process. There is Graphics & Media Control Panel Icon.
If it is not avaliable there, you can get it from "Windows Control Panel". It'll open your "Graphics & Media Control Panel".


Step 2 :

Click Graphics & Media Control Panel icon and choose Graphics Properties...


Step 3:

Under Intel Graphics and …