SEO important of robot.txt for search engine

What is the Robot.TXT?

Robot file in txt extension  alway exit in root directory where is the home page or index page put.when submit your  site url in search engine the search engine look for robot file and act search engine according to robot file instruction.So robot txt it,s control the way of search engine in the site which contain the robot file.

what is contains?

it,s contain instruction written in english when search engine read these lines it,s act according to it,

1- determine which Search engine can crawl the site.

why i should specific Search engine why i can not allow all search engine, i prefer you specific the famous and know search engine to crawl your site. cause many  search engine crawl your site make it slowly.

also can spam site crawl your site links and harm you by make fake backlinks that affect your rank in search engine.

2- prevent search engine from crawl specific link or directories to prevent appear in  search engine like setting files and others (images,videos,...ect).

3- can put site map so can fast crawl your links.

4-other things can do it.

here is site can help you in generation Robot.txt

click generation Robot.txt

click another  generation Robot.txt

Ahmed Ata Almahallawi

Freelancer

IT

IT Help Desk,
SEO experience,PHP,C#,ASPX

Al alamee st

gaza -jabalia, Gaza Strip

Palestine

ahmed.almahallawi@gmail.com

DOB: 05/10/1984
by +Ahmed Almahallawi 

23/11/2013