Gsitecrawler Application - Sitemaps

Also, if you have a website and are trying to increase your search engine ranking, there are another two things you need to do. First, you need to create a robots.txt file so that search engines can crawl your site. A general robots.txt file which allows a search engine to crawl through everything would look like this:

User-Agent: *
Allow: /

Some handy tools for this are Google’s Webmaster Tools.
And of course, you will want to submit your site to search engines. This combined with the crawling .txt are bound to increase your rankings and traffic. Some good sites for this are http://www.submitexpress.com and http://freewebsubmission.com/.