New SEO Tools: Robots.txt Builder, Term Targeting & Crawl Test

Three new SEO tools came out this week that are worth a look. Dave Naylor released a robots.txt builder tool, while SEOmoz released term targeting and crawl test tools. Some more about the tools, below:

The Robots.txt Builder Tool David Naylor, the creator of the tool, explains that one of his clients accidentally blocked all robots to his site and found himself out of the search engines. So he decided to build this tool to make it easy for webmasters to make valid robots.txt files.

Term Targeting SEOmoz created this tool for their premium members and say they may open it up later to all users. The tool “analyzes the content of a given page and extracts the terms and phrases that appear to be targeted at search engines.”

Crawl Test Tool This tool from SEOmoz, previously only available to premium members, now is free for the public. They say the tool can “test how accessible your site is to search engines and can help you quickly diagnose potential crawling issues and give you an overview of your site’s search friendliness.”

Related Topics: Channel: Search Marketing | SEM Tools


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Asia

    I usually read SEOMoz when I see something on this site that entices me to read more. This past week I tried the new services and signed up for premium services, however within an hour, I unsubscribed.

    The crawl test software is flawed, it didn’t find our DMOZ listings on either Yahoo or Google, yet a quick manual search told me otherwise. I immediately decided the software wasn’t of value nor credible at all. If I’m going to pay for premium service, it would be preferable that the service actually worked, I should ask for my money back, but to be honest, I was so disappointed in the Premium service that I had no energy to fight for the $40.

    The Term Targeting software was of no use either. Everything it told me, I already knew – this is why I optimize for these specific terms, and therefore the software should have found it.

    What was useful, were links to software not within SEOMoz.

  • Oatmeal

    This is Matt from SEOmoz, I created the Crawl Test tool along with the rest of SEOmoz’s tools.

    First off: you are confusing the Crawl Test tool with the Page Strength tool (which is completely free, anyone can use it) The Crawl test does not fetch DMOZ results, it tests how accessible your site is by search engines and examines on-page factors.

    Second, the term targeting tool examines a page and determines what terms appear to be important to search engines. If you ran it on your URL and it returned results you were already aware of, then bravo, your site is search optimized. However, try running it on a client or a competitor’s domain and see what kind of results you get. It’s incredibly useful for competitive research as well as catching problems on a site that hasn’t been well optimized.

  • Rhea

    Loving the robot builder tool from Dave!

  • Aaron Shear

    Hey Matt (Oatmeal),

    Keep up the great work with your tools, they are amazing!

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide