Posted by Shiva Shivakumar, Engineering DirectorWe're undertaking an experiment called
Google Sitemaps that will either fail miserably, or succeed beyond our wildest dreams, in making the web better for webmasters and users alike. It's a beta "ecosystem" that may help webmasters with two current challenges: keeping Google informed about all of your new web pages or updates, and increasing the coverage of your web pages in the Google index.
Initially, we plan to use the URL information webmasters supply to further improve the coverage and freshness of our index. Over time that will lead to our doing an even better job of delivering more search results from more websites. (Danny Sullivan interviewed me about this if you want to read
more.)
This project doesn't just pertain to Google, either: we're releasing it under the
Attribution/Share Alike Creative Commons license so that other search engines can do a better job as well. Eventually we hope this will be supported natively in webservers (e.g. Apache, Lotus Notes, IIS). But to get you started, we offer
Sitemap Generator, an open source client in Python to compute sitemaps for a few common use cases. Give it a whirl and give us
your feedback.