Tuesday, January 22, 2008

Feeling lucky at PubCon

Last month, several of us with Webmaster Central hit the "good times" jackpot at PubCon Vegas 2007. We realize not all of you could join us, so instead of returning home with fuzzy dice for everyone, we've got souvenir conference notes.

Listening to the Q&A, I was pleased to hear the major search engines agreeing on best practices for many webmaster issues. In fact, the presentations in the duplicate content session were mostly, well, duplicate. When I wasn't sitting in on one of the many valuable sessions, I was chatting with webmasters either at the Google booth, or at Google's "Meet the Engineers" event. It was exciting to hear from so many different webmasters, and to help them with Google-related issues. Here are a few things that were on the minds of webmasters, along with our responses:

Site Verification Files and Meta Tags
Several webmasters asked, "Is it necessary to keep the verification meta tag or HTML file in place to remain a verified owner in Webmaster Tools?" The answer is yes, you should keep your verification file or meta tag live to maintain your status as a verified owner. These verification codes are used to control who has access to the owner-specific tools for your site in Webmaster Tools. To ensure that only current owners of a site are verified, we periodically re-check to see if the verification code is in place, and if it is not, you will get unverified for that site. While we're on the topic:

Site Verification Best Practices
  • If you have multiple people working on your site with Webmaster Tools, it's a good idea to have each person verify the site with his or her own account, rather than using a shared login. That way, as people come and go, you can control the access appropriately by adding or removing verification files or meta tags for each account.
  • You may want to keep a list of these verification codes and which owner they are connected to, so you can easily control access later. If you lose track, you can always use the "Manage site verification" option in Webmaster Tools, which allows you to force all site owners to reverify their accounts.
Subdomains vs. Subdirectories
What's the difference between using subdomains and subdirectories? When it comes to Google, there aren't major differences between the two, so when you're making that decision, do what works for you and your visitors. Following PubCon, our very own Matt Cutts outlined many of the key issues in a post on his personal blog. In addition to those considerations, if you use Webmaster Tools (which we hope you do!), keep in mind that you'll automatically be verified for deeper subdirectories of any sites you've verified, but subdomains need to be verified separately.

Underscores vs. Dashes
Webmasters asked about the difference between how Google interprets underscores and dashes in URLs. In general, we break words on punctuation, so if you use punctuation as separators, you're providing Google a useful signal for parsing your URLs. Currently, dashes in URLs are consistently treated as separators while underscores are not. Keep in mind our technology is constantly improving, so this distinction between underscores and dashes may decrease over time. Even without punctuation, there's a good chance we'll be able to figure out that bigleopard.html is about a "big leopard" and not a "bigle opard." While using separators is a good practice, it's likely unnecessary to place a high priority on changing your existing URLs just to convert underscores to dashes.

Keywords in URLs
We were also asked if it is useful to have relevant keywords in URLs. It's always a good idea to be descriptive across your site, with titles, ALT attributes, and yes, even URLs, as they can be useful signals for users and search engines. This can be especially true with image files, which otherwise may not have any text for a search engine to consider. Imagine you've taken a picture of your cat asleep on the sofa. Your digital camera will likely name it something like IMG_2937.jpg. Not exactly the most descriptive name. So unless your cat really looks like an IMG_2937, consider changing the filename to something more relevant, like adorable-kitten.jpg. And, if you have a post about your favorite cat names, it's much easier to guess that a URL ending in my-favorite-cat-names would be the relevant page, rather than a URL ending in postid=8652. For more information regarding issues with how Google understands your content, check out our new content analysis feature in Webmaster Tools, as well as our post on the URL suggestions feature of the new Google Toolbar.

Moving to a new IP address
We got a question about changing a site's IP address, and provided a few steps you can take as a webmaster to make sure things go smoothly. Here's what you can do:
  1. Change the TTL (Time To Live) value of your DNS configuration to something short, like five minutes (300 seconds). This will tell web browsers to re-check the IP address for your site every five minutes.
  2. Copy your content to the new hosting environment, and make sure it is live on the new IP address.
  3. Change your DNS settings so your hostname points to the new IP address.
  4. Check your logs to see when Googlebot starts crawling your site on the new IP address. To make sure it's really Googlebot who's visiting, you can verify Googlebot by following these instructions. You can then log into Webmaster Tools and monitor any crawl errors. Once Googlebot is happily crawling on the new IP address, you should be all set as far as Google is concerned.
  5. To make sure everyone got the message of your move, you may want to keep an eye out for visits to your old IP address before shutting it down.
A few webmasters were concerned that proxy services are being indexed with copies of their content. While it's often possible to find duplicate copies of your content in our results if you look hard enough, the original source is most likely going to be ranked higher than a proxy copy. However, if you find this not to be the case, please drop us some URLs in the Webmaster Help Group. There are many Googlers including myself who monitor this group and escalate issues appropriately.

It was great talking with webmasters at the conference -- we hope those of you unable to join us found this post useful. If you want to continue to talk shop with me, other Googlers, and your fellow webmasters, join the follow-up conversation in the Webmaster Help Group.

Update: Additional PubCon notes from Jonathan Simon are available in our discussion group.

Tuesday, January 15, 2008

Sitemaps FAQs

Last month, Trevor spoke on the Sitemaps: Oversold, Misused or On The Money? panel at Search Engine Strategies in Chicago. After receiving a lot of great questions at the conference in addition to all the feedback we receive in our Help Group, we've pulled together a FAQ:

Q: I submitted a Sitemap, but my URLs haven't been [crawled/indexed] yet. Isn't that what a Sitemap is for?
A: Submitting a Sitemap helps you make sure Google knows about the URLs on your site. It can be especially helpful if your content is not easily discoverable by our crawler (such as pages accessible only through a form). It is not, however, a guarantee that those URLs will be crawled or indexed. We use information from Sitemaps to augment our usual crawl and discovery processes. Learn more.

Q: If it doesn't get me automatically crawled and indexed, what does a Sitemap do?
A: Sitemaps give information to Google to help us better understand your site. This can include making sure we know about all your URLs, how often and when they're updated, and what their relative importance is. Also, if you submit your Sitemap via Webmaster Tools, we'll show you stats such as how many of your Sitemap's URLs are indexed. Learn more.

Q: Will a Sitemap help me rank better?
A: A Sitemap does not affect the actual ranking of your pages. However, if it helps get more of your site crawled (by notifying us of URLs we didn't previously didn't know about, and/or by helping us prioritize the URLs on your site), that can lead to increased presence and visibility of your site in our index. Learn more.

Q: If I set all of my pages to have priority 1.0, will that make them rank higher (or get crawled faster) than someone else's pages that have priority 0.8?
A: No. As stated in our Help Center, "priority only indicates the importance of a particular URL relative to other URLs on your site, and doesn't impact the ranking of your pages in search results." Indicating that all of your pages have the same priority is the same as not providing any priority information at all.

Q: Is there any point in submitting a Sitemap if all the metadata (<changefreq>, <priority>, etc.) is the same for each URL, or if I'm not sure it's accurate?
A: If the value of a particular tag is the same for 100% of the URLs in your Sitemap, you don't need to include that tag in your Sitemap. Including it won't hurt you, but it's essentially the same as not submitting any information, since it doesn't help distinguish between your URLs. If you're not sure whether your metadata is accurate (for example, you don't know when a particular URL was last modified), it's better to omit that tag for that particular URL than to just make up a value which may be inaccurate.

Q: I've heard about people who submitted a Sitemap and got penalized shortly afterward. Can a Sitemap hurt you?
A: Only if it falls on you from a great height. (Seriously, though: if it ever happened that someone was penalized after submitting a Sitemap, it would have been purely coincidental. Google does not penalize you for submitting a Sitemap.)

Q: Where can I put my Sitemap? Does it have to be at the root of my site?
A: We recently enabled Sitemap cross-submissions, which means that you can put your Sitemap just about anywhere as long as you have the following sites verified in your Webmaster Tools account:
  • the site on which the Sitemap is located
  • the site(s) whose URLs are referenced in the Sitemap
Note that cross-submissions may not work for search engines other than Google. Learn more about Sitemap cross-submissions.

Q: Can I just submit the site map that my webmaster made of my site? I don't get this whole XML thing.
A: There's a difference between a (usually HTML) site map built to help humans navigate around your site, and an XML Sitemap built for search engines. Both of them are useful, and it's great to have both. A site map on your domain can also help search engines find your content (since crawlers can follow the links on the page). However, if you submit an HTML site map in place of a Sitemap, Webmaster Tools will report an error because an HTML page isn't one of our recognized Sitemap formats. Also, if you create an XML Sitemap, you'll be able to give us more information than you can with an HTML site map (which is just a collection of links). Learn more about supported Sitemap formats.

Q: Which Sitemap format is the best?
A: We recommend the XML Sitemap protocol as defined by XML Sitemaps have the advantage of being upgradeable: you can start simple if you want (by just listing your URLs), but—unlike a text file Sitemap—you can easily upgrade an XML Sitemap later on to include more metadata. XML Sitemaps are also more comprehensive than an Atom or RSS feed submitted as a Sitemap, since feeds usually only list your most recent URLs (rather than all the URLs you want search engines to know about).

Q: If I have multiple URLs that point to the same content, can I use my Sitemap to indicate my preferred URL for that content?
A: Yes. While we can't guarantee that our algorithms will display that particular URL in search results, it's still helpful for you to indicate your preference by including that URL in your Sitemap. We take this into consideration, along with other signals, when deciding which URL to display in search results. Learn more about duplicate content.

Q: Does the placement of a URL within a Sitemap file matter? Will the URLs at the beginning of the file get better treatment than the URLs near the end?
A: No, and no.

Q: If my site has multiple sections (e.g. a blog, a forum, and a photo gallery), should I submit one Sitemap for the site, or multiple Sitemaps (one for each section)?
A: You may submit as few or as many Sitemaps as you like (up to these limits). Organize them in whatever way you find easiest to maintain. If you create multiple Sitemaps, you can use a Sitemap Index file to list them all. Learn more.

If your question isn't covered here, you can find even more questions and answers in our Sitemaps Help Group.

Wednesday, January 9, 2008

Remove your content from Google

Confused about the best uses of robots.txt, nofollow, URL removal tool? Wondering how to keep some of your pages off the web? Our webspam lead, Matt Cutts, talks about the best ways to stop Google from crawling your content, and how to remove content from the Google index once we've crawled it.

We love your feedback. Tell us what you think about this video in our Webmaster Help Group.

* Note from Matt: Yes, robots.txt has been around since at least 1996, not 2006. It's hard for me to talk for 12-13 minutes without any miscues. :)

Update: for more information, please see our Help Center articles on removing content.

Monday, January 7, 2008

Growing the Webmaster Help Groups Team

One of the ways in which we continuously work to improve our service for users everywhere is to localize and adjust our tools for different regions. In November, we had a chance to introduce our international Webmaster Help Group Team and its guides from Dublin. Today I'm very happy to announce that we've launched three more Google monitored groups in Hebrew, Turkish and Hungarian.

Keeping with tradition, the new guides want to introduce themselves, so please meet our new Webmaster Help Group guides :-)

Turkish Webmaster Help Group
Merhaba! I am Bahar and I am very happy about our brand new Turkish Webmaster Help Group. I will be helping out on any topics related to this new group. Having an engineering background, I've worked in different parts of the IT field as a C and C++ developer and now I work for Google in Dublin. It was quite a big change for me to move here, leaving sunny Istanbul behind. But it was very easy to get used to this new life since I have met really friendly people here who are very enthusiastic about the job they have been doing. You're going to meet a couple of them as well when you keep reading further. Please do not hesitate to have a look at the group and drop your posts either to introduce yourselves or to discuss Webmaster related issues with me and hopefully soon many other helpful, open-minded members!
- Bahar

Hebrew Webmaster Help Group
I’m Alon and I’ll be looking after the Hebrew Webmaster Help Group. Originally coming from the wonderful Israeli city of Holon, now I spend my days and nights in the green island called Ireland. So let me tell you a bit about myself by ranking keywords and their importance in my life: family, social activism, work, web, friends, Tel-Aviv, coffee and rollerblading, to mention a few. Prior to joining Google I graduated in economics and worked as a freelancer in the IT field. I love the online sphere, and for a couple of years now I've been volunteering for an organization facilitating international webmaster communication. I’m looking forward to seeing many of you in our Hebrew Webmaster Help Group.
- Alon

Hungarian Webmaster Help Group
Sziasztok, my name is Tibor, and I am very excited about the launch of the Hungarian Webmaster Help Group. I grew up in the heart of Hungary, in the prosperous and pleasant city of Székesfehérvár, right between Lake Balaton and Budapest. I have an engineering and management background, seasoned with marketing and design experience, and have worked in various assignments, including dozens of web related projects (the first in '94). Today I am a happy owner of a small blog, but have some ambitious plans to start some bigger projects too, to get first hand experience of the constantly growing arsenal of Google's Webmaster Tools! I work primarily on search, dealing with Turkish and Lithuanian languages besides Hungarian. I also oversee trends in a dozen other languages scattered across Europe, the Middle East and Asia. Just like my colleague János, I also enjoy old-school times with StarCraft, and my "weaknesses" also include Guitar Hero and the Nintendo Wii. As Webmaster Tools is now available in Hungarian, I look forward to hearing from you in the Hungarian Webmaster Help Group and to engage myself in vibrant discussions :-)
- Tibor

Hungarian Webmaster Help Group
Sziasztok! My name is János and I'm in charge of the Hungarian Webmaster Help Group, too. I studied linguistics and worked in the IT industry before I ended up with Google in Ireland. Currently my primary work focus is search in several languages. In the past I've been lucky to support a wide range of different markets including Russian, Turkish, Polish and many more. Well, I like learning languages, as you might guess. I'm running my own website where I develop content, sometimes I need to troubleshoot it as well - no crawling/indexing issues yet though - I'm a happy Webmaster Tools user ;-) When I don't work, I'm usually at home surfing the Internet or leading my zealous Protoss forces against other races in the StarCraft universe. Please drop by the Hungarian Help Group and introduce yourself, too! My guide colleague Tibor and I are eagerly waiting to help you with webmaster issues.
- János

Thursday, January 3, 2008

Looking back on 2007

Written by Liza Ma, Webmaster Tools Team

2007 has been a great year of change and growth for the Webmaster Central team. Without the webmaster community helping us drive our vision to create better tools we wouldn't have accomplished much. All of us with the Webmaster Central team would like to say a big thank you to everyone! Before we look ahead to 2008 and begin to incorporate your feedback from the blog posts and discussion groups, we would like to take a minute to reflect upon some of our bigger feats of the last twelve months.

Our blog has been very active since its launch in August 2006, and we hope you've found it helpful. For those who are new readers, you can find useful information about our Webmaster Tools updates here, as well as articles on how to make your sites more search engine-friendly and user-friendly. Outside of the US, we have readers from the UK, France, Canada, India and Germany, just to name a few—it just goes to show what a big and widespread community you are! In fact, we also have a dedicated group of readers for our Webmaster Central German and China blogs. For those keeping track, 47.28% of you are using Firefox to read our blog and 45.06% are using Internet Explorer, and our most popular post this past December answered your questions on meta tags and web search. Here's a snapshot of what our top search queries look like for December:

  • Made preparations for our big debut (see February)





    • Introduced Subscriber stats
    • Revamped the UI
    • Added support for 2 new languages: Romanian & Turkish

  • Went all out for Halloween in true Google fashion
  • Launched new features in Webmaster Tools—and a lot of them:
  • Improved data freshness to keep you better informed about your site's performance


  • Another Webmaster Tools update loaded with new features:
    • Improved Sitemap summary pages to help you identify issues in your Sitemaps
    • Content analysis
    • Video Sitemaps
    • Added support for 2 new languages: Czech and Hungarian

It was a busy year, but we had a lot of fun. I hope you'll stick with us as we embark on more great adventures in 2008. Remember, you can always reach out to us through our discussion group or at various search conferences throughout the year and around the globe!