Google Delisting: Lessons Learned

Running a hosted blog is supposed take a lot of the hassle out of website maintenance. But as I have twice found out, there is still a lot each blog owner has to be on the lookout for. The Disney Blog was launched in June of 2004. Its traffic, community, and reputation grew steadily every month until it was ranked in the top 5000 blogs on Technorati. Then I ran into a website owner’s worst nightmare, my site was delisted from the Google Index in August 2006.

In the course of a couple days I researched what I thought had caused
the delisting, posted a plea for help, and took corrective action.
Thankfully a few very helpful people pointed me in the right direction
and I figured out the rest from there. I opened up a Google Sitemaps
account, uploaded a sitemap, and submitted a reinclusion plea. At the
time I thought the main cause was a javascript re-direct I had set up
that pointed a closed blog of mine to The Disney Blog. The blog host
service I use, Typepad, doesn’t support 301-redirects, so I used the
javascript redirect instead. To solve that problem, I turned off the
redirect and password protected the archived blog.

I also took the delisting as an opportunity to domain map my old domain
thedisneyblog.typepad.com to the shorter www.thedisneyblog.com domain.
That’s something I should have done a lot earlier, but since Typepad
doesn’t allow 301-redirects, I was worried about the damage it could
cause to my pagerank/quality rank. Once you’re delisted, that’s no
longer a problem. I also asked as many people as I could to update
their links to reflect the new shorter domain.

As it turns out Typepad does something worse in my opinion. They allow
both URLS (name.typepad.com and www.name.com) to resolve rather than
directing one to the other. So a visitor/bot following a link to the
old domain will still see it as an existing site and index it, rather
than be transferred to the new domain.

[aside: another problem with Typepad is that you have to host your blog
in a subfolder. So Technorati and others end up tracking ‘two’ versions
of your blog based on how people like to you (full URL or short). One
at the short domain (www.thedisneyblog.com) and one at
(www.thedisneyblog.com/tdb/). There appears to be no resolving the two
into one on Technorati’s end. So it’s up to Typepad to figure it out.
Perhaps allow their ‘Pro’ level users the ability to use subdomains for
their domain mapped blogs instead of folders or ‘root’ level blogs with
multiple domain names (as well as they way they have it now).]

About 28 days after I was delisted, and about 25 days after I submitted
the plea for reinclusion, I was magically back in the Google Index
(with both URLs showing in searches). I must have done something right.
But there was no word from Google on what had gone wrong in the first
place, so the delisting could have just been an error.

The traffic I received was still lower than what I had been getting. It
took submitting a few more complete sitemaps, a return of my previous
pagerank (actually, I think I still might be down one number from
pre-August), and some popular posts to finally return traffic to the
previous levels. That was about sometime in November 2006. 

December ’06 was a very good month, thanks in large part to the Scary
Mary video. Then last week on Friday January the 5th, I’m checking my
stats and notice a sudden traffic drop-off. I logged in immediately to
Google Webmaster Tools and my suspicions were correct, I was out of the
index. But strangely there was no notice explaining that I had been
dropped, just a generic message saying the site was not in the index.
It took a few days for that notice to change to something that said I’d
actually done something wrong (but didn’t specify what).

In early December I had closed one blog (Movieland) and merged those
posts into The Disney Blog. But I left the old blog up as a reference
until I could get to manually forwarding each post. So my mind
immediately turned to that as the culprit based on my experience in
August. I took swift corrective action and submitted a plea for
reinclusion on Friday night. I also posted a headsup to my readers that
The Disney Blog had been delisted again.

It was a good thing I made that post. Because what I thought was wrong
wasn’t it. I suspect The Disney Blog made it back into the index in
August due to a combination of factors that included the domain mapping
and time. For whatever reason Google only delists you for 30 days or so
before you re-enter the index. Since I had submitted a sitemap with a
new domain that already had people linking to it, I was reindexed no
questions asked.

This time, when I made the plea for help a few SEO experts were
listening and posted about my plight on their blogs. In turn, someone
from Google read those posts and then made the extra effort to post a
comment on my blog explaining exactly what was happening.

Turns out that the real problem was hidden text that GoogleBot
considered spammy. The truth is, I knew about this hidden text but considered it a feature of Typepad. Since meta-tags are often ignored by
indexing bots, having a div tag that holds a description of your
website in an H level markup seemed like a great idea. It was only
hidden because I had chosen an option in Typepad that let me use an
image instead of text for my header. It was only two lines of text and
it wasn’t trying to spam keywords or anything. It was the actual
description of the website. Useful information for an index bot as far
as I’m concerned.

Anyway, I removed that hidden text from the blog. Emailed Six
Apart/Typepad and let them know that Google cracked down on me for
something they offer as a standard feature and let Google know that
they might end up delisting 400,000 or so blogs for the same reason. As
of the morning of Tuesday January 9th, The Disney Blog was back in
Google’s index with numbers very similar to what I had prior to the
delisting. Whew.

So what lessons have I learned from this whole mess?

I have never deliberately used any SEO tricks for The Disney Blog
(being a blog that covers a very specific topic gets me ranked high
enough in Google as it is) and since I’m not using the ‘advanced
templates’ on Typepad, I can’t add any meta-tags. I was trusting that
the hidden div & h2 tag in Typepad’s code met Google Quality
Standards. I don’t think even Typepad was aware that it didn’t. So this
tells me it was possibly some recent change in Googlebot’s behavior (as
some have suggested).

Perhaps Webmaster tools should allow you to add a site description and
prohibit the Googlebot from accessing certain directories there, since
not all website owners have access to the robot.txt file, even on
Google’s on Google Pages tool. In the meantime, know what’s in your
template/raw html. Don’t allow hidden text to sneak through. Google
will find it.

One of the points in the comment left by Google was that they did try
and contact me via email a week before they delisted the site. If I had
received that email I would have taken care of it immediately. Alas,
the only emails they sent to were all su, webmaster, owner, help, info
@ thedisneyblog.com. At no point did they look for contact info on the
website itself, or use contact info I had provided through the Google
Webmaster Tools or Google Analytics. So I’ve added an email for
[email protected], and I would recommend every blogger do the same.
It would be nice if at the same time Google sends out the one-week
warning, they also highlight the offending blog in the Webmaster Tools
(and Google Analytics if that’s being used too) and then also send an
email to the account used to configure either of those tools. I think
that would cover all the bases.

I still don’t know if the duplicate content was causing problems with
the indexing. Perhaps that’s not as big a sin as hidden text. I hope
Typepad, Technorati, and Google can get together and figure out how to
have just one site from a blog show up in their respective indexes no
matter if it’s domain mapped, forwarded, hosted in a folder or at the
root, or via RSS.

Conclusion: Turns out that on the internet, what you don’t know can
hurt you. But thankfully, if you can make yourself heard via your blog,
then you can attract the support of others and find solutions.

I want to thank everyone who has contacted me or offered to lend a
hand. You are very much appreciated. Please let me know if you have any
further questions and I’ll try and answer them.