Raam Dev’s Weblog

I have often advocated designing websites to support as many screen resolutions and browsers as possible, including tiny mobile devices and text-based web browsers like Lynx (though I prefer Elinks myself). Whether I’m under-the-gun at work, or just designing a new WordPress theme for my blog, when time and resource constraints force me to decide on a minimum resolution I look for statistics to back up my choices. In a perfect world, all sites would be usable with all browser resolutions (and CSS would play nice with all browsers and 1px would be 1px… don’t get me started).

It wasn’t too long ago that a lot of people were still browsing the web with an 800×600 screen resolution. For most of us now though, the only time we see that screen resolution is when we don’t have the correct video drivers installed. W3CSchools keeps lots of interesting web stats based upon visitors to their site, including Browser Display Statistics. Here you can follow a clear trend to higher browser resolutions through the years:

This lead me to check my the visitor browser resolutions for this blog through Google Analytics. I’ve been keeping stats on this blog since September 2007 and with over 40,000 visits I figured I could get some good metrics:

These stats seem to indicate the same thing as W3CSchool’s stats: 1024×768 is a clear winner. This got me thinking though. How many of the popular sites out there are viewable in 1024×768? What about 800×600, or even 640×480?

Yet Another Window Resizer is an excellent Firefox add-on that allows you to easily resize your browser to various resolutions. I visited various popular sites and tested their minimum browser resolutions:

[1024x768] - Amazon.com
[1024x768] - eBay
[1024x768] - Yahoo!
[1024x768] - Wikipedia
[1024x768] - CNN
[1024x768] - YouTube
[1024x768] - Apple
[1024x768] - Microsoft
[1024x768] - Digg
[1024x768] - Delicious
[1024x768] - Facebook
[800x600] - Twitter
[800x600] - WordPress 2.7 Admin Dashboard (screenshot)
[640x480] - Google Search (screenshot)
[640x480] - Google News
[640x480] - Google Maps

Clearly most sites have chosen to ignore smaller browser resolutions in favor of supporting only the most popular as a minimum. (Although this blog has a 1024×768 minimum requirement, the main content area is fully visible in 800×600.) The nature of the content presented by the site dictates a lot of the size requirements. For example, Google search presents textual content that can easily be (and most certainly should be) confined to a small area. In fact, the Google search results maintain a 640px width even when you’re using a much higher resolution (leaving a lot of white space to the right, but maintaining readability).

There is much to be said about textual readability on the web, but I’ll leave that for another post.

I’ve been hearing a lot about IntenseDebate recently, in various blog posts and news sites (they were acquired by Automattic, the company behind WordPress, back in September). IntenseDebate is basically a set of tools that allows for enhanced commenting on your blog or website. Two features that caught my attention were Comment Threading and Reply-by-email.

Comment Threading was of particular interest to me because I find that being able to respond directly to a specific comment keeps the comments organized and relevant. Normally the way you respond directly to a comment without threading support is to prefix your reply with the persons name (e.g., “@raamdev That’s a great idea!”), but constantly searching the list of comments for the comment being replied to gets tiresome.

I don’t really have enough comment traffic on this blog to deal with the aforementioned problem on a regular basis, but I decided to install the IntenseDebate WordPress plugin to see how easily I could integrate it. I was impressed with how smoothly the process went. All my old comments (1,300+) were synced to IntenseDebate and there was nothing I needed to change on my theme templates — the entire commenting system was replaced with IntenseDebate’s commenting system, with threading support, a comment reputation point system, and all those other fancy features.

In the end, however, I decided to deactivate the plugin. I didn’t like that my comments were being loaded from an outside source, which both increased the average load time for each page and added yet another dependency on an external service. (In contrast, my Twitter sidebar uses Twitter Tools, which actually syncs tweets to the same database as WordPress and loads them from there. Dependency problems solved! To be fair though, the IntenseDebate plugin maintains a copy of all the comments in the WordPress database, just in case you decide to stop using it.) Customizing the look and feel of the comments also meant modifying CSS through my IntenseDebate account and right from the start I felt the need to customize and make the comment system less “busy” and self-advertising. There were no themes to choose from either. Ugh.

While doing all this thinking about the disadvantages of outsourcing my comments, I came across a post on Robert Nystorm’s blog where he decides to literally outsource all of his comments. Instead of having any form of commenting system, he simply provides a link to reddit, Digg, and StumbleUpon where he encourages people post their comments on his post. His rationale behind this is that people already use these other sites for commenting and those commenting systems are specifically tuned to handle comments. If you can’t beat’em, join’em.

But that’s extreme. Although I can sympathize with some of Robert’s frustration, I think a commenting system, even if only a simple one, is important. While some people may be familiar with reddit, Digg, and StumbleUpon, not everyone will use those sites regularly or, for that matter, want to go through the trouble of creating an account just to post a comment.

Back in September, I wrote about how excited I was to hear that Comment Threading was coming in WordPress 2.7. I hope the features and functionality of IntenseDebate are somehow integrated into WordPress down the road, instead of indefinitely relying on an external service to provide the same functionality.

Sites and services like Facebook and Twitter ask you questions. In fact they can even be configured to pester (i.e., “nudge”) you if for some reason you feel a moral, religious, or social obligation to keep the vast world of unknown people updated with your every mood, feeling, thought, current activity, or location.

I sat down in front of my computer with a splitting migraine, a migraine that had been piercing through my head the entire day, like an alien parasite whose only job was to disrupt, distract, and otherwise make my day as mentally challenging as possible. As I turned on the monitor, I saw that I had left the Twitter website open. “What are you doing?” the screen pestered, as if instantly joining forces with the alien parasite to make my brain work harder. But it had been 22 hours since I last wrote something on Twitter! Oh no! The world is going to end! I must update everyone immediately!

I suddenly realized that as I begin to use more social-networking sites and write more things on my blog, I am increasingly being pestered with questions and buttons that contain definitions of what I should be writing:

Write a New Page
Write a New Post
Write something…
Write another comment…
What are you doing right now?
Write something about yourself.
What are you doing?

How annoying! I mean really, it’s annoying. No, that’s not the alien parasite talking. This stuff is right in line with Google’s growing visual clutter, only worse. It’s bad enough that I come up with an interesting thing to search the Internet for only to have Google’s suggestions magically chase the original query away, but when everywhere I look for an outlet to express myself I find a question or a definition of what I should be writing, we’re bordering on mental invasion and theft!

These subtle things are killing the creative thought process and subtly removing truly free expression. I feel as though I’m fighting to keep my thoughts to myself just long enough to express them genuinely to whomever might be listening (even if that person is only a future version of myself).

Google’s latest “feature” is nothing short of annoying. I fell in love with Google Search for the clean, textual layout of the search results. The colored text I can deal with, but not visual buttons next to every single result! To make matters worse, Google doesn’t provide a way to disable this feature either, so your only two options are logging out of your Google account or installing a Greasemonkey extension.

Oh, and my rant doesn’t end there. Another feature that was recently added, Google Suggest, has been more trouble than it has help. I can’t even count how many times I’ve went to Google something only to have a big list of suggestions instantly erase the original search query from my head. There are hacky ways to disable that too, but come on Google! There should be options to disable this stuff!

So there I was sitting in my C/Unix class at Harvard barely paying attention to the professor as he talked about HTML forms (!) when I heard him start talking about the history of the HTML radio button. I often wondered why they were called “radio” buttons so I shifted my attention and listened.

He started by trying to explain to a room full of people a third his age how car radios did not always have tiny touch-sensitive buttons and that they used to be single mechanical buttons that when one was pressed, the other would come out (much like the old cassette-based walkman’s).

This little fact fascinated me because I have been using HTML radio buttons for so long and until now, I have been so oblivious to the history behind their name. A quick search on Wikipedia confirmed my professor’s story:

A radio button or option button is a type of graphical user interface widget that allows the user to choose one of a predefined set of options. They were named after the physical buttons used on car radios to select preset stations - when one of the buttons was pressed, other buttons would pop out, leaving the pressed button the only button in the “pushed in” position.

When one of my hosting clients complained about continuously running out of bandwidth on his low-traffic site, I took a peek at the access logs and discovered that Googlebot was indexing every single possible day on a simple calendar addon for the phpBB2 forum software installed on the site. (Googlebot is the program that crawls the web indexing everything so you can search for it using Google.)

A quick peek at the access logs showed thousands of Googlebot requests for a forum calendar:

66.249.71.39 - - [01/Sep/2008:17:09:12 -0400] "GET /forums/calendar.php?m=7&d=21&y=1621&sid=79b643b30eer7140adcd2ba76732688a HTTP/1.1" 200 44000 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:09:33 -0400] "GET /forums/calendar.php?m=4&d=2&y=2188&sid=e4da1ee0a488096e3897a8f15c31cea2 HTTP/1.1" 200 43997 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:09:44 -0400] "GET /forums/calendar.php?m=12&d=4&y=1624&sid=cc5d5084d158457ce3c7a9d38263f553 HTTP/1.1" 200 44076 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.41 - - [01/Sep/2008:17:10:05 -0400] "GET /forums/calendar.php?m=10&d=15&y=1621&sid=a4e8af0d20715g965b3e616ae6f95004 HTTP/1.1" 200 43751 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.41 - - [01/Sep/2008:17:10:15 -0400] "GET /forums/calendar.php?m=9&d=13&y=2187&sid=80c79b2491ddf3d8d46076d48a6282d1 HTTP/1.1" 200 43896 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:10:26 -0400] "GET /forums/calendar.php?m=5&d=30&y=1618&sid=f0619ba6517an57bcd6a7e9ca6289a32 HTTP/1.1" 200 43820 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.39 - - [01/Sep/2008:17:10:38 -0400] "GET /forums/calendar.php?m=11&y=2189&d=30&sid=97c0a58bbd2b3914dbf255ea0a2b1a4c HTTP/1.1" 200 44107 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

A quick Google search turned up many others who’ve had the same problem:

Just found exactly the same on one of my client’s sites. They were complaining that despite being a small site, they’d apparently used all of their bandwidth within 4 days.

They had one of these PHP calendars on their site, where you click the day and it tells you what’s on. Googlebot had tried to index EVERY SINGLE POSSIBLE DAY. And, in the first four days of September, had used up all this site’s bandwidth, clocking up an impressive 19,000 hits and 800MB of bandwidth.

You can use robots.txt to tell all decent robots to push off. I’ve just done that. Let’s see if it works!

So I added a file to the root web directory for the site and named it robots.txt. Inside, I put the following:

User-agent: *
Disallow: /forums/calendar.php

Sure enough, the next time the Googlebot came through it ignored /forums/calendar.php and didn’t use up ridiculous amounts of bandwidth indexing something that need not be indexed.

I can’t blame the Googlebot though. It was just doing its job. The fault goes to the creators of the calendar addon. What they should have done was add a rel="nofollow" to all the links in the calendar. You can add a nofollow tag to individual links to prevent Googlebot from crawling them. Google started using the nofollow tag as a method of preventing comment spam back in 2005.

I’m sure some of you must have seen this warning when you tried to visit my site. Fear not, I have fixed the problem. There was an old file on my domain that had a link to a site that was defined as “malicious” by Google, so they basically added my entire domain to the watch list. I removed the file and, after asking Google to check my site again using Google’s Webmaster Tools, they removed my domain from the list.

So, how did I find the few pages (among thousands of files on my site) that contained a link to the malicious site Google was blocking me for? I logged into my site via SSH and ran a command like the following:

for i in `find . -name "*.ht*"` ; do echo $i; cat $i | grep 195\.2\.252; done