Raam Dev’s Weblog

Avatar

Life is an aftertaste. Drink up and wait for it...

Two useful iPhone features you might not know about

Here are two iPhone features I discovered entirely by accident and that I now find myself using several times a day. They’re not terribly exciting, but you may find them useful if you’re not already aware of them.

Easily Access the First Page of Apps

The first one appears to be a newly added feature included in the latest software update. If you have several pages of application icons on your iPhone, you may find yourself constantly going back to the first page to access things like the Camera, Calculator, or Safari apps (which cannot be moved from the first page). Now instead of flicking back through your 5, 10, or 15 (!) pages of apps to get to the first page, you can do it with one click.

Press the home button on the bottom of the phone one time to immediately go to the first page of apps.

Delete Email Gesture

The second feature is a little more interesting and it has been around for a while. I discovered it by accident while scrolling through my list of emails:

From the email list, you can use a single-finger gesture to bring up a delete button on a per-email basis. Simply flick your finger across an email to the left or right (I found going to the left works better) and a Delete button will show up next to the email (see screenshot below).

HOWTO: Disable OS X Window Drop Shadow for Screenshots

One of the awesome features of OS X is the built-in screenshot mode. It allows you to easily select areas of the screen to capture (press Cmd+Shift+4 to activate screenshot mode), or select an entire window to capture (while in screenshot mode, press the spacebar to switch to window-capture mode).

I frequently post screenshots on this blog and I hated that the window screenshots included the drop shadow added by OS X around the window. This meant the width of the resulting screenshot was actually 50px bigger all the way around.

Luckily, there is a way to disable the drop shadow when taking screenshots. Simply open the terminal (Applications -> Utilities -> Terminal.app) and run the following command:

defaults write com.apple.screencapture disable-shadow -bool true

Now logout (or reboot) and when you login again your screenshots won’t include the drop shadow! For more helpful shortcuts and commands for screenshots in OS X, check out Taking Screenshots in Mac OS X.

HOWTO: Count Files Recursively with Exclusion on Linux

Find all files in this directory, including the files in sub-directories, and exclude all files that start with a period (dot files) and any directories named .thumbs. Then pass the list of results to the wc command to get a total count:

find . ! -name ".*" ! -path "*.thumbs*" -type f | wc -l

HOWTO: Make iTunes Read Ogg Files

After downloading the only available torrent of Hang Drum music I could find, I was shocked to discover that iTunes wouldn’t read the Ogg files it contained. I was so close to losing a ton of respect for Apple until I searched Google for a solution. Hoorah for the xiph.org open-source community!

Simply visit their site and download QuickTime Components binary package. After opening the .dmg file (Windows users should be able to just download and run the .exe file), copy XiphQT.component to ~/Library/Components (user-only) or to /Library/Components (system-wide).

If iTunes is open, restart it and wallah! You’ve got .ogg support in iTunes!

Mounting HFS+ with Write Access in Debian

When I decided to reformat and install my Mac Mini with the latest testing version of Debian (lenny, at the time of this writing) I discovered that I couldn’t mount my HFS+ OS X backup drive with write access:

erin:/# mount -t hfsplus /dev/sda /osx-backup
[ 630.769804] hfs: write access to a journaled filesystem is not supported, use the force option at your own risk, mounting read-only.

This warning puzzled me because I was able to mount fine before the reinstall and, since the external drive is to be used as the bootable backup for my MBP, anything with “at your own risk” was unacceptable.

I had already erased my previous Linux installation so I had no way of checking what might have previously given me write access to the HFS+ drive. A quick apt-cache search hfs revealed a bunch of packages related to the HFS filesystem. I installed the two that looked relevant to what I was trying to do:

hfsplus - Tools to access HFS+ formatted volumes
hfsutils - Tools for reading and writing Macintosh volumes

No dice. I still couldn’t get write access without that warning. I tried loading the hfsplus module and then adding it to /etc/modules to see if that would make a difference. As I expected, it didn’t. I was almost ready to give up but there was another HFS package in the list that, even though it seemed unrelated to what was trying to do, seemed worth a shot:

hfsprogs - mkfs and fsck for HFS and HFS+ file systems

It worked! I have no idea how or why (and I’m not interested enough to figure it out), but after installing the hfsprogs package I was able to mount my HFS+ partition with write access.

Creating a Bootable OS X Backup on Linux: Impossible?

I’ve had plans for a while now to set up a backup system using a Debian Linux server and rsync to back up my MacBook Pro laptop. At first glance, it seemed like it would be pretty straight forward. I’ve been able to make a bootable copy of my entire MBP using nothing but rsync (thanks to some very helpful directions by Mike Bombich, the creator of the popular, and free, Carbon Copy Cloner software). And by bootable copy I mean I could literally plug in the USB drive and boot my MBP from the drive (hold down the Alt/Option key while booting). Restoring a backup is as simple as running the rsync command again, but in the reverse direction. I know this solution works because I used it when I upgraded to a 320GB hard drive.

To start, I needed to create a big enough partition on the external USB drive using Disk Utility (formatted with Mac OS Extended (Journaled)). I then made a bootable copy of my MBP with one rsync command:

sudo rsync -aNHAXx --protect-args --fileflags --force-change \
--rsync-path="/usr/local/bin/rsync" / /Volumes/OSXBackup

But my dream backup system was more unattended. I wanted something that would periodically (a couple times a day) run that rsync command over SSH (in the background) and magically keep an up-to-date bootable copy of my MBP on a remote server.

I love Linux and I jump at any opportunity to use it for something new, especially in a heterogeneous network environment. So when I decided to set up a backup server, I naturally wanted to make use my existing Debian Linux machine (which just so happens to be running on an older G4 Mac Mini).

So, after making a bootable copy of my MBP using the local method mentioned above, I plugged the drive into my Linux machine, created a mount point (/osx-backup), and added an entry to /etc/fstab to make sure it was mounted on boot (note the filesystem type is hfsplus):

/dev/sda /osx-backup hfsplus rw,user,auto 0 0

All that’s left to do now is to run the same rsync command as earlier but this time specifying the remote path in the destination (root@myserver.example.com:/osx-backup/). This causes rsync to tunnel through SSH and run the sync. Unfortunately, this is where things started to fall apart.

OS X uses certain file metadata which must be copied for the backup to be complete (again, we’re talking about a true bootable copy that looks no different than the original). Several of the flags used in the rsync command above are required to maintain this metadata and unfortunately Linux doesn’t support all the necessary system calls to set this data. In particular, here are the necessary flags that don’t work when rsyncing an OS X partition to Linux:

-X (rsync: rsync_xal_set: lsetxattr() failed: Operation not supported (95))
-A (recv_acl_access: value out of range: 8000)
–fileflags (on remote machine: –fileflags: unknown option)
–force-change (on remote machine: –force-change: unknown option)
-N (on remote machine: -svlHogDtNpXrxe.iL: unknown option)

According to the man page for rsync on my MBP, the -N flag is used to preserve create times (crtimes) and the --fileflags option requires chflags system call. When I compiled the newer rsync 3.0.3 on my MBP, I had to apply two patches to the source that were relevant to preserving Mac OS X metadata:

patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff

I thought that maybe if I downloaded the source to my Linux server, applied those same patches, and then recompiled rsync, that it would be able to use those options. Unfortunately, those patches require system-level function calls (such as chflags) that simply don’t exist in Linux (the patched source wouldn’t even compile).

So I tried removing all unsupported flags even though I knew lots of OS X metadata would be lost. After the sync finished, I tried booting from the backup drive to see if everything worked. It booted into OS X, but when I logged into my account lots of configuration was gone and several things didn’t work. My Dock and Desktop were both reset and accessing my Documents directory gave me a “permission denied” error. Obviously that metadata is necessary for a viable bootable backup.

So, where to from here? Well, I obviously cannot use Linux to create a bootable backup of my OS X machine using rsync. I read of other possibilities (like mounting my Linux drive as an NFS share on the Mac and then using rsync on the Mac to sync to the NFS share) but they seemed like a lot more work than I was looking for. I liked the rsync solution because it could easily be tunneled over SSH (secure) and it was simple (one command). I can still use the rsync solution, but the backup server will need to be OS X. I’ll be setting that up soon, so look for another post with those details.

WHM Whitelist to Exclude from Exim Sender Verify Callbacks

Sender verification is an important feature used by email servers to help prevent spam. When sender verification is enabled, the receiving email server checks to make sure the sender exists. Various email servers have different ways of handling this feature. Exim, for example, uses a mechanism called ’sender callouts’ or ‘callbacks’. (When the sending server does not accept a verification request, it does not comply with RFC 2821.)

However, in the event that the network route from the receiving email server to the originating email server is broken (or a firewall blocks the connection), the result can be a bit confusing. The receiving email server treats a failed verification as a failed verification, regardless of whether or not it could even connect to the originating server. This means the email never comes through to the recipient. After all, as far as the email server knows, it’s spam.

One of my hosting clients was experiencing this “lost email” problem and a quick grep at /var/log/exim_mainlog confirmed the problem (hosts and IPs changed for obvious reasons):


2008-11-17 15:02:27 [30121] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36752 I=[69.161.211.25]:25 sender verify defer for : could not connect to customer.example.com [163.112.75.15]: Connection timed out
2008-11-17 15:02:27 [30121] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36752 I=[69.161.211.25]:25 F=<administrator@customer.example.com> temporarily rejected RCPT <raam@mydomain.com>: Could not complete sender verify callout
2008-11-17 15:02:27 [30120] H=relay1.example.com (qsv-spam1.example.com) [67.26.151.59]:36751 I=[69.161.211.25]:25 incomplete transaction (RSET) from <administrator@customer.example.com>

As you can see, the email server was unable to connect to customer.example.com to verify the existence of the sender (administrator@customer.example.com). This doesn’t mean the sending server doesn’t verify callbacks, but rather that the network connection from my server to the sending server could not be established.

Most of the stuff I found online related to solving this problem on a server running WHM (here and here) explain how to modify exim.conf to add special whitelist rules. Luckily, my server is running WHM 11.23.2 and has a whitelist option that makes it really easy to exclude a particular IP address from sender verification without any manual changes to exim.conf:

1. Click Service Configuration -> Exim Configuration Editor
2. Under Access Lists, find “Whitelist: Bypass all SMTP time recipient/sender/spam/relay checks” and click [EDIT]
3. Add the IP address for the sending server for which you wish to skip sender verification (as the note at the bottom explains, hosts cannot be used in this list)
4. Click Save
5. Click Save again near the bottom of the Exim Configuration Editor page

That’s it! Now any emails from that IP that were failing to come through because of a sender verification failure will come through without a problem (again, you can watch /var/log/exim_mainlog to confirm).

  • I’ve been trying to create a new partition on the 250GB drive I installed in my G4 (PowerPC) Mac Mini but I could not for the life of me find a Live CD that would boot. Finally this helpful post pointed me to Ubuntu 6.06 (Dapper Drake). After downloading the ‘Mac (PowerPC) desktop CD’ and burning it, I was pleasantly surprised to see it boot the Mac Mini beautifully (I used the live-powerpc kernel at the boot: prompt). Apparently the later PowerPC distributions of Ubuntu don’t come with the necessary ATI drivers for the G4 Mac Mini! (0)

HOWTO: Install md5sum & sha1sum on Mac OS X

I was a bit surprised to learn that my Mac didn’t have the md5sum and sha1sum tools installed by default. A quick search and I found a site that provides the source. The sources compiled successfully on my Mac (OS X 10.5.5, xCode tools installed).

The only quirk appears in the last step:

$ ./configure
$ make
$ sudo make install
cp md5sum sha1sum ripemd160sum /usr/local/bin
chown bin:bin /usr/local/bin/md5sum /usr/local/bin/sha1sum \
/usr/local/bin/ripemd160sum
chown: bin: Invalid argument
make: *** [install] Error 1

The make install command tries to change the ownership of the files to the bin user. Since that user doesn’t exist on my system, the command fails. This isn’t a problem though, as both binaries work perfectly. By default, they are installed to /usr/local/bin/.

HOWTO: Exclude songs when shuffling iTunes

I have a bunch of audio books and other non-music files in my iTunes library. When I set iTunes to shuffle through the songs in my library it naturally ends up playing one of those non-music files, causing me to stop whatever I’m doing and advance iTunes to the next song (using my iPhone remote, ha!).

When I realized how common a problem this must be for people, I looked around the settings in iTunes for a solution. Sure enough, you can tell iTunes to skip a file when shuffling! Just select the file (or group of files), right click, and choose Get Info -> Options -> Skip when shuffling.

Googlebot Relentlessly Using Bandwidth

When one of my hosting clients complained about continuously running out of bandwidth on his low-traffic site, I took a peek at the access logs and discovered that Googlebot was indexing every single possible day on a simple calendar addon for the phpBB2 forum software installed on the site. (Googlebot is the program that crawls the web indexing everything so you can search for it using Google.)

A quick peek at the access logs showed thousands of Googlebot requests for a forum calendar:

66.249.71.39 - - [01/Sep/2008:17:09:12 -0400] "GET /forums/calendar.php?m=7&d=21&y=1621&sid=79b643b30eer7140adcd2ba76732688a HTTP/1.1" 200 44000 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:09:33 -0400] "GET /forums/calendar.php?m=4&d=2&y=2188&sid=e4da1ee0a488096e3897a8f15c31cea2 HTTP/1.1" 200 43997 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:09:44 -0400] "GET /forums/calendar.php?m=12&d=4&y=1624&sid=cc5d5084d158457ce3c7a9d38263f553 HTTP/1.1" 200 44076 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.41 - - [01/Sep/2008:17:10:05 -0400] "GET /forums/calendar.php?m=10&d=15&y=1621&sid=a4e8af0d20715g965b3e616ae6f95004 HTTP/1.1" 200 43751 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.41 - - [01/Sep/2008:17:10:15 -0400] "GET /forums/calendar.php?m=9&d=13&y=2187&sid=80c79b2491ddf3d8d46076d48a6282d1 HTTP/1.1" 200 43896 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.40 - - [01/Sep/2008:17:10:26 -0400] "GET /forums/calendar.php?m=5&d=30&y=1618&sid=f0619ba6517an57bcd6a7e9ca6289a32 HTTP/1.1" 200 43820 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.39 - - [01/Sep/2008:17:10:38 -0400] "GET /forums/calendar.php?m=11&y=2189&d=30&sid=97c0a58bbd2b3914dbf255ea0a2b1a4c HTTP/1.1" 200 44107 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

A quick Google search turned up many others who’ve had the same problem:

Just found exactly the same on one of my client’s sites. They were complaining that despite being a small site, they’d apparently used all of their bandwidth within 4 days.

They had one of these PHP calendars on their site, where you click the day and it tells you what’s on. Googlebot had tried to index EVERY SINGLE POSSIBLE DAY. And, in the first four days of September, had used up all this site’s bandwidth, clocking up an impressive 19,000 hits and 800MB of bandwidth.

You can use robots.txt to tell all decent robots to push off. I’ve just done that. Let’s see if it works!

So I added a file to the root web directory for the site and named it robots.txt. Inside, I put the following:

User-agent: *
Disallow: /forums/calendar.php

Sure enough, the next time the Googlebot came through it ignored /forums/calendar.php and didn’t use up ridiculous amounts of bandwidth indexing something that need not be indexed.

I can’t blame the Googlebot though. It was just doing its job. The fault goes to the creators of the calendar addon. What they should have done was add a rel="nofollow" to all the links in the calendar. You can add a nofollow tag to individual links to prevent Googlebot from crawling them. Google started using the nofollow tag as a method of preventing comment spam back in 2005.

Google Reported Attack Site

Google Reported Attack Site

I’m sure some of you must have seen this warning when you tried to visit my site. Fear not, I have fixed the problem. There was an old file on my domain that had a link to a site that was defined as “malicious” by Google, so they basically added my entire domain to the watch list. I removed the file and, after asking Google to check my site again using Google’s Webmaster Tools, they removed my domain from the list.

So, how did I find the few pages (among thousands of files on my site) that contained a link to the malicious site Google was blocking me for? I logged into my site via SSH and ran a command like the following:

for i in `find . -name "*.ht*"` ; do echo $i; cat $i | grep 195\.2\.252; done

This basically searched every single .htm or .html file inside my public_html directory and returned anything that contained the IP address I was looking for. Whenever there was a match, the filename that preceded the output was the offending file. I’m sure there’s a more elegant way of doing this, but hell, I just wanted to fix the problem!

Although this was annoying to deal with, it made me feel good that Google is actually keeping track of these things and, with the help of Firefox, is warning people of such sites. Site owners must be vigilant in fixing such problems or they risk losing loads of traffic from Google (and from visitors with Firefox).

Escaping Filename or Directory Spaces for rsync

To rsync a file or directory that contains spaces, you must escape both the remote shell and the local shell. I tried doing one or the other and it never worked. Now I know that I need to do both!

So lets say I’m trying to rsync a remote directory with my local machine and the remote directory contains a space (oh so unfortunately common with Windows files). Here’s what the command should look like:

rsync 'raam@example.com:/path/with\ spaces/' /local/path/

The single quotes are used to escape the space for my local shell and the backslash is used to escape the remote shell.

Switching to suPHP; What a Mess!

When one of my users reported problems deleting files he had uploaded using a PHP script, I quickly discovered all the files being uploaded were owned by the user running the web server: nobody. This meant only the root user could delete those files.

Apache suEXEC is commonly used to resolve this problem. It allows Apache to run as the user who owns the domain being accessed. This way, files created by PHP would be owned by the user owning the site instead of the default nobody user.

However, Apache suEXEC only works if you’re using CGI as the PHP handler. The PHP5 handler on my server was set to use CGI, but I have PHP4 configured as the default PHP version and it was configured to use DSO. When I tried changing PHP4 to use CGI as the handler, most of the domains on my server displayed this:

Warning: Unexpected character in input: ‘’ (ASCII=15) state=1 in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ‘ in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ‘ in /usr/local/cpanel/cgi-sys/php4 on line 772
Warning: Unexpected character in input: ‘ in /usr/local/cpanel/cgi-sys/php4 on line 772
Parse error: syntax error, unexpected T_STRING in /usr/local/cpanel/cgi-sys/php4 on line 772

OK, that looks like a problem with cPanel. I don’t have time to debug cPanel’s problems.

suPHP, like suEXEC, is used to run Apache as the user who owns the domain. I decided to try recompiling Apache and PHP with suPHP enabled to see if that would fix the problem.

File Ownership Hell

suPHP worked, except now the sites using PHP sessions were trying to access stored session data in /tmp/ that was owned by the user nobody! So I deleted all the session data and that allowed the PHP sites to create new session data with file ownership of the user owning the domain.

But then I tried accessing my WordPress admin page and started getting permission denied errors in /wp-content/cache/. Same problem: the cache files that had been created before I enabled suPHP were owned by the user nobody and now the user who owns my domain couldn’t access them. A quick chown -R raamdev:raamdev /wp-content/cache/ fixed that problem.

Yeah, I could simply chown -R [user]:[user] /home/[user] for each of the users on the server, but there’s something about running a recursive command on files I’ve never seen, and know nothing about, that makes me uncomfortable.

More suPHP Limitations

I was beginning to worry that this was going to be more difficult than simply enabling suPHP and I wondered how many other sites I’m hosting could have similar problems. I tried accessing one of the high priority sites I’m hosting and discovered it was broken and displaying an “Internal Server Error”.

After a little research, I discovered that you cannot use php_value directives in .htaccess files with suPHP. The .htaccess file included with (created by?) Joomla! contained this at the bottom:

#Fix Register Globals
php_flag register_globals off

I already knew register_globals was turned off in the global PHP configuration, so I simply commented out that line and the site started working again.

Conclusion

It was at this point that I concluded it was too risky to just blindly enable suPHP while hosting over 50 domains, many of which I am not at all familiar with what’s being used or hosted. I will need to take the time to carefully crawl through all the sites making sure their .htaccess files don’t contain anything that might disrupt suPHP and then confirm all the sites are still working properly.

Lesson learned: Setup suPHP before you’re hosting 50+ domains.

Accessing the VMWare Fusion BIOS

As noted in the Known Issues section of the VMWare Fusion Release Notes, the VMWare Fusion BIOS goes by way too fast to give you a chance to access it. The only way you can access it is by modifying the .vmx configuration file for the VM:

The VMware Fusion BIOS posts too quickly to access.

Most users will not need to access the BIOS, but advanced users might want to do so, to change the boot order, set a boot password, or enable a second floppy drive. To work around this problem, use a text editor to add the following line to the configuration (.vmx) file of the virtual machine:

bios.forceSetupOnce = "TRUE"

The next time you boot up the virtual machine, it will automatically boot into the BIOS. This configuration option then reverts to FALSE. You must set the option to TRUE each time you want to boot the virtual machine into the BIOS.