The manual and even the install script eventually helped with part of it which was the custom install of Apache. I don't use the default Apache install on OS X and instead compile my own software. By setting a couple environment variables I was able to help Passengers install script find the right apache:
>> export APXS2=/path/to/bin/apxs
While the install script liked this and helped get it going it eventally coughed up during compile of the module. I noticed it was mentioning an invalid function used when MACOSX_DEPLOYMENT_TARGET was set to 10.1. Wait...I'm on 10.4 so I set this env variable to 10.4:
>> export MACOSX_DEPLOYMENT_TARGET=10.4
Voila! It works! Thanks for mod_rails Phusion guys!