Midnight Oil

One of the most irritating things about CloudFront is the lack of SSL support. It’s incredibly frustrating to install an SSL certificate, get all your routing set up, then watch the browser freak out because one teeny-tiny image comes through without encryption. A major pain in the ass.

Anyways, it’s possible to sidestep the issue by requesting the image directly from S3 instead of CloudFront. You are no longer leveraging the CDN, but in my case I’d rather have the page load slightly slower than have the browser complain about security flaws.

CloudFront Helper

I wrote the following helper to make it all easy:

module CloudfrontHelper
  # Will return a URL to an S3/Cloudfront image. If the current request is HTTPS, then it will return
  # an HTTPS URL (ie. S3) and if it is HTTP then it will return a Cloudfront URL.
  def cf_img_url(s3_image, *params)
    if request.ssl?
      s3_image.s3_url(*params)
    else
      s3_image.public_filename(*params)
    end
  end
end

SSL Config in amazon_s3.yml

The final step is to turn on SSL support for attachment_fu

production:
  bucket_name: my-bucket
  access_key_id: asdf
  secret_access_key: xxxx
  distribution_domain: [my-cloud-distribution]
  use_ssl: true

Example Usage

Now, anywhere you need to display an image that’s hosted on S3/CloudFront, just use the cf_image_url helper and it will automatically route to either the S3/https version or the CloudFront/http one depending on the protocol for the request. Simple!

< %= image_tag(cf_img_url(@user.profile_pic)) %>

Hosting a Rails app has always been a bit of a pain. Standard shared hosting was always pretty much out of the question, so most people naturally migrated to some sort of VPS. The obvious upside of being able to do whatever you want is a big draw. However, there is a dark side to every VPS: sysadmin work. Updating with patches, installing software, setting file permissions, and all that plumbing takes a lot of time. Then, there’s the big-daddy of them all: managing your own SMTP server. Sure, turning on Postfix or Exim is pretty easy. I just hope you don’t mind it when all your emails disappear into spam boxes.

For Obsidian Portal, we have been happily hosting with Slicehost for a while. Their uptime has been great, and we appreciate their services like automated backup and DNS management. However, we have been handling all our sysadmin tasks ourselves. So, when Ryan recently met one of the representatives from Blue Box Group, and he said that they’re happy to handle all that for us (at no additional charge!), we were quite interested.

Since we’re currently spinning up DoLeaf, we decided it would be a good time to see what we can get with a more hands-on hosting provider. And, after just a few days, we’re very impressed.

Choice of Distro

Blue Box’s basic installation is some flavor of Red Hat. Ryan and I are Ubuntu guys, so this was a problem. A single email to the support staff: “Hey, can you rebuild our server with Ubuntu server edition?” Answer: “Yep, it’s done.” Awesome.

Firewall

Anyone else think iptables is a bit of a pain? I can use Firestarter or Webmin, but when I try to manage it manually, forget it. Luckily it only has to be done once. Or, in the case of BB, never. Just send an email with the ports you want open. They set it up and replied, plus added a few standard ports that we forgot.

Rails Stack

BB supports several different Rails stacks. We’re used to Apache2 + Ruby Enterprise Edition + Passenger + Rails 2.3.2 + MySQL 5. Even though this kind of setup is pretty straightforward, it always seems to take me at least 1-2 hours just to get things set up in the most basic configuration. Now? You guessed it. Send an email and I’m done.

Functioning SMTP – The Great White Whale

We’ve always had problems with emails disappearing en-route. Getting Postfix set up is a 5 minute job, but your emails randomly get flagged as spam, especially by the big boys like Google and Yahoo. There are many different ways to counter this, like SPF and DKIM, but they are a pain to set up, and it’s always tough to know if you got it correct. This was the tipping point for me with BB. They have an SMTP server preconfigured with SPF and DKIM and you can use it for sending emails from your app. They limit you to 750 emails / hour, which is way more than most apps need, especially in the beginning. Considering that we had evaluated similar solutions that wanted to charge $0.01 / email, having this included for free in our hosting plan was mind blowing. Never again will I host anywhere that doesn’t do this.

And the cost?

The most surprising thing is that I haven’t been charged $0.01 for additional support. I’m still testing the waters to see exactly where that line is, but so far, I haven’t hit it. Plus, their prices are quite competitive. Slicehost is a little bit cheaper, but when you factor in the time saved on setup, I’d say we’re already way ahead.

We’ve only just started with Blue Box, and things may go south at some point, but for now we’re ecstatic with what we’ve received. If you want to spend more time coding and less time on irritating sysadmin tasks, then I highly recommend taking a look!

I’ve recently added a patch to attachment_fu that allows for serving out your S3-stored files via Amazon CloudFront. So, if you want an instant CDN, it doesn’t get much easier.

(Note: For Paperclip users, check out this writeup on intridea)

Get the plugin

Get the latest version of attachment_fu from github.

script/plugin install git://github.com/technoweenie/attachment_fu.git

Setup CloudFront

You’ll need to sign up for CloudFront. It’s as simple to join as any of the other AWS’s.

Once you’re signed up, you’ll need to create what are called distributions. These are essentially mappings from S3 buckets to CloudFront domains. You can use the command-line API if you want, but I found it easiest to just use S3Fox.

In S3Fox, find the S3 bucket that you want to serve out, right click, and choose Manage Distributions. You will see a screen like the one below:

Give the distribution a comment, make sure enable is checked, and click on Create Distribution. It will be InProgress for a few minutes, so just keep refreshing. Once it transitions to Deployed, you’re done!

Configuration

We’ve added one new option to the amazon_s3.yml file that comes with attachment_fu, distribution_domain. Copy the Resource Url (something like http://XXXXX.cloudfront.net) from S3Fox and enter it in your amazon_s3.yml file as the distribution_domain. Make sure to only get the domain and strip off the ‘http://’

Next, you’ll need to update all your has_attachment models that you wish to serve out. For each of these, add a cloudfront option like so:

class MyClass
has_attachment(
:content_type => :image,
:storage => :s3,
:cloudfront => true,
:max_size => 1.megabytes,
:processor => "Rmagick"
)
end

You’re Done!

That’s it! Now, everywhere you use myobj.public_filename, it will use the CloudFront domain and serve it through the CDN.

Bonus Points: CNAME

If you think the XXXX.cloudfront.net domain is ugly, you can set a CNAME on the distribution and do some DNS monkey-business to choose your own CNAME. I’ll do this at some point (and update this blog post), but so far it hasn’t been important to me.

Update I finally bit the bullet and went with Mailchimp. It was a great decision and it’s worth every penny. I’ll leave the original post up in case it’s useful to others.

In a nutshell, mailcar makes it (a little) easier to send a mass emails from your Rails app.

Get the plugin!

Newsletters, feature updates, or God forbid, security breach notices all require sending potentially thousands of emails. There are plenty of service providers who will do it for a fee, but if you’re a small-ish site run by stingy admins (like Obsidian Portal), then you don’t want to drop $100 every time you announce a new feature.

Mailcar is my attempt at simplifying this process a little and making it slightly fault tolerant. You can initiate a mass mailing, let it run for a while, stop it, resume it, and generally not worry about double (or infinite) sends or getting irreversibly interrupted.

In it’s current state (0.1) it does not work out of the box. You’ll need to edit the code directly to insert your own ActionMailer model, and add a function for extracting the list of email addresses. There’s probably other stuff too. I still wanted to post it because it’s a good starting point and the code is very simple, so you should have no trouble finding where to make modifications.

If anyone finds this plugin useful, please take a moment to refactor it and make it more reusable. Send me a pull request or a patch and I’ll put your name in the credits. Hopefully we can get it to a works-out-of-the-box state.

One final warning: sending thousands of emails without knowing what you’re doing will almost certainly land you on a spam blacklist. Double and triple check that your SMTP server is set up correctly with RDNS, SPF, HELO, and other acronyms that I don’t know. Don’t ask me about it because I don’t know. I’d just rather roll the dice on getting blacklisted than pay the bulk sender fees.

On January 23, Gabe over at Penny Arcade gave a fairly glowing review of Obsidian Portal. It was short and simple, but basically said “They get it” when it comes to managing a D&D campaign. Minutes later, the PA masses started streaming in, and thus began the craziest couple of hours in Obsidian Portal’s history, and what I will forever call PA Day.

Beware the Cave of Tits!

The Epic Tail

Our traffic spiked by at least 1000% pretty much immediately. Page load times skyrocketed from a second or so into nearly a minute range. To compound the issue, I was at work plugging away on a hard deadline of “right now!” I couldn’t just ditch and go work on Obsidian Portal, so I watched helplessly as we were wanged.

Luckily, Ryan was a little more flexible than me, and he got to work. Logging into our slicehost panel, he immediately started spinning up a slice with 2GB of memory. All the while he was giving me updates via IM. It was like something out of a BSG episode, with the heroes spinning up the FTL drive while the Cylons are swarming them. (Note: We don’t consider the Penny Arcade users to be Cylons, although they might like that.)

Ryan: It’s at 30%
…
Ryan: 60%
…
Ryan: 90% almost done
…
Ryan: 100% – New slice is ready. It’s syncing and doing the switch
Ryan: come on…come on…
Ryan: We’re up!

All the while, I was terrified that something would go wrong in the switchover causing both the original and the new slice to be inoperable. Further, what if there was a problem with the DNS, causing requests to go to the old slice instead of the new one? I thought I was going to throw up, I was so nervous.

But within only minutes our new slice was up and running and doing a much better at handling the influx of traffic. Still, just to be sure, Ryan went in and started tweaking Apache / Passenger settings to make sure that everything was in tip-top shape. As we all know, Apache is pretty damned complex and it’s pretty hard to get the settings just right for your particular instance, especially on the fly. Things were running much better, so I was finally able to start relaxing.

The Fallout

Obsidian Portal had a massive spike in both visitors and signups. Our overall traffic was up 1,000% from the previous day, but even better, thanks to Gabe’s endorsement, the Penny Arcaders were signing up at a rate twice as high as regular visitors. Compare that to your average digging, reddit, or slashdotting, where visitors browse through and possibly leave a hate-filled comment, then move on.

By the time the dust settled on Monday afternoon when Penny Arcade posted a new story on their homepage, we had received thousands of new signups and hundreds of new campaigns. All in all, it was a fantastic weekend.

Visits were way up, but even better…

Pageviews stayed high even as visits declined. That means we hooked some of them!

Lessons Learned

Success or Failure?

Campaign updates were streaming in.

When the first wave hit us and our server started collapsing, it seemed as though we had failed. Here were thousands of people trying to get to our site, and they just plain couldn’t. Ryan especially seemed very frustrated.

However, when I thought about it later, I realized that it was a success, even if the site would have crashed and burned. Why? Simply because thousands of people were trying to get to our site. People were eagerly attempting to connect to our server and in the meantime were spreading the word about our site through the blogosphere, twitter, and others, letting all their friends know how the guys at Penny Arcade thought Obsidian Portal was so awesome. They also found resources in other locations such as previous reviews, Obsidian Portal’s Twitter account, Obsidian Portal’s Facebook Page, and even Michael Harrison’s Viemo Video World Building with Obsidian Portal.

The lesson? Maybe you scale, maybe you don’t, but at least you proved your idea. It’s a lot easier to take a popular site and make it faster than it is to take a screaming fast site and make it popular. The technology issues are by far simpler than the social ones.

Break out the credit card immediately

The very first thing Ryan did was to spin up a bigger slice, and that had by far the biggest impact on the number of requests we could serve. We could have tweaked MySQL settings, adjusted Apache processes, or experimented with exotic caching. Maybe it would have helped, maybe not. Optimization is a tricky business. When you’re at the point where your server is on its knees, you don’t have time to figure everything out, so spend money instead of time. Call your host and say “Gimme the biggest one you got!” You can always downgrade later.

Make basic optimizations ahead of time

Go ahead and spend a couple hours doing basic optimizations. For example, on Obsidian Portal, we had already applied some aggressive caching to the home page prior to PA Day. It took Ryan a couple hours to figure out. That’s a couple hours we didn’t have on PA Day, so we’re glad we spent them up front. Lesson? Don’t go nuts and try to optimize everything, but go ahead and do the basics. Do it now! Don’t wait!

Always have two sysadmins

I’m glad I have a day job (and a paycheck!), but that means I’m not always available to handle issues with the site. On PA Day, I was in crunch time at work and therefore pretty much helpless for Obsidian Portal. Luckily, there are two of us who can essentially do everything. That way, there’s a much better chance that if something happens, one of us can get to it and fix it before it causes too much trouble. It’s important to have failover backups for people as well as hardware, and someone to double check your ideas before implementing them in a hurry.

Yay Slicehost

Getting bigger iron was our fastest, safest, and best course of action. Slicehost made it easy. We were able to spin up a new identical slice with double the RAM and CPU in only a few minutes. Essentially, Ryan flipped a switch, waited 15 minutes, and we were on a beefed up server. No service call, no email, just raw speed. Perhaps other hosts offer this solution, I don’t know. What I do know is that Slicehost has always preformed exceptionally, and we thank them for that.

Yay Passenger

When Ryan wanted to move from Mongrel to Passenger, I was a little skeptical. I’m always hesitant to “fix” that which isn’t broke. Seems a little like running in place to me.

However, Passenger made scaling a snap, at least as far as server management went. No editing the mongrel cluster. No managing and restarting via God or Monit. Just leave Apache running and it will do its thing when the requests start rolling in. It’s just one less thing to worry about.

Going forward

Things have calmed down a bit now, and we’re back to the sure and steady growth that we’ve been seeing all along. Our new Penny Arcade fans have been successfully hooked and are steadily churning out new storylines, characters, and campaigns.

On our end, we’ll keep working on the site, adding features, fixing bugs, and improving server performance to be better prepared for more days like PA Day. The continual good feedback we get keeps us energized and hopeful that it will continue to grow. We may not be rich (yet), but we’re creating something that people love, and that feels great.

And one last thing. It’s a site written in Ruby on Rails and we made it scale!

You’re here because you want to be fully RESTful with Flex, found as3httpclientlib, and are now having some troubles. It could be because of the wonky Flash TCP security policies, problems with maintaining Rails sessions, or any one of a handful of other problems. Well, let me take away your burden, friend: Give up now and descend back into the darkness that is GET and POST.

Adobe, in their infinite wisdom, saw fit to only cover the basics with their built-in HTTP services. GET, POST, 200, 404, 500. Maybe there’s more, but not much. Stray into such strange territory as 201 (Created) at your own risk. Luckily, an enterprising 3rd party came along and figured out a way to get around this, creating the as3httpclientlib. Cleverly using XML sockets and connecting to port 80 allows for building your own web service controls inside of Flash. Brilliant! Unfortunately, it’s useless (or nearly so) for use in a real Flash app with a Rails backend. Give up!

I wrestled with as3httpclientlib for a few days, hoping that I could pair up Rails’s RESTful view of the HTTP with Flex. The first hurdle (as you know doubt know by now) was dealing with the Flash TCP security policy files. This stymied me for a few hours, as I had trouble serving the file from localhost to localhost. Eventually though, I figured it out. Still, requiring access to port 843 is a big red flag, as it immediately knocks out shared hosting and many other situations. So, unless you have control of every single server your Flex app will ever need to connect to, you’re screwed. One red flag…

Following my limited success with the policy files, I decided to push ahead. Immediately, I ran into problem two: session management. Without going into too much detail (since I don’t really understand it myself), the session information is not preserved across subsequent calls using the as3httpclientlib. URLLoader uses the browser’s connection to make the requests, and the browser handles transferring the session information back and forth. So, except in certain specific cases, using the URLLoader means that your Rails sessions just work. Not so with the as3httpclientlib. I didn’t test in detail, but it looks like a total game ender. I don’t think it’s possible (or at least easy) to read the cookies from Flash, and you’re going to have to manage sending that information yourself anyway. Red flag number two goes up…

Take my advice: Give up now and go back to URLRequest and URLLoader. The RESTafarian in me hates to say that, but you necessarily have to give up a lot of ivory-tower purity when you descend into Flex development.

Note: This is not a knock against the as3httpclientlib. From what I can tell, it’s a great library. I lay the blame for my problems at Adobe’s doorstep.

If you’re using the Facebooker SSH tunnels along with Phusion Passenger, you may run into an issue where you have multiple Rails apps running on your machine at different subdomains. Unfortunately, when facebook makes a request to your tunnel, it’s passed through on port 80 to your local machine and Apache chooses the first defined virtual host to serve the request, which may or may not be the one you want.

An easy solution is to use a ServerAlias directive in your vhost file that matches up with your tunnel domain.


<VirtualHost *:80>
ServerName my.local.rails.app.localhost
ServerAlias my.tunnel.domain
DocumentRoot /path/to/rails/app/public
</VirtualHost>


This works because the tunnel will pass the request through to your machine without changing any of the request headers. So, they come through with the Host header set to the tunnel domain, and Apache will match it to your server alias.

Update: (later that day…) A full stomach changes everything. Getting some breakfast lessened my git hatred quite a bit. Still, it manages to irk me on a daily basis, and I hate people who make their ill-thought-out blog posts disappear, so I will leave my hate-filled rant for posterity and a warning to myself not to post before I eat breakfast.

Update 2: (Feb 2010) I’ve been meaning to update this for a long time. I’m a 100% convert to git now. I’ve had my religious epiphany. I couldn’t understand how great it was at first, but over the past year I’ve finally wrapped my head around it. So, to all you haterz out there like me, just keep persevering. The light is there at the end of the tunnel. Now, back to the original rant…

Ok, I’m angry, so I’ll go ahead and make a ranty, heretical statement:

But wait! Micah, aren’t you the ultimate Rails fanboy? Don’t you jump on the bandwagon for everything the rest of the Rails lemmings do? RESTful routing? .html.erb instead of .rhtml?

Yes, in general, I do follow all the other Rails conventions and try to stay on the bleeding edge. I hate to feel like I’m falling behind, and that’s an easy feeling to get in the Rails community. So, I try to follow all the trends and at least educate myself on what it means and how it works. In most cases, like being RESTful, I eventually come to understand the underlying concept and appreciate the change. Learning new things often involves suppressing your own skepticism and the urge to stick to the known path.

So how is git different? Well, I’ve been using it for a few months, and the list of cons just keeps growing, while the list of pros stays nice and small. Sure, it’s fast, merging is dead simple, and I can scoff at all my friends still using subversion. But, every single time I want to do something, I have to look up a whole slew of commands. Why can’t I just memorize them? Because they’re horribly arcane and obtuse.

Want an example? Try deleting a remote branch. This seems like something one would do often, after merging a feature branch into the master. However, rather than using something simple, like “delete the remote branch”, you have to cast some arcane spell. For me, deleting a remote branch involves Google, every goddamn time.

The spark that set me off today was my attempt to connect to multiple repos on Unfuddle. For organizational reasons, I need to have multiple user accounts. Juggling usernames and passwords is bad enough. Luckily, they’ll allow me to use the same OpenID for both accounts. Still, when I tried to add my public ssh key, it complained that you cannot use the same public key for two accounts. Apparently, Github has the same policy.

Well, that was two hours ago, and I’m still trying to figure out how to get git to use my shiny new ssh key that is not named id_rsa. A little stumbling around got me some instructions on Github, but right at the top it says that it’s not an ideal solution and should only be attempted by advanced users. Hmm, using multiple different keypairs for different source control repositories does not exactly sound like an advanced feature. Do they recommend we use the same password for all our websites as well? Seems like a pretty basic concept to me.

Update: I was called “borderline retarded” in the comments for blaming git for the policies of github and unfuddle. Harsh, but true. I conflated git with hosts who provide git, and that’s not fair. So, github and unfuddle, my anger shifts to you!

Further, the recommended solution requires editing configuration files for ssh. So, I have to be an experienced ssh user to use git effectively? That’s ridiculous. Seriously, how many developers ever modify their ssh settings? I’ll muck around in git config files without worrying too much. Change settings in ssh without extensive research? Forget about it. Mess that up and you’ll lock yourself out of your machine, or worse, open it up wide to script kiddies who know more about ssh than you.

From a usability perspective, git sucks, and that’s that. If anyone feels compelled to defend git, please do so in the comments. However, don’t tell me how to do the things I’ve bitched about here. Tell me why it has to be so goddamn hard. To make an analogy, it reminds me of the Rails vs Java mentality. Java web apps are horribly convoluted. Sure, you can write tons of arcane XML and eventually maybe get your WAR deployed into Tomcat correctly. But why? Why waste time on this crap when it can be so much easier? Why spend minutes and hours trying to understand the obscurities of git when subversion, the ugly stepchild that it is, is still easier to use?

Subversion has its problems, but dear God why did the Rails fanboys have to jump to a worse solution? Now, in order to keep my certified fanboy card, I need to spend my precious development time tinkering with git, and then writing blog posts about how much I hate it.

If you allow rich text from your users in a Rails app, then you’re probably familiar with Textile. Either that or you’ve chosen the dark side, aka. Markdown. If that’s the case, I hate everything you stand for and I’ll bet you put your curly braces on the wrong line!

Assuming you’re one of The Chosen who has selected Textile as your input syntax of choice, you’ve probably got plenty of users who are turned off by the syntax. It may be easy for your average web developer to pick up yet another new syntax, but there are lots of people on the web who have never clicked the edit tab on Wikipedia, or think that links are specified with [URL=http://bbcode.com].

For these people, some sort of help is necessary. Quick tip sidebars, tutorials, and help pages only go so far. To make it truly useful you’ll probably have to dip into your Javascript bag of tricks. Luckily for you, a few others have already blazed that trail and created a handful of Textile utilities.

Textile Editor Helper (TEH)

NOTE: We’ve forked the Textile Editor Helper plugin to have more features and to conform to Rails 2.1 conventions. Our fork is available on github here:
http://github.com/felttippin/textile-editor-helper/tree/master

The Textile Editor Helper is a very simple toolbar that sits at the top of your textareas, much like the Wikipedia toolbar. It provides many of the most basic Textile options, like bold and italics, plus some text alignment. Unfortunately, it doesn’t include linking and image includes, which are really easy to do with Textile.

Supposedly, you can add your own buttons, but I haven’t confirmed this. Plus, I think the whole plugin is based around the idea of prefixes/suffixes for text (like * for strong, or _ for em), so anything that requires more than that might not be possible. So, TEH is a good solution if all you need is some basic formatting help.

Live Textile Preview

The Live Textile Preview (aka SuperTextile) is simply a Javascript implementation of the Textile parser. So, you can convert Textile to HTML on the client side. This allows users to experiment with the syntax and quickly see what the changes will look like.

Unfortunately, from my cursory testing, it only implements most of the Textile syntax. Basic text changes, images, and links are covered. However, advanced text alignment and CSS styling elements seem to be beyond its capabilities. So, it’s not perfect. Still, if you only need the basics, SuperTextile can make life bearable for your users.

Sanskrit WYSIWYG editor

Sanskrit is a true WYSIWYG editor, and the only one I’ve seen for Textile. However, it only does the absolute basics, so I can’t recommend it. Plus, I don’t think it’s a great idea to hide the raw Textile from your users. I’m a fan of the Wikipedia / Mediawiki way of things, where you provide some editing help, but your users are always looking at the raw text. Eventually, they’ll get the hang of things and won’t need the crutch of the editor buttons anymore.

markItUp!

markItUp! could be the most promising of all the solutions. A general purpose syntax parser written on top of jQuery, markItUp! can translate any syntax into HTML, provided you write the correct parser. Further, someone has already gone through the trouble and created a Textile parser!

I haven’t installed or tested markItUp!, but I was very impressed after playing around with the Textile demo. It seemed capable of most of the Textile syntax (including links and images), and even included a handy-dandy preview feature (you must render on the server side and return the HTML, I believe). So, it includes the best of all worlds: users work on raw Textile, but can get a live preview at the click of a button.

Update I finally got a chance to play with markItUp and it is hands down the best editor out of the bunch. Unless you need a true wysiwyg (in which case Sanskrit is the only one, AFAIK), then skip all the others and go straight to markItUp.

More?

Well, that’s all the Textile help I’ve found so far. Each one has its plusses and minuses, but any of them is probably better than a bare textarea with little or no help.

If I’ve left anything out, please let me know and I’ll add it to the list. With a little help, we can put together a comprehensive list of all the Textile tools available for Rails programmers.

I submitted my first Rails core contribution a while back, and I just happened to check in and see that it was accepted!