Midnight Oil

For Obsidian Portal, we wanted the ability for users to upload their own maps. The simplest way to do this would be to allow them to upload an image, then display it statically. While not a terrible feature, it has some definite limitations.

What we really wanted was a way to allow users to upload maps and navigate around them as everyone is accustomed to with Google Maps. Luckily, the guys at Google make just such a thing possible. As they say, a picture is worth a thousand words, so here are some examples of what we’ve accomplished. I will add more as users begin uploading.

• Kensing This is the main island for my D&D 3.5 campaign.
• Caedwyr Isle. This is another D&D 3.5 campaign, run by one of our users.

Note: Sorry if any of those links are 404s. The users can delete the maps as they please.

Over the course of a 3-day weekend, I was able to go from nothing to everything. I had a lot of stumbling blocks, though, and with this tutorial it should go much faster.

So, without further ado, let’s get some background on the technology that drives Google Maps: tiles!

What’s a tile?

The data set for Google Maps is just a giant collections of 256×256 images called tiles. When you view a map in your browser, the smarts in the system determine what part of the Earth is visible on your screen and requests only the tiles for that portion. The tiles are then laid out in a grid pattern to make a seamless image. Moving the map around spawns off asynchronous requests back to the server to get more tiles. In this way, the maps load quickly, and allow unlimited scrolling.

By default, the map pulls the tiles from the main server at Google. This makes sense as most people using the API want to display actual images of the Earth, or the nicely made street maps that Google does. The clever guys at Google, however, made it possible to switch this up and request the images from anywhere, thereby allowing developers to serve up their own tiles, yet rely on Google Maps to lay them out and stitch them together appropriately.

Tile cutter? Already written…

Ok, so now we know that we can tell Google to get the images from somewhere else, but now we need to make the images. It’s a lot to ask from your users to take their images and cut them into 256×256 chunks. So, we will need a tile cutter, or program that slices an image into the tiles.

Luckily, a ruby tile cutter does exist, and it works perfectly with Google Maps straight out of the box. The tile cutter does not have a lot of options, but for basic tile cutting it’s perfect.

Tile server? We choose S3.

We’ve got tiles, but now we need to serve them out. Mongrel is great for Rails apps, but not so great for serving images. A lot of sites (including Obsidian Portal) use some funky rewrite rules to try to get Apache to serve the static images and such. Still, serving tiles is a fairly intense task. Every time the user moves the map or zooms in and out, it will make several image requests to the tile server. For a mongrel server, this means it’s serving images rather than handling Rails requests, which is a big no-no.

Riding to our rescue is Amazon’s S3 (Simple Storage Service). Every object stored there can be made publicly accessible, and S3 will serve up the content with the correct content-type header. That’s perfect! If we can build our URLs correctly, then S3 becomes our tile server, thereby offloading all the heavy lifting to them instead of our Mongrel server.

From what I’ve seen so far, S3 is an excellent tile server, at least in terms of speed. Tiles are served faster from S3 than Google’s servers, in my subjective experience. Further, it was even faster than Mongrel serving the tiles from localhost to localhost. Your maps will load quickly, and be very responsive.

Finally, utilizing S3 allows us to store an unlimited number of tiles, which is very important if you have several zoom levels. The ceiling is determined only by the limit on your credit card

attachment_fu, bringing order to chaos

Every S3 bucket is a wild, untamed jungle, and things can easily get lost. Without directory structures or meaningful modes of organization, it can be very easy for objects to go in and never come out. Therefore, it’s very, very important to keep track of everything that is being placed in S3. For us, that means having a database record for every file in S3.

Luckily, that’s exactly how attachment_fu sees things as well. Every file managed by attachment_fu has a corresponding record somewhere, and when that record is deleted (using the destroy method), attachment_fu handles deleting the associated file. This is extremely handy for a map that may have 64, 256, or even 1024 associated tiles images.

Further, S3 communication is built right in to attachment_fu, so you, the developer, barely have to learn anything at all about interacting with S3. Just let the plugin do it.

Enough chit-chat! Show us some code!

Before you jump in and start coding, there are some steps to take first. I won’t go into detail, since it’s outside the scope of this article, but here they are:

1. Sign up for a Google Maps API key
2. Sign up for an Amazon S3 account
3. Install attachment_fu
4. Install RMagick (it’s needed by the tile cutter)

Get the modified tile cutter

The YM4R tile cutter is a command-line tool by default. Since it’s written in ruby, there’s really no reason not to make the calls directly, rather than resorting to using the shell. I slightly modified the tile cutter for this purpose, as well as to enclose it inside a module. I tried to retain its ability to be used from the shell, but I didn’t test extensively. It works for our purposes, though.

Get the modified tile cutter here: tile_image.rb Drop it into your /lib directory and everything will work like magic.

The MapImage and MapTile models

# Copyright (c) 2007 AisleTen, LLC – micah @aisleten.com
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software
# and associated documentation files (the “Software”), to deal in the Software without
# restriction, including without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

require ‘fileutils’
require ’tile_image’
require ‘aws/s3′
class MapImage < Image
include Tiler
include AWS::S3

has_attachment :content_type => ["image/jpeg"],
:storage => :s3,
:path_prefix => “map_images”,
:size => 1.kilobytes..500.kilobytes,
:resize_to => “1024×1024!”,
:thumbnails => {:thumb_512 => “512×512!”, :thumb_256 => “256×256!”}

after_attachment_saved {|x| x.send(“create_tiles”) unless x.parent_id }

validates_as_attachment

has_many :map_tiles, :dependent => :destroy

protected

# Creates the resized images needed for tile creation.
# Then calls the tile cutter.
def create_tiles
tmpfiles = []
[:thumb_256, :thumb_512, nil].each do |t|
# Pulls down the image from S3
key = self.full_filename(t)
logger.debug(“Retrieving #{key} prior to tiling.”)
data = S3Object.value(key, @@bucket_name)

# Store to a tempfile
tmp = Tempfile.new(“tile”)
tmp.write(data)
tmp.close
tmpfiles < < tmp
end

tmp_paths = tmpfiles.collect {|f| f.path}

tp = Tiler::TileParam.new(Tiler::Point.new(0,0), 0, Tiler::Point.new(0,0), 1)
zoom = 0..2

# Tile them in a temp dir in /tmp
tile_dir = File.join("/tmp", self.id.to_s)
FileUtils.mkdir_p(tile_dir)
logger.debug("Creating tiles in #{tile_dir}")
get_tiles(tile_dir, tmp_paths, tp, zoom)
create_tile_model_objects(tile_dir)
FileUtils.remove_entry_secure(tile_dir, true)
logger.debug("Deleting temporary tile directory: #{tile_dir}")
end

# Creates a MapTile object for each tile. This allows us to track the tiles individually
# in the database to prevent them from being orphaned in S3.
def create_tile_model_objects(tile_dir)
Dir.foreach(tile_dir) do |tile_name|
if tile_name.include?('.jpg')
t = MapTile.new
t.map_image = self
t.filename = tile_name
t.content_type = "image/jpeg"
t.temp_path = File.join(tile_dir, tile_name)

logger.debug("Saving MapTile for #{tile_name}")
t.save
end
end
end
end

This is a model for a map with 3 levels of zoom. As required by the tile cutter, we create 3 different size images: 256x256 for zoom 0, 512x512 for zoom 1, and 1024x1024 for zoom 2. Since I'm lazy, I just let attachment_fu create the images by resizing the original to 1024x1024 and then using the built in thumbnail functionality to get the other sizes.

After the images are created and sized we use tile_image (via the get_tiles function) to create all the tiles in a subdirectory of /tmp. These are then used to create MapTile objects (shown below), which get uploaded back to S3.

Astute readers will note that any non-square image will get distorted by this. One solution is to pre-pad the image with extra space to make it square. That is left as an exercise to the reader (Note: If someone comes up with a good way to do this with RMagick, send me the code and I'll post it and credit you.)

Astute readers will also notice that I'm wasting time and bandwidth pulling the images back from S3 after uploading them, rather than breaking in to the attachment_fu upload cycle and tiling the images before they're uploaded. I'm lazy and it works. Send the code if you know how to do it better.

Note: MapImage subclasses from Image, which is an abstract base class I use for all my images managed by attachment_fu. Single-table inheritance allows me to store multiple kinds of images in the same "images" table this way. This is not necessary to get the custom maps to work, it's just how I like to do things.

# Copyright (c) 2007 AisleTen, LLC - micah@aisleten.com
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software
# and associated documentation files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

class MapTile < ActiveRecord::Base
has_attachment :content_type => ["image/jpeg"],
:storage => :s3

belongs_to :map_image

# We overwrite the base_path method for the tiles in order to place them
# in the tiles “directory” under their parent map ID on S3
def base_path
File.join(“map_images”, self.map_image_id.to_s, “tiles”)
end
end

The MapTile class is very simple, thanks to attachment_fu. The only real trick is to overwrite the base_path method. This allows us to modify the S3 key (ie “file path”) given to the MapTile image when it is stored. Instead of being stored using its own ID, we would like to store it using the ID of its parent MapImage. This is the structure that will allow us to serve up the tiles.

The View…now with added JavaScript!

google.load(“maps”, “2″);

var map_z_offset = 2;
var map_zoom_levels = 3;

function load() {
var customType = customMap();
var map = new google.maps.Map2(document.getElementById(“map”), {mapTypes: [customType]});
var newCenter = new google.maps.LatLng(79.1, -135);
map.setCenter(newCenter, (map_z_offset));
map.addControl(new google.maps.SmallMapControl());
}

function customMap() {
var copyCollection = new google.maps.CopyrightCollection(”);
var copyright = new google.maps.Copyright(1, new google.maps.LatLngBounds(new google.maps.LatLng(-90, -180), new google.maps.LatLng(90, 180)), 0, “”);
copyCollection.addCopyright(copyright);

var high_level = map_z_offset;
var low_level = map_z_offset + map_zoom_levels – 1;
var tilelayers = [new google.maps.TileLayer(copyCollection, high_level, low_level)];
tilelayers[0].getTileUrl = CustomGetTileUrl;

var custom = new google.maps.MapType(tilelayers, new google.maps.MercatorProjection(low_level + 1), “Chart”, {errorMessage:”"});
return custom;
}

function CustomGetTileUrl(point, zoom) {
var img_id = “< %= @map_image.id %>“;
var bucket = “< %= @map_image.bucket_name %>“;
return “http://s3.amazonaws.com/” + bucket + “/map_images/” + img_id + “/tiles/tile_” + (zoom – map_z_offset) + “_” + point.x + “_” + point.y + “.jpg”;
}

google.setOnLoadCallback(load);

Warning: I will be the first to admit that I don’t know squat about JavaScript. I’m learning as fast as I can, but it probably looks terrible to someone who really knows what they’re doing.

Most of this code was lifted, almost untouched, from the Mapki page on creating custom maps (see references below). The Google Maps code specific portions of creating custom maps has been understood for a long time now. Still, I’ll explain some of the things we’re doing that are special.

I am offsetting the map by 2 zoom levels. So, when we have a tile for zoom 0, we will display it at zoom 2. This will effectively give some buffer space around the image. If you display exactly according to the zoom level, it will look a little strange. At zoom 0 (all the way out), it assumes you’re looking at the entire world (which is roughly spherical), and therefore will place the same tile multiple times in a line. This works for a world map, but looks strange for anything smaller. So, we offset a bit by fooling the system into thinking that we’re zoomed in. That way, it will pad around our image with blank space.

Near the top, you will see that I am setting the Lat/Lng to a seemingly arbitrary number. This is related to the zoom offset. Tile location 0,0 is somewhere in the arctic ocean. When the map first displays, I want zoom level 0 (a single tile) to be visible in the center. I didn’t find a quick and easy way to calculate from tile X,Y to Lat/Lng, so I just started experimenting with values. (79.1, -135) works well for zoom offset 2, while (66.66666, -90) works well for zoom offset 1. If you want to offer arbitrary zoom offsets, you’ll need to come up with a better way of handling this. Send me the code and I’ll post it.

Finally, the real meat! GetCustomTileUrl is where all the magic happens. This is how we tell Google Maps to use our custom tile server instead of the default one. The parameters are an X,Y point and a zoom. Our tile cutter names the tiles perfectly for this, and we placed them in S3 according to the ID of their parent MapImage, so all we have to do is construct the URL based on this ID.

That’s It!

Now we’ve put all the pieces together. attachment_fu handles uploading and resizing, tile_image handles cutting the tiles, and S3 handles serving them out.

Issues

Image processing on the mongrel thread

If you were paying attention, you’ve realized that when uploading a map everything happens during a single HTTP request. By my estimation, uploading to S3, downloading the resized images, running the tile cutter, and then re-uploading the images takes between 5-10 seconds for a map with 3 zoom levels. In my case, the end total is 26 images that have to be created and moved. For any reasonable size site, tying up the web server for 10 seconds is out of the question. You can counter this with more mongrels, but that’s an approach that won’t scale. Plus, the time increases exponentially with more zoom levels. Tiling 6 or 7 zoom levels could take several minutes, even on a beefy machine.

Fortunately, there’s no reason the image processing and uploading has to be done on the main thread. Simply allow the users to upload the image and push the single image to S3. In the database, flag it as “not tiled” Then, asynchronously, have a script that periodically wakes and scans the database for maps that need to be tiled. It does the tiling work and uploading and then flips the flag in the database to “tiling finished.” For the user, they upload their map and then are taken to a screen that says, “We are currently preparing your map. Please wait a few minutes for the process to complete.” In reality, if the number of maps is low, and the zoom level is small, it could only take 10-20 seconds, assuming your script wakes up frequently enough to check.

Maps must be square

Because of the way the tile cutter works (or at least how I understand it), the starting image must be perfectly square. This can be dealt with by padding the uploaded image with a neutral (or transparent, for PNG) background until it has square dimensions, then send it to the tiler. Like I said before, if someone has a good way of doing this with RMagick, send the code and I’ll post it and give credit.

No map markers

This isn’t technically an issue, it’s just another feature I want Next up on the feature list is the ability to add markers to your map, and drag them around. By using Google Maps as the backend, this should be quite easy to do, at least I hope so. I’ll cover that in part 2, if I ever get around to it.

Resources

These are required resources for this to work at all. If for some reason you are unable to use any one of the following, the entire approach falls apart.

• AWS::S3 Library – The main page for the Amazon S3 Ruby library. It has great examples and documentation.
• Amazon S3 Homepage – Go here to sign up for Amazon Web Services (AWS). Make sure to also sign up for S3, since AWS is comprised of many services, and you have to sign up individually for each one.
• Mike Clark’s attachment_fu tutorial – Pretty much the tutorial/howto for attachment_fu
• YM4R – Includes the tile cutter. While not strictly necessary if you use my modified tile cutter, they still deserve the credit and so are listed in the required resources.
• Google Maps API Key signup – You will need a Google Maps API key, otherwise you can’t use the map code. It’s free, so don’t worry about it.
• RMagick – This is a ruby interface to the ImageMagick (or GraphicsMagick) libraries. It is needed for all the image resizing and cutting.

These are some resources that I found incredibly useful when trying to figure all this stuff out. They give some good background.

• Google Mapki – A wiki devoted to using Google Maps in interesting ways.
• The Mapki page on creating custom maps – Where I lifted almost all of the JavaScript
• MapWoW.com – A World of Warcraft map that convinced me the idea was possible.
• How to make a site like MapWow – Has a good graphical explanation of the exponential growth of tiles when zooming in.

Credits

Thanks to Jordan Bethea (a player in my D&D campaign) for suggesting the feature, and thanks to Scott Turnbull for guiding me to MapWoW.com and giving me tips on how to implement it.

Thanks for reading, and if you liked the article, please consider Digging it or voting for it on your favorite social bookmarking site.