OpenStreetMap is highly addictive. You think you start with fixing the data for your neighborhood and a couple of days later you are modifying routing configuration on a major highway. You've been warned.
I've gone crazy. For almost two months I did not read any of my RSS feeds, I stopped posting much to diaspora/twitter, I bought a rangefinder, began to look suspicious, and talked to a lot of people.
I got hooked on OpenStreetMap.
OpenStreetMap was started by Steve Coast, inspired by Wikipedia and originated from his frustration around the pricing and secrecy of public map information in the UK.
Now the data from OpenStreetMap is used by a number of commercial companies providing various SDKs and functionality to compete with well-established big players, and this is a big deal.
At some point the project started to gain so much attention that TomTom even issued a warning (taken down, link to slashdot) saying that using the maps produced by crowd is not as precise as a commercial offering.
So, what did you actually do?
Back in Ukraine, it was a standard feature of every map to have the building number prominently displayed. Not the case in the US. For example, Google will not disclose this information unless you specifically search for a building, but will show you the numbers of the buildings in European(-ish) countries:
So, I made a plan to add house numbers to every building in Jamaica Plain, a neighborhood of Boston, MA, since I lived there. Having spent a number of days walking around trying to decipher the numbering schemes (spoiler alert - there aren't any, heck, official neighborhood boundaries are not even defined in Boston), I started to look into ways to automate the merge of the data from official sources and OpenStreetMap.
Upon finding the more-or-less reliable source I realized that a lot of Boston building polygons in OpenStreetMaps cannot be used as-is if I want to uniquely associate a building with its outline. You can put an address node approximately where the building is, but IMHO, it looks ugly as it does not allow you to associate the building boundaries when you are looking at it from the street. Originally, the building data was imported from high-definition LIDAR mapping done by MassGIS, and lasers are not really good at differentiating building outlines among terraced houses. A lot of densely populated Boston area blocks consisted of a single building polygon. A single biiig polygon. So I decided to go big and update as many buildings as possible, manually processing around 10,000 of them across all Boston neighborhoods, hoping that by the time I finish, I'd get an approval from City of Boston to use their data.
Unfortunately, as of May 5, 2016 this has not happened yet, but I have all the files available for immediate usage on OpenStreetMap wiki page for Boston SAM import.
While preparing for the merge, I went through quite a few streets that were misplaced, incorrectly shaped or named, and fixed the data based on the aerial imagery, various Boston City documents and location survey. Eventually I got carried away and demolished the bridge at the Casey Arborway Project location. It is slightly messy in Google Maps now - the roads are marked as closed, while in reality they were physically moved. The current state of construction is properly reflected in OpenStreetMap.
Relying on identifiers in OpenStreetMap long term is a bad idea, since they are expected to change. If you want to reference a feature, you should be using the Nominatim service instead. See Fresh Copy and Carolina Place nominatim links for examples.
There are three things you need to know about before you go crazy with OpenStreetMap:
Everything in OpenStreetMap consists of these three primitives. And you can do wonders with these.
Now, wonders are great, but you won't believe in one unless you see it, and here's another thing you need to know about OpenStreetMap: its main asset is the data, not the actual representation you see on https://openstreetmap.org.
Yes, the image tiles you get from openstreetmap.org may look dull, may not show e.g. offices of the companies or turn restrictions (no u-turns, no left turn etc.), but these are all there in the database. The data behind OpenStreetMap can be presented in various different ways, and it's up to the consumer to render the things they want.
Who are the consumers? We will get back to this at the end of the post.
Technically, all the data of openstreetmap.org is currently stored in PostgreSQL tables, but there are two formats used to transfer data in an out of OpenStreetMap - XML and Google Protocolbuffer Binary Format, PBF. XML variant is human-readable, so I will use that in the examples below.
Look at the code below:
<node id="4033939477" lat="42.3075132" lon="-71.1153142"> <tag k="addr:housenumber" v="64"/> <tag k="addr:street" v="South Street"/> <tag k="contact:website" v="http://freshcopyjp.com"/> <tag k="description" v="print, copy, fax, scan"/> <tag k="name" v="Fresh Copy"/> <tag k="opening_hours" v="Mo-Fr 09:00-19:00; Sa 10:00-14:00"/> <tag k="shop" v="copyshop"/> </node>
This describes a node I've added for a print shop I frequently used back in Jamaica Plain and you can see it on the web site as http://www.openstreetmap.org/node/4033939477 (provided that nobody deleted it yet).
Only nodes have the coordinates (lat - latitude, lon - longitude), and nodes are the building blocks of everything else.
The power of OpenStreetMap is in the free-form tags - a node with shop=copyshop (k="shop" v="copyshop", but usually it's displayed as key=value) is a Copy Shop. A node with barrier=gate is a gate which somehow restricts your movement. amenity=bicycle_parking is a place you can park your bike. Even though you can technically create a node and say it's building=yes, it's a wrong way.
A way is an ordered collection of nodes that can be either open (roads) or closed (areas). Technically, you can create an open way and tag it as a building, but it will be wrong, weird, and the validator will complain.
So, open ways are represented as:
<way id="8649811"> <nd ref="61413002"/> <nd ref="61413016"/> <tag k="attribution" v="Office of Geographic and Environmental Information (MassGIS)"/> <tag k="condition" v="deficient"/> <tag k="highway" v="service"/> <tag k="lanes" v="2"/> <tag k="massgis:way_id" v="131450"/> <tag k="name" v="Carolina Place"/> <tag k="source" v="massgis_import_v0.1_20071008193615"/> <tag k="surface" v="unpaved"/> <tag k="width" v="7.6"/> </way>
As you can see, there are two references to nodes - 61413002, 61413016, and those nodes in turn may be referenced by another way, forming a tree. Sometimes you can find an unconnected road in the middle of nowhere, but that usually means this location needs fixing (as I am going through adding sidewalks as separate ways now, I am making a lot of "islands" that will later be connected via driveways).
Closed ways, such as a building look like this:
<way id="408707735"> <nd ref="4037635242"/> <nd ref="326578904"/> <nd ref="4037635245"/> ... <nd ref="326578936"/> <nd ref="4037635244"/> <nd ref="4037635242"/> <tag k="addr:housenumber" v="57"/> <tag k="addr:street" v="South Street"/> <tag k="building" v="yes"/> <tag k="disused:shop" v="grocery"/> </way>
This way describes a former grocery store. The way is closed, since the last node reference is the same as the first one. It's that simple. Closed ways represent various kinds of areas - parks, buildings, lakes, fields, etc.
A relation can tie nodes, ways and even other relations together, forming something new, such as bus routes, highways, turn restrictions, coastal lines, buildings with courts inside, and many different things.
A bus route can be created by setting up a relation that ties together roads the bus goes through with type=route tag. If some intersection forbids a left turn, this is implemented as a relation with type=restriction, where role=from will be one road, role=to will be another road where you must not turn to, and role=via will be the shared node that you should not pass. Routing engines (such as OSRM or Mapzen) then take that into account when building the route.
Building with a hole is represented as a relation with tags type=multipoligon and building=yes, where the outer part is represented by a closed way with role=outer while the the "hole" is defined as another closed way with role=inner. The data consumers (apps, services) then understand what you wanted to do and will cut a hole in the area:
<relation id="63984"> <member type="way" ref="29631300" role="inner"/> <member type="way" ref="29637939" role="outer"/> <tag k="type" v="multipolygon"/> <!-- tag relation, not the outline: --> <tag k="building" v="yes"/> </relation>
So now we are done with primitives, time to put that all together and see how it makes things easier for you.
Making the map
In order to create a map of the features on the ground, you need to have some source, and the most popular are the aerial/satellite imagery, GPS tracks and manual survey.
Mapbox, a company contributing quite a lot of things to OpenStreetMap has also purchased Aerial imagery that can be used to trace the roads/buildings and everything else. In 2010 Microsoft gave their permission to use Bing aerial imagery for OpenStreetMap tracing purposes.
Sometimes you will find that a local government provides some data in a usable format but you must be sure that the data has a compatible license (as I have learned with Boston SAM Import the hard way).
GPS tracks consist of data points that describe your location, elevation based on GPS signal. You can use a separate GPS device or your phone to generate these. GPX (GPS Exchange Format) is a popular XML-based format for GPS tracks.
Manual survey means physically going to a location, making the measurements, writing down the location of the items with GPS coordinates, taking pictures, obtaining the building and road information. It's the most time-consuming way, but you'll get the most accurate data that no aerial imagery can provide (trees are surprisingly good at hiding everything underneath).
You can also use a service called Mapillary that crowdsources street photos. The coverage is spotty, but you can contribute to their project using your phone, so that you or other mappers will have more data. Mapillary explicitly allowed OpenStreetMap to use their data for mapping purposes.
Since the map consists only of nodes, ways and relations, it is extremely important to know what tags you can use with what values. New tags emerge and old ones get deprecated, you can find all that in OpenStreetMap wiki.
What about Google?
Okay, at this point you may be wondering why this all matters if mighty Google is going to save us all and provide the most up-to-date maps for free?
First of all, anything that goes into Google Maps stays there as the property of Google and you can't just download a snapshot of Google Maps data. You get routing, image tiles, building information that is proprietary to Google, and if you want to do some derivative work, you are entering legal waters.
Additionally, as much as Google Maps are awesome, they lag behind a lot in not-so-popular areas. For example, Casey Overpass bridge was demolished and alternative way was built, but Google Maps will direct you via an unnecessary detour:
Of course, if nobody updates OpenStreetMap, then it will be also outdated, but here's the power - you can fix the map issues yourself.
What about accuracy?
So, a month ago I was updating building geometries in North End of Boston, MA. Having surveyed a number of buildings I found an extra node on one of them that contributed nothing to a building, so I removed it w/o checking that a filter is applied that would hide a highway.
This caused a tunnel exit to be missing for about 6 hours until I realized my mistake and corrected it. Boston routing is hard, but you can only avoid making mistakes by not doing anything.
Of course, reputable GPS navigation providers don't make mistakes, do they?
There's even a GPS.gov page on reporting the issues to Google, Here, TomTom, Garmin and OpenStreetMap.
Please remember that navigation software is not magic and you should never trust it blindly, whether that is a community project or a commercial solution. To be extra-prepared, analyze the aerial imagery of the roads you want to take, look at street-level photos from Google, Bing, or even Mapillary. You'll get general idea of the road state and that will make it easier to reach your destination.
In case somebody maliciously changes the data, OpenStreetMap Data Working Group (DWG) can be contacted to revert the changes, and it is a good idea to become a long-term contributor, adopt a location and monitor the changes to make sure that the map is as correct as possible at any time. Like becoming a mayor of a place on Forsquare (or is it Swarm now?), but actually making a difference.
Back to OpenStreetMap
So when you edit the map, you most likely will want to have it modified visually, and for that you can use a number of editors:
- JOSM - desktop application
- Merkaartor - desktop application
- Potlatch 2 - flash-based in-browser editor
I am using JOSM for all my editing needs.
Can't comment on other editors, but here's the workflow:
- Find a spot on the map that needs correction.
- Download OSM data from the server.
- Make necessary adjustments, retrace the roads, update buildings, set height, dig tunnels, build bridges. This starts feeling like a game at some point, but if done well, you are actually making the world a better place.
- Commit the changes, specify the commit message and sources you've used.
- Rinse, repeat.
OpenStreetMap community does not generally welcome imports of external data into the map. I found that when I encounter a misplaced church or a playground in the middle of a highway that was a result of an import from incorrect data, so the attitude is justified.
Before you upload your changes, JOSM will run validation of the data you submit. This won't catch all the mistakes, but some important ones - crossing roads without intersection (is that a tunnel/bridge or there should be an intersection?), node tagged as a building, and many others will let you fix the data before it leaves your machine.
Afterwards, you can use services such as keepright and Geofabrik tools to make sure that map is as correct as it is possible. Note, that a map is never complete, changes happen here and there, so your job is to make it as useful for you as possible, and that will make it useful for others too.
Now, you've mapped your city, corrected intersections, added fire hydrants, or went completely crazy and measured the buildings around with a range finder and added 3D information to the buildings. The map looks great, but now what?
There is a lot you can do, actually. You can use OSRM to build routes (demo installation is at map.osrm-project.org), you can use iOS and Android applications that use OpenStreetMap data, you can download the data from the servers and do local processing. You can do almost whatever you want as long as you comply with ODC Open Database License (ODbL).
If you're on iOS and Android, you can use OpenStreetMap on your phone/tablet:
- OsmAnd - maps update monthly (my review).
- Maps.Me - maps update every time new application version is released.
- Navmii - claims maps update every quarter (but currently is a year behind).
- MapFactor Navigator - claims to update OSM maps every month.
- Telenav Scout - schedule unknown, and Telenav Scrobbler - maps update every time the app is updated.
- Eraser Map (beta, requires setting up a beta-tester account) - last tile update was a week ago.
- Magic Earth - update schedule unknown.
- Locus Map - ditto.
- BeOnRoad - ditto.
Since OpenStreetMap updates every second, there is no reason for the maps to be so outdated, so, developers, could you please invest some time in providing updated maps more often and publish the update schedule? This will show us, the users, that you have plans to stay relevant, and not just came up with some shiny software you have no plans to support. After all, map data should not be tied to your business logic.
I will start publishing reviews for each of these apps shortly since I had varying degrees of pleasure to use all of them during our recent relocation.
Go Map the World!
Joining https://openstreetmap.org made me want to go outside and explore places, talk to people around, note the different building types, street names, new businesses or interesting architecture...
And then go back home, research GIS extensions to an SQL database (yes, with PostGIS and MySQL/MariaDB you can store points, lines, polygons in a database and do calculations with them), play with QGIS, add the obtained data using JOSM, straighten the streets and fix intersections, and mark places to go next.
If you've always been a stay at home person who was shy to talk to other people (or you keep regretting you can't remember the street names around you), go map your street, neighborhood, city. You'll have a perfect excuse to talk to other people (it will look suspicious at first) and you can even get a couple of interesting stories about the building you've passed a thousand times.
I'm ryebread on OpenStreetMap. Go map the world, start from your street.