Now that the election is over and I am well rested, I want to look back at what we have worked on all these months at msnbc.com. A few people wrote me asking about the process of creating a large scale project in a fast paced environment. Here is a longer answer to that question. Since this post will be a bit long, I will break it into several sections.
We put a huge amount of effort into the Decision â€™08 Dashboard data explorer map. Many people had been preparing interactive content for our election coverage for months before I arrived. By the time I started in July 2008, most of the core functionality of the map had been developed. My first assignment was to support other developers and designers, by preparing data and writing small utility classes. Next, I created the polling data display and helped with designing the fundraising data display.
We wanted to map every data set we could find that came with geographic information. There were at least 10 data sets we wanted to map, but we launched our politics front map with only six data sets. We added user opinion, polling, candidate appearances and fundraising data over the next several weeks.
Most of our data sets were huge and we needed the technology to support and serve them to a large audience. This is where the team that builds our publishing platform came in. They built a new, user-friendly database tool just in time for us to use for the election.
The amount of information that the maps show is enormous. We have state and county level demographics data such as age, race and education from the U.S. Census and voting history of states and districts back to the 1980â€™s. We also mapped individual contributions to the presidential candidates as well as every single candidate for a seat in the Senate and House of Representatives. Not a lot of our competitors have that kind of depth. In fact, I canâ€™t recall one that has this much variety of information either.
Of course, there are a few more things we wish we had done, but given the deadlines that we had, I think we did a good job
Election night plans
While I was not heavily involved with the design and implementation of the pre-election maps, my responsibilities changed as we started working on the election results maps. With Chris Cast, a fellow producer from msnbc.com, we was responsible for planning and designing the pre-election maps. We spent some time thinking about the data, and trying to come up with the best way to visualize it. We also thought about what the competitors might do. Chris was more heavily involved with designing the mock-ups while I thought about the code that we would have to write to make things happen.
The Creative Director of msnbc.com, Ashley Wells, had a desire to dominate the web with msnbc.comâ€™s election coverage (not too shabby for a goal =)). To make that happen, we had to tackle a few challenges. First, we had to make sure our servers could handle the traffic that was expected on election night. We also had to come up with an effective way to distribute our content around the Internet.
To meet this very ambitious goal, we did a few things. First, msnbc.comâ€™s web operations team team spent a great deal of time optimizing our server capacity and stability. Second, we created maps for our homepage and the homepage of MSN that could drive traffic deep into our site where more results and maps were displayed. Third, we made portable maps that could be embedded on other peopleâ€™s Web sites and blogs that also linked back to more detailed results. Last but not least was the results map on the Decision â€™08 Dashboard. We also worked with MSN global to create a flexible version that they could translate into many different languages and place on MSN global homepages and news channel pages. The results of several months of work look like this:
While the whole designing and planning process was still going on, I started to build the code foundation upon which most of our election night maps relied. The Politics front results map was the only map that used a slightly different code base. Although I adapted a lot of the code that had been written for our pre-election maps, this job was a challenge for me. It was a challenge because I had to optimize our maps to reduce their file sizes and make sure they only requested the data that they needed. Although we were confident about the serversâ€™ ability to handle the predicted traffic, we wanted to minimize the amount of database queries that the maps made, especially because we were making the calls every two minutes to ensure users had the lastest results. In the end, we eliminated a couple of minor features so that we didnâ€™t put unnecessary stress on the system.
Another challenge we faced was the MSN US map. We had to be extra careful with the file size because of the amount of traffic that would hit the map. Chris Cast and my manager Paige West, who is the interactive director at msnbc.com, spent a great amount of time optimizing the map. Things were a bit easier with MSN global. I was quite happy after seeing the screenshots of the different localized versions.
The biggest problem that we faced throughout this project was with the data. There were a lot of inconsistencies and special cases in the results feed. It probably took 30% of our development time dealing with special cases. It was hard to check the data ahead of time because of its size and the low rate at which these inconsistencies occurred, so until we actually started testing with sample data, we didnâ€™t realize how many problems it had.
Testing testing testingâ€¦
The team spent a lot of time testing the maps to make sure that all the features worked appropriately. In the last few weeks leading up to the election, we participated in several transmission tests and dress rehearsals in which both the editorial and the technical staff at NBC and msnbc.com practiced for every conceivable scenario on election night. We put together a prioritized list of bugs and spent the last couple of days fixing them. Although the bugs list was relatively big, we managed to fix every single one that we found. With five people working on fixing bugs at the same time, the list disappeared in almost no time =).
As more and more people began to see the finished product we had a few requests for additional features or improvements to the existing functionality. Although these requests came late in the game, the team managed to push them through by working in shifts over 24 hrs for several days leading up to the election.
We released the portable maps a few days before the election and hoped that they would get picked up by the social media. We also distributed these maps to a few NBC affiliates, some small newspapers, and other media outlets that did not have the resources to develop this kind of data-based map. Because we considered this project to be highly experimental in nature, we had fairly reasonable expectations. In the first few days after release, the amount of installs grew quickly, but on the day of the election the number of installs skyrocketed.
The maps on our site and all the MSN sites went up in the afternoon of the election day. We continued to make small improvements right up until the maps went live, but over all I think it went pretty well. This fact was later confirmed by the traffic stats. Here are a few excerpts from that report.
Nielsen Online shows that for election day msnbc.com was #1 in total minutes and time per person, doing the best job at meeting usersâ€™ needs and holding their attention.
The combined total in page views on msnbc.com was nearly half a billion: 471 million.
Interactive maps and the one-stop Decision â€™08 Dashboard drove over 100 million page views on 11/4.
Our best-of-breed news widget was shared by users all over the web, driving an unprecedented 12 million widget page views, and driving over 8 million click backs to msnbc.com. The widget was used by approximately 2 million unique visitors over the 2-day period.