An Ode to Dark Sky

This week, the final curtains are closing on one of my absolute favorite online resources for weather data, Dark Sky. A startup originally founded to provide high-resolution rain forecasts for customers like airlines and event coordinators, Dark Sky had created a number of tools that were readily available for other app developers and weather enthusiasts.  The most remarkable of these, in my opinion, was a worldwide map interface that would display highly detailed colormaps of temperature, humidity, precipitation, wind, and other meteorological variables.  Their data accounted for topography more rigorously than any other resource I had seen, adjusting temperature and pressure for elevation differences, even estimating wind channeling effects over complex terrain.  Individual point values within these datasets were made available for use in other programs through the Dark Sky API. After the company was bought by Apple Weather in 2020, these free resources were no longer in the interests of the ownership and were gradually phased out.  While the map interface has long since disappeared from their site, the Dark Sky API has lasted until its scheduled termination date of March 31^st, after which developers must update their code to another source because this data will disappear for good.

I have some history with Dark Sky: in late 2019/early 2020, I was very interested in collaborating with (or even working for) the company, having several calls with team members over the course of a few months.  I saw their terrain heatmap as an ideal base layer for my tornado prediction model and thought that my tornado risk data (instantaneous and eventually aggregate) could be beautifully displayed with the same mapping interface.  I exchanged some code and sample data with a project manager.  In our conversations, the developers behind Dark Sky repeatedly gave me the disclaimer that their algorithms were not grounded in meteorology or physics, emphasizing that they were simply programmers using purely numerical methods. As I interviewed for the title of Data Engineer, I proposed running some of the same analyses on forecast validation that I had applied to my tornado probability model, knowing that it takes years of testing and documentation to be accepted by meteorologists as a dependable predictive tool. Ultimately our conversations wound down as Dark Sky finalized their sale to Apple, a deal that immediately constrained what these guys could share or work on outside of their core products.

It’s a little saddening to look back on what could have been – instead of merging my spatial predictive algorithms for severe weather with the sophisticated mapping and testing interface of Dark Sky, I began working in Texas shortly thereafter and all but abandoned my efforts on the project. I am grateful for their interest in my work, and I completely understand their decision to take the multimillion-dollar opportunity with Apple. After all, a corporate buyout is the end goal for most tech startups, and private weather analytics is an increasingly lucrative industry valued at over 17 billion dollars. However, as the market becomes saturated with products from the big players like Apple, Accuweather, Baron, and IBM (parent of The Weather Company), it has become more challenging for individual developers to break through without IP challenges. Coupled with the lobbying efforts that have weakened NOAA in favor of public-private partnerships, I worry that this general trend will have a negative impact on the amount of scientifically useful information that will be accessible for independent developers and scientists going forward.

First Nor’easter as a Northeasterner

I know that a certain groundhog said that spring is around the corner…but first, the most significant winter storm of the year is hitting the northeast. At the end of a warmer-than-usual winter season that has only brought about 2 inches of snow to New York City and 4 inches of snow to Boston, this storm is likely to double both of those totals while dumping up to a couple feet of wet snow on northern and western parts of New England. When I was a kid in Maine, snow totals like these were common within storms from any which direction, so no Nor’easter stands out in my memory. I’m watching this one from my front window, enjoying the serene sight of snow falling (mostly in a sideways manner) with an eye on my laptop as forecasts evolve.

Winter storm forecasts are fascinating to me because there’s such uncertainty, especially when conditions are close to the freezing line. While a Nor’easter was in the forecast more than a week in advance, snow predictions for my location ranged from nothing to over a foot (though always with a wait-and-see disclaimer) depending on temperature and the path of the low. The temperature stayed above freezing in my location throughout the storm, but somehow the meteorologists predicted spot on that the rain would change to snow around 1pm. The low crossed the base of Cape Cod and passed near my south shore location, so the characteristic northeasterly wind gusts only lasted a short time before ceding to lighter, variable winds. The snow slowed after the first hour, and I expect only 1-3 inches of light accumulation when the storm moves away tomorrow morning – luckily for the Boston area, this storm didn’t quite live up to the hype. Where I’ll be skiing in Vermont this weekend has received 2 feet and counting, though, which I’m pretty stoked about!

I haven’t lived in Massachusetts for long, but there is a very clear ‘snow line’ that generally defines winter weather here. The Atlantic Ocean regulates the temperature near the coast in the 30s throughout most of the winter months, which means we get some snow but arctic blasts are very infrequent (one cold snap this February set records as the first time in decades that Boston fell below 0 ^oF). However, a combination of the Appalachian mountain ranges and inland jet stream effects tend to leave areas north and west of a line between Lowell, Worcester, and Springfield with significantly more snow (that stays on the ground longer, too) than I will get just south of Boston. Unless there’s a Nor’easter with slightly colder conditions, as there was in 2018, then everybody on the eastern seaboard can get a blizzard. I will continue to be interested in the weather here in New England, which has interesting local differences posed by mountain ranges, coniferous and deciduous forests, lakes and oceans that contribute to a surprising variety in the climate.

Average yearly snow totals from 1981-2010, NOAA.