ECMWF Eeurope

Clouds Predicted in the Cloud: How ECMWF Distributes Big Weather Data Using CloudSigma

Customer Profile

The European Centre for Medium-Range Weather Forecasts (ECMWF) is an independent intergovernmental organisation supported by 34 states. It is both a research institute and a 24/7 operational service, producing and disseminating numerical weather predictions to its Member States. This data is fully available to the national meteorological services in the Member States. The Centre also offers a catalogue of forecast data that can be purchased by businesses worldwide and other commercial customers. The supercomputer facility (and associated data archive) at ECMWF is one of the largest of its type in Europe and Member States can use 25% of its capacity for their own purposes.

The organisation was established in 1975 and now employs around 300 staff from more than 30 countries. ECMWF is one of the six members of the Co-ordinated Organisations, which also include the North Atlantic Treaty Organisation (NATO), the Council of Europe (CoE), the European Space Agency (ESA), the Organisation for Economic Co-operation and Development (OECD), and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT).

Choosing CloudSigma enabled them to provision large volume near real-time data sets generated at ECMWF to CloudSigma’s public cloud where Member States can access those ECMWF data sets. Without any ECMWF intervention their Member States are able to go directly to the CloudSigma cloud platform, set up their own account, and start receiving the required data.

“With CloudSigma, today we can offer to our member states something we could not offer before, which is to have their own virtual machine, with their own software, they take care of their own licences for that deployment, where we co-locate the data that they need.”Ricardo Correa, Network Applications Expert @ ECMWF

The Challenge

The main objective of ECMWF is to send their weather forecast products to the Member States. Since many of those products are of quite large volume (hundreds of gigabytes, even up to a terabyte), sending them over the internet is not an option.

“The internet speed was so slow that by the time the product arrived, the data in the products was useless and was not worth anything. We’re talking about weather forecast data, after all. You either deliver it within one hour or less, or you don’t deliver it at all.”

Therefore, ECMWF was looking for a solution, in which they can process the products and then make them available to the Member States via a virtual machine in the cloud. Since ECMWF does not provide virtual machines themselves, they had to consider a public cloud provider as a partner.

ECMWF searched for contacts via Helix Nebula – the Science Cloud Initiative. After testing with a couple of providers, CloudSigma proved to be the only one able to meet their requirements.

“We did a proof of concept with those providers that agreed to offer a networking test for us, and then the best one was CloudSigma, able to offer the bandwidth we needed.”

The Solution

The key success criteria for the ECMWF use case is to be able to get the data to the site of the Member States on time for it to be useful for any potential third party customers, since the value of weather forecasts falls very quickly with time.

Hence, the major requirements were fast networking, sustaining up to 3Gbps for real application data transfers from ECMWF with fast storage able to write data quickly. Then fast computing is required to process that data in time.

“Fast networking and top disk performance are the main reasons for choosing CloudSigma over other providers.”Ricardo Correa, Network Applications Expert @ ECMWF

In addition, the products generated by ECMWF have a very restricted daily schedule and ECMWF is not allowed to release any product before a scheduled time. With CloudSigma both ECMWF and their customer Member States have their own accounts and ECMWF is able to send the products to the CloudSigma server of the Member State at exactly the scheduled time. After receiving them,the Member State can then post process the products in their virtual machine and expose the results to their own customers.

“We stage these products at CloudSigma, because we get a fantastic network bandwidth to our data center – up to 6 Gbps. As a result, we are able to send the products to our Member States at exactly the scheduled time.”

Using the commercial internet providers, ECMWF could not achieve the performance they needed. The maximum performance was 2 Gbps and there was a lot of packet loss. At CloudSigma the solution was enabled by using a private peering session with GEANT via SWITCH.ch with a dedicated bandwidth of 10Gbps (with a commit of 3Gbps 95th percentile).

Another key requirement for the ECMWF use case is to ensure a highly available architecture to avoid single points of failure at the infrastructure level. This was achieved by using the ‘avoid’ functionality for defined servers offered within the CloudSigma functionality and combining this with the node failure tolerant clustered storage system used to drive CloudSigma’s SSD storage offering. The clustered storage system used by CloudSigma offers high availability and an ‘always live’ environment that can survive hardware and software life cycles uninterrupted. This allows true high availability cluster setups to be provisioned.

The Impact

Both ECMWF and their customers are very happy with the service performance, and using the cloud for other things apart from delivering or receiving this particular forecast service. The initial customer that was onboarded today serves as an example for promoting this new ECMWF service. ECMWF is in the process of increasing its collaboration and partnership with CloudSigma in order to benefit further from the expansion of their own service.

“Our customers are very very happy with the service and they are planning to automatically scale horizontally doing all kinds of things.”

A year after the cooperation between ECMWF and CloudSigma started, ECMWF are planning to benefit from our global exposure. On a closely related project ECMWF has commercial customers in the U.S. requiring data to be delivered to their sites.

By using our Cloud infrastructure ECMWF is able to transmit the data from within the U.S. lowering network latencies and therefore increasing data transfer performance. These customers used to be served from a proxy host in Canada but recently the whole infrastructure was successfully migrated to CloudSigma data center in Washington DC.

“The service is much better than I expected. I really did not expect this networking performance.”Ricardo Correa, Network Applications Expert @ ECMWF

The Future

A next step would be to consider ECMWF customers in Asia Pacific, where they could use CloudSigma’s locations in Manila, Perth or Melbourne and dramatically improve the latency of the current service they’re offering.

“We’re trying to deliver our forecast to the commercial customers worldwide as fast as possible. However, especially for places like Malaysia, China, or Singapore the bandwidth from Europe is really really bad. A small number of dropped packets will bring the performance to very low level. With your new cloud locations in SEA and Australia we will be able to achieve a much higher performance in the whole Pacific area.”

Another planned step is to use a process to automatically create all virtual machines at CloudSigma and use the API for building and provisioning the virtual machines with all the software and tools required by a certain ECMWF customer. In this way customers of ECMWF will be able to choose a customized VM image from the CloudSigma cloud marketplace that has been pre-configured by ECMWF. In addition, the ECMWF tools that help customers interpret and number crunch the obtained data can also be integrated onto the Services Marketplace of CloudSigma.

Download PDF

About Vanya Nikova

Vanya is leading the Global Customer Development Team at CloudSigma. Beside that, she is responsible for a number of big data and big science partners and projects at CloudSigma. She has a Masters in Business Administration from the University of Mannheim, Germany with 10 years of work experience in sales and consulting services.

Leave a Reply