SWITCH Network Blog


Leave a comment

Frequency Dissemination in the SWITCHlan network

In collaboration with METAS, University of Basel and ETH Zurich, SWITCH is very happy to announce the successful implementation of the first ultra-stable SI-traceable frequency dissemination in Switzerland, on top of the SWITCHlan network, between Bern, Basel and Zurich and parallel to the existing data traffic. The project, started in 2019, has been progressing methodically over the past year, successfully meeting the milestone of the first transmission of a frequency signal by the end of August 2020. Since then, the transmission parameters have been optimised and the overall performance evaluated. These results have now been published.

Many applications in physics research require a precise measurement of fundamental quantities. One of the most important quantities in the SI-system is the second, which can also be derived from a frequency signal. While there exist many technologies to obtain this quantity, they offer varying degrees of accuracy. The most accurate option currently available in Switzerland is the FoCS-2 primary frequency standard at METAS. This project thus aims to disseminate this frequency standard to the labs at the University of Basel and the ETH Zurich, in order to be available for physics research, e.g. for molecular spectroscopy.

To achieve this, the researchers in Basel and Zurich teamed up with METAS and SWITCH in a Swiss National Science Foundation (SNSF) Sinergia project to realize this frequency dissemination. To transmit a frequency, we do not use a data transmission, but the physical properties of the laser itself. In simplified terms, a narrow linewidth laser is made extremely accurate using an ultra low expansion (ULE) cavity. This laser might be very stable, but its absolute frequency is not predetermined. Instead, it is compared to the primary reference which is itself part of the international system for defining the SI units and the international atomic time (TAI). The laser light is then sent over the fiber to the first lab. However, the signal arriving at the remote site is no longer stable, for the fiber is subject to mechanical vibrations leading to small changes in the frequency; an effect known as doppler shift. To compensate for this, the laser is sent straight back using a mirror. At the originating side, this return light is compared to the original laser, which is then shifted in frequency to precompensated for the error, such that the light will arrive at the remote site with the correct frequency. For all the fine details, please refer to the publication.

While the technology is quite complex, it is not a new invention and it has been implemented by multiple metrology institutes in different countries. By drawing on the experience of the respective implementers, especially INRIM and Cesnet, we were able to adapt it on the SWITCHlan network. In order to save the prohibitively expensive fiber lease costs for this project, the frequency signal had to be multiplexed with the other optical channels used for data traffic. To avoid any issues with the spectrum management of the data traffic signals (for the frequency signal will be fixed), especially in anticipation of future data transmission technologies which will use the spectrum much more dynamically, the frequency signal was moved outside the C-band used for the data traffic. The final selection of 190.7 THz (1572.06nm) was mainly chosen based on parts availability, as the implementation requred custom made lasers and other components.

The results now available show a very good performance of the frequency dissemination. With an accuracy of $2*10^{-18}$, the error added by the transmission is 3 orders of magnitude below what the frequency standard provides, enabling an almost perfect reference at the remote site. This shows readiness for the next generation of frequency standards expected in the future. At the same time, we also measured the effect on the data transmissions, comparing the forward error correction (FEC) performance used in 100Gbit and 200Gbit signals with the frequency signal enabled and disabled. These measurements showed no correlation to the frequency signals confirming that multiplexed transmission is possible without any impact on the data traffic.

Now that this implementation has been successfully demonstrated we are looking forward to extending the frequency dissemination network to other locations. One of the prime candidates will be CERN, where a number of projects could benefit from this, and where there is an opportunity to compare the FoCS-2 frequency reference to other frequency standards, aiding in the international definition of TAI.

This project it funded by the Swiss National Science Foundation (SNSF), Sinergia grant nr. CRSII5 183579.

FM (Text)

P.S.: Also read the SWITCH Story High-precision frequency for research.


Leave a comment

SWITCHlan 2020—Year in Review

When the community met in Schaffhausen on January 21-22, 2020 for the Network and Security WG, I don’t think any of us thought it would be the last such event for a long time.

Impact of COVID-19

The first measures were taken at SWITCH in early March. Among other things, some employees were sent to the home office as a “field force” to be deployed in urgent cases in case the disease spread to the SWITCH head office.

Then, on March 13, the Federal Council announced measures to contain the coronavirus for the whole of Switzerland.

At that point it became clear that the network team—like most other teams at SWITCH—would no longer be meeting in the Head Office until further notice. We had no idea that this would remain the case until now, ten months later, with two exceptions.

Since we already had a long-term remote worker in the team, and some of us had been sent home as part of the aforementioned “field force”, we were not entirely unprepared. Also, the fact that most of us had been working together for a few years (or decades) helped with the transition to ~100% remote work. In addition to email, we use internal text chat for exchanges.

Part of the NeWo team in a meeting

Our colleagues in the MAPS and Collaboration teams had already set up a Jitsi infrastructure in no time at all at the beginning of March in order to be able to offer the SWITCH community an uncomplicated solution for videoconferencing (in the meantime, commercial services such as Zoom, Teams, WebEx or Meet mostly cover this need). In our team, we have been using this “meet.switch.ch” intensively since the first lockdown, for traditional work meetings but also for informal exchanges in the form of “coffee breaks”—twice a day, because networkers are known to consume a lot of coffee.

Impact on network traffic

From one day to the next, SWITCHlan’s usage patterns changed drastically. Normally, during normal (research and teaching) working hours, we transfer a lot of traffic from the general Internet to the campus networks of the universities. This traffic suddenly disappeared as virtually no one was on campus. In its place, demand for lecture streaming and other collaboration and video solutions skyrocketed. The users were (and are) mainly connected to the broadband networks of operators such as Swisscom, UPC, etc., with whom we maintain peerings (direct interconnections). Since many users apparently use the VPNs of their respective institutions, most of this traffic then traverses the campus networks again—only it is sent on from there via SWITCHlan in the direction of the aforementioned broadband providers.

Emergency upgrades

Traffic between SWITCH and Swiss exchange points/peers

As a result, traffic on our peering links with Swiss broadband providers rose sharply. We immediately set about upgrading the most important of these, and thanks to energetic assistance from our partner providers and the operators of the exchange points, we were able to put some of these upgrades into operation in the first few days of the lockdown. Among other things, SWITCHlan’s connection to SwissIX is now running redundantly and at 100 Gb/s.

In addition to the Swiss access providers, there were also other peers with a sharp increase in traffic, especially those hosting the large video services (Zoom, etc.). Here we had to do some “traffic engineering” in some cases. A lucky coincidence was that after long months (or was it years?) of planning, we were finally able to put a 100Gb/s link to DE-CIX in Frankfurt into operation in mid-March.

Geneva, March 16: Racing to the next SWITCHlan PoP for HUG upgrade

Some SWITCHlan connections also needed emergency upgrades; so on March 16—the day the Federal Council declared an extraordinary situation—we were able to help the Hôpitaux Universitaires de Genève (which includes the National Reference Center for Emerging Viral Infections) prepare for the onslaught by upgrading to redundant 10Gb/s.

Projects

Even though the COVID-19 pandemic has shaken up a lot, we have been able to work on some interesting projects. Here are some highlights.

Time and frequency distribution

Highly accurate time and frequency sources, such as those maintained by national metrological institutes (in Switzerland: METAS), can be verified or made even more accurate by synchronizing them with each other. This is possible via high-precision methods over optical fibers. These processes are quite different from those we use to transmit data. But with a few tricks, it’s possible to use the same fibers for both.

SWITCH was actively involved in two projects on this topic in 2020: Together with METAS, we set up a triangle Bern (METAS in Wabern) – University of Basel – ETH Zurich. The first METAS publications with measurements on this system should be published soon. A special feature of the system is that we use bidirectional links here (outbound and inbound signals on the same fiber), which requires the use of special components. Thus, some bidirectional optical amplifiers have been in use since this year, and we are now working out their teething troubles with the manufacturer.

Bern, May 4: bidirectional amplifiers for the Time/Frequency project

At the European level, GÉANT and some national metrological institutes are also working on time/frequency distribution. The discussion about suitable approaches for the coexistence between frequency transmission and “normal” data transmission is much more complex. We have been able to contribute valuable experience from building the Swiss triangle.

See also the story “High-precision frequency for research,” September 2019.

SCION

In the SCION project for secure interdomain routing, we were also involved in two projects: One is SCI-ED (SCION for the ETH Domain), in which all institutions of the ETH Domain get SCION connections. On the other hand SSFN (Swiss Secure Finance Network), where the SCION concept is being tested under the aegis of the Swiss National Bank for its suitability as a future infrastructure for Swiss interbank clearing. Other partners include SIX and two commercial Swiss ISPs, as well as the ETH spinoff Anapaya.

P4/RARE

In the EU project GÉANT (officially GN4-3), the SWITCH network team is involved in the area of time/frequency (see above), but also in activities around novel “programmable dataplanes”. P4 is the state of the art here and is considered an improved successor to OpenFlow. In the GN4-3 subproject RARE (Router for Academia, Research and Education) an already existing open source software platform (FreeRouter) was equipped with a backend for P4 compatible devices.

Zurich, April 30. Flow generator with freshly installed P4 packet broker

SWITCH was able to contribute important experience here, as we are already using P4-based platforms productively for a special application: As part of our Netflow/IPFIX traffic data collection, we programmed a “packet broker” that aggregates the data from several router ports to be measured onto a few fast server ports. (These servers run a Netflow/IPFIX exporter, also from our kitchen). Alex Gall presented this system at a workshop on Telemetry and Big Data of the GÉANT project in November.

Data Center Interconnect (DCI)

We built a DCI (Data Center Interconnect) solution for a major data center relocation project in southern Switzerland. For this we introduced a new optical platform, which seems to be suitable for further applications in the backbone in the future. The data center migration itself started towards the end of the year. The challenges involved are considerable and will continue to keep us busy in 2021.

“Business as usual”

Schaffhausen, January 21: Oriel sightseeing in the evening program of the Net/Sec WG

Besides all these more or less extraordinary activities, the “normal” business had to go on. The aforementioned meeting of the Network/Security WG in Schaffhausen in January was a nice (but, as mentioned, unfortunately also the last for a while) opportunity to exchange experiences, opinions and ideas within the SWITCH network community.

In addition to the new 100Gb/s link to DE-CIX in Frankfurt (see also under COVID-19), there were, as usual, various upgrades. For example, in November we upgraded the link to GÉANT from 100Gb/s to 2*100GB/s. In this context, the existing Cisco ASR9000 was also replaced by a newer NCS 55A1 router. This comparatively very compact and energy-saving platform is proving to be the new standard for sites with a need for (many) 100Gb/s links.

In our optical platform (ECI), we are also benefiting from technical innovations that allow us to build new N*100Gb/s links relatively inexpensively on the main routes where the need exists.

Personnel changes

Long-time NeWo employees Ulrich Schmid and Markus Wittmer have moved to the newly created DTec (Development & Technology) team. We will continue to benefit from their expertise.

After an excursion of several years into the field of cloud infrastructure, Simon Leinen returned to the network team.

Kurt Baumann was heavily involved in SWITCH’s newly launched work on research data management in 2020 and will join the Business Development team in 2021.

Finally, we had one retirement to mourncelebrate, that of Ernst Heiri, who left after 25 years of service to SWITCH, including 17 years with the network team.

Outlook for 2021

In the year that has just started, we are looking for a new member to join our team. This could be someone fresh out of university who wants to become a learned networking practitioner in a stimulating but relatively stable environment—job ad here; please forward to any interested parties!

For March 11 we plan to invite to a meeting of the Network WG, which will take place virtually for the first time.

On the technical side, we will build on what we have achieved and continue to work on the projects mentioned above. Also, in 2021 we want to evaluate candidates for a new backbone router platform that also supports 400Gb/s connections—as cost effectively as possible and hopefully with (at least) the current level of stability.

For 2021, we wish the entire SWITCH community every success and look forward to continued good cooperation. If a little normality returns, so much the better. Even so, we certainly won’t be bored!

SL (Text & Photos), KC (Photos)