Global Ring Network for Advanced Applications Development

Overall System Design

The GLORIAD expansion includes plans to build a Hybrid Backbone (HB) [1] to interconnect the national S&E IP-routed networks of the US, Russia, and China, as well as create a switching infrastructure at layer-two (L2) (Ethernet) to serve special communities, unique applications with particularly heavy service requirements, and to provide an infrastructure for network research and experimentation. This L2 switching infrastructure will, towards years four and five of the project, evolve into a similar infrastructure for switching full light paths - layer-one (L1). The GLORIAD network plan includes special Points of Presence (PoPs) in Chicago, Amsterdam, Moscow, Novosibirsk, Khabarovsk, Beijing and Hong Kong.

Figure 1 provides an overall picture of the interconnected S&E IP networks, as well as the separate organizations and networking centers associated with the HB. In describing the overall system design, it is convenient to consider L3 and L2 elements separately. These two components share a Point-to-Point Transport Layer. The L3 component includes the User and Routing layers, while the L2 component, the User and VLAN layers. Of course, both layers represent the GLORIAD base but the content, devices and management associated with the two layers are different.

A.1 User Layer

For the routing-backbone, users are represented with national S&E networks: Abilene (US) [2], ESNet [3], NASA networks, other federal networks, the RBNet (Russia) [4], and CSTnet (China) [7]. To facilitate exchange of traffic with any of the networks connected to GLORIAD’s IP backbone service, routing equipment of a national S&E network should establish a direct link to one of GLORIAD’s routers for the external BGP connections [37] with, IP addresses for interfaces on the routers, and a list of Autonomous System (AS) numbers, which that network is going to advertise through the backbone to other networks.

For the switching backbone, users are represented by S&E organizations, institutions, laboratories, etc. Users requiring the L2 switched services must be capable of bringing their Ethernet links/payloads (10/100/1000 Mbps) (perhaps in multiples) to the GLORIAD PoP (or to facilities interconnected with the GLORIAD PoP). At the PoPs, the user’s links are terminated on the Ethernet switch ports.

A.2 Layer 1

At layer 1, routing- and switching-backbones share the same transport core of the HB – consisting of Multi-Service Provisioning Systems (MSPP) [10,11,12] interconnected with leased high speed links, physically forming a line topology during the first phase of the project (Beijing, Hong Kong, Chicago, Amsterdam, Moscow) that will later evolve into the physical ring topology with the addition, during Phase II of the program, of the segment from Moscow-Novosibirsk-Beijing. At L1, point-to-point (p-t-p) connections are provided between the GLORIAD PoPs, terminating on the Ethernet ports of the MSPPs. This is illustrated in Figure 2.

Backbone Link Requirements

MSPP requires that all links used for interconnecting MSPPs use consistent technology (SONET’s STS or SDH’s STM) [19] along the entire network topology. Unfortunately, different countries use different transport technologies. Table 1 lists a number of countries and the different technologies in common use [20].

Table. 1.

While providers working in the international market can typically provide a link utilizing either of the technologies (SONET’s STS-n or SDH’s STM-m), national providers generally do not. As some of the segments of the GLORIAD backbone are located entirely within the territory of one country (for example, China and Russia), the choice of the technology is necessarily dictated by the provider’s possibilities in these countries. Fortunately, China and Russia use the same technology - thus, SDH-m links will be deployed on the GLORIAD backbone. The choice of beginning the project at phase I with 2.5 Gbps circuits on the HB was established using mini-max criteria suggesting minimum reasonable speed dictated by anticipated applications and network use, and maximum link speed that could be purchased from telecommunications providers given the anticipated financial resources.

Another key technical question that has been addressed is what type of STM service (“channeled” or “concatenated”) must be utilized. Since MSPP SDH equipment uses channels (STM-1, STM-2, etc.) of an STM link for inserting different types of user payloads for transport through SONET/SDH networks, the STS/STM links must be channeled.

To summarize, the requirements for the backbone links to be used in the HB have been established: at the demarcation points, the provider’s equipment must hand over SDH’s channeled STM-16 links for Phase I of the project (upgrading to STM-64 for Phase II) or 2.488 Gbps wavelength of DWDM system (9.9 Gbps wavelength).

Multi-service Provisioning Platforms (MSPP)

The networking industry achieved a real breakthrough with the market release of a class of devices called Multi-service Provisioning Platforms [13, 14, 15, 16, 17, 18]. These devices provide a SONET/SDH hierarchy by combining OC-3/STM-4 to OC-192/STM-64 ADM (Add and Drop Multiplex) with a cross-connect functionality in a single, compact, flexible chassis. Some versions have a wide range of interface modules such as, leased lines, Ethernet, and ATM - the payloads of which can be mapped, cross-connected, and transported upon the SONET/SDH hierarchy. From the point of view of an end-user (at a work station), the MSPP provides unique opportunity for direct access to high speed of a backbone link. For the GLORIAD HB, it is mandatory that all PoPs are equipped with MSPP equipment of compatible vendor, type and functionality/operation. With this requirement satisfied, an efficient centralized management system will be developed - an essential for creating compatibility with the management systems of switched services to which GLORIAD must interconnect. As this is written, Cisco’s version of the MSPP has been chosen as an essential building block for the GLORIAD HB (CISCO ONS 15454 SDH version). However, this decision can be changed as the project progresses, as there are other options with similar functionality that the network engineering team will continue to evaluate.

A.3 Switching Backbone

With the switching backbone, users will be able to provision and use dedicated VLANs [36] that will be able to interconnect with circuits satisfying specific requirements with regards to network capacity and quality. In practice, this will be achieved with high performance Ethernet switches located at the PoPs and connected to the Ethernet ports (10/100/1000 Mbps) on the MSPP equipment. Figure 2 illustrates the use of VLANs for interconnecting computational centers, offices, and devices (such as, workstations or other Ethernet connected devices). The user’s Ethernet links are connected to the LAN Ethernet ports on the provided switches and the ports are assigned to VLANs. For example, if only two ports are included in a VLAN, then this VLAN is considered a p-t-p connection (p-t-p VLAN). VLAN traffic is transported through the trunk links. All switches are interconnected with trunk links through the MSPPs, while the trunk capacities are adjusted at the switches with the included EtherChannel mechanism [35]. The number of the trunk links between two switches will be one or more. If, a for a p-t-p VLAN, a dedicated chain of trunk links is configured between a pair of switches, this configuration will supply the maximum possible quality provided with the switching backbone (i.e., if a full 100 Mbps Ethernet is assigned to a VLAN, the user will have full access to that capacity between the pair of configured switches). This structure provides an extraordinary flexibility when providing interconnection schemes and provisioning the HB resources for various projects.

A.4 The Routing Level

The routing-backbone will provide for routing of IPv4 and IPv6 unicast/multicast traffic between national S&E networks [21, 22]. The capacity dedicated to the routing-backbone will be adjusted to meet demand as the project develops management of the HB for all included services.

The routing-backbone is provided with border/core routers placed at the GLORIAD PoPs and connected to the line Ethernet ports on the MSPP equipment during Phase I of the project (Fig. 3), and to the LAN ports on the switches during Phase II (Fig. 4). In Phase II, the links between routers at the PoPs will be provided with p-t-p VLANs and dedicated trunk lines (see 2.1.8.3) that will be shaped with EtherChannel.

GLORIAD’s routers will only advertise routes received from the connected networks consistent with the backbone routing policy (generally all S&E institutions connected to the national S&E networks). Before connecting a new network to the backbone, the list of Autonomous System numbers that are to be advertised from that network through the backbone must be approved by the GLORIAD network policy working group and GLORIAD management. This will effectively prevent use of the backbone services by public and commercial networks - as well as help prevent the spread of harmful network activity originating in the networks outside the international S&E community.

A.5 Structure of Points of Presence

The structure of a typical GLORIAD PoP is shown in Figure 3 for Phase I and in Figure 4 for Phase II. Every PoP in the HB will be built upon fully compatible MSPP equipment (preferably of the same vendor and type). This does not change for either Phase I or Phase II of the project. At the most basic layer, the MSPPs are interconnected with STM links. During Phase I, a router and Ethernet switch are connected to the line Ethernet ports on the MSPP’s interface modules. During Phase II, the router will be connected to the switch and the switch connected to the MSPP, which will allow using the VLAN-Trunk-EtherChannel (see A3) technique for interconnecting routers and switches in the routing- and switching-backbones.

The collection of GLORIAD monitoring systems, a mandatory element of each GLORIAD PoP, will be used for monitoring availability of services, availability of hosts in the network connected to the IP backbone, collection of statistical data, and the monitoring of traffic through the L3 service for certain classes of behavior identified as potentially harmful or abusive by the GLORIAD security working group.

A more detailed diagram of the GLORIAD PoP at StarLight, built with Cisco equipment, is shown in Figures 5 and 6 (for Phase I and Phase II respectively). It is important to note that the Ethernet trunk link - interconnecting the GLORIAD and StarLight switches - joins two switching infrastructures so that users of both can benefit from using VLANs that span the port of both networks [20, 38, 23].

A.6 Reliability, Scalability, Compatibility

Reliability

Achievement of a high level of reliability of the GLORIAD HB will be realized with SDH switched protection of the links between MSPP systems. This guarantees 50 millisecond interval switching time from a broken working path to a reserved one from NYC-Amsterdam and Seattle-Hong Kong (note that this only applies, for the initial phase of the project, on the more expensive transatlantic and transpacific links - which, if they fail, can require long periods of time to restore operation). While the full L1 level protection is expensive, it is considered important until the larger set of services associated with the emerging TransLight and GLIF projects emerge, in which case protection service can be engineered via bandwidth swaps with other partnering network providers. For financial reasons, circuits transiting the US (from NYC to Chicago and Seattle to Chicago) do not include full restorable service. There are enough alternative routes that the down time experienced should be minimally disruptive.

GLORIAD will provide redundant equipment for key items such as processing modules, interface cards and power supplies.

GLORIAD will clearly benefit from its own ring topology (that will be fully operational in year three) that provides two paths between any nodes in the ring. If a link between two of them is down, traffic can still flow between them using the second path.

Scalability

Scalability is considered an essential service requirement. Technically, the proposed GLORIAD structure is easily scaled to satisfy growth of requirements for the basic link capacity and the number of connections to the switched part of the backbone (Fig. 7). As more users are connected, additional interface modules can be installed into the Ethernet switch - or a new switch purchased if necessary. The addition of new switched capacity to one of the backbone’s switches is accomplished with a trunk link that is simply configured so that the new switch is joined to the backbone’s management domain. If the basic capacity on one of the backbone links must be increased - and this cannot be done by upgrading just the STM span itself (such as, upgrading from 2.5 G to 10 G wavelength), then additional MSPPs must be provisioned and installed - and the additional telecommunications service acquired from a provider to create a new backbone segment. It is not necessary that the new segment create an exact ring topology; any topology may be employed as the new segment is not physically joined at the Point-to-Point Transport Level, but at the VLAN level - i.e., the Ethernet port of the MSPP connecting the segment is connected to the LAN ports on an Ethernet switch. When creating a new segment, if one of the MSPPs is placed at a new location, this is equivalent to the creation of a new PoP. It is important that the vendor and type of MSPPs used in the new segment be fully compatible with those in current use. When adding new segments to the backbone, the resulting physical topology should remain simple in order to preserve ease of management of the backbone as a complete structure.

Compatibility

GLORIAD’s partners in GLIF represent some of the most innovative networks and exchange facilities in the world. They employ a similar structure as what we have described here for GLORIAD - thus, a merger of two or more of them can be described within the context as a scaling process. At a physical level, this is accomplished by interconnection of Ethernet switches belonging to the merged networks. As mentioned in the above section on scalability, the vendor and type of the MSPPs used in the merged networks can differ from each other. To summarize, GLORIAD’s HB will be highly compatible with these other networks considering structures deployed, however, the management systems employed by distinct networks are not as easily compatible as their structures. This is important for collaboration between the networks and more specifically, the network engineering teams.

A.7 Management System and Engineering Support

Management and engineering support are two elements of the same goal – the effective functioning of the GLORIAD HB.

Engineering support will include installation and configuration of hardware and software, as well as troubleshooting. This support will be provided by the international engineering group. To coordinate the activity of this group, and provide user service, the distributed international NOC will operate on a 24/7 schedule with immediate GLORIAD team members in the US, Russia, and China (but in close cooperation with other NOC facilities for providing true end-user-to-end-user support). The management system will consist of both Monitoring and Scheduling systems. The monitoring services (see 2.1.8.5) will focus on detecting and preventing the deterioration of GLORIAD services and help motivate hardware and software configuration changes promptly [24, 25, 26]. Indeed, a unique feature of the GLORIAD PoP will be the very close integration of its various monitoring systems (including, utilization monitoring, performance monitoring, and security monitoring) with the GLORIAD trouble ticketing system, BUGS, resulting in immediate notification of NOC staff of not only service outages, but also of suspicious network behavior or poor performance. The trouble tickets created (and stored in an SQL database) will be rated as to severity, will include as complete a history as is obtainable from the automated monitoring systems, and will trigger other alarms as necessary. The GLORIAD team will work closely with the TransLight and GLIF consortium partners on developing both scheduled and dynamic provisioning of the switching-backbone services giving the service window time and HB resource requirements of such facilities as VLANS and Ethernet trunks shaped with the EtherChannel mechanism.

This scheduling system is expected to develop in two stages. During the first phase (not necessarily corresponding to Phase I of the physical link topology), a database of users' requests with calendar functionality and web interface will be deployed. At this period, configuring VLANs will be accomplished manually by network engineering staff. When this system proves successful through an agreed-upon testing period, an interface, likely based on SNMP commands [27], will be introduced for configuring VLANs and switching capacity automatically. A real challenge in deploying this system is ensuring that the ability to provision VLANs end-to-end can cross network domains (i.e., from user network through MREN, through GLORIAD or other international service provider and back down through MREN and to user network). This is a nontrivial problem suggesting the essential nature of close cooperation with all other partnering networks (both domestic and international).

A.8 Security

GLORIAD’s HB will be deployed with a security policy that every participant establishing GLORIAD PoP(s) will agree to [28, 29, 30]. The security policy will be considered mandatory for all participants but additional independent security measures may be introduced by participants according to the specific requirements of their countries. The only caveat is that the security additions should not block the service provisioning mechanism for the entire backbone. The policy will cover the following areas: 1) The level of physical security provided at the facilities where equipment is located; 2) The use of secure protocols for accessing configurable equipment through public networks; 3) Provisioning passwords for access to the network equipment, management scheduling systems, as well as the authentication procedures for communications with phone calls or a messaging system between personnel working at different locations; 4) Support of a list of personnel authorized to install, configure, and support equipment directly at the facilities or with access to the management scheduling systems; 5) A disaster recovery plan.

GLORIAD’s Point of Presence at Translight/SL Facility, Chicago, Illinois, US PHASE I

The diagram of the PoP at the StarLight facility in Chicago is illustrated in Figure 5 and the list of equipment in Table 1. Equipment has already been selected for Phase I of the project - suggesting use of STM-16 spans for interconnecting Multi-service Provisioning Platforms (CISCO ONS 15454 SDH). The hardware configuration of the ONS has been chosen to utilize full capacity of the span for what is equivalent to 1 Gbps Ethernet payload and a set of 10/100 Mbps Ethernet payloads per each STM-16 span. During the first phase of the project, one of the cards of the ONS will serve as a router. The CISCO Catalyst 6506 provides L2 functionality (see A.3) and will contain a quantity of LAN 10/100 Mbps Ethernet ports - sufficient to provide physical connections to Ethernet ports on the ONS - one port or more for trunk connection with StarLight switch and about 20 ports for user connections.

The monitoring system will comprise a set of X-Servers or Unix servers for collecting of statistical utilization data, NLANR AMP machine to monitor state of network paths to network hosts, and a Packet Shaper 9500 used to monitor IP traffic used for experimentation (initially) and evaluation for detailed application level performance analysis and, perhaps at some point in the future, use of its traffic shaping services.

GLORIAD’s Point of Presence at Translight/SL Facilty, Chicago, Illinois, USA PHASE II

At the second phase, the PoP will be upgraded for utilizing STM-64 links (Fig. 6, Table 2). The number and type of ONS cards are defined based on the number Gigabit and Fast Ethernet payloads that can be mapped into the STM-64 span. As the link dedicated to the IP routing backbone is increased to one payload of Gigabit Ethernet and more, and in order to have more slots available at the ONS, the router should be installed as an autonomous unit (suggesting a CISCO 7603). According to changes in the hardware configuration of the ONS, the CISCO Catalyst 6506 will be upgraded also. As a result of the upgrade, the six LAN Gigabit Ethernet and 24 Fast Ethernet ports will be available for user connections.

In the monitoring system, an additional Packet Shaper 9500 may be acquired and the quantity of spare cards grown to seven.

GLORIAD’s Point of Presence at Translight/PW Facilty, Seattle, USA PHASE II

The STM-16 link from Hong Kong is converted into 10 Gigabit Ethernet link towards the PoP at Chicago. GLORIAD’s Cisco 7507 router (in current service on the NaukaNet/”Little GLORIAD” project will be deployed for routing service.