As of this page is under development. Contact Paul Allen with questions.
Executive Summary
- Cornell AWS accounts using Direct Connect will be migrated to a new network architecture in January 2023.
- There will be no interruption in Direct Connect connectivity during this migration.
- Cornell AWS account owners/administrators will be solicited for feedback directly via email several times throughout this process. But, in most cases, this input is not required for the migration to proceed.
- AWS accounts having VPCs with numerous subnets and route tables, or where network resources are configured by infrastructure-as-code will need to provide input to complete this migration.
- The new Direct Connect architecture will cost AWS account owners about $36/mo.
Introduction
This document provides details about the Direct Connect architecture migration we will be executing in early 2023.
Rationale
Cornell AWS accounts using Direct Connect for private access to Cornell networks will be transitioned to using Internet 2 Cloud Connect (I2CC) as the Direct Connect provider.
That change will simplify configuration and management of Direct Connect for AWS accounts.
This switch to I2CC Direct Connect also will allow private Cornell network traffic in AWS and Azure to flow between clouds without transiting campus, as it does prior to this migration.
Scope
As of , 65 Cornell AWS accounts were configured to use Direct Connect. During this migration, all those AWS accounts will have their existing Direct Connect connectivity updated to use new pathways and AWS resources to connect the Cornell campus network to AWS.
Within those 65 Cornell AWS accounts, only the network resources within VPCs using Direct Connect will be affected. Other VPCs in those Cornell AWS accounts will not be affected.
Nomenclature
We use the following terminology:
- customer – Cornell AWS account owners/administrators
- Version 1 (v1) architecture – This is the network architecture used by Cornell AWS Direct Connect networking prior to the 2023 migration.
- Version 2 (v2) architecture – This is the network architecture used by Cornell AWS Direct Connect networking after the 2023 migration.
- VPC – Virtual Private Cloud
- DC – Direct Connect
- TGW – Transit Gateway
- VGW – Virtual Private Gateway
Architectures
These diagrams show the network resources within Cornell AWS accounts that connect a VPC to the Cornell campus network via Direct Connect.
Version 1 (pre-2023)
draw.io source: dc-arch-legacy.customer.v2.drawio
Version 2 (2023)
draw.io source: dc-arch-2023.customer.v2.drawio
What Is Changing?
Before the migration is executed, a set of resources in Cornell AWS accounts will be tagged with details about the migration. In addition, a small set of new resources that support the v2 architecture will be created in Cornell AWS accounts. After the migration is complete, a few resources not used in the v1 architecture will be deleted.
Cornell AWS customers will have the opportunity to provide feedback before migration and any resource deletion that affects their AWS accounts.
New Resources
Resource Groups
New AWS resource groups collect references to relevant AWS account resources in one place (per Cornell AWS account) for easy reference and review:
Resources can and will appear in multiple resource groups!
- cit-dc-arch-migration-affected-resources – These resources will be directly affected by this migration. These resources include:
- new resources that support the v2 architecture
- resources that support the v1 architecture and will no longer be needed for the v2 architecture
- resources that will remain, but will have their configuration changed to support the v2 architecture
- Transit Gateway Attachments should appear in this list, but a limitation of AWS Resource Groups appears to exclude these resources.
- cit-dc-arch-version-1-resources – All network resources that support or utilize the v1 architecture
- cit-dc-arch-version-2-resources – All newly-created resources that support the v2 architecture
- Transit Gateway Attachments should appear in this list, but a limitation of AWS Resource Groups appears to exclude these resources.
After the v1 → v2 migration is complete, v1 resources will either be deleted (if they are not used in the v2 architecture) or relabeled as v2 resources (if they continue to be used in the v2 architecture).
These Resource Groups will be created during the Preparation phase of the migration. See Migration Process.
Route Tables
The AWS Transit Gateways used in the v2 architecture require different routing rules than the Virtual Private Gateways (VGW) used in the v1 architecture. Each VPC Route Table that references a Virtual Private Gateway will be duplicated and, in the new Route Table, rules referencing a VGW will be replaced with rules referencing a TGW Attachment.
These new Route Tables will be created prior to the migration, but will not actually be utilized until the migration is executed. When migration is executed, subnets associated with the v1 Route Tables will be re-associated to the corresponding v2 Route Tables. Similarly, if the "main" Route Table for the VPC references a VGW, the corresponding v2 Route Table will be set as the "main" Route Table for the VPC.
These Route Tables will be created during the Migration phase of the migration. See Migration Process.
Transit Gateway Attachments
Transit Gateway Attachments are the mechanism that VPCs connect to Transit Gateways. The Transit Gateways we use in the v2 architecture reside in a central AWS account, and a TGW Attachment is what links the VPC in a Cornell AWS account to those central TGWs.
Unlike Virtual Private Gateways, TGW Attachments connect to specific subnets in a VPC. We will be making these TGW Attachments to multiple private subnets in your VPCs. For best resiliency, we will select private subnets in multiple Availability Zones (AZs) for the TGW Attachments. In most Cornell AWS accounts, each private subnet resides in a unique AZ. If your VPC contains more than one private subnet in a given AZ, we will consult with AWS account owners to determine the best private subnet to select for the TGW Attachments. This is because each AZ can accommodate exactly one TGW Attachment.
TGW Attachments will be created during the Migration phase of the migration. See Migration Process.
Tagging
For this migration, we are tagging AWS resources to provide information about how the each resource is involved in the migration itself, the v1 architecture, and the v2 architecture.
| Tag Key | Tag Values | Description | VPC | Subnets | Route Tables | Transit Gateway | Virtual Private | Direct Connect Virtual Interfaces |
|---|---|---|---|---|---|---|---|---|
| cit:dc-arch-migration-target | yes/no | Will this resource itself be affected as part of the migration? |  | | | | | |
| cit:dc-arch-migration-description | prose | Description of planned changes to this resource | | | | | | |
| cit:dc-arch-version | v1/v2 | Is this a v1 or v2 architecture resource? After migration, v1 resources utilized in the v2 architecture will be relabeled as v2 resources. | | | | | | |
| cit:dc-arch-migration-new-resource | yes/no | Is this a new resource specifically created for the v2 architecture? | n/a | n/a | | | n/a | n/a |
| cit:dc-arch-migration-replaces | resource ID | If this v2 resource will be replacing a v1 resource, this ID references the resource that will be replaced. | n/a | n/a | | n/a | n/a | n/a |
| cit:subnet-type | public/private | Is this a private or public subnet? Public subnets are those with a route to an Internet Gateway. Private subnets are all subnets that are not public. | n/a | | n/a | n/a | n/a | n/a |
| cit:tgw-attachment-target | yes/no/guidance-required | Will a Transit Gateway be attached to this subnet? If "guidance-required" then account owners will be consulted about the TGW Attachments. | n/a | | n/a | n/a | n/a | n/a |
Resource Deletion
After migration is complete, a few resources will be deleted during the Cleanup phase of the migration. These are:
- Virtual Private Gateways (VGWs)
- Direct Connect Virtual Private Interfaces (DCVIFs)
Neither VGWs nor DCVIFs have a role in the v2 architecture.
V1 Route Tables will not be deleted, but will not be used in the v2 architecture. Cornell AWS account owners can delete the v1 Route Tables if they wish. Once the VGWs are deleted, those v1 Route Tables will not be all that functional.
Costs
The management and routing simplification offered by the v2 architecture comes with a modest change in costs seen by Cornell AWS accounts using Direct Connect.
V1 Architecture
Cornell AWS accounts using Direct Connect will no longer see these charges:
- Bandwidth charges for traffic from VPCs to campus using the Direct Connect is charged at $0.02/GB.
- Bandwidth charges from traffic TO VPCs from campus is free.
V2 Architecture
These charges are new to Cornell AWS customers using Direct Connect:
- Every VPC connected to the Transit Gateway will be charged $0.05/hr.
- Bandwidth charges for traffic from the VPC to the TGW is $0.02/GB.
- Bandwidth for traffic from the TGW to the VPC is free.
Summary
In total, Cornell AWS accounts using Direct Connect should not experience any significant change in bandwidth-related charges for VPC traffic bound for the campus. But, these AWS accounts will see an increase in Direct Connect-related costs of about $36/mo for each VPC connected to the v2 Direct Connect architecture.
If you have concerns about these cost changes, please contact cloud-support@cornell.edu.
Migration Process
draw.io source: direct-connect-migration-process.v2.drawio
| Phase | Stage | Timeframe | Activity | Impact on Cornell AWS Account VPC Network |
|---|---|---|---|---|
| Preparation | Data Collection | November 2022 |
| none |
| Resource Tagging | December 2022 |
| none | |
| Resource Groups |
| none | ||
| Customer Input #1 |
| none | ||
| Migration | Transit Gateway Attachments | January 2023 |
| none |
| Customer Input #2 |
| none | ||
| VPC Routing Updated |
| VPC-to-campus traffic will be routed through the v2 architecture | ||
| Campus Direct Connect Routes Updated |
| campus-to-VPC traffic will be routed through the V2 architecture | ||
| Cleanup | Customer Account Cleanup | Jan/Feb 2023 |
| none |
| Campus Direct Connect Cleanup |
| none |
FAQs
How do I tell if my AWS account will be affected by this change?
The list of AWS accounts affected by this migration is here: Cornell AWS Accounts Affected by 2023 Direct Connect Architecture Migration
You will receive multiple emails to the email address associated with the root user of your Cornell AWS account. These emails will make announcements and ask for your input during the migration process.
Will there be any interruption in Direct Connect connectivity during this migration?
As of , our testing indicates that we will be able to complete this migration without any interruption in overall Direct Connect connectivity. VPCs should not experience any outage of connectivity to or from campus.
How will this change affect my AWS account costs?
Cornell AWS accounts will see an increase of about $36/mo for each VPC connected to the v2 Direct Connect architecture.
For more details, please see the Costs section above.
Does this change affect VPC peering?
VPC peering is not affected by this change.
Future Peering Changes
Since the Transit Gateway in the v2 architecture is configured to fully interconnect attached VPCs, most VPC peering among Cornell AWS VPCs could be removed eventually. When VPC peering is removed, VPC-to-VPC traffic that formerly used a peering connection would use the Transit Gateway instead. All traffic would remain in AWS, and the traffic would take two (2) hops to reach the target VPC instead of the one (1) hop that the peering connections support.
There are two cases where VPC peering would need to remain in place:
- When Security Groups in one VPC reference Security Groups in a peered VPC, that peering cannot be removed without adjusting the security group to use CIDR blocks instead of the referenced Security Group. TGW Attachments do not support this type of cross-VPC Security Group referencing.
- Peering between VPCs where one of the VPCs is not using Cornell Direct Connect. VPCs not using Direct Connect cannot replace peering with the TGW Attachment in the v2 architecture.
Reducing the amount of peering amongst Cornell AWS VPCs will take place later and customers will be contacted separately about that. No peering changes are planned as part of the Direct Connect architecture migration.
Does this change affect which campus network CIDR blocks are routed to/from my private and public subnets?
Our goal with this migration is that the routing of traffic between your VPC and Cornell public and private CIDR blocks will remain effectively unchanged between the v1 and v2 architectures. I.e. the Direct Connect routing option that you chose when your Direct Connect connectivity was established will remain in place. Those routing options are "private network extension", "hybrid", and "all campus" routing. For details on those options see Cornell AWS Direct Connect Routing Diagrams.
The exact pathways that Direct Connect traffic takes will change between the v1 and v2 architectures. But, the starting point (e.g., your VPC) and endpoint (e.g. a campus VLAN) of this traffic will constant.
When, specifically, will this migration occur?
- We expect to complete all stages in the Preparation phase in December 2022.
- We expect to complete all stages in the Migration phase in January 2023.
- More specific dates for each stage are forthcoming.
What if I use Terraform or a similar tool to manage the network resources in my AWS account?
If you use Terraform or a similar tool to manage your VPCs, please inform the Cloud Team during the Customer Input #1 stage. You would have several options to update your infrastructure-as-code configuration to support this migration. You could import the new Route Tables and TGW Attachments, or you could create your own Route Tables and TGW Attachments.
I don't really use the Direct Connect in my AWS account. Can I remove that feature?
Yes, you can. Just contact cloud-support@cornell.edu. Depending on the timing of things, we may need to migrate your Direct Connect connectivity to the v2 architecture before removing Direct Connect from your account entirely.
References
- Cornell AWS Accounts Affected by 2023 Direct Connect Architecture Migration
- 2022-11-02 Upcoming AWS Direct Connect Changes
- Cornell AWS Direct Connect Routing Diagrams