Warning |
---|
This page is being retained for historical purposes, but is no longer maintained. All relevant Direct Connect information about the current (2023 and after) Direct Connect architecture has been migrated to primary customer Direct Connect documentation, Cornell AWS Direct Connect. |
Info |
---|
Executive Summary
|
...
Within those 65 Cornell AWS accounts, only the network resources within VPCs using Direct Connect will be affected. Other VPCs in those Cornell AWS accounts will not be affected.
Terminology
We use the following terminology in this document:
For VPCs using Direct connect, the following table identifies the impact of this migration on specific types of network traffic
Connectivity | Changing? |
---|---|
VPC connectivity to the Internet | not changing |
VPC-to-VPC peering | not changing |
VPC to campus addresses via the Direct Connect |
|
VPC to campus addresses via the Internet | not changing |
Terminology
We use the following terminology in this document:
- customer – Cornell AWS account owners/administrators
- Version 1 (v1) architecture – This is the network architecture used by Cornell AWS Direct Connect networking prior to the 2023 migration.
- Version
- customer – Cornell AWS account owners/administrators
- Version 1 (v1) architecture – This is the network architecture used by Cornell AWS Direct Connect networking prior to the 2023 migration.
- Version 2 (v2) architecture – This is the network architecture used by Cornell AWS Direct Connect networking after the 2023 migration.
- VPC – Virtual Private Cloud
- DC – Direct Connect
- TGW – Transit Gateway
- VGW – Virtual Private Gateway
...
Resource | Filtering | |
---|---|---|
Source | Direct Connect Virtual Interface | — |
↓ | Virtual Private Gateway | — |
↓ | NACL of Subnet containing EC2 instance | inbound rules of NACL |
↓ | EC2 Instance Security Group | inbound rules of SG |
↓ | EC2 Instance Elastic Network Interface | — |
Destination | EC2 Instance | — |
Version 2 (2023)
draw.io source: dc-arch-2023.customer.ALTERNATE.10-8.v2.drawio
Paths and Traffic Filtering in Version 2 Architecture
Inbound Traffic – From TGW to EC2 Instance NOT Residing in a Subnet Attached to TGW
Resource | Filtering | Notes | |
---|---|---|---|
Source | TGW | — | |
↓ | TGW Attachment | — | |
↓ | TGW Attachment Elastic Network Interface | — | |
↓ |
—
Inbound Traffic – From TGW to EC2 Instance NOT Residing in a Subnet Attached to TGW
NACL of Subnet attached to TGW | outbound rules of NACL attached to utility subnet | The NACL bound to the utility subnets will allow all traffic in and out. | |
↓ | Route Table of Subnet attached to TGW | — | |
---|---|---|---|
↓ | NACL of Subnet containing EC2 instance | inbound rules of NACL for destination subnet | |
↓ | EC2 Instance Security Group | inbound rules of SG | |
Destination | EC2 Instance |
Elastic Network Interface | — |
—
What Is Changing?
Before the migration is executed, a set of resources in Cornell AWS accounts will be tagged with details about the migration. In addition, a small set of new resources that support the v2 architecture will be created in Cornell AWS accounts. After the migration is complete, a few resources not used in the v1 architecture will be deleted.
Cornell AWS customers will have the opportunity to provide feedback before migration and any resource deletion that affects their AWS accounts.
...
What Is Changing?
Before the migration is executed, a set of resources in Cornell AWS accounts will be tagged with details about the migration. In addition, a small set of new resources that support the v2 architecture will be created in Cornell AWS accounts. After the migration is complete, a few resources not used in the v1 architecture will be deleted.
Cornell AWS customers will have the opportunity to provide feedback before migration and any resource deletion that affects their AWS accounts.
Anchor | ||||
---|---|---|---|---|
|
For this migration, we are tagging AWS resources to provide information about how the each resource is involved in the migration itself, the v1 architecture, and the v2 architecture.
Tag Key | Tag Values | Description | VPC | Subnets | Route Tables | NACLs ‡ | Transit Gateway | Virtual Private |
---|
For this migration, we are tagging AWS resources to provide information about how the each resource is involved in the migration itself, the v1 architecture, and the v2 architecture.
Tag Key | Tag Values | Description | VPC | Subnets | Route Tables | Transit Gateway | Virtual Private | Direct Connect Virtual Interfaces | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
cit:dc-arch-migration-target | yes/no/guidance-required | Will this resource itself be affected as part of the migration? For subnets, if "guidance-required" then account owners will be consulted. | ||||||||||||||
cit:dc-arch-migration-description | prose | Description of planned changes to this resource | ||||||||||||||
cit:dc-arch-version | v1/v2 | Is this a v1 or v2 architecture resource? After migration, v1 resources utilized in the v2 architecture will be relabeled as v2 resources. | ||||||||||||||
cit:dc-arch-migration-new-resource | yes/no | Is this a new resource specifically created for the v2 architecture? | n/a | n/a | n/a | n/a | ||||||||||
cit:dc-arch-migration-replaces | resource ID | If this v2 resource will be replacing a v1 resource, this ID references the resource that will be replaced. | n/a | n/a | n/a | n/a | n/a | n/a | ||||||||
cit:subnet-type | public/private/utility | Is this a private or public subnet? Public subnets are those with a route to an Internet Gateway. Utility subnets will be created specifically to use for TGW Attachments. Private subnets are all subnets that are not public and are not utility subnets. | n/a | n/a | n/a | n/a | n/a | n/a | ||||||||
cit:tgw-attachment-target | yes/no/guidance-required | Will a Transit Gateway be attached to this subnet? If "guidance-required" then account owners will be consulted about the TGW Attachments. | n/a | n/a | n/a | n/a | n/a | n/a | ||||||||
cit:dc-arch-exclude-tgw-attachment-guidanceroutes | yestbd/attach/no-attach | This tag provides a place for a human reviewer to provide guidance about whether a TGW attachment should be made to the tagged subnet.
Of all these tags, this is the only tag whose value should be updated by customers. | n/a | n/a | n/a | n/a | n/a | is applied only to the new Route Table that is created for use by the TGW Attachment utility subnets | n/a | n/a | n/a | n/a | n/a | n/a | ||
cit:dc-vgw | yes/no | cit:dc-vgw | yes/no | Does this Route Table contain rules referencing a VGW? | n/a | n/a | n/a | n/a | n/a | n/a | ||||||
Cost Center | R524755 | This tag added to TGW Attachments will result in CIT paying for the $0.05/hr cost of attaching a VPC to a TGW. | n/a | n/a | n/a | n/a | n/a | n/a |
‡ Only the NACL created for use by utility subnets will be tagged.
Direct Connect Gateways are also involved in the Direct Connect Gateways are also involved in the migration but cannot be tagged.
...
- New Resource Groups are an easy way to see the lists of affected resources.
- New Route Tables will have routes that replace Virtual Private Gateway destinations with Transit Gateway Attachments destinations.
- New utility Subnets, one for each AZ where the VPC is active.
- New permissive Network ACL to be used only by the new utility Subnets.
- Transit Gateway Attachments will connect VPCs to the v2 architecture.
...
If you use Terraform or other infrastructure-as-code tools to manage your VPC, you may need to add configuration to instruct those tools to ignore these new tags. For example, we have specific guidance for Terraform: Terraform Configuration Guidance for 2023 Direct Connect Architecture Migration.
Note |
---|
Some resources that belong in these Resource Groups will not be present because of limitations in the the resource types that Resource Groups can handle. These resource types are:
|
...
The AWS Transit Gateways used in the v2 architecture require different routing rules than the Virtual Private Gateways (VGW) used in the v1 architecture. Each VPC Route Table that references a Virtual Private Gateway will be duplicated and, in the new Route Table, rules referencing a VGW will be replaced with rules referencing a TGW Attachment. These The new Route Tables will not include "blackhole" routes (i.e. routes to resources, like old peering connections, that no longer exist) from the original Route Tables.
These new Route Tables will be created prior to the migration, but will not actually be will be created prior to the migration, but will not actually be utilized until the migration is executed. When migration is executed, subnets associated with the v1 Route Tables will be re-associated to the corresponding v2 Route Tables. Similarly, if the "main" Route Table for the VPC references a VGW, the corresponding v2 Route Table will be set as the "main" Route Table for the VPC.
These Route Tables will be created during the Migration phase of the migration. See Migration Process.
Transit Gateway Attachments
Transit Gateway Attachments are the mechanism that VPCs connect to Transit Gateways. The Transit Gateways we use in the v2 architecture reside in a central AWS account, and a TGW Attachment is what links the VPC in a Cornell AWS account to those central TGWs.
Unlike Virtual Private Gateways, TGW Attachments connect to specific subnets in a VPC. We will be making these TGW Attachments to multiple private subnets in your VPCs. For best resiliency, we will select private subnets in multiple Availability Zones (AZs) for the TGW Attachments. In most Cornell AWS accounts, each private subnet resides in a unique AZ. If your VPC contains more than one private subnet in a given AZ, we will consult with AWS account owners to determine the best private subnet to select for the TGW Attachments. This is because each AZ can accommodate exactly one TGW Attachment.
TGW Attachments will be created during the Migration phase of the migration. See Migration Process.
Resource Deletion
After migration is complete, a few resources will be deleted during the Cleanup phase of the migration. These are:
- Virtual Private Gateways (VGWs)
- Direct Connect Virtual Private Interfaces (DCVIFs)
- Direct Connect Gateways that supported the v1 architecture.
Neither VGWs nor DCVIFs have a role in the v2 architecture.
Info |
---|
V1 Route Tables will not be deleted, but will not be used in the v2 architecture. Cornell AWS account owners can delete the v1 Route Tables if they wish. Once the VGWs are deleted, those v1 Route Tables will not be all that functional. |
...
Utility Subnets
New "utility" subnets will be created in each VPC. The sole purpose for these subnets is to be used to make TGW Attachments. One new subnet will be created for each AZ where the VPC has private or public subnets. (This provides the best resiliency for the Direct Connect connectivity through the TGW.)
In order to create these subnets, VPCs will have an additional CIDR block associated with it. The new subnets will be created with /28 CIDR blocks from the new CIDR attached to the VPC. These tiny subnets (~16 IPv4 addresses) should not be used for anything else. The Route Tables and NACLs associated with these subnets make them unsuitable for general use.
Network ACL
The Network ACLs already in customer VPCs will not be affected by this migration. However, a new NACL will be created in each VPC and associated with the new utility subnets. The NACL will be permissive (allowing all traffic in and out) and named in such a way to discourage use for other purposes.
Transit Gateway Attachments
Transit Gateway Attachments are the mechanism that VPCs connect to Transit Gateways. The Transit Gateways we use in the v2 architecture reside in a central AWS account, and a TGW Attachment is what links the VPC in a Cornell AWS account to those central TGWs.
Unlike Virtual Private Gateways, TGW Attachments connect to specific subnets in a VPC. We will be making these TGW Attachments to the utility subnets in VPCs which were specifically created for this purpose.
TGW Attachments will be created during the Migration phase of the migration. See Migration Process.
Resource Deletion
After migration is complete, a few resources will be deleted during the Cleanup phase of the migration. These are:
- Virtual Private Gateways (VGWs)
- Direct Connect Virtual Private Interfaces (DCVIFs)
- Direct Connect Gateways that supported the v1 architecture.
Neither VGWs nor DCVIFs have a role in the v2 architecture.
Info |
---|
V1 Route Tables will not be deleted, but will not be used in the v2 architecture. Cornell AWS account owners can delete the v1 Route Tables if they wish. Once the VGWs are deleted, those v1 Route Tables will not be all that functional. |
Anchor | ||||
---|---|---|---|---|
|
The management and routing simplification offered by the v2 architecture comes with a shift in costs seen by Cornell AWS accounts using Direct Connect, but the overall impact to Cornell AWS account costs should be negligible.
V1 Architecture
Cornell AWS accounts using Direct Connect
The management and routing simplification offered by the v2 architecture comes with a shift in costs seen by Cornell AWS accounts using Direct Connect, but the overall impact to Cornell AWS account costs should be negligible.
V1 Architecture
Cornell AWS accounts using Direct Connect saw these Direct Connect-related charges:
...
draw.io source: direct-connect-migration-process.v2.drawio
Anchor | ||||
---|---|---|---|---|
|
Phase | Stage | Timeframe | Status | Activity | Impact on Cornell AWS Account VPC Networks | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Preparation | Data Collection | November 2022 |
| none | ||||||||||||||||||
Resource Tagging |
|
| none | |||||||||||||||||||
Resource Groups |
| none | ||||||||||||||||||||
Customer Input #1 | - |
| none | |||||||||||||||||||
Migration | Transit Gateway Attachments | - |
| none | ||||||||||||||||||
Customer Input #2 | - |
| none | VPC Routing Updated |
|
| VPC-to-campus traffic will be routed through the v2 architecture | Campus Direct Connect Routes Updated |
|
| campus-to-VPC traffic will be routed through the V2 architecture | Cleanup | Customer Account Cleanup | - |
| none | Campus Direct Connect Cleanup |
| none- |
| none | |
v2 BGP Updated | 7am |
| Azure-to-AWS-VPC traffic may begin to use the v2 architecture (in just the one direction). This is limited only to Azure-to-AWS-VPC traffic due to Cornell's network architecture. | |||||||||||||||||||
VPC Routing Updated | 9am |
|
| |||||||||||||||||||
Campus Direct Connect Routes Updated | 9am |
|
| |||||||||||||||||||
Cleanup | Customer Account Cleanup | - |
| none | ||||||||||||||||||
Campus Direct Connect Cleanup |
| none |
Anchor | ||||
---|---|---|---|---|
|
Customers have the option to request that migration for their VPC(s) occur during the week of Jan 9-13 instead of the default migration dates of January 17 and 18. This is especially encouraged for customers that have a separate sandbox or development VPC that needs to be migrated. We can also support taking both migration steps on the same alternate day, but we'd like to leave a 1-4 hour gap between migration steps to confirm that the "VPC Routing Updated" step was successful before continuing to the "Campus Direct Connect Routes Updates" step.
Anchor | ||||
---|---|---|---|---|
|
Both the "VPC Routing Updated" and the "Campus Direct Connect Routes Updated" steps have simple rollback mechanism. If you discover problems with networking in your VPC after either step and think the change needs to be rolled back, send an email to cloud-incident@cornell.edu and ping Paul Allen (pea1) on Teams.
- The rollback for the "VPC Routing Updated" step is to reassign the original Route Tables to the public and private subnets. This will rollback takes effect immediately.
- The rollback for the "Campus Direct Connect Routes Updated" step is to the cancel the failover of the Direct Connect Virtual Interfaces that we triggered to initial the campus routing updates. This rollback takes 5-20 minutes to complete.
FAQs
How do I tell if my AWS account will be affected by this change?
...
When, specifically, will this migration occur?
...
See detailed schedule above.
Is there any flexibility in migration dates?
Yes. See Alternate Migration Days above.
Can the migration be rolled back?
Yes. Each of the two active migration steps ("VPC Routing Updated" and "Campus Direct Connect Routes Updated") can be individually rolled back for each migrating AWS VPC. See Rollback above
...
.
What if I use Terraform or a similar tool to manage the network resources in my AWS account?
...
Yes, you can. Just contact cloud-support@cornell.edu. Depending on the timing of things, we may need to migrate your Direct Connect connectivity to the v2 architecture before removing Direct Connect from your account entirely.
References
- Cornell Documentation
- Cornell AWS Accounts Affected by 2023 Direct Connect Architecture Migration
- Terraform Configuration Guidance for 2023 Direct Connect Architecture Migration
- Cornell AWS Direct Connect Routing Diagrams
- Announcements
- Cornell AWS Accounts Affected by 2023 Terraform Configuration Guidance for 2023
- 2023-01-10 AWS Direct Connect Architecture Migration Customer Review and Feedback #2
- 2022-12-20 AWS Direct Connect Architecture Design Update
- 2022-12-15 AWS Direct Connect Architecture Migration Customer Review and Feedback
- 2022-12-09 AWS Direct Connect Architecture Migration Preparation Continues
- 2022-11-02 Upcoming AWS Direct Connect Changes
- External Documentation
...