Expert · Incident · 9 of 180:00 / 20:00

The firewall broke everything

Problem

Spoke 1 can't reach the API in Spoke 2.

Your job
  • Get Spoke 1 reaching the API on vm-api (443) through azfw-hub
Without breaking
  • Keep SSH from Spoke 1 to vm-api blockedbroken
  • Don't peer the spokes to each otherbroken
Background

"Since the new firewall went in last night, Spoke 1 can't reach the API in Spoke 2." The API runs on vm-api on 443, and vm-spoke1 calls it. The plan: all spoke-to-spoke traffic goes through azfw-hub, an Azure Firewall at 10.0.1.4. HTTPS from Spoke 1 to Spoke 2 is allowed, everything else between the spokes is blocked, and the spokes are never peered to each other.

How to work it
  1. 1See it fail

    Prove the problem yourself before you change anything. Reports are often wrong.

    VM commands forVMs
    1. ssh vm-spoke1
      What's this?

      Opens a shell on vm-spoke1, so the next commands run on that VM. In this lab it's a shortcut. In real Azure you'd connect through something like Azure Bastion, or SSH to an address you can reach.

    2. nc -zv 10.2.1.4 443
      What's this?

      netcat, a standard Linux tool. Tries to open a TCP connection and reports whether it worked.

      • -z only check whether the port answers. Send no data
      • -v print the result in words
      • 10.2.1.4 where to connect
      • 443 the port to try

    Ticks off when the test runs on vm-spoke1.

  2. 2Find where it stops
  3. 3Name the cause
  4. 4Fix it and test again
Sources on Microsoft Learn (3)
0/3 changes
Network map
?
  • Resource group, the outer dashed frame
  • Virtual network, with its address space top right
  • Subnet, with range and usable IPs (5 reserved)
  • Network security group. A small shield on a VM's NIC means the NIC has one too
  • Route table on the subnet
  • NICVM Appliance Private endpoint
  • Peering Connected Initiated or access off
  • Goal working Goal broken, X marks where it stops
  • Private DNS zone, dotted lines go to linked VNets
  • Orange pulse: what your last command changed. Drag to pan, Ctrl and scroll to zoom.
Resource grouprg-labvnet-spoke1: tap to inspectVirtual networkvnet-spoke110.1.0.0/16subnet-app: tap to inspectsubnet-app10.1.1.0/24 · 251 usablert-spoke1vm-spoke1 10.1.1.4: tap to inspectNICvm-spoke110.1.1.4vnet-hub: tap to inspectVirtual networkvnet-hub10.0.0.0/16AzureFirewallSubnet: tap to inspectAzureFirewallSubnet10.0.1.0/26 · 59 usableazfw-hub 10.0.1.4: tap to inspectFWazfw-hub10.0.1.4vnet-spoke2: tap to inspectVirtual networkvnet-spoke210.2.0.0/16subnet-api: tap to inspectsubnet-api10.2.1.0/24 · 251 usablert-spoke2vm-api 10.2.1.4: tap to inspectNICvm-api10.2.1.4Connected: tap to inspectpeeredConnected: tap to inspectpeered
  • vm-spoke1 → vm-api:443should work: not working
  • vm-spoke1 → vm-api:22should stay blocked: blocked
Goals 0/3
Cloud Shell · bashVMsrg-lab · eastus2
cloudshell:~$