Expert · Incident · 18 of 180:00 / 25:00

Nothing works on launch day

Problem

The client team can't open the shop.

Your job
  • Get shop.corp.internal resolving to 10.1.1.10 and opening on 443 from vm-client
  • Get vm-web1 and vm-web2 both passing the health probe
Without breaking
  • Keep SSH from vm-client to the web servers blockedbroken
Background

"Launch is in an hour and the client team can't open the shop at all." Clients are in vnet-client, like vm-client, which is peered with vnet-shop. They open shop.corp.internal, which points at lb-shop at 10.1.1.10, in front of vm-web1 and vm-web2. Both web servers have to be in rotation for launch traffic. Clients never get SSH to them.

How to work it
  1. 1See it fail

    Prove the problem yourself before you change anything. Reports are often wrong.

    VM commands forVMs: Linux
    1. ssh vm-client
      What's this?

      Opens a shell on vm-client, so the next commands run on that VM. In this lab it's a shortcut. In real Azure you'd connect through something like Azure Bastion, or SSH to an address you can reach.

    2. nslookup shop.corp.internal
      What's this?

      Asks the VM's DNS server what IP address a name points to. Run it on the VM that has the problem, because each VM can get a different answer.

    3. nc -zv shop.corp.internal 443
      What's this?

      netcat, a standard Linux tool. Tries to open a TCP connection and reports whether it worked.

      • -z only check whether the port answers. Send no data
      • -v print the result in words
      • shop.corp.internal where to connect
      • 443 the port to try

    Ticks off when the test runs on vm-client.

  2. 2Find where it stops
  3. 3Name the cause
  4. 4Fix it and test again
Sources on Microsoft Learn (4)
0/3 changes
Network map
?
  • Resource group, the outer dashed frame
  • Virtual network, with its address space top right
  • Subnet, with range and usable IPs (5 reserved)
  • Network security group. A small shield on a VM's NIC means the NIC has one too
  • Route table on the subnet
  • NICVM Appliance Private endpoint
  • Peering Connected Initiated or access off
  • Goal working Goal broken, X marks where it stops
  • Private DNS zone, dotted lines go to linked VNets
  • Orange pulse: what your last command changed. Drag to pan, Ctrl and scroll to zoom.
Resource grouprg-labvnet-shop: tap to inspectVirtual networkvnet-shop10.1.0.0/16subnet-web: tap to inspectsubnet-web10.1.1.0/24 · 251 usablensg-webvm-web1 10.1.1.4: tap to inspectNICvm-web110.1.1.4vm-web2 10.1.1.5: tap to inspectNICvm-web210.1.1.5lb-shop-fe 10.1.1.10: tap to inspectLBlb-shop-fe10.1.1.10subnet-client: tap to inspectsubnet-client10.1.2.0/24 · 251 usablevnet-client: tap to inspectVirtual networkvnet-client10.2.0.0/16subnet-client: tap to inspectsubnet-client10.2.1.0/24 · 251 usablevm-client 10.2.1.4: tap to inspectNICvm-client10.2.1.4Connected: tap to inspectpeeredcorp.internal: tap to inspectDNSPrivate DNS zone · corp.internal · 1 link · shop → 10.1.1.10
  • vm-client → shop.corp.internal:443should work: not working
Goals 0/3
Cloud Shell · bashVMs: Linuxrg-lab · eastus2
cloudshell:~$