* Set a rolling upgrade policy so that changes to the worker node
pool are rolled out gradually. Previously, the VMSS model could
change, but instances would not receive it until manually replaced
* Align Azure node pool behaviors more closely with AWS and GCP:
* On AWS, worker instance template changes trigger an instance refresh
* On GCP, worker instance template changes roll out via proactive
* Define Azure automatic instance repair using Application Health
Extension probes to 10256 (kube-proxy or Cilium equivalent) to match
the strategy used on Google Cloud
* Azure Load Balancers charge by load balancer rues (5 included)
so its useful to provide ways to stay under that number, either
by dropping support for port 80 traffic or IPv6 traffic. When
using global proxies, you can usually serve IPv6 or http->https
redirects separately anyway
* Update Google Cloud TCP proxies from classic to current
* Google Cloud TCP proxies no longer restrict which frontend
ports may be used
* Switch apiserver to listen on 6443 to match other cloud
platforms
* Switch the HTTP (port 80) proxy to a TCP proxy to match
what's done for HTTPS traffic to ingress/gateway controllers
* Add a variable `enable_http_lb` to make TCP/80 IPv4/IPv6
forwarding rules optional. Default to false. Google Cloud
charges by forwarding rule, so dropping support for plaintext
http traffic can save costs. And if you front traffic with
global load balancer providers, you may handle http->https
redirects there anyway, so there's no loss
* Using spot instances, when an instance is deleted it actually
lowers the desired number of nodes in the VMSS so the node is
not replaced
* Restore the auto-scale setting needed to maintain a consistent
desired number of workers while spot instances come and go. This
was mistakely removed in refactoring
* Azure Load Balancers include 5 rules (3 LB rules, 2 outbound) whether used or not
* [#1468](https://github.com/poseidon/typhoon/pull/1468) added 3 LB rules to support IPv6 load balancing,
raising the rules count from 5 to 8 and added ~$21/mo to the cost of the load balancer. If you use an edge
(e.g. Cloudflare) a cluster does not need to load balance IPv6, so this additional cost can be avoided
* I noticed this because my load balancing costs were up for the last
few months. The gotcha is that outbound rules count toward the 5 rules
included with the base cost of the LB (~$18/mo)
Docs: https://azure.microsoft.com/en-us/pricing/details/load-balancer/
* flannel and Cilium default to UDP 8472 for VXLAN traffic to
avoid conflicts with other VXLAN usage (e.g. Open vSwith)
* Aligning flannel and Cilium to use the same vxlan port makes
firewall rules or security policies simpler across clouds
Rel: https://github.com/poseidon/terraform-render-bootstrap/pull/403
