Part-1 Just getting started
This is written from the perspective of a 20 year IT veteran and Lifelong “iron” infrastructure enthusiast and engineer.
Some Notes for context
- Took a job as the Primary Engineer/Architect for a pure azure webapp infrastructure (all server/saas/paas services would be my baby, Network, Server, Storage, Security)
- I was to build originally 4 Environments when later turned into six and then were condensed down to 3 (I also add this was my first real DEV side of the house job, most of my life had been spent consulting on Back office HQ IT)
- This was a “temporary LIFT and SHIFT” we were moving the legacy application to azure from a “on-prem” style datacenter deployment
- This was to be PURE cloud, all on prem-services will be terminated at completion of project
Biggest (most important) Lessons learned.
My number #1 is this, if your going cloud GO CLOUD, my first and biggest mistake was trying to build “Iron” in the cloud. I was building cloud infrastructure like I would a Datacenter. The old spin up a server for whatever service you need. It made things expensive cumbersome and less then agile. Biggest lesson I learned was USE SaaS/PaaS in the cloud as much as humanly possible. This process took about 2 months to figure out.
The originally legacy product that we had to support was a mostly iis based .net app and some classic asp, this was all driven by a ms enterprise sql cluster. So being a dumb ass I start to build db clusters and iis clusters and just start “racking” up the hardware. My CTO went and talked with another company that is owned by the same Investor as us. They called us out on our design and told us our flaw.
So I tore up my plans and started from scratch (this wouldn’t be the only time I did this lets not count!) First things first the biggest and most expensive compute we will have will be our DB servers and our AI boxes. Of those two we can get rid of the compute and infrastructure overhead of the DB servers by going to azure SQL Managed Instance. This takes management of the sql cluster and all its network and resource neediness “out” of your realm of concern. So that’s where we started. On a personal note managed instance is really neat, you give it a empty subnet inside your vnet it preps it and bang managed instance is inside your vnet just like you had servers inside it. No need to do the nasty external endpoint (out thru the web)
Next up was to get rid of firewalls in every environment (Ill go into actual network design in my next blog) With out putting to much detail in which I will save for the next one, what we basically did was made separate subscriptions with separate vnets and then vnet peered them in azure. There was only going to be one fire wall needed at that point and that would go into the “MGMT” network. This firewall would be used for one thing and one thing only SSL VPN for admins and Devs. Each over the other environments would peered to the MGMT vnet and there fore allow traffic from vpn over the vnet peer. I am sure your asking soooo how are your clients going to get to the webapp? Instead of a traditional firewall, We used Imperva, (cloud waf and firewall NOT CHEAP but amazing) How this worked was, I put in a azure external LB with a front end PIP on it and then used azure NSGs (the firewalls of azure) to only allow traffic from Imperva inbound on that public ip range on a specified port
So with just those two changes We wacked at least 3 HA firewall pairs and 4 DB clusters, that is a METRIC FUCK TON of money in licensing, not mention storage and compute, and of course network
Almost as Important
So this one is more of a mind set or a soft skill then a technical skill. It might sound cliché but be agile and fluid. To be blunt keep a open mind at all times.
Agility use tools like Ansible, Chef, Puppet, Terraform the list goes on and on. Automation tools in other words. And this will also be another part to my blog series. Learning to write infrastructure from code, never been a dev in my life guess what I learned to write yaml (maybe not well but I get it done) We chose to go with ansible. I owe some people big kudos for that first few months of learning to use automation on that level, I think there is a few good friends that know who they are that kept me from jumping off Spaghetti Junction at Rush hour (Look here if your not from Atlanta) Once I figured out how git/ansible and azure CLI worked it became a lot easier. I could now build machines in minutes without ever touching the portal.
Id say the biggest pitfall in the infra as code area was MS would change the AZ CLI applets that ansible used to build your machines in azure. This brings me to the “being fluid” part, always be willing to learn a different way of doing it. Azures dev cycle in some cases is weeks or days. Ive seen weeks where there has been multiple updates and changes, thing appear and disappear. Azure is a teenager in puberty its got a really good jump start on who its going to be, but at some points it gets a little over zealous. So you have to stay very very flexible with your code and never get complacent.
So I am going to end Part one here. I could blab on for pages and pages about other take away’s (which I will do in following parts) But to me these were the big ones that I wanted to really start my story with. These are the two real pivot points of what was done. If I can save someone else from just a little bit of the “pain” I went thru it will all be worth it. So with that being said I humbly request for your feedback and questions, If there is anything I can talk about or share as I go thru my journey please ask!