I started with a 3" drone I built $175, analog box goggles $150 that had poor range and lousy quality, and a $100 controller. It takes a lot of practice to learn to fly, batteries lasted 4-5 mins. I've since bought a DJI Avata, which you'll be able to fly from day 1, with incredible HD goggles and you can work up to flying manually if you care to. You could probably find a refurb Avata with the 1st gen Goggles which I have and get into it for a bit less. I bought my Avata from a friend for $850, Avata, Goggles, Motion controller, & FPV Remote. It was a really good deal. I bought 2 more batteries and some extra props.
I have a DJI P4P and an Air2s, which I use for work. I wanted the capability to offer interior fly throughs so I slowly started with the 3", now it sits on the shelf. If you can swing it, I'd go for the Avata.