Yes, if you operate your drone outdoors under the Special Rule for Model Aircraft (which applies to drones flown recreationally) then you must fly the drone within your line-of-sight. While the FAA is not permitted to make rules regarding model aircraft or recreational drones, the Special Rule for Model Aircraft passed by Congress outlined a few limitations. One of the three defining elements of a “model aircraft” flight is that the unmanned aircraft be “flown within visual line of sight of the person operating the aircraft.”
According to the FAA, the line-of-sight mandate prohibits the use any type of vision enhancing devices (binoculars, night vision goggles, etc) other than eyeglasses or contact lenses. US Drone Law notes that the Special Rule for Model Aircraft does expressly prohibit vision enhancing devices. It is conceivable that an operator could maintain visual line of sight while utilizing a vision enhancing device.
The line-of-sight requirement for recreational use of drones does seem to eliminate the possibility of using first-person view goggles to fly a drone. Unless you fly under the Part 107 rule (which means you need to have a remote pilot certificate and comply with all other requirements in the Part 107 rule) and request a waiver to allow the use of first-person view goggles. In short, without a waiver or COA from the FAA, the only way you can use first-person view goggles with a drone is to operate the drone indoors.