I suggest that you try scanning a simple, known-good load - specifically, a good 50-ohm dummy load. Then, try a 100-ohm resistor, and then a 25-ohm resistor. Then, a small capacitor in series with each of the above. If you still get inconsistent and noisy readings, you can probably figure that you have a duff unit.
Please do understand that trying to do a VNA or other analyzer run on an antenna, when the antenna is up in the air, is always fraught with complications. The problem is that the antenna will be picking up over-the-air signals, and these will mix with the return-loss signal coming from the VNA. If you've got any hams operating 75-80 meters within a few miles of you, their signals could be stronger (as seen by a VNA's receiver) than the part of the VNA's own test signal reflected from the antenna.
Any in-band noise source can cause this sort of problem... and 80 meters is a notoriously noisy band.
