Yes, diving is very relaxing, and you don't really need to do anything really hard once you are in the water. However, you need to consider what you have to do on land before you get into the water. The diving gear itself is heavy and getting into a tight-fitting wet suit is a big job. You have to haul your diving gear around on land and put it on before you get into the water and even bringing it from the dive shop or base camp to the water can be a real chore.
Another thing to consider is emergency situations. When an emergency situation comes up while diving, your free floating fun has generally come to an end. You will likely have to swim hard and fast and you may need to pull someone else with you. You will need to be able to respond quickly, do what needs to be done, which might include cutting someone out of entanglement or any number of other things. Divers need a good level of physical fitness in order to perform these functions so that they will have the strength to do what they need to do and so they will not become so out of breath that they use up the air in their tank too quickly.
Lastly, you will be diving in a number of different conditions. It won't always be that brochure dive in calm tropical water. You may have to deal with strong currents or you may have to swim a long distance on the surface to get to shore or the boat once your dive is finished, all with your heavy diving gear on.