Does being physically fit mean I have to workout a lot?

No, being physically fit does not mean you have to be ready for a weight lifting competition or to run a marathon.  What it does mean is that you should exercise regularly, the basic type of exercising that your doctor recommends, and that you are in good health.