I have a job that's physically demanding that does a lot of lifting throughout 9 hours. When i come home i take a nap then wake up to do minimal bodyweight training but its not enough to get me bigger.

If I workout five times a week there will be no room for growth because my job stil uses those muscles the next day. I don't want to injure myself but should I just focus on my job and do minimal lifting? Or how is else can I safely balance the two?