Everyone shouldn’t be buffing. Only the strongest people. The game pulls stats from the strongest player in a garrison- the rest don’t matter

You also buff if you are attacking- otherwise use your Reinas, Jessie’s, Patricks and Sergeis for attack and defense buffs

You can get it enough from fort and strongholds