No drawbacks for either. Perhaps better handling on tarmac with a thinner tire.
Some say better handling off-road with a thicker tire but I've sometimes found that to be a very false case of confidence.
Thinner will almost always be better for a faster ride if that's important to you. Your question is too wide depending on what type of riding you are doing. Like knobbies vs. 50-50 tires in mud.
What type of riding you do will determine plus or minus's.