@MadisonRaine Hard to say without seeing for oneself, but it definitely sounds like shyness to me, especially if he's looking down or away from you. Maybe you intimidate him, or maybe he does like you and he's nervous that he doesn't want to say the wrong thing to lose you. One might argue that he's bored and subconsciously looking for something else to keep busy with, however if he is still engaging in active conversation with you, I would say it's the opposite. The feet are always a good indicator of where their head really wants to be so pay attention to that when you're talking to him.
If he has known these "other people" for a long time that he's not like this with, it's because he's more comfortable around them, which is normal. First rule of dating someone new is to hang out with them when they're with their friends. This would give you a clearer sense of who they are and how they act, because they've been themselves in front of them for a lot longer than they've known you. If he doesn't know these other people at all and is not nervous around them, one could say it's because he really doesn't care about what they think of him, so why be shy around them?
Another sign that I've learned about is fondling items when in conversation with someone. Women do this especially. They play with the straw in their drink, or they twist their necklace around, all good signs of what's REALLY on their mind, but of course everything is about context so don't jump to any conclusions.
Hope this gives you something to go on, or sheds light in a way you haven't already thought of. Good luck!