As BWD says it is about inertia.
Not as much the center og gravity as it can be the very same on two boards, but if one is an either longer board, or a heavier board, the moment of inertia (or rotational inertia) is bigger.
Moment of inertia is how difficult it is to "turn" something, so if really low, you got a foil that feels superlight to turn
For the very same reason we all know, that if you rotate on a chair with arms/legs out, and bring them in to your body, you will accelerate like crazy
We turn approximately around the center of effort (COE) on our hydrofoils if we just focus on yaw, COE is around 15 cm / 6 inches in front of the mast on many foils, the point between our feet.
If you got a longer board, nose longer, everything else being equal, even if board total weight is the same, the weight of the nose of the board sit further away from this COE.
The moment of inertia is proportional to the distance to the nose, SQUARED. And proportional to the weight of the nose, but not squared.
Meaning if you have a 120 cm board, the nose is say 75 cm in front of the COE, and with a 130 it is 85 cm away.
If you put the same quite heavy weight (a lot more than the board) on the very nose of these two boards, the big one will feel 28% slower to turn than the 10 cm shorter one.
In reality the difference is a lot smaller, my estimate would be a few percent only, as most of the mass sits closer to the COE, but my point is, that the longer boards will feel rapidly harder to turn because "distance" is squared.
And we humans are extremely sensitive to feel these small differences, so no doubt it can feel like A LOT
Having said that, I honestly believe it is also a mental thing, our minds help us feel smaller boards are a lot more agile than they really are sometimes
My 150 cm board is extremely light, and got almost no weight at the nose, so my 118 and 130 cm boards are not that much faster to turn, but mentally you will most likely get an aversion against the bigger ones, that will multiply any difference hugely, or even ADD a difference, even if there is none ha haa, that is how we as humans work.
Usually there WILL be a difference though, and can be easily felt, the shorter ones, same type of build will be much faster to turn!
In normal conditions I can not see wind on the board having any noticeable effect at all, compared to the moment of intertia which is a huge factor regarding how fast they are (to turn).
So extremely light at the nose, doesnt matter that much at the mast really, and a shorter board gives you the most lively board