I am asking about possible drift of personality traits.

 

If we have a small founder population (200-500 individuals) who just happen to say, less neurotic or more extroverted (I'm talking about Big Five traits which are hereditary) would it be possible that the population will shift in distribution of said traits?

Conditions are likely to favour offspring that are less neurotic or introverted despite the skewed initial population. It would not be "drift" - which refers to levels of random mutation, that, if benign or beneficial can passed to offspring, to become prevalent and ultimately fixed within a population. I'm not convinced neuroses and introversion are likely candidates for becoming fixed traits.