I remember it being explained a long time ago, that Planck time is an indivisible unit. Hypothetically speaking if one acknowledges an indivisible unit of space (Say at point A) then plank time is the distance to the adjoining unit of space (Point B) with no other distinct unit/units of space between. If the smallest unit of space can be subdivided then Planck time could be subdivided also I suppose.
A couple of hypothetical thought experiments come to mind. The first is how ancient Hindu philosophers (2nd century B.C. http://en.wikipedia.org/wiki/Vaisheshika) postulated that matter may not be infinitely divisible. They argued if a small rock is infinitely divisible then we arrive at infinite particles and with these infinite particles we can re-construct any other matter including the Himalaya's. Since this can't be, a finite indivisible entity must underlie the construction of all matter, they called this hypothetical particle paramanu, in present day Indian (Devangiri) this is transliterated to atom, however I don't believe they meant present day atom, as much as "..That smallest indivisible particle".
The other thought experiment is Aristotle's I believe (Someone correct me, if I have this wrong). I think the premise of the hypothetical experiment is when two runners run towards each other at equal speed, then they would meet in the middle. If we now had them do the same with the new halved distance then theoretically they could run into each other, half way, ad infinitum. What happens then when Planck time is reached? The runner at point A and point B can either meet up at point A or point B and not in-between, there being no space in-between to run through.
Another way of considering the query "Why can't we have a smaller unit of time?" would be "Why can't we have half or a quarter photon?"