I'm afraid "you are not thinking 4 dimensionally", as John Lithgow often said in "Back to the future".
1) It is not Space that is curved, it is SPACETIME (4 dimensions).
2) If at rest in a three dimensional space, be it curved as, e.g., the sides of a pit, one needs a force to get him/her rolling downwards.
3) In 4 dimensional SPACETIME, on the other hand, it s a principle of its GEOMETRY that paricles/bodies not influenced by anything except the gravitational field, move along TIMELIKE GEODESIC CURVES. This is a GEOMETRIC PROPERTY, as particles/bodies are themselves GEOMETRIC FEATURES of Spacetime, according to Einstein's Gravitational Field Equations. In other words: The geometric features of spacetime called material particles satisfy the equations of timelike geodesics:
[math]\frac{d^2x^\alpha}{ds^2} + \Gamma^{\alpha}_{\beta\gamma}\frac{dx^\beta}{ds}\frac{dx^\gamma}{ds}=0 \Leftrightarrow \frac{d^2x^\alpha}{ds^2} = - \Gamma^{\alpha}_{\beta\gamma}\frac{dx^\beta}{ds}\frac{dx^\gamma}{ds}[/math]
the indices running over the values: [math]0[/math] (time coordinate) and [math]1, 2, 3[/math] (spatial coordinates) whereas summation is understood over repeated indices ("dummy indices"). The second half of this relation gives the acceleration ([math]\frac{d^2x^\alpha}{ds^2}[/math]) of the material particle.
4) Nowthen: Suppose Spacetime is curved, i.e. the quantities [math]\Gamma^{\alpha}_{\beta\gamma}[/math] (Levi - Civita connection coefficients) are not all zero. In that case, even if the spatial velocity is initially zero ([math]\frac{dx^\alpha}{ds} = 0, \alpha = 1, 2, 3[/math]) , this is NOT the case with the temporal velocity ([math]\frac{dx^0}{ds} \neq 0[/math] - no material particles can stand still in time!) Hence, from our equation above:
[math]\frac{d^2x^\alpha}{ds^2} = -\Gamma^{\alpha}_{00}\frac{dx^0}{ds}\frac{dx^0}{ds} \neq 0[/math] for [math]\alpha = 1, 2, 3,[/math] i.e., even if, initially, the particle is at rest (in space) it will accelerate, in general (that is, if [math]\Gamma^{\alpha}_{00}[/math] are NOT zero for all spatial values [math]\alpha = 1, 2, 3[/math]) as a result of the GEOMETRY of spacetime.
5) Motion along geodesic curves manifests itself in 3 dimensions by falling down the gravitatonal field.
I hope this has helped you out!