If I would hit wall(with fist) very very hard (I mean with strength drastically greater than human can possess) and that wall wouldn't crush would I move backward (like few meters backward or other noticable distance) ?

Yes. A collision where you stick to the target is called completely inelastic. In any other collision there will be a recoil of some sort, separating the two bodies that are colliding. The important quantity here is momentum (mass x velocity, making it a vector). If there is no net external force, momentum will be conserved.

 

A wall is very massive (or effectively so, as it's attached to the earth), so you would recoil backwards in any collision that is not completely elastic.