To return to the original question, time has, within the realm of human understanding, a start point (The Big Bang) and can be thought to have an end point as well; the heat death of the universe, when all all energy has been evenly distributed between all existing particles, at which point all motion will cease. Humans mostly have a pretty tenuous understanding of time outside the scale of a few lifetimes anyway; we have a hard enough time contemplating the world we're going to leave for our grandchildren, let along comprehending a geological epoch.
Matter has a similarly enormous but finite bound. There are only so many hydrogen atoms. Some of them are alone and surrounded by vast gulfs of emptiness on all sides, but on an atomic scale EVERYTHING is surrounded by vast gulfs of emptiness on all sides anyway.
Distance is probably the closest property to being infinite; we only have a finite amount of time in which to travel and there's a fixed upper boundary on our speed. In functional terms, that means that space is almost completely filled with "You can't get there from here." Putting aside the fact that it's ridiculously difficult for us to get off the rock we live on, we can only make something move at the tiniest fraction of light speed, and even at that speed, we can only travel some finite distance within the span of time remaining in our universe.
So... space.