Most are being tracked by various institutions around the world. These entities have NEO monitoring down to a fine art. They can track the trajectories of these objects flying through space with  a high degree of accuracy.

     Great you say, at least they can predict if and when something might impact the earth and how much time we have.

     Not so fast! Now there is a theory that something called a Primordial Black Hole, a remnant of  "Big Bang",  might interfere with the the calculated orbits of this junk and possibly  knock  them into  a collision course with Earth. These PBH's can be minuscule in size or the size of a golf ball but have the mass of the earth. You  can imagine something like that  zooming thru the field of asteroids and upsetting their existing orbits.  Some scientist have calculated that  possibility of a large asteroid who's orbit was disrupted and colliding with  earth could happened every 190 million years. Geological studies seem to  concur with that calculation.

     It would seem that the question is not if a large asteroid will impact the earth but when. The good news is that  you  have a few million years to get  your stuff in order.

You can find out more by visiting Universe Today