Imagine there’s been some apocalyptic event – say a cataclysmic solar flare has fried all electronics around the world. There’s no communication and cars are useless. After days of confusion subside to desperation, your nice neighbor is standing at your door with a gun. You don’t have a gun, so decide to give him all your food. It dawns on you that the city is becoming “uncomfortable”. You pack some essentials together and use your bike as a carrier to head toward the mountains where it might be safer. Along the way, you get mugged for your bike and tent and continue not-so-merrily on your way, relieved that you still have clothes. You make it to a secluded place and quickly become adept at bushcraft. You’ve set up an almost-comfortable camp which your new neighbors like so much they choose to keep it. So repeats this Groundhog Day cycle for a few months until you’ve settled far away. Your cave offers optimal shelter for the winter. You decide to decorate a bit, and with charcoal from the fire, you process months of trauma with some art therapy. You draw a picture of people chasing you with spears, and another of you hunting some sheep. As you contemplate your situation, it dawns on you “I’m a caveman”.
This is how it would have been for many tribes for some after the chaos at Babel.
Such a nomadic existence excludes metal technologies and fine stone-crafting. Metallurgy requires a permanent settlement to develop the infrastructure to mine, smelt, cast and forge. After a few generations of nomadic existence, these unused technologies are lost. Colonial history demonstrates this among native Americans, Polynesians and Aboriginals. It’s not that their ancestors were always primitive, but they lost “civilized” technologies while advancing in more practical hunter-gatherer ones. The native Americans pitied the Europeans who were incapable of survival. During the New Zealand land wars, the Maori were formidable because of their mobility. Europeans could be defeated by destroying their base. The Maori just built a new fortified village within a few weeks, which would be no great loss if destroyed.
The Bible records several people who for a time dwelt in caves.
Genesis 19:30 Lot and his two daughters left Zoar and settled in the mountains, for he was afraid to stay in Zoar. He and his two daughters lived in a cave.
Also Samson, David, Elijah. Jesus was buried in one.
See also https://creation.com/bible-cave-men.
Neanderthals were one such group with peculiar deformities through bad nutrition and prominent jawbones to support muscles from using teeth as tools. Our ancestors readily interbred with them as is evident in DNA ancestry. Among such established “missing links” is less diversity that we see in modern humans around the world.