Legend has it that this is the very place where Bhartrihari, the step brother of Vikramaditya and a very renowned poet used to live and meditate after relinquishing all the luxuries of life.
The caves are situated on the banks of River Shipra, adjacent to the temple of Gadkalika, this is the reason the Caves were a serene place to be in and hence accentuating Bhartrihari's concentration while meditating.