Formed by the constant dripping of limestone-impregnated water over many thousands of years, the caverns contain stalagmites and stalactites, tinted stones and tapestries, and walls of marble. For ages a favourite area of the Gundungurra people, the natural wonder has attracted geologists, adventurers and tourists ever since Europeans first explored the site in the early 19th century.

An account of one such visit was provided in the Goulburn Herald of December 30, 1871.

“These caves, situated about 42 miles from Goulburn, have not been thoroughly explored. They have just been visited by a party of young men of Goulburn who left town at the end of last week for the purpose of eating their Christmas dinner in “marble halls”. And truly they were not disappointed, for not only had they the pleasure of doing so, but of also spreading their blankets on marble floors. 

Under the guidance of Mr Charles Chalker, the keeper of the caves, the party traversed the old cave from the end known as the “Whombeyan Church”, where the creek enters the side of the range, to the point at which the watercourse emerges from the ground on the opposite side, passing along the bed of the stream and wading through a deep circular basin known as the well. Afterwards the upper regions of the cave were visited. Those are partially separated into different rooms known as the ball room, crystal palace and glass cave etc, all of them beautifully ornamented by stalactites. 

The party next visited a smaller cave situated lower down the creek. The stalactites in this cave are very numerous and beautiful, and when struck by the hand make a most melodious sound very similar to that of a piano, and being of different lengths and thickness, there is a difference in the tone and pitch of each. This cave is known as the musical box. 

The cave recently discovered by Messrs Chalker and Mylecharane was next proceeded to, and in freshness of colour and beauty of form, it really far surpasses those before known. Almost in the centre lies the skeleton of a dingo; and in some parts the floors are so covered with bones of wallabies, which in the process of petrifaction have been firmly embedded in the limestone, that these portions may be said to be paved with bones. 

The party had the good fortune to discover another room in this cave which had before escaped observation, and which may he considered to be the gem of the whole collection. Upon a sloping surface about 20 ft. in height, semi-circular basins, resembling in appearance a swallow's nest, have been formed, it is supposed by the continual dropping from above, varying in depth from 1½ inch to 5 or 6 foot, of a beautiful white. The ceiling is here low and beautifully decorated with stalactites, a large cluster of which is suspended from the top into the centre of one of the basins, branching out and having the appearance of a large chandelier. 

After seeing the sights, all but three of the party took their departure for home on Monday night. Those who stayed were on Tuesday rewarded for their perseverance by discovering two new caves nearer to Mr Chalker's house, one of which is of considerable size, the other smaller and much resembling the "glass cave”, but far surpassing it in beauty of formation, the sides being crystallised and sparkling like diamonds; and another which they were prevented from exploring to a greater depth than about 150 foot on account of not having a sufficient length of rope. 

The party found Mr Chalker exceedingly kind and obliging, and we believe no more suitable person could be found in the colony to lead a party of visitors, both on account of this, and also on account of daring and intrepidity which it is impossible to praise too highly. 

Very much sport was afforded to the party by wallaby shooting. Those animals are here exceedingly plentiful, the place being actually alive with them. 

The last of the party bid farewell to Mr Chalker and the caves, and very reluctantly left a locality which, though it required some exertion to travel to, afforded them very much pleasure and good healthy enjoyment.”