It certainly felt safe, arriving late into Muscat after a dawn flight from London via Doha. Smooth highways took us past gleaming government ministries and the architectural masterpiece that is the Royal Opera House before depositing us at the Intercontinental Hotel just in time to answer the question that had been consuming us the entire journey. Would we be able to get a proper drink on this holiday? The answer was yes, but only expensive ones and only inside big licensed hotels, meaning most of our time outside the capital would have to be powered by non-fermented sugary highs.

The following morning we started as we meant to go on, loading up on pancakes drizzled in date syrup from the breakfast buffet. Then it was time for a jet lag-jettisoning speedboat ride from Marina Bandar for a dolphin-spotting group tour. There are so many frolicking pods that it’s practically impossible not to get lucky. We leapt from one side of the boat to the other, as more and more fins and noses arched out of the water, doing the dolphin equivalent of Mexican waves around us. But it was during the second half of the tour, when we moored in a shallow coral reef cove off Bandar Khiran Bay, that the magic really happened. The children put on snorkel masks and goggles for the first time and couldn’t believe what they were seeing as scores of neon-coloured butterfly, parrot and angel fish flitted around them. Our youngest vanished off under the care of one of the guides, reappearing half an hour later filled to bursting with wonder and a derring-do tale of following two sea turtles through the turquoise waters.

Strolling among Omani families on Muscat’s Water-Front

That evening we strolled among elegant Omani families out for dinner on the Water-Front, a fancy shoreline development of restaurants and cafés in the business district, feeling rather shabby amid the men and boys, in their beautiful white dishdasha robes and embroidered kuma caps, and the women, in their abayas.

The Sultan Qaboos Grand Mosque is open to non-Muslims

GETTY IMAGES

Though most public areas in the city require women only to cover their shoulders and wear clothing below the knee, some have stricter policies. In hindsight I probably should have anticipated that the Sultan Qaboos Grand Mosque, which we visited the next morning, would be one of them. I arrived dressed at my most demure yet still managed to scandalise the security guard with three inches of uncovered forearm and was directed to a nearby modesty kiosk to buy emergency sleeves (like tights for arms). Sultan Qaboos is the only mosque in the country accessible to non-Muslims, and only before 11am, every day apart from Friday. There are many impressive details here: the world’s second-largest Persian prayer carpet, handwoven by 600 women; a chandelier the height of a house; and some glorious gardens. But in 35C heat, with tights on my arms, it was nice to get back into the air-conditioned 4×4 we’d picked up to explore beyond Muscat.

Driving through Oman is easy — just watch out for camels

The straight, wide roads are a dream to drive on, though frantic undertaking seems to be a national sport and stray goatherds remain a hazard. At one point we couldn’t work out why an approaching car was flashing us furiously, until we saw an un-manned camel ambling across the A road in front of us.

Our first stop was spending a few hours in the ancient city of Nizwa at the foot of the magnificent Al Hajar mountains, once Oman’s capital and key trading hub in the 6th and 7th centuries. Despite Nizwa calling itself “the most visited monument in Oman”, its fort was quiet when we arrived, so much so that we practically had its 17th-century ramparts and towers to ourselves, as well as a performance of the rhythmic Al-Razha sword dance by a troupe of male musicians.

The Nizwa Fort is one of Oman’s most popular sites

GETTY IMAGES

From there we drove into the mountains, clinging on around every hairpin bend until we reached the village of Misfat al Abriyeen, or Misfah for short, 915m (3,000ft) above sea level. Entry to the village is only by foot and once we’d parked the car on the outskirts we understood why. There are no roads, just narrow stone-paved alleys that zigzag between its ancient dwellings. We stayed at the enchanting Al Misfah Hospitality Inn, with topsy-turvy staircases seemingly reaching to the sky. Next door the owner has turned the home he grew up in into a little museum dedicated to the old ways of life, though truly the whole village is a living archive. Follow Misfah’s maze-like paths downwards and you’ll soon hear the rushing noise of the “falaj” water system. For some 2,000 years a version of this system has been irrigating the small patches of terraced land, packed tight with mangos, pomegranates and fig and olive trees.

The Jebel Shams’ Wadi Ghul canyon

Jebel Shams’ Wadi Ghul is Oman’s answer to the Grand Canyon and its Balcony Walk is on every must-do list but it’s not for the faint-hearted. The drive there is heavy-going, with the final six miles steep and unpaved. Once you’ve parked up, though, the views across the canyon are astonishing.

Wadi Ghul is known as the Grand Canyon of Arabia

GETTY IMAGES

We shuffled along the rocky mountain ledge for 45 terrifying minutes before a combination of spindly eight-year-old legs, baking heat and not unreasonable vertigo (the gorge below is 1,000m deep) got the better of us. Had we kept going I’m told we would have been rewarded at the base with a deserted ghost village and a natural pool for swimming. Still, all was not lost: when we got back to the car two goats had jumped on the roof to munch away at an overhanging tree. It remains the funniest thing the kids have ever seen, and in their later retellings of this holiday, apparently a lot more memorable than the natural wonder they’d just been conquered by.

Stars Gate Camp at Wahiba Sands is a highlight

A room with a view at Stars Gate Camp

Next it was time for my highlight, the new Stars Gate Camp at Wahiba Sands, and not just because of those marble floors. Even getting there was fun. Before leaving the main road we had to make an easy pitstop to let our tyre pressure down before we could drive into the sands.

It’s the newest of a handful of luxury encampments in the heart of the desert, two hours south of Muscat. It was beautiful and twinkly at night with 15 tent-like huts arranged around a swimming pool for cooling down in the daytime heat. As sunset approached we booked a driver to take us slaloming up and down mighty sand dunes in a 4×4 like we were in a James Bond movie. Then we whizzed off to find the most majestic spot to watch the sky turn apocalyptic orange as the sun made its final descent.

Back at camp we sat up late on the veranda, stargazing into the brilliant sky. A few hours later we awoke to watch the colours changing again as the sun rose, accompanied only by the sounds of a distant cockerel and the fire of a solitary hot-air balloon passing slowly overhead.

• 16 of the best hotels in Oman

The harbour town of Sur has dhow-building factories

The next day we headed for lunch at the harbour town of Sur, where the highlight was a visit to its boat-building dhow factory. The magnificent wooden boats have been hand-carved here since the 6th century, when they would sail as far as India, east Africa and China with their cargoes of precious stones and spices.

Modern shipping means there are few left but the local craftsmen still use ancient techniques to produce two or three of these enormous works of art a year, which you can climb inside as they’re building them.

Traditional wooden dhows in the harbour town of Sur

GETTY IMAGES

Back in Muscat, we got another blast of tradition at Mutrah souk, one of the oldest markets in the country. Inside a covered arcade, the air thick with frankincense, vendors hawk jewellery and clothes for locals, and handicrafts and dagger souvenirs for the tourists. Prettiest of all is Ali Baba Gift Town, a trove of colourful trinkets that would be an Instagrammer’s paradise if photos weren’t banned.

We combined the souk with a sunset stroll along the Corniche as the street lights lit up around the harbour’s edge. Then we grabbed an elaborate juice concoction from one of the many street bars and jumped in a taxi to Ramssa, a lively restaurant where we had the choice of sitting at a table (boring!) or on the floor in our own booth with our feast laid out like a picnic in the middle (yes please!). Like cowards we avoided the camel cheeseburger, instead ordering falafel platters, soupy fish bablo and shrimp skewers, alongside enamel-stripping homemade lemonade and a fabulous selection of traditional puddings.

Less authentic but no less comfortable for it was the Hilton Muscat Al Bandar, our hotel for the last few nights. It is actually three hotels — two family-friendly and one adults-only — grouped together around a beautiful private beach. Here, we jet-skied, pool-lounged and checked out the hotel’s in-house classic car museum.

­­­­For the majority of the time, though, we just went round and round in circles in inflatable rings on its 500m-long lazy river, a modern nod to Oman’s traditional falaj water system and, in the children’s minds, up there with the hungry goats in terms of everlasting memories. Absolute genius. There, I said it again.
Krissi Murison was a guest of Experience Oman (experienceoman.om) and Stubborn Mule Travel, which has seven nights’ B&B from £2,350pp, including flights, car hire, excursions and some extra meals (stubbornmuletravel.com)