Curiously, the only people I know who have come down it didn't like it much. It's fast, lots of time on the brakes, tiring for the hands, shoulders, neck, etc., and you have to deal with a fair amount of auto traffic that can be challenging to interact with because your speed is often pretty close tot theirs.
And yeah, going up is only for the serious cyclist. The 60+ miles doesn't deter me--I'm happy doing that, too--but the near 10,000 feet of climbing, all in one shot, is daunting.