The technology gives users choice in how secure they want their systems, but most people will stick with the more safe default mode that runs only trusted code.
Users will have the ability to set the level of protection they want with Mountain Lion's new Gatekeeper feature.
Many people complain about Apple controlling what apps can run on the iPhone, but with Mac OS X Mountain Lion, the company has struck the right balance between security and freedom, experts say.
"Users can opt to turn this off and allow any software to be installed with the click of a button," said Dino Dai Zovi, chief technology officer at security firm Trail of Bits. "There'll be no need to jailbreak your Mac."
Apple unveiled details last week about Mac OS X 10.8, the latest version of its Macintosh operating system, which is due out in the middle of the year. One of the new features, aptly dubbed Gatekeeper, is designed to protect users from downloading malicious applications. Users have three settings to choose from: only allowing apps from the Mac App Store; allowing apps from the online store as well as apps from registered developers; or any apps at all.
The default is the second option, which uses a signature system that checks with Apple to make sure the app was created by someone with a valid Apple developer ID. When a user tries to install that app for the first time, the system will check to make sure it has not been previously identified as malware, has not been altered since being signed and that the developer is not a known distributor of malware.
The system will not uninstall apps that have not been signed or prevent people from using apps they've already installed. And the system has been designed to allow people to manually override the protection measures and install software that hasn't been signed, even if the settings are turned all the way up to allow apps from the App Store only.
Gatekeeper won't block apps that are already on the system, files installed using DVDs and USB drives, or scripts and other code that isn't executable, Rich Mogull, CEO of Securosis, wrote in a blog post. In addition, Gatekeeper looks for files that contain an extended file attribute set called the "quarantine bit", and if the apps don't have that then they aren't affected by Gatekeeper; nor are users protected from Flash and Java malware, he wrote.
Apple also is requiring developers distributing apps through the Mac App Store to use sandboxing technology designed to limit the ability of buggy or malicious code to harm the system by isolating it.
This level of protection is unprecedented, although it sounds like Microsoft might be headed in a similar direction with Windows 8 and its Metro app store. A Microsoft spokesperson did not immediately respond to a request for comment.
"Apple is advancing the farthest in the level of control of their platform, which is a very Apple thing to do and does provide security benefits," Dai Zovi said.
"Gatekeeper is a very big deal. Combined with the other recent security enhancements in OS X, it virtually eliminates the possibility that Macs will ever see the sort of malware epidemic that Windows users have dealt with," Mogull told CNET. "I'm not saying it will stop all kinds of trojan-based attacks, but that it will prevent that from happening on a wide, long-term scale ... for average consumers this could be game changing."
Apple has marketed the Macintosh system as being safer than Windows, and security experts agree but mostly because malware writers tend to target the dominant platform to get maximum impact. By allowing iPhone users only to run apps that Apple has pre-approved, the company has managed to protect iPhones from malware that has hit Android users. So, in one way, Apple is extending its iPhone security model to the Mac OS, while also tacitly acknowledging that Mac users aren't immune from malware on the desktop.
"They're looking ahead and seeing that malware might be a problem for Mac OS X," said Charlie Miller, security researcher at Accuvant. "It's really the only long-term solution to malware. They've seen that in the iOS the lockdown approach works and for most people it's OK. I think they're now trying to move OS X toward the iOS security model and I think that's smart."
Miller said he just hoped that Apple wouldn't change the feature in the future to remove the option to allow any app to run. "I'm worried the next version of the 'opt out' switch will be gone," he said.
"Apple is not in denial that OS X is vulnerable to things like malware just like any operating system and is doing things to protect users," said Michael Gartenberg, an analyst at Gartner. "It's a good, balanced approach."
Apple has been making strides not just in the security protections it adds to its software but also in how it works with the community of security experts and developers.
"While I still don't agree with how they handle everything, I've noticed a massive change in the past two years. With Lion, Apple for the first time invited certain security researchers to evaluate pre-release software (albeit under NDA) without forcing them to pay for a Developer Program subscription. With Mountain Lion they pre-briefed an outside security type for the first time ever. They have hired a bunch of very smart and experienced security experts," wrote Mogull on his blog.
"I believe Apple recognises they aren't the kid in the corner hanging out with all the artsy types any more," he wrote. "They know that the bottom line will be affected if users no longer feel safe on their products. So they are taking security much more seriously."