LIGHTS OUT! Outage impacts Salesforce and marketing automation as people are left with a lot more access than originally intended.
On Friday 17th May 2019, Salesforce experienced a significant outage after a faulty database script accidentally gave users broader access to data than intended.
The error allowed Salesforce users to see all of their company’s data, whether they had permission to see it or not.
Used by sales and marketing teams around the world, Salesforce is one of the most widely used enterprise cloud platforms.
Salesforce told customers Friday that it is experiencing a “major issue with its service” and is attempting to resolve it as quickly as possible.
“To all of our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you,” Salesforce Co-Founder and CTO Parker Harris wrote on Twitter at 12:40 p.m. ET Friday. “Please know that we have all hands on this issue and are resolving as quickly as possible.”
While this issue only impacted customers that use the Pardot service or previously used the Pardot service, to protect our customers, Salesforce blocked access to all instances that contain affected orgs.
Haven’t checked up on our previous Salesforce release instalment? You can check it out here ‘Salesforce April 2019 release summary’
“To protect our customers, we have blocked access to all instances that contain affected customers until we can complete the removal of the inadvertent permissions in the affected customer orgs,” the company wrote on its Trust status website. “As a result, customers who were not impacted may experience service disruption.”
Customers have been reporting issues with Salesforce since Friday 17th May from 9:22 a.m. ET, according to downdetector.com, with the number of outage reports peaking at 3,262 shortly before 1 p.m. ET. An outage map on downdetector.com indicates significant problems with Salesforce in Boston, New York, Washington D.C., Atlanta, Chicago, Los Angeles, and San Francisco.
Salesforce users took to Twitter to complain about the outage.
As of May 18, 2019, access for users with a System Administrator profile has been restored to customers affected by the database script issue, and full access has been restored to customers unaffected by the database script issue.
Salesforce has supposedly been working over the weekend to restore access to customers, however, they as of Monday 20th May, platform access issues are still arising following the incident.
An update from 19th May 2019 states, “We restored administrators’ access to orgs affected by the recent permissions issue. In parallel, we ran an automated provisioning fix to restore user permissions to where they were before the incident occurred.“
“The automated provisioning has been run on most instances, including NA57. We continue to work on a fix for the remaining instances. If you are still experiencing issues with your user permissions, we have prepared a set of instructions for admins to manually restore them.”
“While the permissions issue did not affect every Pardot customer, we did disable jobs that sync data between Pardot and the Salesforce application in order to protect the integrity of our customers’ data.”
“As we work to restore permissions to affected orgs, we are also working to reenable sync between Pardot and the Salesforce application for those orgs that have not been affected or have had their permissions fixed. Data from interactions with Pardot is being queued and will resume processing back to the CRM once it is safe to do so.”