Marketing Enterprise portals unable to access some public pages, workflows, emails, and reporting.
Incident
Report for HubSpot
Resolved
3:00 PM EDT - We have resolved the major outage and restored the cascading effects of the outage. All workflows, reporting, email, and website functionality has been restored. There are some outstanding issues our team is continuing to work through on a case by case basis. The current status of those issues are below.
We have published the results of a thorough retrospective here alongside the original post from our executive team.
Websites: Our engineering team has been able to restore all affected customer sites that we were able to rebuild programmatically. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage purposefully. If you’re experiencing issues related to missing pages, logos, or menus, please contact support.
We have published the results of a thorough retrospective here alongside the original post from our executive team.
Websites: Our engineering team has been able to restore all affected customer sites that we were able to rebuild programmatically. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage purposefully. If you’re experiencing issues related to missing pages, logos, or menus, please contact support.
Monitoring
6:45 AM EDT - We have resolved the major outage and restored the cascading effects of the outage. All workflows, reporting, email, and website functionality has been restored. There are some outstanding issues the engineering team continues to work through on a case by case basis. The current status of those issues are below.
Later today we will publish the results of a thorough retrospective alongside the original post from our executive team.
Websites: Nearly all affected customer sites are fully restored. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage purposefully. Our engineering and support teams are also working to identify any more cases of missing menus or pages. If you’re experiencing further issues like these, please contact support.
Email: The final Smart Send emails will be sent this morning at the appropriate local time for the recipients.
App Integrations: We have identified integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We have reached out to all affected app integrators directly.
Later today we will publish the results of a thorough retrospective alongside the original post from our executive team.
Websites: Nearly all affected customer sites are fully restored. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage purposefully. Our engineering and support teams are also working to identify any more cases of missing menus or pages. If you’re experiencing further issues like these, please contact support.
Email: The final Smart Send emails will be sent this morning at the appropriate local time for the recipients.
App Integrations: We have identified integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We have reached out to all affected app integrators directly.
Update
6:22 PM EDT - This will be the final update of the day. We’ll continue to work with affected customers on an individual basis to clear up any remaining issues and will post the next update by 9:30 AM EDT.
Tomorrow, we’ll conduct a thorough retrospective and will publish the results in the afternoon alongside the original post from our executive team.
Our team is still restoring sites to some Marketing Enterprise portals and addressing any other areas of the product that were affected. Here’s the current status of these features:
Websites: Nearly all affected customer sites are fully restored. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage. Engineering is continuing to investigate sites that are not rendering properly. If your site was affected and you’re still experiencing issues, please contact HubSpot support so they can help.
App integrations: We have identified instances where we incorrectly failed access token refresh requests from integrators and, as a result, those integrations may have detected some applications as having been uninstalled from HubSpot. We will be contacting those integrators directly and working with them to resolve these issues.
Hapikey access: Hapikeys that were deactivated as a result of this issue have been re-enabled and those customers have been notified directly.
Tomorrow, we’ll conduct a thorough retrospective and will publish the results in the afternoon alongside the original post from our executive team.
Our team is still restoring sites to some Marketing Enterprise portals and addressing any other areas of the product that were affected. Here’s the current status of these features:
Websites: Nearly all affected customer sites are fully restored. Logos, favicons, and menus have been restored, except in cases where customers manually changed those assets and settings during the outage. Engineering is continuing to investigate sites that are not rendering properly. If your site was affected and you’re still experiencing issues, please contact HubSpot support so they can help.
App integrations: We have identified instances where we incorrectly failed access token refresh requests from integrators and, as a result, those integrations may have detected some applications as having been uninstalled from HubSpot. We will be contacting those integrators directly and working with them to resolve these issues.
Hapikey access: Hapikeys that were deactivated as a result of this issue have been re-enabled and those customers have been notified directly.
Update
As of 3:38 PM EDT our team is still in the process of restoring sites to affected Marketing Enterprise portals, and addressing other areas of the product that were impacted. Here’s the current status of individual features:
Websites: As of 3:38 PM EDT, most customer sites are fully restored. Engineering is investigating the remaining sites not rendering properly. Some websites have homepages that were unpublished and we have identified and fixed all of the ones known to support. We are working to determine if any other sites’ home pages were affected so we can fix them. Other websites had logos, favicons, or menus reset. We anticipate restoring logos and favicons for most customers in the next 30 minutes and are working to restore menus.
Email: The final Smart Sends are now being sent. In rare cases, they might not send until their next Smart Send window, which could be as late as tomorrow. These will show as “processing” in the app. There were some SMTP and Single Send API attempts that failed during this window. Due to the nature of these requests, we are unable to retry them.
App integrations: We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators directly.
Hapikey access: Hapikeys for some customers were disabled as a result of this outage. We have re-enabled those hapikeys and are working to notify those integrators. We are also working to backfill those audit logs.
We are continuing to review all privacy-related features you may use in the product to confirm full functionality.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Websites: As of 3:38 PM EDT, most customer sites are fully restored. Engineering is investigating the remaining sites not rendering properly. Some websites have homepages that were unpublished and we have identified and fixed all of the ones known to support. We are working to determine if any other sites’ home pages were affected so we can fix them. Other websites had logos, favicons, or menus reset. We anticipate restoring logos and favicons for most customers in the next 30 minutes and are working to restore menus.
Email: The final Smart Sends are now being sent. In rare cases, they might not send until their next Smart Send window, which could be as late as tomorrow. These will show as “processing” in the app. There were some SMTP and Single Send API attempts that failed during this window. Due to the nature of these requests, we are unable to retry them.
App integrations: We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators directly.
Hapikey access: Hapikeys for some customers were disabled as a result of this outage. We have re-enabled those hapikeys and are working to notify those integrators. We are also working to backfill those audit logs.
We are continuing to review all privacy-related features you may use in the product to confirm full functionality.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Update
As of 2:20 PM EDT our team is still in the process of restoring sites to affected Marketing Enterprise portals, and addressing other areas of the product that were impacted. Here’s the current status of individual features:
Websites: Most customer sites are fully restored. Engineering is investigating sites that are missing pages or are rendering improperly. A number of sites have missing favicons and broken menus and we are working to restore those now. Some websites have homepages that are saved as drafts rather than published. We are working to restore those automatically but they can also be manually published by customers.
Email: As of 2:09PM batch email sends, RSS sends, and blog update sends were all complete. There was a small number of A/B sends that should have been sent during this window but were not. We are not automatically restarting these sends so we don’t alter the timing and impact test results. There were also a small number of Smart Send emails that were not send during this window. We expect these to send very soon.
Reporting: All tracking settings and reports have been restored. We’re continuing to monitor the event completion data recovery. Reports will continue to update as the status of other systems improves.
App integrations: We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators directly.
Workflows: All workflows have been reactivated. Please see prior updates for more information.
We are currently reviewing all privacy-related features you may use in the product to confirm full functionality.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Websites: Most customer sites are fully restored. Engineering is investigating sites that are missing pages or are rendering improperly. A number of sites have missing favicons and broken menus and we are working to restore those now. Some websites have homepages that are saved as drafts rather than published. We are working to restore those automatically but they can also be manually published by customers.
Email: As of 2:09PM batch email sends, RSS sends, and blog update sends were all complete. There was a small number of A/B sends that should have been sent during this window but were not. We are not automatically restarting these sends so we don’t alter the timing and impact test results. There were also a small number of Smart Send emails that were not send during this window. We expect these to send very soon.
Reporting: All tracking settings and reports have been restored. We’re continuing to monitor the event completion data recovery. Reports will continue to update as the status of other systems improves.
App integrations: We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators directly.
Workflows: All workflows have been reactivated. Please see prior updates for more information.
We are currently reviewing all privacy-related features you may use in the product to confirm full functionality.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Update
As of 1:19 PM EDT our team is still in the process of restoring sites to affected Marketing Enterprise portals, and addressing other areas of the product that were impacted. Here’s the current status of individual features:
Websites: For impacted portals, any website page, blog, or landing page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. As of 1:00 PM EDT most customer websites are functioning properly. We have restored domain access and have updated content automatically where possible. Engineering is investigating sites that are missing pages or are rendering improperly. Customers who are seeing pages down which they manually updated during the outage should re-publish those pages. Customers can also remove URL mappings they created to work around the outage.
Email: We are preparing the batch email, smart send, time zone, blog update, and RSS sends that were not sent during this window this morning. They will begin sending shortly.
Reporting: Lead Revisit notification settings, cookie banner settings, analytics and tracking code settings, custom event definitions, attribution reports, legacy contact and company reports have been restored. We're continuing to work to restore sources reports and event completion data.
App integrations: Some integrations may have tried to refresh access tokens during the outage, resulting in potentially declined or misconfigured tokens. Any integrators that require enterprise-only scopes and received declined tokens, will need to refresh. We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators. For any integrations that refreshed the tokens successfully during the outage, they may have received misconfigured tokens, with a lower level of access. Those tokens were automatically restored at 1 PM ET.
Workflows: All workflows have been reactivated. Please see prior updates for more information.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Websites: For impacted portals, any website page, blog, or landing page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. As of 1:00 PM EDT most customer websites are functioning properly. We have restored domain access and have updated content automatically where possible. Engineering is investigating sites that are missing pages or are rendering improperly. Customers who are seeing pages down which they manually updated during the outage should re-publish those pages. Customers can also remove URL mappings they created to work around the outage.
Email: We are preparing the batch email, smart send, time zone, blog update, and RSS sends that were not sent during this window this morning. They will begin sending shortly.
Reporting: Lead Revisit notification settings, cookie banner settings, analytics and tracking code settings, custom event definitions, attribution reports, legacy contact and company reports have been restored. We're continuing to work to restore sources reports and event completion data.
App integrations: Some integrations may have tried to refresh access tokens during the outage, resulting in potentially declined or misconfigured tokens. Any integrators that require enterprise-only scopes and received declined tokens, will need to refresh. We are working to identify integrators whose access token refresh requests were declined and have not retried. These integrators may have interpreted these declined requests as customers uninstalling their integration. We will be reaching out to these integrators. For any integrations that refreshed the tokens successfully during the outage, they may have received misconfigured tokens, with a lower level of access. Those tokens were automatically restored at 1 PM ET.
Workflows: All workflows have been reactivated. Please see prior updates for more information.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Update
As of 12:20 PM EDT our team is still in the process of restoring sites to affected Marketing Enterprise portals, and addressing other areas of the product that were impacted. Here’s the current status of individual features:
Websites: For impacted portals, any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 12:20PM EDT we have restored domain access and are continuing to update and republish live content. We expect a majority of content to be available again in the next 30 minutes and engineering is investigating sites that are missing pages.
Email: We have identified a small number of batch email, smart send, time zone, blog update, and RSS sends that were not sent during this window. We will not re-send any missed emails until all websites are functional. This will not cause any duplicate sends, but may cause some duplicate events to appear in post send pages for emails that are not sent due to normal reasons (unsubscriptions, prior bounces).
Workflows: All workflows have been reactivated. Please see prior updates for more information.
Reporting: Lead Revisit notification settings have been restored. We're continuing to work to restore analytics and tracking code settings, including cookie policy settings. We’re also continuing to restore Enterprise reports like events, attribution, and legacy contacts and companies reports.
App integrations: Some integrations may have tried to refresh access tokens during the outage, resulting in potentially declined or misconfigured tokens. Any integrators that received declined tokens, will need to retry and refresh. For any integrations that refreshed the tokens successfully during the outage, they might have received misconfigured tokens, with a lower level of access. Those tokens will be automatically restored by 1PM ET. We are still working to understand any additional impact for integrators.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Websites: For impacted portals, any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 12:20PM EDT we have restored domain access and are continuing to update and republish live content. We expect a majority of content to be available again in the next 30 minutes and engineering is investigating sites that are missing pages.
Email: We have identified a small number of batch email, smart send, time zone, blog update, and RSS sends that were not sent during this window. We will not re-send any missed emails until all websites are functional. This will not cause any duplicate sends, but may cause some duplicate events to appear in post send pages for emails that are not sent due to normal reasons (unsubscriptions, prior bounces).
Workflows: All workflows have been reactivated. Please see prior updates for more information.
Reporting: Lead Revisit notification settings have been restored. We're continuing to work to restore analytics and tracking code settings, including cookie policy settings. We’re also continuing to restore Enterprise reports like events, attribution, and legacy contacts and companies reports.
App integrations: Some integrations may have tried to refresh access tokens during the outage, resulting in potentially declined or misconfigured tokens. Any integrators that received declined tokens, will need to retry and refresh. For any integrations that refreshed the tokens successfully during the outage, they might have received misconfigured tokens, with a lower level of access. Those tokens will be automatically restored by 1PM ET. We are still working to understand any additional impact for integrators.
Please see the public post from our executive team here: https://blog.hubspot.com/customers/were-sorry
Update
As of 10:45 AM EDT our team is in the process of restoring sites to affected Marketing Enterprise portals, and addressing other areas of the product that were impacted. Here’s the current status of individual features:
Websites: For impacted portals, any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 10:45 AM EDT we are programmatically updating and publishing live content and working to restore domain access and content access. We will update this as soon as that process is complete.
Email: A small number of scheduled email sends, smart sends, and time zone sends did not send as expected between 4:40 AM and 8:10 AM EDT. We have identified the batch email, smart send, and time zone sends that were not sent during this window and will resend them once all websites have been restored. We are adding systems to prevent any cloned or manually re-sent emails from being sent again. That said, it’s best to not try and re-send any emails as we work through this fix as this could lead to duplicate email sends.
Workflows: All workflows have been reactivated. Contacts are flowing through again as expected. If a contact was already enrolled, but a step was set to execute between approximately 4:00AM and 10:26AM EDT, the contact will have skipped the step. We have a list of those contacts for each HubSpot customer, and are investigating additional remediation. Contacts will not be re-enrolled in the same workflow a second time and multiple emails will not sent to the same contact through a reactivated workflow.
Reporting: We're continuing to work to restore analytics and tracking code settings, and restore Enterprise reports like events, attribution, and legacy contacts and companies reports.
Websites: For impacted portals, any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 10:45 AM EDT we are programmatically updating and publishing live content and working to restore domain access and content access. We will update this as soon as that process is complete.
Email: A small number of scheduled email sends, smart sends, and time zone sends did not send as expected between 4:40 AM and 8:10 AM EDT. We have identified the batch email, smart send, and time zone sends that were not sent during this window and will resend them once all websites have been restored. We are adding systems to prevent any cloned or manually re-sent emails from being sent again. That said, it’s best to not try and re-send any emails as we work through this fix as this could lead to duplicate email sends.
Workflows: All workflows have been reactivated. Contacts are flowing through again as expected. If a contact was already enrolled, but a step was set to execute between approximately 4:00AM and 10:26AM EDT, the contact will have skipped the step. We have a list of those contacts for each HubSpot customer, and are investigating additional remediation. Contacts will not be re-enrolled in the same workflow a second time and multiple emails will not sent to the same contact through a reactivated workflow.
Reporting: We're continuing to work to restore analytics and tracking code settings, and restore Enterprise reports like events, attribution, and legacy contacts and companies reports.
Update
As of 9:30 AM EDT our team is in the process of restoring sites to affected Marketing Enterprise portals, as well as to remedy other areas of the product impacted. Here’s the current status of individual features:
Websites: For impacted portals any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 9:30 EDT some websites may still redirect to the staging domains as we work to restore access across all affected subdomains. Customers can continue to re-connect or re-publish content to take their sites live - we are working on programmatically doing this for all pages as well.
Email: Many scheduled email sends, Smart-Sends, and Time Zone sends did not send as expected between 4:40AM and 8:10AM EDT. We are working to identify the specific emails that should have sent, but didn’t. Once this is complete we will be re-sending them once all domain and website fixes are in place. Customers should not re-send any email on their own as this could lead to duplicate email sends.
Workflows: We are working to re-activate workflows. When they resume, contacts will be enrolled as normal moving forward, but from approximately 4:00am EDT until the time that workflows are reactivated, contacts were not enrolled in workflows. Additionally, contacts who had a workflow step scheduled during that time will have skipped the step. We are getting a full list of contacts who were impacted.
Reporting: We're working to restore Analytics & tracking code settings, in addition to restoring Enterprise reports like Events, Attribution, and legacy Contacts & Companies reports.
App integrations: We are investigating any affect this issue might have had on app integrations, like causing the integration to temporarily suspend actions.
Please see public post from our executive team here - https://blog.hubspot.com/customers/were-sorry
Websites: For impacted portals any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 9:30 EDT some websites may still redirect to the staging domains as we work to restore access across all affected subdomains. Customers can continue to re-connect or re-publish content to take their sites live - we are working on programmatically doing this for all pages as well.
Email: Many scheduled email sends, Smart-Sends, and Time Zone sends did not send as expected between 4:40AM and 8:10AM EDT. We are working to identify the specific emails that should have sent, but didn’t. Once this is complete we will be re-sending them once all domain and website fixes are in place. Customers should not re-send any email on their own as this could lead to duplicate email sends.
Workflows: We are working to re-activate workflows. When they resume, contacts will be enrolled as normal moving forward, but from approximately 4:00am EDT until the time that workflows are reactivated, contacts were not enrolled in workflows. Additionally, contacts who had a workflow step scheduled during that time will have skipped the step. We are getting a full list of contacts who were impacted.
Reporting: We're working to restore Analytics & tracking code settings, in addition to restoring Enterprise reports like Events, Attribution, and legacy Contacts & Companies reports.
App integrations: We are investigating any affect this issue might have had on app integrations, like causing the integration to temporarily suspend actions.
Please see public post from our executive team here - https://blog.hubspot.com/customers/were-sorry
Update
As of 8:16AM EDT our team is continuing to work to restore access to features to Marketing Enterprise portals affected by this issue. Here’s the current status of individual features:
Websites: For impacted portals any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 8:10AM EDT some websites may still redirect to the staging domains as we work to restore access to affected subdomains.
Email: Many scheduled email sends, Smart-Sends, and Time Zone sends did not send as expected between 4:40AM and 8:10AM EDT. We are working to understand if we can automatically send the sends that did not go out as expected.
Workflows: Some workflows were deactivated and we are working to re-enable them. For actions that were disabled during this time we are looking to replay these safely and are still looking into total impact.
Salesforce integration: No known effect on salesforce integration for these portals.
Reporting: The engineering team is working on restoring Custom Event tracking for affected portals.
Websites: For impacted portals any website page, landing page, or site page that was connected to a HubSpot hosted domain as of this morning redirected to a staging domain (hs-sites). In some cases, the website pages were not loading. We have identified and remedied the root cause of this and are working to restore the pages and enable subdomain access. As of 8:10AM EDT some websites may still redirect to the staging domains as we work to restore access to affected subdomains.
Email: Many scheduled email sends, Smart-Sends, and Time Zone sends did not send as expected between 4:40AM and 8:10AM EDT. We are working to understand if we can automatically send the sends that did not go out as expected.
Workflows: Some workflows were deactivated and we are working to re-enable them. For actions that were disabled during this time we are looking to replay these safely and are still looking into total impact.
Salesforce integration: No known effect on salesforce integration for these portals.
Reporting: The engineering team is working on restoring Custom Event tracking for affected portals.
Update
We have fixed the issue that caused some portals to lose enterprise features, and we are working to restore website functionality. Our team is also monitoring the situation to see if there are any other impacts.
Identified
We have identified the cause of the issue and are working to restore functionality to all portals.
Update
We are continuing to investigate the issue. Currently we're seeing issues with marketing enterprise features, ad-ons such as ads and website tools, and some live pages are not loading.
Investigating
We are currently experiencing an issue with marketing enterprise features. Our development team is investigating the issue.