Continuing to process data from March 28th & 29th
Incident
Report for HubSpot
Resolved
This issue is now resolved. We have caught up on the remaining backlog of data from March 28 and March 29 ET.
Customers can expect their analytics data from March 28 and 29 to be up to date.
We have identified the cause of the issue and are taking steps to prevent this from happening again. You can view the full retrospective here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
Customers can expect their analytics data from March 28 and 29 to be up to date.
We have identified the cause of the issue and are taking steps to prevent this from happening again. You can view the full retrospective here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
Update
We have restored the vast majority of the remaining backlog of data from March 28 and March 29 ET. We are continuing to process and validate a minority of analytics data from March 28 and 29, customers may notice some small variations in their reports from these days.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us later this week.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us later this week.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We have restored the vast majority of the remaining backlog of data from March 28 and March 29 ET. We have two updates regarding workflows and analytics data.
Workflows: Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 has been backfilled, and workflows that enroll based on that data has caught up.
Some workflow enrollments from contact property changes and form submissions during the outage were missed. Affected users have been notified in the Workflow App: https://app.hubspot.com/l/workflows. We have created an in-app system in the Workflow App that will let affected users easily identify affected workflows and create static lists of the contacts that were not enrolled as a result of the outage.
Analytics Delays: We are continuing to process and validate some analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us early next week.
We have restored the vast majority of the remaining backlog of data from March 28 and March 29 ET. We have two updates regarding workflows and analytics data.
Workflows: Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 has been backfilled, and workflows that enroll based on that data has caught up.
Some workflow enrollments from contact property changes and form submissions during the outage were missed. Affected users have been notified in the Workflow App: https://app.hubspot.com/l/workflows. We have created an in-app system in the Workflow App that will let affected users easily identify affected workflows and create static lists of the contacts that were not enrolled as a result of the outage.
Analytics Delays: We are continuing to process and validate some analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us early next week.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. Affected users have been notified in-app via the Email Dashboard: https://app.hubspot.com/email/l/manage/state/all. We have created an in-app system on the Email Dashboard that will let affected users easily create static lists of the contacts that were not sent an email as a result of the outage.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we are working to identify impacted contacts and workflows. We will directly communicate to those affected with next steps.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties have caught up.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us early next week.
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. Affected users have been notified in-app via the Email Dashboard: https://app.hubspot.com/email/l/manage/state/all. We have created an in-app system on the Email Dashboard that will let affected users easily create static lists of the contacts that were not sent an email as a result of the outage.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we are working to identify impacted contacts and workflows. We will directly communicate to those affected with next steps.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties have caught up.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us early next week.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working on adding custom “Not Sent” events for these emails and we will follow up with affected customers later this week.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we are working to identify impacted contacts and workflows. We will directly communicate to those affected with next steps.
Zapier: During the outage from March 28-29, specific Zapier Zaps were not triggered as expected for some portals. An in-app banner notification is live for affected portals. These customers can access a new settings page where they’re able to see how many zaps were not triggered. These zaps are arranged in three buckets: new contact, new contact in list, and contact property change. Affected users also have the opportunity to re-trigger these zaps if they deem it necessary.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us by Thursday, April 11.
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working on adding custom “Not Sent” events for these emails and we will follow up with affected customers later this week.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we are working to identify impacted contacts and workflows. We will directly communicate to those affected with next steps.
Zapier: During the outage from March 28-29, specific Zapier Zaps were not triggered as expected for some portals. An in-app banner notification is live for affected portals. These customers can access a new settings page where they’re able to see how many zaps were not triggered. These zaps are arranged in three buckets: new contact, new contact in list, and contact property change. Affected users also have the opportunity to re-trigger these zaps if they deem it necessary.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
You can expect another update from us by Thursday, April 11.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working on adding custom “Not Sent” events for these emails and we will follow up with affected customers later this week.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email, and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we will directly reach out to those affected with next steps.
Zapier: During the outage from March 28-29, specific Zapier Zaps were not triggered as expected for some portals. We are preparing an in-app settings page to enable customers to re-run some or all of the missed triggers and will communicate via an in-app banner notification and direct email outreach to portals affected.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
You can expect another update from us on Wednesday, April 10.
We have completed a retrospective of the March 28 and 29 outage which you can read about in more detail here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Email: All available email analytics data is up to date. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working on adding custom “Not Sent” events for these emails and we will follow up with affected customers later this week.
Workflows: Workflow enrollments are catching up from backfilled data. Data from analytics, email, and integrations that was missed over the previously reported period of March 28-31 is being backfilled. Workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we will directly reach out to those affected with next steps.
Zapier: During the outage from March 28-29, specific Zapier Zaps were not triggered as expected for some portals. We are preparing an in-app settings page to enable customers to re-run some or all of the missed triggers and will communicate via an in-app banner notification and direct email outreach to portals affected.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
You can expect another update from us on Wednesday, April 10.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We have completed a retrospective of the March 28 and 29 outage, you can read it here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. We had hoped to identify all affected customers by today, but some areas will not be identified until next week- namely Workflows, Email and Zapier. Our team is continuing to work on the following parts of HubSpot:
Emails: Some email analytics data is still delayed, but we expect that to be backfilled. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working through the weekend to identify which emails were not sent, and we will follow up with affected customers next week.
Workflows: As the data from analytics, email and integrations that was missed over that period is being backfilled, workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we will reach out to those affected directly with next steps.
Integrations & Webhooks:
Salesforce backfill is now complete.
We have finished backfilling Eventbrite timeline and contact data from March 28 and 29, and new/current data is not delayed.
Missed webhooks have now been replayed.
Some Zapier Zaps were not triggered as expected during the outage. As these are often time-sensitive we won't automatically re-run the triggers. We are identifying impacted portals and will reach out to those affected directly next week.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
You can expect another update from us on Monday, April 8.
We have completed a retrospective of the March 28 and 29 outage, you can read it here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to restore the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. We had hoped to identify all affected customers by today, but some areas will not be identified until next week- namely Workflows, Email and Zapier. Our team is continuing to work on the following parts of HubSpot:
Emails: Some email analytics data is still delayed, but we expect that to be backfilled. A small percentage of marketing emails were not sent to all recipients during the outage. The team is working through the weekend to identify which emails were not sent, and we will follow up with affected customers next week.
Workflows: As the data from analytics, email and integrations that was missed over that period is being backfilled, workflows that enroll based on that data will be triggered. Workflow enrollments from contact property changes and form submissions during the outage were missed, and we will reach out to those affected directly with next steps.
Integrations & Webhooks:
Salesforce backfill is now complete.
We have finished backfilling Eventbrite timeline and contact data from March 28 and 29, and new/current data is not delayed.
Missed webhooks have now been replayed.
Some Zapier Zaps were not triggered as expected during the outage. As these are often time-sensitive we won't automatically re-run the triggers. We are identifying impacted portals and will reach out to those affected directly next week.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
You can expect another update from us on Monday, April 8.
Update
All HubSpot apps are operational, and all current/new data is processing normally.
We have completed a retrospective of the March 28 and 29 outage that outlines what led to the issue, the ensuing events, our response, and what we’re changing as we operate moving forward, available here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to make progress on restoring the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Form Fills: Form fills between March 28 and 29 are now processed.
A minority of form fills (less than 0.2%) between March 28 and 29 were not submitted. Most of these are recoverable. We’ve identified affected customers and have begun reaching out today.
Emails: Some email analytics data is still delayed, but we expect that to be backfilled. A small percentage of marketing emails may not have sent to all recipients during the outage. The team is working hard to identify which emails were not sent, and we will follow up with affected customers.
Workflows: As the data from analytics and integration events is being backfilled, workflows that enroll based on either of these data sources will be triggered. Additionally, a minority of contacts that should have enrolled in workflows between March 28-31 will not have processed correctly. We’re working to identify these contacts and will follow up with affected customers.
Integrations & Webhooks: We are still backfilling Salesforce data and Eventbrite timeline and contact data from March 28 and 29, new/current data is not delayed.
We are ready to replay any missed webhooks, and are communicating to integrators prior to start due to the higher than normal traffic that will be generated. We expect to start this tomorrow morning EDT.
Some Zapier Zaps were not triggered as expected during the outage. As these are often time-sensitive we won’t automatically re-run the triggers. We are identifying impacted portals and will reach out to those affected directly with next steps.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
If you notice data inconsistencies in your account from March 28th and 29, know that we are working through the backlog and fixing these.
We have completed a retrospective of the March 28 and 29 outage that outlines what led to the issue, the ensuing events, our response, and what we’re changing as we operate moving forward, available here: https://blog.hubspot.com/customers/last-weeks-product-outage-what-happened-how-were-changing-as-a-result
We are continuing to make progress on restoring the remaining backlog of data from March 28 and March 29 ET. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlog finishes processing, data from the time of the outage may be incomplete for some customers. Our team is continuing to work on the following parts of HubSpot:
Form Fills: Form fills between March 28 and 29 are now processed.
A minority of form fills (less than 0.2%) between March 28 and 29 were not submitted. Most of these are recoverable. We’ve identified affected customers and have begun reaching out today.
Emails: Some email analytics data is still delayed, but we expect that to be backfilled. A small percentage of marketing emails may not have sent to all recipients during the outage. The team is working hard to identify which emails were not sent, and we will follow up with affected customers.
Workflows: As the data from analytics and integration events is being backfilled, workflows that enroll based on either of these data sources will be triggered. Additionally, a minority of contacts that should have enrolled in workflows between March 28-31 will not have processed correctly. We’re working to identify these contacts and will follow up with affected customers.
Integrations & Webhooks: We are still backfilling Salesforce data and Eventbrite timeline and contact data from March 28 and 29, new/current data is not delayed.
We are ready to replay any missed webhooks, and are communicating to integrators prior to start due to the higher than normal traffic that will be generated. We expect to start this tomorrow morning EDT.
Some Zapier Zaps were not triggered as expected during the outage. As these are often time-sensitive we won’t automatically re-run the triggers. We are identifying impacted portals and will reach out to those affected directly with next steps.
Other Delays: We are continuing to process and validate analytics data from March 28 and 29, customers may notice some small variations in their reports over the coming days. Calculated properties and dynamic lists based on email activity and form submissions may also be delayed.
If you notice data inconsistencies in your account from March 28th and 29, know that we are working through the backlog and fixing these.
Update
All HubSpot apps are now operational, and all current/new data is processing normally.
This issue affected certain data in HubSpot's systems between March 28 and March 29. We’ve restored most of that data and are working hard to restore the rest. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlogs are fully processed, data from the time of the outage may be incomplete for some customers, within the following parts of HubSpot:
Form Fills: A minority of form fills between March 28 and 29 were not submitted. Most of these are recoverable, and identifying who has been affected by this is our first priority. We’ll be following up directly with customers who have been affected when we have that information.
Emails: Some email analytics data is still delayed, but should be backfilled. A small percentage of marketing emails may not have sent to all recipients during the outage. The team is working hard to identify which emails were not sent, and we will follow up with affected customers.
Imports: Some imports from the time of the outage may have not been completed. You can see which imports failed, if any, and re-import them by going to the Imports tool.
Workflows: As the data is backfilled, workflows that enroll based on that data will be triggered.
CRM Updates: Some bulk property updates in CRM may not have been applied to all records. If you bulk updated a property between March 28 and 29, we encourage you to check that the property update was made and complete that update again if necessary.
Salesforce Integration: Some customers may see delays. We recommend resyncing any data to Salesforce that might have not been created/updated.
Other Delays: We are reprocessing analytics data from the time of the outage. Reports may look incomplete for data during that time period until the data finishes processing. Calculated properties and dynamic lists based on email activity and form submissions may be delayed.
If you notice data inconsistencies in your account from March 28th and 29th, know that we are working through the backlog and fixing these.
You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
This issue affected certain data in HubSpot's systems between March 28 and March 29. We’ve restored most of that data and are working hard to restore the rest. Data that was already in the system prior to March 28 or was added after March 29 was not impacted.
Until the backlogs are fully processed, data from the time of the outage may be incomplete for some customers, within the following parts of HubSpot:
Form Fills: A minority of form fills between March 28 and 29 were not submitted. Most of these are recoverable, and identifying who has been affected by this is our first priority. We’ll be following up directly with customers who have been affected when we have that information.
Emails: Some email analytics data is still delayed, but should be backfilled. A small percentage of marketing emails may not have sent to all recipients during the outage. The team is working hard to identify which emails were not sent, and we will follow up with affected customers.
Imports: Some imports from the time of the outage may have not been completed. You can see which imports failed, if any, and re-import them by going to the Imports tool.
Workflows: As the data is backfilled, workflows that enroll based on that data will be triggered.
CRM Updates: Some bulk property updates in CRM may not have been applied to all records. If you bulk updated a property between March 28 and 29, we encourage you to check that the property update was made and complete that update again if necessary.
Salesforce Integration: Some customers may see delays. We recommend resyncing any data to Salesforce that might have not been created/updated.
Other Delays: We are reprocessing analytics data from the time of the outage. Reports may look incomplete for data during that time period until the data finishes processing. Calculated properties and dynamic lists based on email activity and form submissions may be delayed.
If you notice data inconsistencies in your account from March 28th and 29th, know that we are working through the backlog and fixing these.
You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
All HubSpot apps are now operational, and all current/new data is processing normally.
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, data from the time of the outage may be incomplete for the following: calculated fields, dynamic lists based on email activity and form submissions, analytics activities, and imports. For imports, you can see which imports failed, if any, and re-try them by going to the Imports tool. We will update this page again, after 4:00pm unless there are any significant changes that require an update before then.
You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, data from the time of the outage may be incomplete for the following: calculated fields, dynamic lists based on email activity and form submissions, analytics activities, and imports. For imports, you can see which imports failed, if any, and re-try them by going to the Imports tool. We will update this page again, after 4:00pm unless there are any significant changes that require an update before then.
You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
At this point, all HubSpot apps are operational, and all current data is processing normally. Search and filtering functionality has been fully restored.
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, reports, analytics, and CRM updates may be incomplete, and dynamic list updates and workflow enrollments from the affected time period may be delayed.
We will update this page again, after 1:00 PM ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, reports, analytics, and CRM updates may be incomplete, and dynamic list updates and workflow enrollments from the affected time period may be delayed.
We will update this page again, after 1:00 PM ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
At this point, all HubSpot apps are operational, and all current data is processing normally.
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, reports, analytics, and CRM updates may be incomplete, and dynamic list updates and workflow enrollments from the affected time period may be delayed.
We will update this page again tomorrow, Monday 1st April, after 9:00 AM ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We are continuing to make progress to restore and validate any remaining data that was delayed between March 28th and 29th. Until the backlogs are fully processed, reports, analytics, and CRM updates may be incomplete, and dynamic list updates and workflow enrollments from the affected time period may be delayed.
We will update this page again tomorrow, Monday 1st April, after 9:00 AM ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
We are continuing to make progress on the processing and updating of data that was delayed between March 28th and 29th.
All current and/or new data is processing normally. As a result, we are moving the status of several individual HubSpot apps to operational. You can see the full list of apps below this update.
We continue to work through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics, and email reporting data may be incomplete until the backlogs are fully processed, and therefore some reports may be missing data from the time of the outage. Any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours. We expect email sending to be completed within the next 16 hours.
We will update this page again after 3:00 PM ET today unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
All current and/or new data is processing normally. As a result, we are moving the status of several individual HubSpot apps to operational. You can see the full list of apps below this update.
We continue to work through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics, and email reporting data may be incomplete until the backlogs are fully processed, and therefore some reports may be missing data from the time of the outage. Any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours. We expect email sending to be completed within the next 16 hours.
We will update this page again after 3:00 PM ET today unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
We are continuing to process and update a backlog of data from March 28 and 29, the CRM is functional at this time. All current and/or new data is processing normally.
We continue to work through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage. Additionally, any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours.
We will update this page again after 12:00 PM ET today unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We continue to work through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage. Additionally, any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours.
We will update this page again after 12:00 PM ET today unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
Although we continue to process and update data from March 28 and 29, the CRM is functional at this time. All current and/or new events are processing normally.
We are also still working through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage. Additionally, any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours.
We will update this page again at 9:00 AM ET tomorrow morning unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We are also still working through a backlog of data from form submissions and analytics. These tools, as well as search and filtering functionality, may experience some delays while we continue to restore impacted data. Reports, analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage. Additionally, any time-zone emails scheduled during the outage will be sent in their correct time zones over the next 24 hours.
We will update this page again at 9:00 AM ET tomorrow morning unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
Although we continue to process and update data from March 28 and 29, the CRM is functional at this time. All current and/or new events are processing normally. Since our last update, workflows, list enrollments and analytics are now operational. We are now working to review and validate data from the outage.
We are also still working through a backlog of data from form submissions and analytics. These tools may experience some delays while we continue to restore impacted data. Reporting and analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage.
We will update this page again after 9:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We are also still working through a backlog of data from form submissions and analytics. These tools may experience some delays while we continue to restore impacted data. Reporting and analytics data and email reporting data may be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage.
We will update this page again after 9:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
Although we continue to process and update data from March 28 and 29, the CRM is functional at this time but CRM search and filters might be behind. All current and/or new events are processing normally. We are still working through a backlog of data from workflows, form submissions, and list enrollments from March 28 and 29. These tools may experience some delays while we continue to restore impacted data. Reporting and analytics data and email reporting data will be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage.
We will update this page again after 6:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We will update this page again after 6:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
Although we continue to process and update data from March 28 and 29, the CRM is fully functional at this time. All current and/or new events are processing normally. We are still working through a backlog of data from workflows, form submissions, and list enrollments from March 28 and 29. These tools may experience some delays while we continue to restore impacted data. Reporting and analytics data and email reporting data will be incomplete until the backlogs are fully processed, therefore some reports may be missing data from the time of the outage.
We will update this page again after 6:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We will update this page again after 6:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
CRM data has correctly been restored, but CRM search and filters are behind. All current events are processing normally. We are still working through a backlog of data from workflows, form submissions, list enrollments from March 28 and 29. These tools may experience some delays while we continue to restore impacted data. Reporting and analytics data and email reporting data will be incomplete until the backlogs are fully processed.
We will update this page again after 3:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We will update this page again after 3:00 pm ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
We are currently working through the backlog processing data across multiple HubSpot tools. All current events are processing normally. As we backfill historical data, some reports may be missing data from the time of the outage, but they will return to normal during the coming days.
We will continue to update this page with information as the remaining issues are resolved. We expect our next update to be around 9 am ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
We will continue to update this page with information as the remaining issues are resolved. We expect our next update to be around 9 am ET unless there are any significant changes that require an update before then. You can read our more in-depth statement on the issue here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
All functionality is now available. New enrollments for workflows are operational, but we are still working through the workflows backlog. Some analytics functions are still delayed, including the email post-send screen. Some reports will be missing data from the time of the outage, but they will return to normal during the coming days. We will continue to update this page with information as the remaining analytics issues are resolved. You can read our more in-depth statement here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Update
Most critical functionality is now available. Workflows which are still experiencing issues. Some analytics functions are still delayed, including the email post-send screen. Some reports will be missing data from the time of the outage, but will return to normal over the coming days. We will continue to update this page with further information. You can read our more in-depth statement here: https://blog.hubspot.com/customers/a-note-on-the-march-28-outage
Monitoring
We are continuing to make progress. Most critical functionality is now available. Emails are sending, but email event data has not caught up, meaning email post send pages will not show any information. Analytics is processing, but is delayed. Workflows are coming back online. Although this functionality is available, it will take some time to recover fully. We will continue to update this page as more functionality becomes available. As mentioned previously, we will be posting a more in-depth statement by the end of the day (ET).
Update
We are continuing to make good progress. Most critical functionality is now available, including creating and updating records (contacts, companies, deals and tickets) and activities (notes, emails, calls, tasks, and meetings), list processing, exports, and search. We’ve made significant improvements to reporting, though some reports will still be experiencing issues. The team is still working to restore imports, marketing email, and calculated properties. We will continue to update this page as more functionality becomes available. As mentioned previously, we will also be posting a more in-depth statement by the end of the day (ET).
Update
We've made significant progress, many tools are back up and running, but we are still working to restore all functionality including email sending. Updates and analytics are still delayed and we are working to restore all data including form submissions, CRM property updates, list enrollments, and messages. We will update here with more information as that data is restored. We will also be posting a more in-depth statement by the end of the day (ET).
Update
Due to a large backlog of data processing, we are seeing an impact on a number of HubSpot tools. Customers may see issues with all email, page publishing, sales tools, integrations, automation, notifications, CRM, reporting, connected inboxes, content tools, and partner services, in addition to processing delays. We are actively working to resolve the issue and will update our Status Page with more information.
Update
Due to a large backlog of data processing, we are seeing an impact on a number of HubSpot tools. Customers may see issues with all email, page publishing, sales tools, integrations, automation, notifications, CRM, reporting, connected inboxes, content tools, and partner services, in addition to processing delays. We are actively working to resolve the issue and will update our Status Page with more information.
Update
Our systems are continuing to recover. Due to the large amount of information in the backlog, we expect system reprocessing to continue over the next few hours. We will post an update in three hours with the latest status update.
Update
Our systems are continuing to recover. Due to the large amount of information in the backlog, we expect system reprocessing to continue over the next few hours. We will post an update in three hours with the latest status update.
Update
Our systems are continuing to recover. Due to the large amount of information in the backlog, we expect system reprocessing will continue over the next few hours. We will post an update in three hours with the latest status update.
Update
This issue was caused by the failure of one of our internal infrastructure components, which resulted in processing delays for multiple applications. It caused a backlog of information that needed to be reprocessed. These systems are still recovering from the data backlog. We are actively working to resolve the issue and will update this page with more information.
Update
We are continuing to work on a fix for this issue.
Update
We are continuing to work on a fix for this issue.
Update
We are continuing to work on a fix for this issue.
Update
We are continuing to work on a fix for this issue.
Update
We are continuing to work on a fix for this issue.
Identified
We've identified the issue that's caused the processing delays for multiple tools to be delayed since 11:32 am ET. We're actively addressing the cause of this issue and will update this page when more information is available.
Update
We're experiencing processing delays across multiple tools in HubSpot including analytics, conversations, workflows, lists, meetings, form submissions, sequences, reporting, business card scanner, ads, and all email sends. We’re actively investigating the cause of this issue and will update this page when more information is available.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating this issue and will update this page when more information is available.
This incident affected: HubSpot APIs and Integrations.