Incident management is the technique developers and IT operations teams use to respond to system failures (incidents) as rapidly as possible and restore regular service operations. This process includes identifying the root cause of the incident, implementing a fix, and testing the fix to ensure that the issue has been resolved.
An incident is an event that reduces the quality or completely disrupts a provided service. In most cases, incidents require an immediate reaction from the development or operations team.
Why is incident management a vital process?
Incident management is critical for any company that wants to provide consistent customer service. It ensures that incidents are dealt with quickly and effectively and that the right people are notified.
It also guarantees that the correct resources are allocated to deal with incidents and that communication is maintained between all parties involved.
The true cost of downtime
Facebook outages in recent years are an excellent example of how significant downtimes can be. In 2019, the Facebook platform was down for 14 hours. This incident is expected to cost about $90 million. In another outage in 2021, the cost was $65 million in only a few hours.
Incidents have high costs, including lost potential revenue and customers, damage to the company's reputation, or increased employee stress.
When incident management is done well, it can help to minimize negative impacts and allow teams to do more things, such as:
- Quickly detect and handle situations
- Notify customers and stakeholders about incidents and take steps to minimize any negative effects.
- Collaborate efficiently to resolve incidents and reduce any obstacles in the process.
- Learn from previous incidents to continually improve and build more stable systems.
Steps of the incident management process
The way that different companies handle incident management can vary greatly, depending on the size and type of the company, the tools and systems they use, and the customers and stakeholders they have. Because of this, no single incident management process fits all companies.
Companies that tend to have software products generally have a more standardized approach when it comes to incident management, with steps such as:
1. Detecting the incident
The key to effective incident management is having a centralized source of information that combines different monitoring and reporting tools. This makes it easier for support and other teams to identify, communicate, and resolve incidents. Incident management tools like Odown communicating with stakeholders can help with this by allowing various team members to collaborate on incident detection, communication, and resolution.
New outages can be started in one of two ways: automatically, through integration with a monitoring tool (like an Odown), or manually, through a customer support ticket.
Automatic monitoring for incidents
When a monitor reports an error, an incident is automatically created in the incident management solution. The team is then alerted of the outage via multiple alert channels such as email, Slack, Webhooks, SMS, telegram, or Discord.
Manually reported incidents
The incident form is used by support and other teams responsible for reporting manual incidents to submit their reports.
2. Communicating with stakeholders
Incident communication is important not only for acknowledging an incident has occurred but also for sharing new findings during the investigation and resolution process. The communication and incident resolution processes work together to ensure that all relevant information is shared promptly.
The best way to communicate about an incident is to use a status page. This allows easy communication between people inside and outside the company.
Internal communication
The internal communication aspect of an incident response plan includes any teams within the company that might be affected by the incident. This can include sales teams that may be giving demos of non-functioning products or marketing teams that are spending money on online ads that bring traffic to a landing page that is down.
The goal of internal incident communication is to ensure that all company operations are aligned in such a way that losses due to an incident are minimized.
External communication
The two key benefits of externally reporting incidents are maintaining customers' trust and saving customer support resources.
Having a company status page that is the go-to place for information during an incident can help reduce the number of questions directed to customer support. When done well, customers may appreciate the frankness of the communication about the downtime.
3. Resolving the incident
The investigation and resolution process can begin after an incident is detected and communicated to the relevant team.
The best-case scenario is that the first team member notified about an incident is also the person best equipped to solve it. According to the "you build it, you run it" philosophy, the developers who build the software should be the same people who maintain it since they have a better understanding of the system and are better equipped to troubleshoot issues.
If the current teammates assigned to the incident cannot resolve it, the incident must be escalated to a more senior or knowledgeable person.
What tools should be used for incident management?
Several different tools can be used for incident management, depending on the organization's specific needs. Common options include help desk software, project management software, and customer relationship management (CRM) software. The most important thing is to choose a tool to help the organization effectively manage incidents and meet its specific needs.
- Monitoring — to detect when anything is wrong with your system. These could be commercial solutions like Odown or open-source ones like Prometheus.
- Incident tracking — to keep track of incidents across multiple services.** Having an incident management tool as a centralized source of truth can be extremely helpful.
- Alerting — to always alert the right person on your team, you need a reliable alerting system with multiple channels. Most incident management tools like Odown have this capability.
- Chat room — Slack or Microsoft Teams help keep a timestamped record of what occurred during an issue and have a real-time conversation platform during the downtime.
- Video call — Tools for video calls like Zoom or Around are important for quick reaction calls with the entire team.
- Status page — to communicate incident updates both externally and internally.
