Security incidents are nighmares to the companies as it significantly impacts reputation and credibility. Recently, CircleCI faced a security incident and forced to alert customers to rotate or revoke their secrets.
I thought of understanding the overall incident in detail and try to analyze the overall timeline to understand why and how it happened.
Before I start, it’s worth to appreciate the transparency extended by CircleCI, they not just alerted their customers and took necessary steps to ensure secured environment but also shared every possible details of the incident on public domain for their users.
Let’s try to understand the incident in detail now.
This attack took place through Malware which was capable to perform actions which can steal, damage or destroy things on Circle CI environment. This Malware was deployed on one of CircleCI engineer’s Machine and from this machine it may intended to impersonate and spread further since it’s having production system reach.
CircleCI reportedly said that it was intended to steal valid 2FA session, execute session cookies theft to get access of their production system.
It’s surprising to know that this malware was not detected by CircleCI’s antivirus software. That’s unfortunate, but what exactly malware is, we need to understand it first to understand what sort of problems it can create.
What is Malware?
Malware is malicious software intended for destruction, theft, encryption, alter of data or component in any system. There are few types which you might have heard of include trojan horse, ransomware, spyware etc. all are kind of malwares. These malwares are keep changed or improved continuously by hackers to be used for aforementioned destructive intentions. Thus some antivirus softwares also can’t detect them.
But how can it reach to your system?
There are different ways, I’m listing few of them,
- Download software from unauthentic websites
- Download attachment from spam
- Connect external drive which has malware
- Join insecure network etc.
Coming back to CircleCI incident, since team was totally in damage control mode, their investigation identified that the malware was capable of doing things which can compromise their production environment since the engineer on whose machine the malware was deployed was having access to the production for his work.
The idea of intrusion can be to impersonate (through cookie hijacking, stealing 2FA session data etc.) and gain production system access.
But why CircleCI asked customers to rotate their secrets?
This step was precausionary to ensure secure and clean system for both CircleCI as well as Customers.
As soon as the CircleCI team detected the unauthorized activity, it immediately took action to turn down this malware from accessing and doing any sort of damage.
By rotating secrets immediately they can get assurance that even if the attacker might have accessed any of the secret information, they will not be able to use them to speard and damage further in system.
Following diagram can help to visualize the incident easily,
What are the steps CircleCI took to avoid this in future?
Now this is very important question since this incident is a lesson to learn, not just for CircleCI but for any other company.
From their report, I can summarize following steps,
- Restrict production access to limited employees
- Monitoring and alerting systems
- Additional authentication on top of 2FA for prodution
- Detect and block malware through Antivirus Softwares
Did it cause any damage?
None so far as per the report.
However, I can give a sample scenario.
Consider we have an application deployed on AWS Infrastructure. For the sake of simplicity you stored your AWS access secrets to your CircleCI environment variables (neither I would recommend to use access key nor to use CircleCI environment variables for storing sensitive information).
Somehow attacker got these AWS access and secret key and now it can access your AWS resources for which this key has access.
Assume we are not aware that CircleCI environment variables are compromised and we keep the secrets in environment variable. Unknowingly, our AWS infrastructure is at risk!
This is just an example to explain the possibility and severity of threat and very unlikely to happen. Real time scenarios can be more complex and destructive in nature when it comes to security attack.
Having said that, the data of which domain is compromised makes a lot difference here as well. Domains like health care can not take chance of even a minor security incident and needs the highest level of surity in terms of security.
Report also mentioned a worthy line regarding security,
Security work is never done.
References
