The following is an in-depth description of how we treat, evaluate and patch CVEs at Arrikto. We will explain our security scanning process, CVE assessment, and what’s our current plan for dealing with existing CVEs.
At Arrikto, we are committed to provide a secure, enterprise-grade version of Kubeflow. As such, we are regularly scanning our images to identify and remove known CVEs. In addition, we review any remaining CVEs and make sure it doesn’t affect our product before releasing a new version. The process we are following is:
- Create a list of all the Docker images EKF uses.
- Check all images for known CVEs using Grype (https://github.com/anchore/grype), an open-source security scanner.
- Compare the CVEs against the CVEs of previous scans, and filter new and updated CVEs. Older CVEs that have not been updated (i.e., have the same severity and fix status) have already been triaged.
- Filter only Critical CVEs from the list of CVEs. In practice, these are the CVEs with any impact in the real world. In the future, we plan to include High severity CVEs as well.
- Assess if they are false-positives, i.e., if the scanner has detected a CVE due to a bad check. If yes, mark them as false-positive.
- If the upstream distribution has released a fix for a CVE, update the image to remove it.
- If the upstream distribution has not released a fix for a CVE, first assess if the CVE can actually affect the image. If it can, then we try to upgrade the image to a newer release of the distribution. Else, according to the CVE's severity, we may postpone an action until a fix is ready, or patch the image ourselves.
- Write an internal report for the triaging of the new CVEs, where we explain our actions for each of them.
This way, we ensure that our software has as few CVEs as possible, and that existing ones do not affect it. We have followed this process for every major (e.g., 1.5) and minor (e.g., 1.5rc4, 1.5.1) release.
In the real world, upgrading the base distribution of an image to the latest version does not erase all CVEs from it. There are various reasons why:
- There's no patch or it has not been backported yet.
- The software is not affected due to build flags.
- The security scanner has detected a false positive CVE.
- A new CVE popped up in the time between upgrading the image and running the security scan (in 2022, ~3 CVEs were submitted per hour).
We have to accept that occasional CVEs will slip through. For these types of CVEs, we assess if they affect our software or not. For our assessments, we take several factors into account:
- Does it affect containerized environments?
- Can a malicious user affect the environments of other users?
- What is the response of the authors or the OS vendor?
- Does our software make use of the vulnerable feature or code path?
If there are CVEs that severely impact our software, we hold the release and find a way to fix them. If they don't, we report why they don't affect us in the release notes, and proceed with the release.