Musings, Thoughts, and Announcements

There are two very different definitions of DevSecOps in broad usage within the U.S. Department of Defense, and almost nobody seems to realize it. Unknowingly, entire organizations have been built around the term, resulting in whole departments talking past each other.

You keep using that word. I do not think it means what you think it means.

Yesterday I spoke at the 2023 DoD Weapon Systems Software, presenting findings from my team’s research into the performance and viability application platforms build with AWS’s managed edge devices—Outposts rack and Snowball Edge. We spent that last year testing the AWS solutions in various networking scenarios, including:

• degraded (moderate bandwidth, high latency)
• very degraded (low bandwidth, very high latency, packet loss)
• fully disconnected

Our platform architecture evolved over the year as we built in resiliency for the various failure scenarios we discovered. In the end, we demonstrated operational stability with AWS Outposts rack across all scenarios, including 200+ hours of fully disconnected operations.

Will Kline and I presented at KubeCon 2022 yesterday. Following up on our presentation at DEF CON 30 earlier this year, we explored the vulnerabilities exploited in the prior to talk and gave tips for hardening Kubernetes to limit the impact each of the vulnerabilities. If the demo cluster we attacked at DEF CON had been hardened properly, none of our exploits would have been possible.

Earlier today, my friend Will Kline and I presented at DEF CON 30. Our talk demonstrated the dangers that vulnerabilities in third-party applications can pose to Kubernetes clusters by attacking a cluster with vulnerable versions of Kiali , Fleet and Longhorn . The vulnerabilities we exploited were based on our own research, and the vulnerabilities were all responsibly disclosed and patched prior to this talk.

I had a great time creating and recording the attack, so I decided to publish a walkthrough for anyone interested in trying the attacks out themselves. I also published the Longhorn exploit , which I’ve named Rustler.

According to an April 15 advisory from Rancher, a vulnerability in go-getter may “expose the configured SSH private key secret if Fleet fails to download the git repo.” The advisory rates the severity as Low, despite the potential of a complete takeover of any cluster managed by the affected repository if an exposure leaks the key to a malicious user. More interesting, though, is Rancher’s finger pointing at go-getter to the tune of going to MITRE as a CNA of Last Resort to assign a CVE. In my opinion, the vulnerability is not in go-getter but in how Rancher Fleet handles errors returned by go-getter. I am the researcher that disclosed the original vulnerability to Rancher, and this is the story of that disclosure.

Longhorn has a vulnerability (CVE-2021-36780 ) in all versions of the 1.2.x branch prior to v 1.2.3 and in all versions prior to v.1.1.3 that allows unauthenticated attackers to access data in a Longhorn replica. An attacker with access to a cluster’s internal network can:

• read data in any Longhorn volume
• write data to any Longhorn volume

This vulnerability was discovered while investigating the vulnerability disclosed in CVE-2021-36779.

Longhorn , Rancher’s distributed block storage for Kubernetes, has a remote-code execution vulnerability (CVE-2021-36779 ) in versions in the v1.2.x branch prior to v1.2.3 and in all versions prior to v1.1.3. An unauthenticated attacker with access to the Longhorn network services can:

• execute, as root, any binary on the host machine or in the Longhorn container
• copy arbitrary data, including executable scripts or binaries, to the host machine or Longhorn container
• enumerate and read all files on the host machine, such as private keys and stored credentials

Because Longhorn is deployed as a DaemonSet, it is present on every node in the cluster, enabling an attacker to gain root access to every node.

A hard-coded signing key in Kiali prior to version 1.15.1 allowed attackers to craft valid authentication tokens if Kiali was configured for password-based authentication. The hard-coded key could be overridden in by configuration, but Kiali’s documentation and website did not mention the option. Red Hat has issued CVE-2020-1764 and the Kiali developers have published a security advisory .

CVE-2020-1762 was discovered during the investigation of CVE-2020-1764, and I’ll do a longer write up for that vulnerability. In short, though, Kiali did not check token expirations prior to v1.15.1. When using password-based authentication, users’ sessions would be based on a JWT token that would effectively never expire. Kiali offers several authentication mechanisms, and I did not take the time to fully investigate the impact of this vulnerability on other authentication mechanisms.

For the past few years I’ve been mulling over a seemingly simple problem: what’s the fastest way to obtain ownership of an arbitrary record in a distributed system? Consider, for example, a global network of multi-party video conferencing solutions. Users are assigned “meeting rooms” that are really just identifiers (e.g., 9813492 or dagan@example.com). When a user starts a meeting, that meeting room needs to be assigned to a specific conferencing server to receive and mix media streams for each user. It needs to happen fast, and it needs to happen only once per instance of the meeting. With conferencing servers distributed globally for high-availability and reduced latency, how can a distributed system ensure exactly once instance of the meeting room exists at any given time without incurring the performance impact of a single source of truth?

I’ve been working on video- and audio-conferencing solutions since I joined Vidtel in 2011, and it’s been exciting to see the space grow so quickly. For the past couple of years, I’ve been working with my colleagues at Fidelity to improve the enterprise experience in video conferencing, and I’m excited to share that we’ve been awarded US Patent 9,118,654 for a system my colleague Eric Anderson and I designed to address the complex requirements that govern communications between various compliance profiles (e.g., traders whose communications must be retained, human resources professional whose communications may need to be retained, and legal counsel whose communications are rarely retained).