Table of Contents
The event formally known as KubeCon + CloudNativeCon Europe 2021 – Virtual was held online from May 5-7. This was my third straight virtual KubeCon and I enjoyed it a lot. I want to share my thoughts on the conference and some of the talks I recommend. Videos of the presentations are still available on the event platform if you registered, but they will eventually move to the CNCF YouTube Channel.
There were two significant differences between this and last year’s virtual EU KubeCon: the schedule and the event platform.
When I first saw the schedule, I have to admit that I was bummed. I’m in the US on Pacific time, and last year I was able to catch most of the program live, including the keynotes. This year the schedule shifted a few hours earlier, and the keynotes started at about 1 AM my time, so that wasn’t happening. I assume that the organizers made this change to fit the conference into more regular hours in Europe. Despite that schedule being less convenient for me, I think it was the right thing to do.
The other significant change was the new virtual conference platform. This year’s platform looked a lot less slick to me at first glance, but what the new platform lacked in visual slickness was very much made up for with its functionality.
That sidebar on the left did so much heavy lifting in terms of the conference experience. With the previous platform, I often had problems finding the next talk I wanted to view. You had to know the track and locate that, and then find the session. This new platform let you add talks to your own schedule. That was a huge usability improvement, and I have to applaud the CNCF and the conference organizers for seemingly putting utility above aesthetics here. Rushing to view a talk and not being able to find it easily in past events was not a great feeling.
Now let’s dig into some of the talks that I watched. Since I didn’t see the keynotes, I won’t talk about them, but there are many other great sessions to share.
#Day One - Wednesday
#We Didn’t Start the Fire: Communication Breakdowns and How to Prevent Them
I was very excited to see this talk from Ian and Kat, two people that I appreciate a lot. The focus was on communication from maintainers to the community, and the speakers dug through specific examples of when that went awry.
The first example was some breaking changes in 1.16. The maintainers didn’t think the changes were a big deal, and they just used their previous methods of communication. But it turned out that the community had grown, and many folks didn’t get the memo. This was one of the recurring themes in the talk: the community is constantly growing and changing, and communication needs to adapt.
A lot of the talk was devoted to something I remember very well, the Dockershim incident. A tweet was made that said Docker in Kubernetes was being deprecated, and a lot of folks panicked when they saw it. The tweet wasn’t even accurate, which caused a lot of the confusion. Dockershim was being deprecated, not Docker. Ian and Kat both tried to help with damage control because they are kind people who care. Ian did some tweeting but had to get back to work. Then Kat wrote a Twitter thread and a blog post.
Eventually, it was all straightened out, but there were a few days with a lot of unnecessary pain for many people.
I spent a short stint as a product manager before I got into DevRel, and I feel like the issues mentioned in this talk overlapped a lot with things that product management and community teams usually handle. Kubernetes is structured differently than a typical software shop, and different teams own those functions, sometimes more than one. The process has improved over time, like with the addition of a Contributor Comms team.
The big takeaways for me were that people making changes need to talk to both internal and external folks and can’t make assumptions about the community.
#Panel: Your Path To Non-code Contribution In The Kubernetes Community
Kaslin Fields, Kat Cosgrove, Matt Broberg, Kohei Ota
This was a great panel discussion with some folks I’m a huge fan of. I didn’t know Kohei, but he lives in Japan and works on Japanese localization for the Kubernetes docs, which is super cool. Localizing docs can be very helpful for folks that don’t speak English natively.
The panel discussed a lot of interesting topics. Dockershim came up again. Later, Kaslin explained that she contributes by using her art to explain technical concepts visually, which is fantastic. I loved her thoughts on learning too.
"Contributing to open source is learning in public." @kaslinfields is super right. We all make mistakes and learn. #KubeCon
— Rich Burroughs (@richburroughs) May 5, 2021
Kohei mentioned that fixing typos in docs is a great way to start, and I agree. I’ve made that type of contribution and I think it’s very helpful, especially if there are mistakes in things like command-line examples. You want users to be able to cut and paste commands and code from your docs and have them work. Otherwise, it’s an opportunity for them to bail on your project.
And Matt discussed the value of beginners to a community.
"Being a beginner is an expertise." @mbbroberg - so true. Using a project for the first time and helping to clarify the docs is super helpful. People writing docs make assumptions. #KubeCon
— Rich Burroughs (@richburroughs) May 5, 2021
I enjoyed this panel a lot.
#Writing for Developers: Take your Project Docs to the Next Level
Continuing on the theme of communication, next I saw Celeste’s talk on writing docs. I’ve written plenty of docs in my career, but hearing from someone specializing in it is always extremely helpful. Celeste used the storytelling concept of the Hero’s Journey as a metaphor, which was great. I think storytelling is a part of many things we do in tech, from doing demos and writing docs to job interviews.
Users all have goals, and there are different types of docs to address them.
Your user always has a goal, and your first job is to understand what it is. You can learn about this by talking to users. You can keep track of questions/bugs that come up repeatedly. You can also outline the docs during the epic. #KubeCon
— Rich Burroughs (@richburroughs) May 5, 2021
Celeste mentioned three steps to explaining a thing: explaining the concept at a high level, illustrating tasks that it’s used for and pitfalls that users may run into, and reference docs. She also commented on the value of new folks.
New contributors can open PRs on docs, make suggestions to the project team (like writing blog posts), and cherishing your beginner's mindset. 💯 #KubeCon
— Rich Burroughs (@richburroughs) May 5, 2021
If you write docs in your role or are interested in it, I highly recommend this talk.
#Day Two - Thursday
Thursday was my second straight day up for talks on Europe time, and it was starting to catch up with me. Thankfully there were some great talks to keep my attention.
#Hacking into Kubernetes Security for Beginners
This talk was amazing. If you could only watch one presentation from KubeCon EU, I’d recommend this one. It was one of the most unique talks I’ve seen at a conference. It was even a talk but a short film with a story and great characters that illustrated security concepts in Kubernetes.
I could write a lot about this talk, but I don’t want to spoil it. You should watch it and hopefully laugh as hard as I did.
LOL at "DevSecOps Enforcement." Trying not to squeal with laughter at 3 AM and wake my neighbors.
— Rich Burroughs (@richburroughs) May 6, 2021
Props to Ellen and Tabby for raising the bar on conference storytelling. I have heard rumors that there may be a sequel.
#Uncovering a Sophisticated Kubernetes Attack in Real-Time
Jed Salazar & Natália Réka Ivánkó
I’ve been interested in security a lot more the last couple of years after seeing Ian Coldwater’s fantastic talk at KubeCon San Diego. So I stuck with the theme and watched this talk about eBPF. If you’re not aware, eBPF is the new hotness, and if you make a KubeCon bingo card, it should be on there at this point. But eBPF is hot because it’s a great technology with a lot of use cases. Visualizing an attack in real-time is a great one.
The talk started with Jed giving an overview of Kubernetes security.
"CNCF Threat Landscape" is good 😂 #KubeCon
— Rich Burroughs (@richburroughs) May 6, 2021
Lots of really good advice here re things like seccomp, AppArmor, image scanning, etc. #KubeCon
— Rich Burroughs (@richburroughs) May 6, 2021
The last part of the talk was the demo that the talk’s title had promised. Natália and Jed used Cilium, an open source tool created by the folks at Isovalent. Cilium emits JSON events that you can ship to your observability tools.
The demo was very entertaining. I’d love to see more content that digs deeper into how eBPF works in this use case.
#Notary v2: Supply Chain Security for Containers
There’s not much of a hotter topic in the software world than supply chain security. Since the Solar Winds compromise, many people have been thinking about securing their build and deploy pipelines to prevent people from injecting malicious code. I spoke with Brad Geesaman about this in the most recent episode of my podcast Kube Cuddle, and we both expected it to be a popular topic at the conference.
I hadn’t heard of Notary, but apparently it’s been around awhile as this talk was about the changes in v2. Since I saw the session, I’ve noticed that it’s one of the tools installed in Docker Desktop.
Justin and Steve gave an excellent overview of the current state of container image signing.
People want to add things after the fact, like signatures to show approval or other metadata. And it's not just containers going in to registries. #KubeCon
— Rich Burroughs (@richburroughs) May 6, 2021
Very few people are using container image signing right now. They want to get standards out there, and make flexible tools that people can build on top of. The libraries can be used in your own tooling. #KubeCon
— Rich Burroughs (@richburroughs) May 6, 2021
The idea of having a signed manifest of what’s in the image is clever, and Docker with the Notary plugin can check signatures when you do a pull. As with anything that involves crypto and signatures, making the process easy for users is a massive part of accelerating adoption. It sounds like the team is very focused on that.
Steve gave a great demo of what they’re working on for v2. If this topic interests you, this talk is worth catching up on when the videos hit YouTube.
That was the last talk of the day for me. After a couple of days of being up at 3 or 4 AM, I was getting fried. I decided to rest up for the last day of the conference.
I’m curled up on the couch watching Gamorrah on HBO Max on my phone. I took a nap for like an hour but I don’t want to sleep too long. Getting up again at like 3 in the morning for the last day of #KubeCon.
— Rich Burroughs (@richburroughs) May 6, 2021
#Day Three - Friday
I felt pretty wiped out Friday morning and decided to skip the 4 AM talks and rest a bit more.
Good morning #KubeCon. I can't remember which button does which on twitter dot com, if you want to know where my brain is at.
— Rich Burroughs (@richburroughs) May 7, 2021
I think it’s essential when attending a conference to feel like you have the space to skip some talks when you need to. Virtual conferences seem even more draining than physical ones somehow, and it’s important to take care of yourself.
#Traces from Events: A New Way to Visualise Kubernetes Activities
I’ve been interested in Kubernetes events since I saw a great talk at KubeCon San Diego from my friend Jesse Dearing. It’s not a topic that I hear discussed a lot, but there’s a lot of great information you can get from events in your clusters.
You can see events with "kubectl get events." (I often forget that this exists.) The individual event objects have more info in them, like the source of the event, the namespace, and the number of times it's occurred. #KubeCon
— Rich Burroughs (@richburroughs) May 7, 2021
Bryan’s presentation was about a tool he built called Kspan, which turns Kubernetes events into distributed traces. It’s super cool. Bryan’s tool uses Jaeger to visualize traces, but you could potentially feed the traces to other OpenTelemetry clients.
This talk was very visual with a great demo. Bryan ran a tool named podinfo in some containers to generate events and then visualized them with Kspan. Just starting up two pods created 14 Kubernetes events. It was fantastic to see them laid out as traces.
Here's an example of the traces from the Kspan README on GitHub. https://t.co/bzkGuo3qZp #KubeCon pic.twitter.com/n2xsSbTtmJ
— Rich Burroughs (@richburroughs) May 7, 2021
Finding some way to visualize events could be super helpful for operating clusters and troubleshooting. Kudos to Bryan for this experiment in how to do that.
#Understanding Isolation Levels in the Kubernetes Landscape
People talk a lot about how difficult Kubernetes multi-tenancy is. People operating Kubernetes clusters have to choose between providing clusters to individual developers or teams and having lots of cluster sprawl or having bigger, shared clusters that are less isolated. I care about this topic so much that I went to work at a company that makes tooling to address the problem.
Jiaqi gave a great introduction to what the issues are.
Jiaqi is recommending single tenant clusters if you just have a few, but multi-tenant clusters if you have more. There has to be some level of trust for multi-tenancy. #KubeCon
— Rich Burroughs (@richburroughs) May 7, 2021
The tradeoff is between cluster management complexity and isolation. #KubeCon
— Rich Burroughs (@richburroughs) May 7, 2021
She also covered things like namespaces and RBAC and what their limitations are. In the latter part of the talk, Jiaqi covered a use case at the University of Chicago. Developers needed to launch Jupityr notebooks, but the cluster admins didn’t want all users to have access to the Kubernetes APIs. They built a tool called Hatchery to address that use case.
If you’re not familiar with Kubernetes multi-tenancy, this talk would be a great starting point for you.
#Making Dynamic Admission Control Even More Dynamic Using WebAssembly
Flavio Castelli & Rafael Fernández López
I couldn’t catch this presentation live as it conflicted with Jiaqi’s talk, but it’s another topic I’m very interested in so I watched the video right away. The tool of choice for enforcing policies and admission control in Kubernetes has been Open Policy Agent. A lot of folks find OPA complex and hard to use, though. It uses its own language called Rego that is not intuitive for everyone.
Flavio and Rafael are on the Rancher/SUSE team that has built a new tool for enforcing policies called Kubewarden. Kubewarden uses WebAssembly, and it’s a very clever take on administering policies. Using WebAssembly means that people can use their preferred languages to write policies and compile them down to Wasm.
As Flavio mentioned in the talk, there are a lot of advantages to defining policies as code. We’ve seen that for a while in the Infrastructure as Code space. Kubewarden was built to serve two personas: cluster administrators and app developers.
This topic interests me because the company I work for, Loft Labs, just released a different open source tool for managing policies called jsPolicy. The approach taken by Kubewarden and jsPolicy and the goals of the projects are very similar. We both worked on these tools independently, but I think it shows there was a need for more options in this space. jsPolicy uses JavaScript and Google’s super-optimized v8 engine.
Kubewarden seems very interesting, but I haven’t had a chance to take it for a spin. I would have loved to see a demo in this talk, but the recording deadline was a while before the conference, and Kubewarden may not have been ready yet.
I have a ton of respect for the Rancher team, and we use k3s in our virtual cluster project vcluster. I’m also super happy to see more alternatives to OPA cropping up. Giving the community options is great.
The bottom line is that you should be using something for admission control. If you have OPA and Rego down and are happy, use that. Or use one of the other options like Kubewarden or jsPolicy. I used to work at Puppet, and I would always laugh when Nathen Harvey from Chef would talk about competition in the configuration management space. Nathen said that folks would ask him if they should use Chef or Puppet or Ansible, and his answer was “Yes.” I feel the same way about managing policies.
But I do think jsPolicy is super dope. If you try jsPolicy, I would love to hear your feedback. You can find me as Rich Burroughs in the CNCF and Kubernetes Slacks or on Twitter.
#Cloudnative.tv
After the final talks of day three, I hung out in the Cloudnative.tv Twitch stream, which was a fun addition to this year’s KubeCon. The hosts rotated daily. Friday’s stream was hosted by POP and Rawkode, who are both very smart and funny. Ellen and Tabby joined to talk about their fantastic short film, and it was a lot of fun. I also caught a bit of the day one stream hosted by Kat Cosgrove and Matt Stratton, who were also very entertaining.
I thought this idea of giving folks a place to have conversations at the end of the day was excellent, and I hope it continues.
#Conclusion
So that was a wrap on the third virtual KubeCon, and possibly the last one. Congrats to the co-chairs, the rest of the organizers, and all of the speakers.
Right now the plan is to have KubeCon North America 2021 in Los Angeles from October 12-15. We’ll have to see if that holds up with Covid still happening, but it might work out with more folks getting vaccinated before then. I’ve heard that the CNCF will be deciding for certain 90 days out, but I didn’t hear that through official channels, and I may be wrong.
I’m not sure how the pandemic will impact folks from outside the US who would like to attend. I think it’s the right call to make these events more focused on the local audiences, though, in the same way that this KubeCon EU was held more on European time this year. One prediction I have about events after Covid is that they’ll be more focused locally. I think the days of big international tech conferences are mainly behind us.
I’m looking forward to a time when I can hang out in person with my friends from the Kubernetes community, whatever happens. There are so many amazing people in the community, and I owe many of them hugs and thanks. No handshakes, though. Fist bumps for everyone from now on.
