Table of Contents
KubeCon + CloudNativeCon North America 2021 was held October 13-15 in Los Angeles, California. It was a hybrid event, meaning people attended both in person and virtually. This was the first KubeCon that had an in-person component since the Covid-19 pandemic began.
I was honestly a mix of terrified and excited to be attending a conference in person. I’d been mainly sheltering in place again since the Delta variant cases got worse in my area, and it was my first time doing air travel since Covid. If my employer hadn’t been sponsoring the event, I think I would have likely attended virtually. But I also knew that I would see a lot of friends in the community there, and I was excited about talking to folks and getting some hugs, too, especially after being alone in my apartment a lot.
The conference organizers and the CNCF did about as good of a job with safety as they could. People were required to show proof of vaccination and have their temperature taken to enter. Masks were required and every other seat was blocked off in the talks to encourage distancing. There were colored bracelets so attendees could indicate how much distance they wanted to keep, and the dining area had some outdoor seating, which was nice.
Overall I felt very safe at the conference, and it was great to see so many amazing people again.
I’ll be sharing thoughts about the talks I attended. I could only watch a small portion of them, but the ones I’m mentioning are well worth watching if you missed them live. The recordings should be showing up on the CNCF YouTube in the next few weeks.
Day One - Wednesday
Keynotes
{{< single-tweet 1448318059454758914 >}}
I grabbed a seat up front to watch the keynotes, and it felt great to be back in this community. Priyanka Sharma started off talking about how fantastic it was to be meeting in person again. She shared some statistics about the growth in the CNCF, which went from 44 projects to 144 since 2019, and 137,000 contributors.
There’s a new certification called the Kubernetes and Cloud Native Associate that’s meant to teach people cloud native basics. I think it’s a fantastic idea. Many folks work around the cloud native space aren’t developers or SREs, and it’s great to have an on-ramp for them to get some hands-on experience with cloud native.
And Priyanka shared that the Kubecon Scholarship Fund has been renamed the Dan Kohn Scholarship fund. I didn’t know Dan, but it’s pretty clear that he had a massive impact on the community and many people in it. We’re closing in on the one year anniversary of Dan’s death, and it felt very fitting to honor him again.
Next up was Tim Pepper, and his section of the keynote was one of my favorite parts of the entire conference. The event was held the week of Indigenous Peoples’ Day, and Tim shared a bit about what it’s like to be an indigenous person in tech.
{{< single-tweet 1448325426078642179 >}}
Tim was joined by some folks from one of the local indigenous tribes, and they performed a song. It was very moving for me. I spoke with Tim a bit later, and he mentioned that when he proposed this to the CNCF, he was unsure how people would react, but it seemed very well received. I was also happy to see the CNCF acknowledging a group that’s very underrepresented in tech.
Some other highlights of the keynotes for me were Katie Gamanji talking about the CNCF End User Community and the Technology Radar, and Cornelia Davis’s explanation of the technical oversight in the CNCF was very helpful. If you are confused about what TAGs and SIGs are, watch her keynote.
We also heard from conference co-chairs Constance Caramanolis, Jasmine James, and Stephen Augustus. Jasmine was a new addition to the group of co-chairs for this KubeCon, and she will be continuing for future events.
I was able to see four of the breakout sessions on this first day, and they were all very well done.
Customizing Kustomize with Client-Side Custom Resources - Katrina Verey & Jeff Regan
I spent several years working as a practitioner with configuration management, and those problems are still there. Kustomize is one of the more popular tools for managing configurations in Kubernetes, and from seeing the talk, it looks very powerful.
{{< single-tweet 1448350126217433090 >}}
The talk looked great, but some of it was a bit over my head, which was totally my fault. It was listed as an intermediate level talk and I went in knowing almost nothing about Kustomize. That said, even as a newbie, I saw the power of the templating and transformations you can do. If you’re getting into Kustomize and want to level up, I bet this talk would help you.
sigstore: How We Started, Where We Are, Where We are Headed - Bob Callaway & Dan Lorenc
Have you heard about supply chain security? I bet you have. It’s been a huge topic since the Solarwinds breach, and with good reason. If an attacker could inject malicious code into a popular open source project undetected, they could potentially gain access to hosts across the internet.
sigstore is a Linux Foundation project that helps with cryptographically signing software. It takes the approach of simplifying the process of signing your code and is meant to help open source maintainers do the tasks involved easily. sigstore was inspired by Let’s Encrypt, which changed the way we use SSL certificates.
{{< single-tweet 1448363140903620610 >}}
The heart of this talk was several demos that dove in and showed the lower-level operations that sigstore abstracts away from users. I think that approach was super smart. By showing how painful the manual process is, the presenters made you appreciate their set of tools that simplify it. I highly recommend watching this talk if this is an area of interest or concern for you.
Kubernetes Exposed! Seven of Nine Hidden Secrets That Will Give You Pause - Ian Coldwater & Brad Geesaman
I’m not going to try to pretend to be objective here. Ian and Brad are two of my favorite people in the Kubernetes community, and there was no way I would miss this talk. It didn’t disappoint.
Ian and Brad discussed some things I knew about already, like Kubernetes overriding Docker’s seccomp settings. But there were some other very cool things discussed, like DNS tunneling and setting up an exfiltrating webhook. I highly recommend this one if you manage Kubernetes clusters.
Exploiting a Slightly Peculiar Volume Configuration with SIG-Honk - Ian Coldwater, Brad Geesaman, Rory McCune, Duffie Cooley
I mentioned how much love I have for Ian and Brad, and Duffie is another of my favorite folks in the community. I haven’t met Rory but I’m guessing he’s pretty rad if he hangs with the rest of this group.
This was a very different talk than the previous one. It walked through the process of looking at a vulnerability in runc, figuring out how to exploit the race condition that triggered it, and then escalating the issue to the community and the cloud providers. If you are a security nerd, this should be right up your alley.
What I really loved about this talk was seeing how this group of folks works together and how they level each other up. There is some Voltron action going on here - they’re more powerful together than each individual.
That was the last talk I attended on day one. I spent some time in the sponsor area and ran into a bunch of friends. I even got a few hugs, which I really needed.
{{< single-tweet 1448378568312954880 >}}
Day Two - Thursday
Keynotes
The keynotes started with Stephen Augustus giving some Kubernetes project updates. Stephen mentioned the Code of Conduct Committee and its transparency report on CoC issues. Their work is an essential part of providing psychological safety for folks in the community. Also, there were shoutouts for a couple of my favorite Special Interest Groups. SIG-Security and SIG-Release. The security posture of Kubernetes is improving all the time, including some new features added in 1.22. And release engineering is an often underappreciated field, which enables all of the other engineers in the project to do their work better. Big thanks to these groups and the other SIGs.
I really enjoyed Constance Caramanolis’s presentation called “Where Do We Go From Here?” As she pointed out, many of us focus on Kubernetes, but it’s one tool among many in the CNCF. This is something I’m guilty of.
{{< single-tweet 1448690025810509830 >}}
Constance had a lot of great thoughts about how we can collaborate better in cloud native.
Jasmine James gave an informative talk on developer experience, which is so important. Many of our customers are internal, and helping them do their work more efficiently makes them more productive and happy. If you’re on a platform team managing clusters or providing other tooling for developers, she explained a lot of ways to gather metrics and feedback from them. I was live-tweeting the talk, and I had trouble keeping up with all of the great information.
Other highlights for me were talks by Lachlan Evenson and Robert Duffy. If you were at KubeCon San Diego, you might have seen a talk by Tim Hockin and Kal Henidak about the project to add IPv6 support to Kubernetes. It turns out that was really hard, and Lachlan gave an update on what’s happening. The initial implementation created tooling that was hard for users to use, so there’s been a lot of refactoring. The new code is in beta now.
Robert discussed how the platform team at Expedia standardized on tools for their pipeline and built allies internally. Both of these talks emphasized listening and collaboration, which I think are key values for working in software development.
Beyond Namespaces: Virtual Clusters are the Future of Multi-Tenancy - Lukas Gentele
Speaking of biases that I can’t hide, I work with Lukas at Loft Labs and I’m very excited about virtual Kubernetes clusters, so I was ready for this one. If you’re not familiar with the concept of virtual clusters, they provide a way to share Kubernetes clusters that’s more powerful than namespace isolation alone. Users are presented with what looks like a full-blown Kubernetes cluster, which means they can create and manage cluster-wide resources like CRDs.
Lukas explained the concept and dove into the open source implementation that we maintain called vcluster, which uses k3s under the hood.
{{< single-tweet 1448713374477467648 >}}
vcluster is also a certified Kubernetes distribution, which means it passes a set of conformance tests from the CNCF.
I think vcluster is super cool and very fun to play with. If you haven’t heard about it before, Lukas’s talk has technical details about the implementation and a demo. So, look for that recording on YouTube.
A Vulnerable Tale About Burnout - Julia Simon
I’m kind of cheating by including this talk in my list as I wasn’t able to attend it. But I heard it was amazing and I will be watching it when the recordings make it to YouTube. I appreciate people sharing their mental health experiences, as it helps normalize them for everyone in the community.
{{< single-tweet 1448727880582647817 >}}
Measuring the Health of Your CNCF Project: Going Beyond Stars and Forks - Dawn Foster
Dawn has been working with open source communities for many years. I saw her great work in the Puppet community firsthand and then had the good fortune of working with her there. She is at the top of the list of folks that I trust when it comes to this topic.
Dawn gave a lot of great ideas about metrics to measure community engagement with your open source project.
{{< single-tweet 1448764850549755908 >}}
Her examples used an open source tool from the CNCF called Devstats, which uses Grafana to visualize GitHub data.
If you want to go beyond a surface-level understanding of your project’s health, this talk is for you.
Cloud Native Superpowers with eBPF - Liz Rice
The last talk I attended on day two got me very fired up about eBPF. I’ve been hearing about eBPF for some time now, but I heard about it initially in the context of performance visualization, from Brendan Gregg, and I don’t think I grasped how powerful it is.
One of my favorite things was seeing the service map that Cilium builds based on network traffic.
{{< single-tweet 1448796679608487937 >}}
I would have legit killed for this sort of tool back when I was working in operations. I worked with maps like this at times that were built manually, but there was never a guarantee that they were still accurate. With eBPF, you can observe all of the network traffic on a system and every syscall. It’s super cool.
Thanks Liz for helping me realize the power of these tools. I’m looking forward to playing more with Cilium.
Thursday night was the all-attendee party at Xbox Square. First, I went with my team to the party sponsored by Teleport and Pulumi, and we had so much fun that we missed a lot of the official party. But we got to take a tour through the Grammy Museum, which was open for KubeCon folks. My favorite part was seeing all of the costumes and the displays for the Motown artists. It was a very nice touch for the conference organizers to include that activity.
Day Three - Friday
Keynotes
By day three I think a lot of folks were dragging a bit. I know I was.
{{< single-tweet 1449046054477787140 >}}
I was happy with my t-shirt selection, though.
{{< single-tweet 1449098436595253248 >}}
(I’m not sponsored by Vans but very open to it, please hmu.)
There was a lot of focus on the sustainability of the community during the Keynotes, which is a super important topic. Christoph Blecker and Paris Pittman discussed the importance of sponsorship and recognizing the work that’s gone into helping folks contribute to the community.
{{< single-tweet 1449051588421505034 >}}
Stephen Augustus gave a very real talk (minus speaker notes) about what it’s been like for him to wear all of the hats that he does. I’ve often wondered how he does it all, and it turns out that it’s been wearing on him. I’m not at all surprised.
{{< single-tweet 1449054097810722819 >}}
Stephen is being more thoughtful about how he participates in the community, and I hope other folks hear his message.
{{< single-tweet 1449054713077403650 >}}
One of the ways he’s doing that is by stepping down as a KubeCon co-chair, along with Constance Caramanolis. I think they’ve run at least four or five KubeCons at this point and I can understand them needing a break. If you’ve enjoyed recent KubeCons, I encourage you all to thank them for what Constance called their “reign of terror.” I’ve attended all of the Europe and North America KubeCons since San Diego, and I’ve gotten so much out of them. Thanks so much, Stephen and Constance.
There also were keynotes on package management and supply chain security. If you don’t know what an SBOM is, check out the recordings when they’re live. There’s a good explanation by Frederick Kautz and Allan Friedman.
Last up was the Community Awards. Anaïs Urlichs was named Top Ambassador for 2021. Nikhita Raghunath was named Top Contributor, and Tim Bannister was recognized as Top Documentarian.
The Chop Wood Carry Water award recognizes people who put in hard work behind the scenes that keeps the community moving forward. This year’s recipients were Emily Fox, Aeva Black, Tasha Drew, Carlos Panato, and Carolyn Van Slyck.
Congratulations to all of the winners. You can read more about the awards and the winners in this blog post from the CNCF.
Beyond Kubernetes Security - Ellen Körbes & Tabitha Sable
I’m not actually sure how to describe this presentation lol. I overheard Tabby talking to the session host beforehand and she described it as “performance art.”
{{< single-tweet 1449087871785193475 >}}
You may have seen Ellen and Tabby’s amazing video from KubeCon Europe, where Ellen played a developer trying to get around the company’s security restrictions. This was not a direct sequel, but one in spirit.
I don’t want to spoil too much, but there’s lots of time travel and enjoyable supporting characters. I think some of the jokes might not land for folks who aren’t involved in security, but I found it very entertaining. As crazy and funny as the story was, there was a real point behind it: showing some of the advances made in Kubernetes security, like the new features in the 1.22 release.
The performance was a lot of fun, and I think it’s especially a must-watch for security nerds.
Beyond printf and tcpdump: Debugging Kubernetes Networking with eBPF - Martynas Pumputis & Aditi Ghag
I was so excited about watching Liz Rice’s talk on eBPF that I wanted to learn more. Aditi presented this talk, but Martynas contributed to it.
In the talk, Aditi introduced an eBPF-based open source tool for network debugging called pwru.
{{< single-tweet 1449143369003794433 >}}
I was very impressed seeing it in action. Aditi did a demo that looked a lot like doing a tcpdump capture, but faster and easier. She also explained several real-world examples of how they used pwru to debug network problems.
As Liz mentioned in her talk, you don’t need to be able to write eBPF tools to benefit from them. Open source eBPF tools like Cilium and pwru can help many people trying to solve complex problems. Also, that mascot with the Go gopher is super cute.
What We Learned from Reading 100+ Kubernetes Post-Mortems - Noaa Barki & Shimon Tolts
This is another shoutout for a talk on my schedule that I didn’t end up attending. I was pretty fried by the end of the day Friday and ended up hanging around the sponsor area instead to chat with friends. I will be watching this later, though. Operating clusters is hard and we should learn from the pain that other folks have experienced.
Conclusion
I had a wonderful time at KubeCon LA. It was great to see so many friends and meet some of my co-workers for the first time. It was perfect timing for me as I’d spent the previous couple of months feeling pretty isolated and in a rut.
I’ve been participating in open source communities for many years, and I think the Kubernetes community is a very special one. There are so many amazing people who give a lot of themselves to help other folks level up. I left this KubeCon feeling very energized.
To everyone who helped make that experience possible for me, thank you.
{{< single-tweet 1449401508533993478 >}}
