KubeCon + CloudNativeCon North America 2022 was held October 26-28 in Detroit, Michigan. It was the second in-person KubeCon NA since the Covid pandemic started, although there was also an option for viewing the talks virtually. Videos of the talks should be posted within a few weeks on the CNCF YouTube channel if you’d like to watch any of the talks I mention below.

This was a very fun KubeCon. I’d not been to Detroit before, and it seemed like a great city. This was also my first KubeCon as a speaker (I moderated a panel on ADHD, more about that later), which was a great experience.

I did hear comments from several sponsors that they didn’t see as much foot traffic as expected. The estimate I’d heard ahead of time was that there were about 7,000 people registered, but it felt like less attended. I haven’t seen an official attendance number yet.

I also participated in some of the pre-conference events for the first time. I enjoyed it, but it made for a very long week. I was battling insomnia for a lot of the week, so making it to things late and participating less than I’d hoped will be a running theme here, but I still had a great time. I always try to prioritize self-care at conferences, especially at one as big and long as KubeCon.

#Pre-conference

Before KubeCon proper, I spoke at Cloud Native Rejekts and attended one of the KubeCon day zero events.

Rejekts was fun, although my time there was limited. I presented a talk titled Virtual Kubernetes Clusters: Tips and Tricks. I had proposed this idea for KubeCon and it had been rejected, so I was glad to find a place to give it. The crowd at Rejekts was small but friendly, and there were some great questions after the talk.

On Monday I attended Cloud Native eBPF Day, where I saw a couple of talks about eBPF. I also had a chance to talk to my friends at Isovalent. I’m a big fan of what they’re doing; Cilium is a tool that helps many people.

I took Tuesday off. I had initially planned to do something fun in Detroit, like visiting the Motown Museum, but I was so tired that I barely made it out of the hotel. It was definitely not related to this tweet:

If you’re staying at the Fort Pontchartrain, note that it’s easy to accidentally press the button on the bathroom door and lock it behind you. I did not do this last night, however. I read about it on Reddit or something. #KubeCon

— Rich Burroughs (@richburroughs) October 25, 2022

Shoutout to Buddy’s Pizza and Lafayette Coney Island for keeping me full of pizza and coney dogs those first few days. I’d tried Detroit-style pizza before in Portland, where I live, and hadn’t liked it. But Buddy’s is the truth.

#Day One - Wednesday

#Keynotes

I was late to the keynotes but managed to make it in time for the emotional moment when Ihor Dvoretskyi stepped on the stage. If you’re not familiar with Ihor, he’s a very well-loved member of the Kubernetes Community who took a step back from open source to fight for his country, Ukraine, when Russia invaded it. I don’t know Ihor but I have a lot of friends who do, and seeing him in person after so long must have been amazing for them. I was pretty choked up myself.

Ihor Dvoretskyi took to the stage at #KubeCon + #CloudNativeCon North America with thunderous applause this morning. It has been incredible watching the open source community rally behind him and his sacrifice.#Ukraine #Kubernetes @razomforukraine pic.twitter.com/ejhE5MUfIl

— The Linux Foundation (@linuxfoundation) October 26, 2022

There was also some great news that the group Razom For Ukraine and the Linux Foundation are partnering to translate LF training courses into Ukrainian. It’s great to know that this training will be more accessible to folks in Ukraine. I met some of the Razom team at KubeCon Valencia and their work is very inspirational.

The CNCF has been helping to raise money to support Ihor’s fight through Operation Dvoretskyi. If you’d like to donate, your employer may match your contribution.

#A Raccoon And a Group Of Turtles Secure Clusters Together! - Pushkar Joglekar & Naadir Jeewa

This was a very interesting talk about the efforts Naadir and Pushkar have made to improve the security of the Cluster API project. The title refers to the animal mascots for some of the projects involved. If you’re unfamiliar with Cluster API, it’s a project for provisioning Kubernetes clusters and managing their lifecycles. A project with that level of access to clusters is bound to be very interesting to attackers, so securing it is very important.

In the talk, Naadir and Pushkar dealt with some of the practical aspects of getting a project like this running. They eventually settled on a mix of Slack chat and meetings, as the team was distributed across several time zones. They also mentioned the importance of finding sponsors. VMware donated some hardware.

The security investigation started with a self-assessment from the OpenSSF, and Naadir did a data flow diagram. Eventually, they got funding from the CNCF to do a security audit, which identified some issues. Some of those issues are still open, and they need help. If you’re a security person involved with Kubernetes and would like to help, watch the talk and reach out to them.

#The 10 Biggest Mistakes You Shouldn’t Make In Open Source - Bill Mulligan & Divya Mohan

Next up I watched a talk by Bill and Divya with some tips about participating in open source communities. I know Bill a bit from Twitter, and Divya has been a guest on my podcast Kube Cuddle. Their advice was great.

One thing they focused on that I think is very important is the importance of non code contributions.

Props to @breakawaybilly for mentioning that he’s not a coder and there are other ways to contribute. #KubeCon

— Rich Burroughs (@richburroughs) October 26, 2022

I don’t code a lot myself, and a lot of my PRs to open source projects have been things like fixes to READMEs. If you’re just getting started with open source, that’s a great way to get your feet wet.

They focused on other ways you can help the community too, outside of commits to projects.

They also have written blog posts, given talks, and been on podcasts. #KubeCon

— Rich Burroughs (@richburroughs) October 26, 2022

Writing or making other content that helps people understand tools is a really great way to share your knowledge.

If you’re new to open source, I think this talk would be a great introduction to getting involved. So check it out when the video is available

#Evening shenanigans

There were some great looking talks in the last slot of the day. I especially was looking forward to Tabby Sable and Paris Pittman’s talk “No One Is Saving Us But Us,” but I was exhausted and ended up heading back to my hotel room for a power nap.

That evening our company Loft Labs sponsored a mixer event called Staging Things with the team at Civo Cloud. It was fun to see the folks that embraced the 80s theme, and I heard reports that it was the best party of the night. I had some really good conversations with some people, but headed back to get some sleep after a couple of hours.

#Day Two - Thursday

Thursday was a really big day for me and I’d been looking forward to it for quite some time.

Big day today :) Our project @vcluster is getting mentioned in one of the keynotes, I’m moderating a panel about ADHD at 11. And I’m giving a lightning talk about vcluster at the RedHat booth as part of a multi-tenancy meetup that starts at 1:30. #KubeCon

— Rich Burroughs (@richburroughs) October 27, 2022

#Keynotes

I had to leave the keynotes early to get ready for my panel, but I had a chance to catch the one I was super excited about, “What a RUSH! Let’s Deploy Straight to Production!” from  Whitney Lee and Mauricio Salatino.

I work a lot with an open source tool called vcluster that we maintain at Loft Labs, and it was used in the demos for the talk. Mauricio had filled me in a bit ahead of time on what would be happening, and it was fun to see it unfold.

This talk was also a really great introduction to the idea of platform teams, and how they should interact with developers. Over the last five years I’ve seen more and more teams treat their platforms as products, like having dedicated product managers and doing user research. Whitney and Mauricio put forth that same view, that the job of platform engineers is to reduce complexity for application developers who use their platforms, and to provide them easy self-service. Listening and collaboration is key. This all fits in completely with my views.

The demos were great too.

Yay demo time. They created an environment with some simple YAML that spins up a @vcluster with Crossplane and Helm, with Knative installed. It was so fast! #KubeCon

— Rich Burroughs (@richburroughs) October 27, 2022

I honestly got pretty emotional watching this talk. I joined our company shortly before vcluster was open sourced, and I’ve watched the project grow so much over the last 18 months. To see it on the KubeCon keynote stage was such a great feeling.

P.S. I interviewed Mauricio ahead of KubeCon about vcluster and the challenges facing platform teams. He had some great thoughts. And the code for the keynote demo is here.

#ADHD: Understanding, Awareness, And Shared Experience - Bart Farrell, Heba elAyoty, Farrah Campbell, Rich Burroughs

I went right from that amazing experience to the panel I moderated. It was my first time speaking at KubeCon, and I really want to thank Bart, Heba, and Farrah for including me. I also especially want to thank Walid Shaari, who had the idea with the panel and got us all connected but then was unable to travel to KubeCon to share the stage with us. If you know Walid you probably know what a kind and generous person he is, and we really missed his presence.

I’ve been talking about my experiences as an adult who was diagnosed with ADHD a lot on my Twitter, and it was great to do that at a CNCF event. It was also wonderful to hear the experiences of the other panelists, which were very similar to mine in some cases, but also sometimes different. ADHD doesn’t present exactly the same in any two people, and it’s important to remember that someone else may have different problems and solutions.

We talked about our backgrounds with ADHD, getting diagnosed, some strategies we have for managing it, and what it’s like to be someone with ADHD in the CNCF and Kubernetes community. I heard from several people who found it very valuable to hear our experiences, and to see someone talking about this subject publicly. The room was pretty packed too.

If you have ADHD or know someone who does, you might find this interesting to watch once the videos are posted. Thanks if you made it there in person, too.

#Platforms Working Group Meetup

After lunch I met up with some people from the newish Platforms Working Group. We were all people from teams that make tools for folks building platforms, and we took turns giving lightning talks about how our tools work. The idea was that we could find ways that the tools can work together, and also places where they overlap. I spoke about our open source tool vcluster.

Thank you Josh Gavant for organizing the gathering. Josh wrote up some thoughts on the Platforms WG and the meetup, if you are interested. I had a lot of fun and also discovered ways to collaborate with some other projects. I hope to participate in the WG a lot more.

#How the Basics Of Kubernetes Auth Scale For Organizations - Leigh Capili

After lunch I had a chance to catch one talk. I’m a big fan of Leigh’s, so I wanted to make it to this one. When I’m talking with people and Leigh comes up, I always mention how great he is at doing live demos. I’ve also seen him bust out his rapping skills a couple of times, and he started out this talk with a rap about RBAC.

This was an introductory talk to authentication and authorization in Kubernetes. Authentication establishes who you are, and authorization controls what you have access to. Leigh focused mainly on the authorization part, and he gave a really great overview of the primitives in RBAC. But he also threw in some more advanced RBAC tips that I wasn’t aware of.

If you have 100 namespaces and want to give someone access to 5O what do you do? You can point a RoleBinding at a ClusterRole instead of giving them access to everything. (I didn’t know this.) #KubeCon

— Rich Burroughs (@richburroughs) October 27, 2022

If you’d like a good intro to RBAC or a few more advanced tips, check out this talk later when the videos go live.

#Evening event

The evening events for this KubeCon were split up between three locations: the conference venue, a nearby boat, and the 25th floor of the Fort Pontchartrain hotel. I was tempted to try the boat, since there was gambling and the opportunity to be outdoors (I’m still working pretty actively to reduce my risks of catching Covid), but I was staying at the Fort Pontchartrain and ended up choosing the easiest option.

Things were a little crowded when I arrived, but there was a great live band and I had a lot of friends there. I was running pretty short on energy though, and just stayed for a bit.

#Day Three - Friday

Friday was my 8th day on the trip, and I was very ready to get home. I managed to sleep better though, and made it to the venue in time for the keynotes.

#Keynotes

The keynote session I looked forward to most was “Cloud Native 101: Motor City Edition” from Jeffrey Sica and Bob Killen, who you may know better as jeefy and MrBobbyTables. This was a fun talk where they used some extremely Michigan things to explain cloud native concepts, including pizza, car manufacturing, and Faygo. They were right, Faygo is everywhere in Detroit.

Stephen Augustus gave a quick update on Cisco’s open source project called OpenClarity, Alvin Estrada talked about doing DevOps in Central America, and Emily Fox did a talk called “How to Become a Cloud Native Mechanic.” Emily mentioned that only five percent of people who use open source projects contribute to them and that this is a danger because maintainers can get burned out.

Last up was one of my favorite parts of KubeCon, the community awards. So many people work hard on Kubernetes and other CNCF projects, and it’s great to see some of them recognized. Chris Aniszczyk from the CNCF presented the awards.

The Top Documentarian award was a tie between Rey Lejano and Catherine Paganini. Congrats to both of them. Carolyn Van Slyck was awarded Top Committer/Maintainer. Carolyn is great, and she’s also appeared on my podcast Kube Cuddle, where she talked about her work on Porter and what it’s like to be a maintainer.

And the Chop Wood and Carry Water award winners were:

Adolfo Garcia Veytia, Xing Yang, Catherine Paganini, Alex Chircop and Patrick Ohly receive Chop Wood Carry Water Award, 2022!#KubeCon #CloudNativeCon pic.twitter.com/Vegh20BO5l

— CNCF (@CloudNativeFdn) October 28, 2022

I was so happy to see my friend Adolfo (who you may know better as @puerco) recognized. He’s one of the folks who implemented signing Kubernetes artifacts with sigstore, which is super important work. I ran into Adolfo and Carlos Panato at dinner that evening, and it was lovely to see them.

I got a little emotional talking to @puerco and @comedordexis tonight about how much I appreciate their work on Kubernetes. I know release management is hard and can be pretty thankless work. It’s so important though.

— Rich Burroughs (@richburroughs) October 29, 2022

After the keynotes I managed to make it to three breakout sessions.

#How Adobe Planned For Scale With Argo CD, Cluster API, And VCluster - Joseph Sandoval and Dan Garfield

This was yet another vcluster talk (I think four talks touched on it somehow), and I was looking forward to it. Joseph works at Adobe, and I’ve been able to talk to him and Mike Tougeron both about their work with vcluster. Dan is a co-founder at Codefresh and an Argo CD maintainer, and in the talk he and Joseph explained their process for designing a new CI/CD platform for Adobe that uses both vcluster and Argo CD.

They had been using Mesosphere but they had scaling problems when they moved to k8s. Some of this was people too, like getting PRs approved. They started looking at the Argo community and thinking about decoupling releases. #KubeCon

— Rich Burroughs (@richburroughs) October 28, 2022

They chose vcluster because virtual Kubernetes clusters are fast and great for testing.

They are able to test operators against multiple versions of k8s (this is really cool). Running a vcluster is incredibly cheap compared to spinning up real clusters in your pipeline. #KubeCon

— Rich Burroughs (@richburroughs) October 28, 2022

The platform is still a work in progress and the Adobe team is still looking for ways to optimize it, but it was interesting to see why they made certain choices and how they validated them.

#Bare-Metal Chronicles: Intertwinement Of Tinkerbell, Cluster API And GitOps - Katie Gamanji

I have zero experience with bare metal Kubernetes, and I attended this talk of Katie’s to catch up on what folks are doing. I’ve known about the Tinkerbell project for a w,hile but I didn’t know how it interacts with Cluster API.

Katie started with excellent overviews of Cluster API and Tinkerbell. Tinkerbell became a CNCF sandbox project in 2020.

These building blocks seem to work great together:

You can point a Tinkerbell server at a Cluster API management cluster to provision clusters. We can also use GitOps and templates to define the clusters that are provisioned. GitOps gives us a versioned history of our clusters’ states. #KubeCon

— Rich Burroughs (@richburroughs) October 28, 2022

Katie did a great live demo. She’s a pro and had a recorded version just in case. She’s a fantastic speaker and explains technical concepts so well.

If you’d like to see how to use this stack of tools to provision bare metal clusters.

#Turn Me On With Cloud-Native Feature Flags! - Alex Jones, Canonical

I was excited to see this talk and it lived up to my hype. I think feature flags are such an important tool for deployment safety, and they can be used in many other ways. One way I’ve benefited from them as a DevRel person has been getting early looks at new features in production. James Governor coined the term “progressive delivery” for the process of deploying with feature flags and things like canaries, and I think it’s such a smart way to roll out software.

And there’s some big news involving feature flags, at least it was news to me. There’s a new project called OpenFeature to standardize how we use feature flags in the same way that OpenTelemetry standardized how we consume metrics. From what I’ve seen OpenTelemetry has been a huge success and great for users, so I’m excited to see where OpenFeature goes. A lot of popular vendors are already participating in OpenFeature, like LaunchDarkly and Harness.

Alex also did a demo of a project called flagd, an open source daemon for feature flagging that uses the OpenFeature spec. Users run flagd in sidecars along with their apps, and it’s been designed to be very simple to use.

As a person who deployed apps for many years, I’m glad to see all of the advances in these areas. I think deployment safety is critical for teams that are pushing out code, and it helps with psychological safety. Deploying new code and then testing with a small set of clients using feature flags is a pattern you should be looking at if you deploy apps and you’re not already using it.

This is the talk I learned the most in I think, as it was a topic I’m familiar with but dealt with advances I wasn’t aware of. Thanks for sharing your knowledge on this Alex.

#Conclusion

If you made it this far, thanks so much for reading. These posts are always very long and it’s great that people appreciate them.

Overall I had a great time at KubeCon Detroit. Thanks so much to the CNCF, the conference chairs, all of the other organizers, the program committee, the speakers, and all of the staff who kept things like the food and coffee coming. Organizing an event like this is a huge amount of work.

It’s been very nice to get out and meet people again in person. I’m very risk averse when it comes to Covid, but I’ve felt pretty safe at the last few KubeCons. I got the chance to see a lot of friends again in Detroit, but also to make some new ones. This community has so many amazing people in it, and I’m happy to be a part of it.

KubeCon North America 2023 has been announced and it will be happening in Chicago. I’m excited about that one. I haven’t been to Chicago in a long time. I’m also really looking forward to the next Kubecon in Amsterdam. I hope to see a lot of you at both of these events.