Table of Contents
KubeCon + CloudNativeCon Europe 2022 was held in Valencia, Spain, from May 18-20. This was the first in-person KubeCon since the Covid-19 Pandemic began. It’s also the first KubeCon Europe that I attended.
I hadn’t traveled to Europe in many years, and I was a bit nervous about flying internationally with the pandemic still going on. My path was to fly from Portland to Seattle, then to Paris, and lastly on to Valencia. Overall, the trip went better than expected, except for a panicked rush through security at Charles de Gaulle airport with other conference attendees.
The conference was better attended than KubeCon Los Angeles, with over 7000 at the venue. While it felt pretty lonely to be in the sponsor booths in Los Angeles at times, there were a lot of great conversations to be had here in Valencia. It’s also a very charming city, and I’m glad I got to visit. The biggest surprise was that there aren’t ride-sharing apps there, and the taxi infrastructure had trouble at times dealing with thousands of people leaving a conference venue.
I arrived Tuesday at about noon after about 20 hours of traveling, and I was very ready to rest up for the first day of the conference.
#Day One, Wednesday
#KubeCon welcomes @razomforukraine, an organization which supports Ukraine. "Everything to stop the bleeding and save lives." While Razom prefers to perform cultural functions, but when emergency response is needed, they step up to help. pic.twitter.com/v0njUpPUAw— Kaslin Fields (@kaslinfields) May 18, 2022
Olena and her husband Yuriy were in the group of KubeCon folks in the security line at Charles de Gaulle with me, and I had bonded with them a bit as we sprinted through the airport. It was great to hear her talk about their work for Ukraine. I had a chance to chat with her later, after her keynote, and she mentioned their technological needs at Razom. It’s an interesting scaling situation, where the demand for their organization’s help skyrocketed with the Russian invasion, as well as their donations. They’re doing critical work, and I’m glad the organizers gave her time on the stage to let our community know about it.
I was also happy to see Taylor Dolezal on the stage. Taylor gave an update on his new role at the CNCF as Head of Ecosystem, focusing on the end user community. Taylor’s known for his sense of humor (he let the puns fly), but he also impressed me with his fashion sense.
Meme & pun master - and my esteemed colleague @onlydole - describes the #enduser opportunity and plugs today's CTO Summit on resiliency in multucloud. Orange you glad he spoke? I am! 🍊😉#KubeCon #CloudNativeCon pic.twitter.com/kv4CXNIAM9— Hilary Carter 🇨🇦🇮🇪🇬🇧🇺🇦 (@TweetFromHilary) May 18, 2022
I also enjoyed the keynote from Kate Mulhall and Emma Collins, both from Intel, who discussed power and the environment.
The computing that we all do has a big impact on the planet, and it’s good to be thoughtful about that.
I’ve felt pretty driven to attend a lot of talks at the last few KubeCons, but this time around, I spent much more time at our booth and just wandering around. I met many amazing people from the community for the first time and saw a lot of friends. I did make it to a couple of great sessions on day one, though.
If you missed any of the talks and would like to view them, they’re still available for a bit on the conference platform (if you registered). If not, they will be uploaded soon to the CNCF YouTube Channel.
#Autoscaling Kubernetes Deployments: A (Mostly) Practical Guide - Natalie Serrino
Natalie works on Pixie, and she gave an excellent overview of the different ways to do autoscaling in Kubernetes. Her talk contained some great demos and a lot of helpful information. She focused mainly on the Horizontal Pod Autoscaler (HPA), which she said is her favorite method for autoscaling.
She even showed off a Turning complete HPA, which she described as “totally deranged.”
I also appreciated the way Natalie handled herself in the Q&A after her talk.
I really like the way Natalie answers questions. She’s not dismissive, very direct, and thorough. #KubeCon— Rich Burroughs (@richburroughs) May 18, 2022
This one is very worth looking up.
#The CRDs that Broke the Camel’s Back - Alper Rifat Ulucinar
I enjoy stories about people who hit scaling issues that many of us would never experience, either because of their scale or specific use case. Alper’s talk focused on the hundreds of Custom Resource Definitions (CRDs) that Crossplane uses for configuring cloud provider resources. Because of this unusual use case, the team ran into both client-side and server-side problems.
Alper walked through a few of these issues, including this kubectl issue.
This is related to the client-go discovery client, which uses a Token Bucket Rate Limiter (tblr). In k8s 1.23 there was a configuration bug that limited the queries per second, it was much lower than intended. #KubeCon— Rich Burroughs (@richburroughs) May 18, 2022
At least a couple of these investigations resulted in upstream patches, so thanks to the Crossplane team for making Kubernetes better for all of us.
That was the last talk of the day for me. I spent some time hanging out with the cool kids at the Sigstore booth and then called it a night.
#Day Two, Thursday
I think this was the first time at a KubeCon where I missed all of the keynotes. It was a combination of moving a bit slow due to jet lag and wanting to spend more time in the sponsor area talking to folks. I plan to go back and watch some of the talks that I missed later on video, but in the meantime, I can recommend some excellent live tweets from Kaslin Fields, Daniel Bryant, and Jessica Anderson.
I did make it to a few breakout talks in the afternoon that were a lot of fun.
#Make the Secure Kubernetes Supply Chain Work for You - Adolfo García Veytia
You may know Adolfo better as @peurco on Twitter. He gave a hilarious talk about some of the weaknesses in a software supply chain and the SIG Release team’s work to better secure Kubernetes releases. Adolfo is a huge Star Wars fan, and there were many funny references. His sense of humor really came across.
LOL at @puerco’s fake TED talk beginning, I was howling 😂— Rich Burroughs (@richburroughs) May 19, 2022
I’m a huge fan of SIG Release, as I’ve said on many occasions. I worked alongside many release engineers in my operations roles, and I know it can be a pretty thankless task. Most people won’t notice when you’re doing release management well, but they will see when you’re not.
The team’s work to head off supply chain issues in Kubernetes has come a long way. They started by adding a Software Bill of Materials (SBOM), and they’re now using some of the Sigstore tools to sign artifacts. Adolfo explained in the Q&A that the signing uses short-lived ephemeral keys, but the public keys are recorded in Sigstore’s transparency log.
Adolfo and some of the other folks tackling this problem are on the Chainguard team, and I want to thank Chainguard for giving them the time to do this vital work. It benefits the entire Kubernetes community.
#Cilium: Welcome, Vision and Updates - Thomas Graf, Liz Rice, Laurent Bernaille, Purvi Desai
Earlier this year, I wrote about 7 Kubernetes companies I’d be watching in 2022, and Isovalent made that list. I am still watching very actively.
There’s a lot of excitement around Cilium right now. It was one of the big takeaways from the conference for me. This talk was in one of the big auditoriums in the Event Center, and it was packed.
One of the big announcements was that the team has open sourced Tetragon, the observability and runtime environment from their commercial product. Cilium can also act as a service mesh now using Envoy, which I’ll cover more later.
Thomas started with some background about Cilium and eBPF. Laurent talked about how they began as Cilium users at Datadog but have become contributors. Purvi described how Google used Cilium in the V2 of their GKE dataplane and the increased adoption they’ve also seen from customers. And Liz gave some great encouragement and info for people who want to get involved with the community and possibly contribute.
eBPF is just so powerful, and as time goes on, the Cilium team keeps finding new ways to take advantage of it. That’s important because writing eBPF programs in C isn’t exactly easy. Many teams won’t end up doing that, and with great eBPF tools like Cilium around, there’s much less need to.
#Registries After Dark, Part 2: Distributed Random Access Merkledags - Daniel Mangum & Jason Hall
I’ve been following Dan on Twitter for a while, and I had a feeling this would be a super fun and nerdy session. I was correct. It was a sequel to a previous talk of Dan’s, which I also saw, but this version was much weirder. Unfortunately Jason wasn’t able to make the trip to Spain, but Dan did a great job carrying the speaking load.
A lot of the fun in this talk came from some unexpected events and topics, so I don’t want to give too much away. The TL;DR is that you can stick a lot more kinds of data into container registries than just container images.
Definitely do look this talk up when the videos are posted, and if you have a chance to do some weird things to container registries yourself, I’m guessing that Dan and Jason would love to hear about it.
That was the last talk of day two for me. I was pretty tired, and the evening event was a way out, so I decided to skip it. I had dinner with Loft Labs co-founders Lukas Gentele and Fabian Kramm at a nice Italian restaurant, where we swapped some fun stories. Did you know it’s possible to run Kubernetes on top of Mesosphere? I don’t recommend doing it, but if you ever run into Lukas or Fabian, feel free to ask them about it.
We also had a chance to wander around the older area of Valencia for a bit, and it was really lovely.
#Day Three, Thursday
Did I make it to the keynotes on Thursday? No. Did I feel bad about it? Yes, but I was running very slow again and wanted to try to meet some folks before the conference ended. I managed to run into a few of them.
I want to thank all of the folks who came up and said hi during the conference. People reached out to tell me about how much they love our tool vcluster or how they listen to my podcast Kube Cuddle, and some even talked about enjoying my writing. It’s great to hear from people who appreciate what I do in the community, so if you ever see me at a conference, please feel free to say hello.
I did make it to several breakout talks in the afternoon.
#Too Much to Choose – Making Sense of a Smorgasbord of Security Standards - Anais Urlichs & Rory McCune
When I saw the schedule this talk jumped out as one to see. Anais and Rory are both great, and this is a critical topic. There’s a lot of buzz around supply chain security right now, and rightly so, but you still need to secure your clusters.
What happens after you do that work, though?
Someone will ask “Is this secure?” And that’s where standards come in, it’s how you can prove something is secure. #KubeCon— Rich Burroughs (@richburroughs) May 20, 2022
Rory explained that security standards are usually checklists (pass/fail) or hardening guides (less prescriptive, and more vague). He talked about some of the specific standards used for Kubernetes and why it’s essential to understand their context.
Some distros have their own standards but many do not. The CIS benchmark may give false negatives or positives depending on your distro. You also need to look at which versions of k8s are covered by a standard. #KubeCon— Rich Burroughs (@richburroughs) May 20, 2022
After Rory’s excellent overview of standards, Anais covered some of the tools people use to validate that they are meeting those standards.
This was a super helpful talk and I really enjoyed it. A big bonus was all of the folks from the Kubernetes security community that were in attendance.
Love you, security nerds ❤️ #KubeCon— Rich Burroughs (@richburroughs) May 20, 2022
#A Guided Tour of Cilium Service Mesh - Liz Rice
Ok, prepare for more of me talking about how rad Cilium is. I mentioned earlier that it now has service mesh capability, and Liz’s talk went into that in more detail.
The sidecar pattern evolved for good reasons, but the reality is that it can be pretty wasteful. Liz asked, “Can we move service mesh to the kernel?” The answer is no, not at the moment, but we are pretty close.
Cilium’s service mesh implementation eliminates the sidecars by routing all of the service mesh traffic to one Envoy instance. The Envoy listeners are configured with a CRD called CiliumEnvoyConfig. Liz did some great demos, and it all looked very slick.
If you are interested in service mesh, this is a talk you should make sure to watch.
P.S. Liz will be my next guest on the Kube Cuddle podcast in June. If you’d like to hear our chat, search for Kube Cuddle in your podcast player and subscribe.
#Navigating the CNCF Landscape, the Right Way - Divya Mohan, Savitha Raghunathan, Kunal Kushwaha, and Saiyam Pathak
This was the last talk I attended, It was great to see the speakers in person. I met Saiyam and Kunal during this last year and I’ve been very impressed with the amount of education they do in the community. I recently found Divya and Savitha through Twitter and they clearly have that same passion for spreading knowledge.
This talk was in the Student track and it introduced the CNCF Landscape, which can be very intimidating to behold. The presenters covered how the landscape is organized, what information is on the cards that make it up, and other info about how the CNCF is organized, like the TOC and TAGs. If you are new to cloud native or Kubernetes and are overwhelmed by some of the CNCF terminologies, this talk would be a great place to start.
I enjoyed meeting the speakers, and I hope they all continue doing their fantastic educational work. The community is a lot better for it.
So that was it for KubeCon proper. After the conference, I joined most of my Loft Labs coworkers for a weekend retreat at a lovely villa in Valencia.
Not a bad morning, thank you @loft_sh. I’m getting some writing in while the rest of the team that’s here explores Valencia. I’m pretty jet lagged and didn’t sleep much so I stayed behind, but I also wanted to write up my #KubeCon thoughts while they’re fresh. pic.twitter.com/Y7upr7jJIw— Rich Burroughs (@richburroughs) May 21, 2022
We’re a small company but also very remote, and it was great for many of us to meet in person for the first time. There were fun team activities like a Segway tour and a hot air balloon ride.
The conference was great for me and also for our team. We heard many stories of people using vcluster in different ways. It’s clear that there’s a lot of excitement about virtual Kubernetes clusters right now and it’s always fun to see what ideas people come up with.
I was nervous about flying home on Monday, especially after seeing stories of people getting stranded in Valencia for days because of problems with their flights. But mine made it out on time, thankfully. The trip home was long but a bit smoother than the previous one.
After I arrived home on Monday I saw a couple of interesting things on Twitter. First, people started sharing their positive Covid test results with the hashtag #KubeCovid.
We need a way to know who’s tested positive for COVID so that the vulnerable people, and those with vulnerable family, can be prepared to isolate at home if they were around you.— David Flanagan (@rawkode) May 23, 2022
If you’ve tested positive and where at #KubeCon, please tweet and tag with #KubeCovid
If you attended the conference it would be worth having a look there to see what your exposure was. There’s also a spreadsheet. A few people I spent time with have tested positive. My tests have all been negative thankfully but I’m planning to isolate and keep testing for a bit since I had some exposure.
If you test positive please email the CNCF at firstname.lastname@example.org, and if you’re on Twitter it would be great to share in that hashtag. Thanks Rawkode for kicking off the hashtag and Tiffany Jernigan for making the spreadsheet.
The other interesting Twitter thread was from Josh Berkus, who shared some information about staffing problems during the conference due to what he described as a “sick out.” Those people being out resulted in some of the logistical problem we saw (lack of water, soap running out, etc.). He’s deleted the thread since because we can’t have nice things.
It wasn’t clear to me if people called in sick for some organized reason, or if they had Covid or some other illness. But it got a lot of us thinking about whether it even makes sense to be doing these big events if they are resulting in folks who don’t make tech salaries catching Covid. I definitely felt some guilt over that and I wasn’t alone.
I'm not here to shame people who went, because I went too, but I think we should all take a moment to think: is this worth it? I'm not sure it is— Ian Coldwater 📦💥 (@IanColdwater) May 24, 2022
I think we should continue examining how we gather in this new world we’re in. I’m planning to be in Detroit in October for KubeCon NA 2022, and I’m excited about attending the next KubeCon Europe in Amsterdam. But I also want to be as responsible as possible to our community and the people outside of it that we impact, so this is a topic I’ll be reflecting on. The CNCF will also be sharing more info about the number of attendee Covid cases in the transparency report. That will be interesting data to see.
If you attended the conference, I hope you had a great time. If you didn’t make it, there will be a lot of great talks to watch on YouTube soon.
Thanks to everyone who made the conference possible: The organizers (CNCF and LF), speakers, sponsors, and everyone else.
Image by Dan Finneran