Architecting Kubernetes clusters: one large shared or multiple small clusters?

Michael Petersen: 8 o'clock. So I guess we can go ahead and get started. We've got a good amount of people that have joined in. Let me do some some starter stuff for people that are just joining in, and then I'll hand it over to to Daniel for the presentation and the workshop. So the blog's up. If you want to follow along and check all this out. The blog is in the chat, and I believe everyone can see it. If you can't see it, let me know, and I'll post. I'll post it again.

Michael Petersen: If you have questions, ask questions in Q&A, and we'll cover it at the end. During QA, we're not going to do it. Live while things are happening, but we will. We will answer it at the end. There's be plenty of time for the QA. QA. Stuff. We just want to make sure that we get through all the content, and then then we'll go there. So if you think of something posted in there, and then we will get to it.

Michael Petersen:  Oh, is the chat all right? So we had a question already in the Q. And A is the chat meant to be disabled? I think maybe we're. I don't know if it's disabled, and maybe we're just doing Q&A. So that we can answer there. So it doesn't get overloaded with being able to post the blog posts or other things that we need to post that or resources. But

Michael Petersen: let me let me also. I'll see you about that, while as soon as we start presenting also.

Michael Petersen: you can join our slack if you want to ask questions too. So slack dot loft, sh! And go into the vcluster channel, and if you have questions we'll be watching for questions in there, too. I think, Daniel, I said. The

Michael Petersen: that that they're in there looking looking for questions to you, just in case.

Michael Petersen: But all right, let me go ahead and introduce. I didn't introduce myself, Mike Peterson. I'm with loft labs.

Michael Petersen: technical marketing, and I'm here with Daniele from Learn K. 8. So you've been doing a lot of learn K8's stuff in like teaching on Kubernetes and teaching the community how to use things right like this is going to be a

Michael Petersen: a single cluster versus each their own right. This is the first question. I think. You see, as soon as you get into Kubernetes, does everybody get their own? But I'll make it over to you, Daniel. It's nice to meet you, and I can't wait to hear your presentation.

Daniele Polencic: Yeah, thank you very much for the intro by appreciate it. So yeah, I think one of the questions that everybody ask is, you know, I've got these classes. Should I create more? You know, and how many more?

Daniele Polencic: And I think the questions, you know, sort of

Daniele Polencic: it's all seem simple at the beginning, but as you explore the topic, at least what I did for for this. You know what I researched for this this presentation. I found that there is so much variety, and then the variety has changed quite a lot. If you look, you know, throughout the years as well.

Daniele Polencic: and that that takes me to to today. So

Daniele Polencic: oops before I start, let just see a few words. I'm like, like I said, I'm I'm Daniela. So I'm one of the instructors that learn humanities. And

Daniele Polencic: I spent quite a lot of time working with Cuba. He's so much time that my colleague Craig, called me d 5 e

Daniele Polencic: I am I present? II was lucky enough to present a kube con twice.

Daniele Polencic: and I was one of the first 500 engineers who got the Cks certification back in 2,000. 2,017.

Daniele Polencic: That was when the exam was 3 h long and not just just 2 h. So, yeah, that's you know, how long have been? I've been doing this. And today's yeah, today, I want to talk about queue multi-tenancy. In particular, when we're talking about multi-tenancy, I think my research has been around 3 sort of categories, free areas.

Daniele Polencic: The first one is isolation. So when I look at multi-tenancy, the kind of questions that I ask myself is

Daniele Polencic: okay, do I trust the people that I'm onboarding in this cluster. And what is the kind of level of trust? Do I trust them because they are in the same? They work in the same company

Daniele Polencic: as B, so I can go in and say, Hello! Can I go to that desk and say Hello! If I onboarding tenants that I don't necessarily

Daniele Polencic: no, so maybe they are external. So there is a team external working with with my company, and I need to provide them an environment for them to play with.

Daniele Polencic: or I'm not writing third party applications, and I don't trust them at all. So I think, you know, there are different sort of levels of of trust when it comes to isolation, and that the other things that I sort of explored is okay. If there are different level of isolation, what? What is the cost?

Daniele Polencic: What is the cost involved in maintaining these different levels of isolations? How does how much does it cost to actually build this platform, but also operate it, and the effort and knowledge and demand power that it takes to to run such thing. And all of these sort of things, then think they come down to you know how much costs and how much time do I need to invest in in building and and maintaining this platform?

Daniele Polencic: So this was basically, you know, sort of the question that I had. And then so I started investigating and

Daniele Polencic: and sort of mapping out what I think are the 3 different scenarios that sort of

Daniele Polencic: sort of depict, how how we can, how we can look! Look at this multi-tenancy. Now, before we talk about multi-tenancy. I just want to do a quick recap on how we deploy and we create Kubernetes clusters. And and this is because we just need to put things into context before we actually dive in, dive, dive into into multi-tenancy.

Daniele Polencic: So generally, what we have is a collection of collection of service, and

Daniele Polencic: in the collection of service. What we do is we have one of these as control plane and the rest of the work at the rest of the fleet joining his worker notes.

Daniele Polencic: And once we join these worker nodes, then we have what we call the cluster. So from a perspective of of a user. Then when we deploy workloads, we deploy them in the cluster. There is no node node as such. But in reality what we have is kubernetes just looking at these workloads and and placing them in different nodes inside the cluster.

Daniele Polencic: You know this, this is probably not not news to you, and you probably what is not used to you is, you know. Sometimes we also find way to sort of make sense of all these deployments. We want to group comings in in a way that is easiest, easier for us to manage.

Daniele Polencic: And that basically has been

Daniele Polencic: for that reason we usually use something like namespaces and namespaces are basically just grouping of things. And most of the time we also think about, you know, namespaces is taking a slice of this cluster and just partitioning it. So we can use it for different, for different teams.

Daniele Polencic: And this is basically where things go become a little bit more complex. Right? So namespaces in essence they, they tend. They lead us to think that

Daniele Polencic: okay, this namespaces for this team and certain namespaces for the other teams. We do sort of mental mapping where we map the tenant to a namespace.

Daniele Polencic: which is, I guess, okay. And but you can also see that sometimes namespaces that are associated with projects. So project A in namespace, a project B in Namespace, B, and so on, and so forth.

Daniele Polencic: So is really a namespace of tenant. Well, it's a little bit tricky, and to explain what is tricky, I want to show you I want to show you something. So let's say you've got an e-commerce website

Daniele Polencic: and an ecommerce website. I've got a team team, A who looks after the checkout. So they create a deployment. They've got some network policies limit range resource quota, and so on and so forth.

Daniele Polencic: And then team A is doing very well, and it's get assigned second project. And the second project is basically just the user user accounts. So when the user can go in and change the details, the billing details and so on. So forth.

Daniele Polencic: Now, team A, you might create a second namespace, and that second namespace could create could contain another deployment, and then it might also contain the same sort of resources, and such as limits range resource quarter, but also roles right and permissions

Daniele Polencic: as as the previous as the previous project. So this team is basically managing tuning spaces because there are 2 different projects in there.

Daniele Polencic: and and the reasoning, and and they could potentially merge everything into a single namespace. But that would be problematic if they need to migrate. So here's the problem. So we've got tuning spaces. They need to duplicate resources.

Daniele Polencic: They don't really want to imagine one because of the way, it's becoming more more complex to manage exceptions.

Daniele Polencic: We really see how there is. There is some sort of tensions about these namespaces being sort of a flat structure that that groups the resources.

Daniele Polencic: So what can we do? Well, there is one project from Seagull which is quite interesting. It's called here. I call namespaces. And what let those this Controller lets you do is basically lets you just nest namespaces, and with next namespaces. What you can create is a namespace for the entire team. And then a child namespace for each project. But the beauty of these controllers basically is that once you create these child namespaces, then spaces will. Basically.

Daniele Polencic: those those namespaces will basically just inherit any sort of

Daniele Polencic: resources that you want from the parent namespace. In this particular case. Imagine they've got a purple limit range of yellow resource quarter. As I create them in the root namespace, those are basically copied. They are basically propagated inside, inside, inside,

Daniele Polencic: inside the children.

Daniele Polencic: And if I create, let's say, a network policy, and I enable the propagation. And I create a child from an extra child. Then those figure resources will be propagated as well. Okay. So in a sense, in access, what we have is we have Kubernetes, namespaces.

Daniele Polencic: But this namespace controller what it does. It gives us some sort of extra layer where we can organize our resources more precisely right and without duplicating them.

Daniele Polencic: And what does it look like when and you actually use it. You basically install this controller, which is which is a component that lives in the cluster.

Daniele Polencic: And then you've got a plug, a kube. Ctl, plugin. which enables you to do this, this kind of commands, where you can create a relationship between 2 namespaces.

Daniele Polencic: So in this particular case. What I what I've done is I've created role in the independent namespace. And then, as I create a child namespaces, then a child namespace that that child namespace will will inherit that particular role.

Daniele Polencic: Which is great. So I don't have to duplicate things. I can just inherit them from from the parent namespace.

Daniele Polencic: This is, this is a good workaround. But but there are some some sort of edge cases.

Daniele Polencic: And the edge cases are basically

Daniele Polencic: this works. I mean namespaces work, because we can name space resources, but not all, and not all resources in Kubernetes can be namespaced. So take a pick, for example, a persistent volume. Persistent volume is a global resource in Kubernetes. So if you look at the Api.

Daniele Polencic: and so all these deployments and paths are basically just Api and qinators. If you look at those things such as paths have got a namespace.

Daniele Polencic: But a persistent volume doesn't have a name space at all. It's just like a resource which is available to to the entire cluster

Daniele Polencic: and other other examples are namespaces themselves, or notes or

Daniele Polencic:

Daniele Polencic: validation, yeah, webex for admission controllers.

Daniele Polencic: And and those basically, those resources are installed are stored inside that city. But what happens when you want to grant access to these resources? Well, if I grant access to for read and write to team A, team B and team C, they will all be able to see this persistent volume, which is sometimes something that I don't want. I want to have this team

Daniele Polencic: to have a single view of the class theory, which is just their own resources. So we're really seeing that how Kubernetes is basically giving us

Daniele Polencic: some levels of isolations. But it's sort of, you know, sort of bending the rules on others. So on. Isolation. You know, this is probably okay. If you're working with, you know, colleagues and teams within within your company, this this level of isolation is probably okay.

Daniele Polencic: But perhaps you know, you're looking at this and say, Well.

Daniele Polencic: this is not really my capital. I want something a little bit more right

Daniele Polencic: before we more, we move to something a little bit more complex.

Daniele Polencic: I just want to touch on something, something related, which is, which is upgrade. So we talked about.

Daniele Polencic: You know how multi-tenancy we're going to measure multi-tenancy on on 3 different sort of axes. We're going to look at

Daniele Polencic: isolation, ease of management and costs. So one interesting things to look at is is upgrades. So this applies to namespaces

Daniele Polencic: and and and the controller we just we just presented. So back in Cuba, S. 1.18. I think we used to have this very old.

Daniele Polencic: and we used to have this this definition of of the ingress, which was the V one beta one. And then sometimes I think in 1 19 it was upgraded to V one and

Daniele Polencic: what I'm interested in is that this. This process of upgrading and and removing Apis and Kubernetes is quite common, right as things move, and as the as these resources mature, then then some of the versions are deprecated. The challenge is

Daniele Polencic: the system. Sometimes these applications are problematic, and the problem came at a time. We had a problem where all of these B one Beta definition for English were basically removed in cumulative 1, 22.

Daniele Polencic: But it was still working in 1 21. And

Daniele Polencic: and this is exactly a good scenario to explore how.

Daniele Polencic: and dependencies in multi-tenant clusters are are problematic. So what happens when you have several teams in your clusters.

Daniele Polencic: all sharing namespaces and one of those teams and developed the app and then just abandon it. I mean, it's still maintaining. Maybe it's it's just in, you know, sort of in a maintaining phase, it's not actively being developed on.

Daniele Polencic: If this team is using an ingress v. One beta.

Daniele Polencic: then this cluster will not be be able to be upgraded, because at the time where we we do upgrade it, then

Daniele Polencic: their resources will go, will go out of sync with their repo, and that might be issue with with the ingress.

Daniele Polencic: But if there is no budget who's going to pray these, this resource, I mean, this is basically where

Daniele Polencic: we are sharing the same resources. So we need to play nicely. You know, we we all need to be playing nicely with with what we call

Daniele Polencic: the right thing to do would be to update the ingress. But you know, that's not always possible. So overall

Daniele Polencic: just using namespaces or something a little bit more than namespaces, like hierarchical controller is okay. You get isolation, which is fairly weak.

Daniele Polencic: But it's okay, generally just, you know, cheap. But you need to be able to go out and interact with teams, and just, you know, make sure that you move all of your tenants and and help them progress through and through their joining with with your cluster.

Daniele Polencic: But the real issue is here is that you're sharing pretty much everything in the cluster. So namespaces are

Daniele Polencic: something something fundamental in Kubernetes, you're sharing the control plane and a sharing the controlled plane basically means that whatever resource you create, like the ingress is created in the same database as everyone else.

Daniele Polencic: And and that's okay. But be able to manage it. And the second sort of

Daniele Polencic: challenge that you get with with this is that if something goes wrong with the Api server.

Daniele Polencic: it's still a shared resource. So it doesn't just apply to resources, but also components of given names. So if someone abusing of the Api server and the Apis have a

Daniele Polencic: and

Daniele Polencic: restrict the number of Api calls, this is going to take. Basically, this is basically going to make the Api. 7 unavailable for for everyone. So no great resurrection. So what can we do to actually isolate this problem?

Daniele Polencic: And and here's a clever solution. So what you could do instead of having a control plane, you could basically create a control plane as a pod

Daniele Polencic: as a regular pod. And instead of issuing commands directly to the control, to, to the, to, to the cluster. Then all these commands are sent to this control plane, which is which is running as a pod.

Daniele Polencic: And this control pane is not. Note. Okay.

Daniele Polencic: but this control plane has got the same components as regular Kubernetes cluster. So if you create a pod.

Daniele Polencic: then that pod will be stored inside a CD or any other sort of story that that you have.

Daniele Polencic: Unfortunately, there are no nodes. So this pod is not creative.

Daniele Polencic: So how can we fix that? Well, what if we actually copy?

Daniele Polencic: What if you actually copied that part from one control plane to the other right? I think that would be enough. The second control plane has got nodes so it could deploy. Actually deploy that pod inside the infrastructure.

Daniele Polencic: And that's basically the idea behind V cluster. So V cluster is basically just a very convenient way for copy. Some of resources inside one control plane

Daniele Polencic: inside.

Daniele Polencic:  inside the other, the other the the host cluster

Daniele Polencic:  And this solves some of the problems that we've seen before, and I think this is where I think Scalia be more interested. So if let's say, you create, we talked about persistent volumes as global resources. So you can create the persistent volumes inside

Daniele Polencic: one of these nested control plane

Daniele Polencic: right? And then what it will happen next is the V cluster controller would basically copy this.

Daniele Polencic: This resource. So you can. You can think about this Yamo file that you store inside the control plane is going to be copied inside the host control plane.

Daniele Polencic: And then that's basically when the persistent volume will be created.

Daniele Polencic: So in essence, we're using this nested control plane just as a way to manage resources inside the database. And when we get ready, we basically copy these resources across to the real cluster.

Daniele Polencic: But the advantages of this is that all of the resources that are being created are basically created inside these nested clusters. So you get good isolation. So if I list, if I go back and list the persistent volumes, then I will be able to see just the persistent volumes in this tenant. There is no way that I can see all the persistent volumes, even if

Daniele Polencic: but the persistent volumes like are just global objects.

Daniele Polencic: So overall isolation is less like so much better than than just plain regular namespaces.

Daniele Polencic: However, there is something to keep in mind here. So if we look at the same example of upgrades, and that we discussed with with ingress controllers, let's see how that plays out with with a nested controller with with vclaster. So with V cluster. So imagine that you've got the nested cluster on

Daniele Polencic: on 1 21, and

Daniele Polencic: the host cluster 1, 21 as well. So same version. If you have an ingress which is in Vb to a one, then the sync will work. However, if use, if you upgrade the must. If you create a main cluster, and the tenant is still an old version.

Daniele Polencic: then what happens next?

Daniele Polencic: Well, in 1, 22, we are not able to read. V. One beta, right? So this is taken straight from from V cluster documentation. So you basically need to go. What is this saying? It is saying that with you need to disable sync

Daniele Polencic: and upgrade the tenant

Daniele Polencic: right? Because you're not able to sync because this control, this control plane doesn't understand that resource anymore.

Daniele Polencic: Now, this is

Daniele Polencic: these highlights, some of the some of the constraints that we're playing with. Because yes, control. vcluster is giving us a full control plane. But this syncing mechanism

Daniele Polencic: carries some extra complexity when it comes to management. So we are trading. This isolation for for free ease of management.

Daniele Polencic: Now let's have a look at something else. What if I overload the Api server? Well.

Daniele Polencic: that's okay. It's just going to break one part, and that tenant is going to be isolated so overall this is, this is giving us more, more isolations than we had before.

Daniele Polencic: However, the node

Daniele Polencic: is still shared.

Daniele Polencic: and and what I mean by sharing is that you know all the paths that are deployed on on that particular node.

Daniele Polencic: Then we'll have the same kubelet. And the kubelet is that agent lives on the control plane which just pulls down information from the control plane and says, Okay, let's create this part

Daniele Polencic: and to create the party goes through 3 steps. It goes through container, runtime, interface, the container network interface and container storage interface.

Daniele Polencic: But these pods. These pods are basically just containers and and containers don't actually exist.

Daniele Polencic: Yeah, just basically, processes with extra current strengths and constraints comes in the form of you're not allowed to use. You know more than this memory, or you're only allowed to use, you know, these amounts of bandwidth, and they also have things such as namespaces which lets you isolate what kind of network they have.

Daniele Polencic: But bottom line. But bottom line is, they're just processes. So you could have someone escaping this container and gaining access to the node, and be able to do

Daniele Polencic: to see all the other parts inside that node.

Daniele Polencic: But a single kernel crash could take down the entire node. So this could be problematic, for you know, because it's going to affect other tenants as well.

Daniele Polencic: And then the other things to keep in mind is that Kubernetes has got 2 fundamentals. Rule for networking, which is any part can talk to any part.

Daniele Polencic: So even if you have these 10 and control like tenants control planes. Then the pods deployed inside of the host cluster. They can still talk to each other.

Daniele Polencic: If if you know the IP address. and that for some. maybe depending on the type of industry you work in might be might be something something to keep in mind?

Daniele Polencic: You know it depends what you do. Just

Daniele Polencic: just keep invite. So how can you avoid this problem? Right? What if you're going to have more isolation. What if you had to have? You don't want to share nodes because you could have node pools? You know what? If you want to have more strict networking. You know what kind of options you have. Well.

Daniele Polencic: you can try and keep to, you know, hardened sort of this this configuration, or you could go to the extreme, and the extreme head is having a dedicated cluster per per team

Daniele Polencic: with. Then you click cluster per team. Then each each team has got their own cluster. And if that sounds overkill, then you haven't heard of Kamata

Daniele Polencic: and Kamato. The way it works is basically is one of those tools that you basically install on what we call a management cluster. And it's got sort of its own control plane

Daniele Polencic: which sits on top of the Kubernetes control plane. And what it lets you do is basically lets you control other and Kubernetes clusters.

Daniele Polencic: And why would you do that? Well, maybe because you want to have a unified view of everything that is deployed across all the tenants.

Daniele Polencic: or maybe because you want to deploy a part on all the cluster that you maintain.

Daniele Polencic: So in this particular case, what commander does is they connect with an agent to the cluster, and they can interact. And and they they can interact with the Api server. So with a single pupil apply, I can deploy a single pod in all clusters. I can have like policies on how this pod is going to be deployed

Daniele Polencic: by the team, the team that got access to the local cluster so they can still interact with it.

Daniele Polencic: and they can still create their own resources.

Daniele Polencic: What's interesting about this is that you've got a management cluster

Daniele Polencic: right? Which is, I think, I think, the way I like to think about this is that it's very similar to what I see with vcluster, but it's taken to a next level by having dedicated cluster. So I still have like

Daniele Polencic: something central to to check what's going on. But then I eventually I just delegate all of all of these, you know, creating parts and and deployments to to the other clusters.

Daniele Polencic: and you know I can have the the manager pushing a pod which is common to all. I can have clusters in the deploying individual pods and their own pods in inside infrastructure.

Daniele Polencic: You know, I can take this to extreme, and I can also have these clusters located in different locations if I want to. Why not? And and that this is kind of very much an extreme. But II could control the pods being deployed there as well

Daniele Polencic: now as we did for and for the previous 2 tools, you know, if you look at upgrades, upgrades, they sort of look

Daniele Polencic: simpler, because what you do is basically you need to find a way to upgrade a cluster, and it could be like creating a second cluster and then replacing it. But I think this is sort of very naive. I think upgrading, upgrading clusters is never such a

Daniele Polencic: easy tasks, so I think I think this is probably a little bit more complicated than it seems.

Daniele Polencic: And what about blacks? Radius? Right? We had a look at the other options. What if something goes wrong? What if we lose the control plane, what if we lose the Api server? What happens next?

Daniele Polencic: Well, in this particular case.

Daniele Polencic: then the the failure will be isolated to the cluster. So if Team B has got a problem, and they, you know, take down the wrong cluster, then it's isolated to the wrong cluster. That's it.

Daniele Polencic: If the manager cluster goes down. Yeah, not a problem. The cluster, the other clusters can still work. We won't. We won't be able to see what's going on.

Daniele Polencic: and we won't be able to push any other parts, but at the very least is all the other themes are independent. so not too bad. so isolation wise. This is great. right?

Daniele Polencic: But when it comes to ease of management, I mean.

Daniele Polencic: we are talking about managing clusters at scale, which is okay. If you have something like Commander where you can push deployments. But what about all the rest?

Daniele Polencic: How you gonna self-serve these clusters, how you gonna maintain the updates. How you, gonna you know, keep them up to date, how you gonna manage them. So I think there is a lot of choice and a lot of things you need to think about when it comes to

Daniele Polencic: adopting something like this.

Daniele Polencic: And let's be serious. We are comparing here. We have. We have dedicated clusters. So yes, you, you get good isolations, but trade off the trade-off you have to pay is is quite high.

Daniele Polencic: and and cost is something that you know I haven't talked about, but I think it's something that, and we should be talking about. So I presented you 3 options. So the Hierarchical Controller.

Daniele Polencic: the cluster and

Daniele Polencic: and you know, if I were to look at those.

Daniele Polencic: then I could almost picture them and say, Okay, I've got a baseline level, which is Kubernetes namespaces, then installing and managing something like the hierarchical controller.

Daniele Polencic: Okay, it's

Daniele Polencic: it's good. It's giving me some isolation, weak. Probably. But overall is basically giving me some

Daniele Polencic: flexibility in organizing my resources and propagating and propagating them. And the cost is not particularly high. I just need to run a controller inside my cluster.

Daniele Polencic: So overall. Ok? And then we look at vcluster in vclaster. The cost is okay. I'm running a cluster on top of that. I need

Daniele Polencic: one control plane pertain and and and the controller, which is going to do to sync as well.

Daniele Polencic: Okay, this is probably manageable. But the more tenants I have, the more control planes. I need to run, so I think the cost wise, still manageable. But it's definitely not as cheap as running any spaces.

Daniele Polencic: And then you have

Daniele Polencic: the extreme, which is basically running one net, one entire plus their per team, right or per tenant. And that's basically, you know, the cost comes from from these separate control planes, as well as extra nodes as well as the ability to not be able to to share resources. So everything is dedicated to that team. So the cost is quite high.

Daniele Polencic: But I think, you know, we're looking at costs. We just look at those, you know most of the time at least during my research, I was just looking at a cost and say, Okay, is this the end of a story? Then this is basically when when I got sort of sidetracked. So

Daniele Polencic: a kube offense. So we, we launched a podcast about Cuban, 80 S, and then there is one question that we ask all, all guests, which is, what are the 3 tools that you install in a brand new cluster?

Daniele Polencic: And then we're running. We've run. We've run about 2020 episodes. So far so it's very young. Podcast. But the kind of answers that we get about these questions are very, very consistent.

Daniele Polencic: So it seems like the 4 top and tools that are mentioned in the podcast are permifus, external Dns, cert manager and Ioc.

Daniele Polencic: and this got me thinking. Let's take, let's take Prometheus right

Daniele Polencic: when you have a multi-tenant cluster. What do you do? Do you have one single Prometheus. or shared across all tenants, or share like one Prometheus tag

Daniele Polencic: per tenant.

Daniele Polencic: I think the answer, of course, is, it depends. But yes, it depends. But this it depends. It actually adds up into costs. But the costs are not added linearly, because if you think I want a single Prometheus, because this is gonna

Daniele Polencic: help me manage the costs. I will say, yes, you're right.

Daniele Polencic: But what about like in the case of Kamata? You can't really go every single Prometheus, even if you install something like Tanos, you need to have one Prometheus per cluster. So the costs

Daniele Polencic: don't necessarily scale linearly with all of the options that you have.

Daniele Polencic: And on top of that you could go multiple Prometheus, where you have one Prometheus per tenant, and that will be, you know, a fair chunk of costs

Daniele Polencic: that that you're going to add. But what you're going to do with it. You're probably going to have something like which is going to centralize the metric as well, which is going to be something else that you need to build, deploy, and maintain somewhere else, which is going to add more costs.

Daniele Polencic: And if you think about it, this is just metrics with Prometheus. What about logs?

Daniele Polencic: Maybe we logs? You do the same right. You have some logs per tenant, and then you aggregate them. So you keep adding costs

Daniele Polencic: as you go down, and it doesn't scale linearly either. So what you end up with is that there are some costs which are direct. So you can't number nodes. You can the number of control planes. There's some costs that are sort of

Daniele Polencic: consequences of what what you decide right? Like Prometheus. Okay, now I need to install one Prometheus per cluster because I chose commander.

Daniele Polencic: and and this takes me to to sort of the end of the presentation. At least.

Daniele Polencic: you know, sort of this summarizes how. I basically came about to think about all of these different options that we have in multi-tenancy. By the way, I present 3 options.

Daniele Polencic: But reality is that in in the community there are more tools, and sometimes they overlap. Sometimes there are some gaps.

Daniele Polencic: But basically this, this is generally you know how I see them now. But I spent so I spent some time researching them. So generally, I would basically divide them in in 2 blocks. The first one is single cluster, like, I want to have everything self-contained and single. Cluster and and second

Daniele Polencic: part is basically dedicated clusters.

Daniele Polencic: And in single cluster, basically have 2 options, a, and and the option is okay, do I want to keep very very light

Daniele Polencic: version of multi chancing?

Daniele Polencic: Or do I want to go and and have like a little bit more isolation? And if I want to have a little bit more isolation. I need something like a dedicated control plane for me.

Daniele Polencic: And on top of the others, I basically have what I call adults.

Daniele Polencic: So I'd also basically wait for me to restrict even further.

Daniele Polencic: you know, to isolate even further my tenant, you know they will cost me some money or some effort, but it will basically give me more guarantees around isolation.

Daniele Polencic: And then on top of that, I basically have got what should I do with monitoring and logging. And you know

Daniele Polencic: also, what should I do with adequate attic? Shared resources like the interest. For example, I can always go, for should I have a single resource for everyone

Daniele Polencic: and share? Or should I have a dedicated one

Daniele Polencic: and then move on.

Daniele Polencic: and then we can play with this and and see where we go? Right. So let's say that my multi-tenancy

Daniele Polencic: is mostly deploying.

Daniele Polencic: So I've got. I've got a few tenants. Let's say I've got, you know, 5 tenants. and they mostly deploy web applications, and those web applications tend to be static web applications.

Daniele Polencic: So if I think about it, then my choice would be should I go with something like a dedicated cluster. Hmm, probably overkilled.

Daniele Polencic: Okay? So I'm like, I'm left with, okay, single cluster.

Daniele Polencic: And then single cluster, do I need shared control plane. Or do I want a dedicated control plane?

Daniele Polencic: I mean, in this particular case, these applications are basically static websites. So I want to go cheap. I want to go very lightweight, so maybe a share control plane is fine for me.

Daniele Polencic: Do I need any dedicated node pool or a sandbox runtime? Probably not right. Maybe I can go with a sandbox, Runtime, if I think that you know the the app could be exploited. Maybe I could find other solutions sort of hard in these containers.

Daniele Polencic: and then I'm left with the choice of you know what kind of monitoring or logging I want.

Daniele Polencic: Maybe I want to go share it. Maybe I want to have like a dedicated interest, because I'm concerned that one of these apps

Daniele Polencic: will actually consume will actually need more, you know, different parameters like, for example, very common reason to have dedicated ingress controllers is that when you're uploading files, then you need to configure your your reverse proxy to actually accept that. You know. 50, you know, 1020, 30 MB of upload, but not everyone might want the same upload size. So you want to be consensus. So if you want to do that, you need a dedicated

Daniele Polencic: inverse controller.

Daniele Polencic: let's have a look, another one. So let's say we have a crude application with a database.

Daniele Polencic: How is this going to change when I think about multitudinency? Let's say I've got, you know, the same 4 or 5 teams. But instead. This time they've got database like impostres, and they've got, you know, a flask application in Python. Should I go with the same choice?

Daniele Polencic: I don't know. Still, I think in this particular case, dedicated clusters

Daniele Polencic: sounds like an overkill

Daniele Polencic: but, to be honest, I think if this dedicated class, you know, if they are deploying an application which is a web app plus a database, and maybe the database is managed using crds.

Daniele Polencic: My my thinking is, maybe maybe you want a little bit more isolation this time, so we'll go with the dedicated control plane.

Daniele Polencic: Do I want? And dedicated node polls? Yes, I do. If you're deploying databases. I want to make sure that there is no

Daniele Polencic: noisen neighbor club problem. Right? So I want to make sure that I separate sort of I isolate the issue where one database could affect others.

Daniele Polencic: or one thing, it could affect others. Maybe I can not use the sandbox, Runtime, because I'm using a buff in front of my apps. and

Daniele Polencic: and I've got choices again. Monitoring a login should go shared or dedicated. Maybe I go with shared ingress and and shared logging, because I know that this app won't be. Maybe they will gonna be used internally so, or I'm sure that there won't be a problem with the ingress if one of these is overloaded.

Daniele Polencic: Yeah. So I think those are like 2 examples I think

Daniele Polencic: they tend to be. I think the the examples I present. They tend to be you know

Daniele Polencic: not a real representation of what you will face in the I think reality will not just have monitoring and logging and interests, you might have more shared components that you need to account of. But at least I think this should give you an idea about how to approach the problem. So this is not meant to be like a definitive framework for evaluating what you should do. But it should give you just enough expertise to look at the problem and say, I know what to do next.

Daniele Polencic: So so takeaways, so what what we learn today. And so I think I think it's important to tackle the problem from to start with the requirements. So when you have a problem, when you want to do multi-thanacy, when you want to build a platform on Kubernetes? I think you should start by looking at what kind of isolation do I want? How much do I trust my users.

Daniele Polencic: and should I trust them? And then you look at okay. what kind of options do I have to make my life easier?

Daniele Polencic: Right should I? Is the isolation, isolations, pushing me to build more tools and to use more complicated tools?

Daniele Polencic: Or can I get away with something different?

Daniele Polencic: And how these choices are going to affect. You know my costs, and when you look at cost, when we look at cost, we look at the cost which are direct. So running, one more part of running, more running. One more cluster, and then we're also looking at and indirect costs.

Daniele Polencic: So things such as you know how many Prometheus do I need?

Daniele Polencic: How many ingresses do I need? And today we'll have a look at what I call namespaces plus plus plus, which is basically all sort of tools

Daniele Polencic: and software which is basically creating multi-tenancy on Q, and A is based on namespaces like capsule or hierarchical name, controller namespace, Controller. So those are like namespaces plus because they are very lightweight. They've got some trade-offs which are basically made of.

Daniele Polencic: They're they're designed to help teams which mostly trust each other share a common cluster, and I think that's a good good choice to make.

Daniele Polencic: Then we had a look at how we can isolate even further within a single cluster. And that's basically this is basically the positioning for for V cluster. I think this is probably where you can get the most isolation in a single cluster.

Daniele Polencic: And then finally, we had these dedicated clusters. So this is a very, very much an extreme. We will look at each team having their own dedicated cluster. And then the cost, the cost for for for managing something like this.

Daniele Polencic: Yeah, like I said, I think even direct costs are one thing. But look, look! Looking at this cost ballooning as as soon as you add more stuff on top, then then really opens your mind, because if you think about it. When you run the cluster

Daniele Polencic: you need to run, you know, extra kubelets and extra control plates. And yes, this goes. It has a

Daniele Polencic: running one more. One more, one wrong, one more convivial, you know. One more open telemetry, you know. Jaeger or

Daniele Polencic: not. Alex. Stack. Yeah, that that is really gonna add up to the costs very, very, very quickly.

Daniele Polencic: Yeah, I think that was basically all I wanted to say today. So thank you very much for

Daniele Polencic: listening to to this talk. I hope you enjoy it. I wanna just say thank you for for love to loft for for having me. Thank you very much. And yeah, that's it.

Michael Petersen: Awesome. Thank you so much. We've got Q&A and stuff. Now we've got a bunch of questions already queued up, we can probably move on to that.

Michael Petersen: You may go and get started with. Well, the great presentation. Thank you so much like that was very informative, and I think every time

Michael Petersen: every time someone gets into Kubernetes the first question is, Do I share it with people? And then you get the first request from a developer. Hey? Can I get my own cluster because I don't wanna overwrite this other person's cluster? And by the time you actually take a look you've got 20 or 30 clusters running.

Daniele Polencic: What does this cost?

Daniele Polencic: And then I think, to to be honest, that I think the the one for me is like, you know, when I started was like, Okay, I can just use a namespace. And, to be honest, I think next week we'll have we have salman. And so we'll basically just decide to hold this, you know, awesome. And I look at how how we gonna

Daniele Polencic: you know how how isolation in namespaces work and why it doesn't work.

Daniele Polencic: So I think, yeah, think that that is gonna be interesting as well.

Michael Petersen: So we had a couple of questions. There's everyone was kind of asking will there be recording? There will be a recording? And then we had another question. Oh, for the recording, there should be an email coming out, probably in like a couple hours, or something like that, saying where to go check out the recording. I think it'll be on our Youtube, but not 100% sure. Also.

Michael Petersen: the question was, will the slides and I'll be available. Do you have somewhere that these slides are? Or should they just look at the blog post? Maybe because that has some of the stuff in there?

Daniele Polencic: I think I can make the the slice available. But all of his slides, you might notice they're quite

Daniele Polencic: visual, so there is no notes or anything so licenses they make. It's a little bit high

Daniele Polencic: at that time. So I think the recording will really help.

Michael Petersen: Yeah. So I can make porting

Michael Petersen: and the blog post. Okay.

Daniele Polencic: yes, let's go.

Michael Petersen: Let's go ahead to the the questions real quick. So I'll start with the first question. So is it possible to restrict access to a particular namespace? For example, I want users to access all namespaces except default.

Daniele Polencic: I don't think you can say except so you will have to say, I want to access to this, this, this and this.

Daniele Polencic: and then just keep adding it as like you add more namespaces they want to have access to. So you have to automate that maybe not just don't include X.

Daniele Polencic: I think the the 1 one solution I've seen on in in this area, and I think this is the only one I've seen, but there might be others. This is, you know, just just sort of a something I'm sharing with that. So I've seen the Capsule project. So the capture project is one of these namespaces, plus plus tools that let you are more

Daniele Polencic: sort of higher level abstractions to manage your tenants, but still based on namespace, and they've got like a proxy

Daniele Polencic: which lets you sort of give access to particular namespaces to to your tenants. So instead of going like I think it works by by connecting to keeps to the Api 7 keeps it. Yeah, I don't know how.

Daniele Polencic: But basically you can do Qcr get namespace, and you only see the namespaces you're allowed to see?

Daniele Polencic: So I so I think that's basically the closest you can do to what what you describe.

Daniele Polencic: Gotcha. Alright. So next one else is also suggesting, yeah. Well done. Yeah. Well, I think that's an option. Yeah. Good. To make sure you do it to everyone, because you're posted it to the, To us, and we're getting the useful information.

Michael Petersen: Let's go to the next one. So what happens if you attempt to upgrade the tenant cluster control plane without disabling sync. If there, if there's exists a depreciated apa resource on the tenant cluster, so I guess they're talking about

Michael Petersen: like, what if you upgrade the tenant cluster and use some like V cluster? Or just, I think, to say, Yeah, I think the sync is big cluster. Do, Mike? Do you mind. sir?

Michael Petersen: I don't know how it's gonna reconcile that right like, if it's already existing, you upgrade everything. And now the Api makes it invalid. How is it? Gonna keep the the spec up to date? Not 100%. Sure. I think I think that you might have issues, because the

Michael Petersen: the Api will have a different

Michael Petersen: spec than maybe what you have. But hopefully, that's only on bigger jumps, right? Like hopefully, that's not gonna be like. You're just

Michael Petersen: making like a huge upgrade from like a very old Kubernetes cluster to a new one. And you still have these same Api resources. But

Michael Petersen: to answer that without giving a real answer.

Michael Petersen: maybe try it on a test cluster or something like that. First, right like, instead of doing production, we got something. Daniel.

Daniele Polencic: Yeah, I was about to say

Daniele Polencic: so. So we usually work with version resources. And when I say, version resources, right is the name ingress v, one b, 21.

Daniele Polencic: However, when you store these resources in sed

Daniele Polencic: the annual research, the end of story like that. Yeah, basically story, we we basically what we call just an ingress, just what Q&A thinks is an ingress.

Daniele Polencic: So as you upgrade that that basically the control plane just upgrades that resource.

Daniele Polencic: And then, when you request that resource, that's basically re serialized in in V one or VV, 2, one. So I think I'm not sure. I'm not sure. 100% sure how it works in in vclaster, because I'm still trying to make sense of all of your syncing. But I suppose that if if you are upgrading every time, and these resources inside the control plane is is upgraded as well, then there won't be a problem. Then we have a problem when things go out of sync.

Daniele Polencic: because you have something, yeah, or you know. And then sometimes you have the problem. When you know you've got resources on your

Daniele Polencic: in your kidity.

Daniele Polencic: in your git repo, right? And it goes out to sync with control plane. So you upgrade the control plane. It's not able to accept. V. One, b, 2, one anymore. But if you want be to one, it's something that you have, and in your git repo. You try to apply. You see an error.

Daniele Polencic: That's that's what

Michael Petersen: you might not notice anything until you try to recreate it or upgrade it or update it or do anything save it. Okay, correct.

Michael Petersen: Alright. So next one, can we control individual elements if they can be synced versus managed outside of sync.

Michael Petersen: yeah, you can. You can specify what you want to sync in vcluster. If we're talking about vcluster.

Michael Petersen: you can. You can specify what you want to sync like, say, you don't want to sync ingress, you can turn that, you can just not enable it. Or if you do want to sync ingress. You can enable that to where it'll use the host clusters, ingress and other crds if you want to sync those so you can control what syncs and what doesn't

Daniele Polencic: in terms of like? Do you want to give access to what's running on the base cluster? Do you want them to install their own Crd's stuff like that. And and I think that's actually interesting. Part of the Ecluster is that you've got all these dials right? You can tune. And that's basically when you can shift your your isolation, you know, to be hard or or to be quicker. And I think this is where I think

Daniele Polencic: no, of all the options this this gives you, the more it's a more. I think it's a more flexible because it gives you this, this ability to tune. How you want. At least this is what II found. I think if I want we, you know, lighter than definitely something like, you know, capsules or namespaces. But this one basically.

Daniele Polencic: at least in my mind, we call this like a broader spectrum.

Michael Petersen: all right. Next question. Kind of similar. How are Crd's handled with the sync. So

Michael Petersen: you're you as the administrator are, gonna decide what syncs right? So you can. You can sync and give access to Crds on the host. Cluster up to the vcluster if you want to, that's gonna be, take some configuration, or there's some built ins, too, that you can just sync like ingress and stuff. But if you're talking about like custom C, or talking about Crds outside of just the the basic stuff, including kubernetes like Crossplane, or getting into that. It's gonna take a little bit of configuration if you want to share that up.

Michael Petersen: Otherwise your users can install Crds into their own virtual cluster because they have their own Api, that's separate from the base cluster Api. So if they want to use a different version, or if they want to use certain crds that that aren't available on the host cluster. They can create it there.

Michael Petersen: So how does it? How does it sync you just install it into the into the b cluster, and it'll sync down like it'll it'll figure out how to configure it on the Kubernetes cluster if you can reconcile it as long as as long as the versions aren't way too far apart, I guess. Right? Like we were talking about earlier.

Michael Petersen: Do you have anything to add, Daniel tech marketing. So I'm answer as much as I can. If I can't answer your question, I said, join join slack, and then some of our developers and stuff and and super technical people will be able to answer a little bit more than I can to. But I'm gonna try to answer before before people are leaving. We should just mention that you know this. This is the first installation of of the series. Next one next one is coming next week with Salman and yeah, I think if you haven't signed up, please do.

Daniele Polencic: And the next one.

Daniele Polencic: yeah, I think first to say it's very much more, you know, foundational on on namespaces. So you will see.

Daniele Polencic: See some some interesting stuff on how Qvc. Works under the hood.

Michael Petersen: So come back next time, too. Yeah, this is just kind of the the intro to to what's going on. And then you're gonna get a lot more

Michael Petersen: deeper stuff next time. Alright. So let me see. Answer that one. Let's go to the next one.

Michael Petersen: What about global resources? A lot of questions about Sir D's, what about global resources like Syrtis, if you sync them in the same way.

Michael Petersen: we don't have real isolation. Am I wrong? Well, so like I said, you can install Crds into the virtual cluster that are separate from the host cluster, or you can install them in the host, cluster and share them up. So it just depends on your configuration about how isolated you want everything, and how much.

Michael Petersen: how much permissions you want to give to your users to be able to install whatever whatever they want in their sort of their cluster, right? Because everyone's gonna wanna try a different version of something, or they're gonna try something new. And you don't wanna have

Michael Petersen: that on the base cluster everywhere if you can handle it. Cause and you, you're gonna have to manage it instead of them.

Daniele Polencic: Yeah, I think I think what's great is that you know all of these things with with something like an isolated control plane is basically you can. They are cheap to create and ship to destroy. Right?

Daniele Polencic: So we look. We had a look at costs, but I think the only way to have, like

Daniele Polencic: dedicated classroom, which is not a full cluster. It's basically to create a control plane. And I think this is the easiest option. Right? You just create. You test the crd and just util down. And it's a very cheap operation. And I think that's basically where you see the ease of you know, we talked about free access and isolation, easel management and cost, and then, ease of management you get, you get easy. It is easier to spin up clusters like we clustered and fully dedicated clusters

Michael Petersen: somewhat like speed, too. Like, it's a lot faster.

Daniele Polencic: Yeah, it's a lot less.

Daniele Polencic: Well, I think, Speed, you know, there are some providers that tend to focus on that other tooling. But I think just the simplicity of just going there and doing, you know, okay, let's create. Let's get a new control plane is just a pod. I think that's unbeatable. I think I think it. I think it's unbeatable.

Michael Petersen: Alright, so to the next one. So how does this differ? If you have multiple clusters?

Michael Petersen: And different teams and deploy on those clusters while controlling on argosy? So yeah, Argosd, great, great, great use case, you would just create the virtual cluster, and then you would some into the V cluster they might be just talking about in general, like what you, what would you do with a multi-tenancy? But, like with Vcluster, you can just get the kube config for the virtual cluster. Add it to Argosd, and then you would just treat like a regular Kubernetes cluster so you could deploy with helm on. There. You could even create virtual clusters through Argos, CD. Using the helm chart. So

Michael Petersen: you can. You can use vert. But the V cluster is gonna make it easier to where you can just

Michael Petersen: stand up a cluster quickly deploy something, test it, and then just delete it quickly, too. So you don't have to worry about the spin-up time.

Michael Petersen: Daniel. I don't know if you have something to add about just in general, like, what about using Argo?

Daniele Polencic: I think. I think to be honest, you know I presented the end, Kamada, as a way to sort of control all these all of these other clusters. But I think Argo is also one of his other tools that lets you sort of reach that level of of automation.

Daniele Polencic: But yeah, I think. And then what I was surprised to see is that you know, if I look at.

Daniele Polencic: If I look at Arguad, you also argue also integrates very well with the names, namespaces plus plus solutions as well as V cluster. So it's a very versatile tool when when it comes to, you know multi-tenancy. I think

Daniele Polencic: my worry is not about. I go as it would be like my my choice for for deploying and creating this tenants rather than you know what it does. What does this class look like like like? What is the multi-tenancy? Sort of configuration for this cluster. So I think that would be my, my first concern. I think I would sort of

Daniele Polencic: just a tick box. Yes, that. Yeah. Yeah.

Michael Petersen: But in general, if you're using anything like Argo or something like that, yes, it would. It would work with vcluster to where you're just creating quick ephemeral clusters that spin up in like a minute, and then you can do whatever you need, and then just delete them. But there are ways to hook it into there, too.

Michael Petersen: let's go to the next one. So how does vcluster work with managed Kubernetes solutions and their integrations? So aws, if you want to create.

Michael Petersen: I am roles being bound to the Eks cluster. Oh, I, DC Provider, as a long 100 and specific services for yeah. Aws access. It's a full project, or for

Michael Petersen: so is it still possible for V cluster managed clusters that don't have an aws Oidc provider. So like we've got a blog post, too, talking about how you set up storage and everything. So you need to set up different roles and access for this, for, like storage, so that you can expose Amazon storage like on eks or something like that. So

Michael Petersen: you're still gonna have to create some of these things for the roles and bind them. Vcluster is like a level up. And then I've only done the very basic stuff on that. So like for a more detailed answer on this might have to go to slack, but

Michael Petersen: I don't know if you've messed with the cluster on eks at all, Daniela, if you have any.

Daniele Polencic: No, but but I just do. I study the architecture quite a lot. And and I think that's like the class has got some design principles on how to develop these controllers and and control planes.

Daniele Polencic: And the the basic idea is that you know everything is simple, as simple as creating just a stateable set. Right? So this is why I think deploying the cluster is so versatile because it doesn't matter what climate cluster you have. You can just deploy as as a simple, you know, simple, stateful sense. And then from there you just create this nested clusters.

Daniele Polencic: I think what becomes a little bit more complex is this syncing mechanism? And that's basically where you know what Mike is saying about workload identities and integration with Eks. This is where it becomes a little bit more complex. But if you want just the vanilla stuff just deploying this, you know simple status that will be enough to create these isolated control planes.

Daniele Polencic: Yeah, for the for the workload identity. I think that's something you would need to go and check documentation, or someone more more expert expert than me.

Michael Petersen: Alright cool. So I got information, too, like the next 2 parts, march seventh, March fourteenth. So I was trying to get that posted in the chat, too, so that we have that information. All right, let's move on to next one. We got a bunch of questions. Isolated control plane can work with aws eks, so I'll say, be cluster. Yes, but you were talking about other options. Have you looked at the other options with

Daniele Polencic: is so isolated control planes? I think. So there is the cluster, and then there is

Daniele Polencic: what I've called, I think, is called V cluster as well. So the project from the official kubernetes sink

Daniele Polencic: which overlaps in in, in name as well as functionality. But I don't know how well it's maintained, but apart from that, I wasn't able to find anything which is exact

Daniele Polencic: replication of the cluster. So I think isolate. The control plane is part. I think the only one doing it.

Daniele Polencic: you know, in my research was was V cluster. I find other projects which can do control plane as a service. So you basically, you know, you have dedicated control plane as parts.

Daniele Polencic: But they cannot be used in inside bunch of services. So solutions like, hypershift or komachi, those can basically create this, you know. So so sort of same feeling as being clustered, but

Daniele Polencic: they sort of require a different setup. that's for sure

Michael Petersen: awesome. And then, on our side, too, on Loft V. Cluster pro will give you access to another kind of isolated control plane where you can deploy a control plane, and then you could tell it which clusters to deploy the workloads and stuff like that, too. So you got V cluster open source, and then V cluster pro, which is the enterprise version that gives you another step up on isolated control plane, where you're not running virtual cluster stuff on the on. The worker nodes. You're just running the

Michael Petersen: the work loads alright. So next question same person, any actual use case, use case for karmata. Do you have any any blog posts or anything that you want to share on that that you've

Daniele Polencic: yeah. So so I presented commodity and and and multiple, you know, multi region clusters. Last year. And then what I did, I basically deployed 3 clusters into a different regions, and I installed A/C on top. And then I shifted traffic from one region to the other, so you could basically request for an applications in London. You will get through a response from Singapore or you know San Francisco.

Daniele Polencic: and then you could also, you know, fight policies where you basically just, you know, prefer the local traffic and then move everything else.

Daniele Polencic: So I think, yeah, II can. I can find where the link is and and share it. But yeah, I tried. I think her mother

Daniele Polencic: is somehow in the same league as ages. CD. And my thinking is, should I use Argos City? Should I use commando? I don't know. At the time I was like more interested to explore this, this different tool, but I think everything you do will come out. I think you can also do real

Daniele Polencic: and august city.

Daniele Polencic: but that's what we use cannot afford only only for that

Michael Petersen: awesome looks like your answer was was good, so we got a thanks. Alright. So let's go to the next one.

Michael Petersen: How does V cluster work with server specials like Sto. Do I need to install it separately in each isolated environment, inside, in the same cluster? Or is there a way to have one installation, and all these like the question, right do I install SEO in my virtual V cluster, or do I install it on the base cluster, and do all that configuration. And I've seen both right like.

Michael Petersen: I think, oh, wow! But like so most for the most part, like, I asked this question on one of our previous

Michael Petersen: previous

Michael Petersen: cluster office hours. And the answer is like, it depends right? Like it depends on what you want to do. I will see. I would have thought that I would just install this to you on the base cluster. Right? But I don't know. It depends on what your use cases, I would say, on the base cluster, but I think we also have some information about how to install it into V cluster, if if you want. But oh, wow! I could be wrong

Michael Petersen: fairly. I can only imagine setting policies. Yeah, I don't. I don't know. I don't honestly know how inside of the the V cluster like I've never tried to do it inside the V cluster. I always do everything on the base cluster, so I don't know how that works like it's beyond me. So we have engineers that can answer some of those questions. I've always known the base cluster.

Michael Petersen: Let's enter another one. Does this apply to openshift or Kubernetes platforms? I think this was asked when you were talking about?

Michael Petersen: Are you just asking? I don't know if this is in general. If this is just in general about multicultural stuff. Yeah, you can do the stuff on openshift and other Kubernetes platforms like, I said, eks and stuff like that as long as it's kubernetes, and you can run a pod.

Michael Petersen: you can pretty much run vcluster unless there's like some

Michael Petersen: crazy thing excluding the ability to run stateful sets or so anything like that. Really, you should be able to run this

Michael Petersen: I don't know if you have other.

Daniele Polencic: If you have feedback, let me know, Daniel, or I'll keep going, because there's there's like 20 questions.

Michael Petersen: all right. So we shared about teams. My question is related to different environments. Let's say, dev stage test prod clusters. How should we approach that?

Michael Petersen: If we're also on aws cloud with first and eks? Second, use case. Let's say, Ka, opposite Cuba. So I'll let you answer this first. So what do you think for like different environments for devs, test test, test, test, test, test, test, stage and prod like, what are you using? Yeah, I think that's a good question, because

Daniele Polencic: I think I think when I think multi multi-tenancy is, you know, a way to separate teams, you know, to separate what they do or separate projects. But I can also see, you know. I also see that some of the clients we work with they use these separate clusters with different for lower environments, right? So we're talking about their staging. QA.

Daniele Polencic: And you know, 1 one of the things, at least in my mind is when I when I tried to figure out how many environments how many teams is, you know the cost of managing all of those, all of those environments.

Daniele Polencic: So I think what I've seen most common is companies doing like lower environments from a single cluster and just sharing it.

Daniele Polencic: and then having, like production, etc. Cluster.

Daniele Polencic: And and when they have this, you know, single cluster, then they use, they define multiple, you know, way to to sort of slicing and dice. It cluster for for multiple tenants. So you know, solution like V cluster, for example, or solution with namespaces depending on on the isolation appetite you've got and would be, would be an option to to create lower environments for different teams.

Daniele Polencic: I think that that could be an option. But again, it goes back to okay. What is my my appetite for managing a lot of clusters and reusing resources? You know what is going to be the costs, how much effort. So it's always, you know, a trade off. And so really, you need to keep buying something to give and and substitute. So

Daniele Polencic: Eddie pines.

Michael Petersen: Yeah, as I say, the cost, the cost is probably the most thing that depends. Oh, cost in the scale, or probably the the cost of scale. Yeah, yeah, it's like, if if I if I if you're just starting out and you've got like a couple of things running in prod, and it doesn't matter that you've got other stuff running like. And you can share

Michael Petersen: the cluster. Yeah, run it all in the same spot like like you're saying, like, maybe split off dev dev test and stuff, and maybe have like dev test and a stage in one cluster. And then you can use Vcluster to do all of those in the same thing.

Michael Petersen: and then maybe your prod's moved off somewhere else. It's a little bit more important to you. But you can use vcluster for each of these things that you're talking about. We're talking about like splitting it up. So there are people using B cluster for dev stage test. And there are people using it for prod, too. We have a.

Michael Petersen: there's actually a video from right before kubecon that's talking about someone using vcluster in production core weave they're using. They're using it for for Gpu access and stuff like that, too. And that's vcluster pro. So you can check that out.

Michael Petersen: All right. Answer that one. Let's go. Is it possible to have different customers from different organizations? Share a cluster of full isolation?

Michael Petersen: Do you think vcluster may be the solution for this? So I mean, since I'm answering the first, let me let you answer, and then I'll give the V cluster spin alright like. So you can talk about like everything. Yeah.

Daniele Polencic: Don't know. It depends how how much you trust those those user. I think it all comes down to, you know we can trade offs, and

Daniele Polencic: you're willing to accept.

Daniele Polencic: And then sometimes you've got like, you know, you might choose

Daniele Polencic: you know. Let's say that your concern is that someone is going to deploy an application which is going to I don't know. It's gonna have.

Daniele Polencic: you know, could escape the node. I don't know it's gonna escape the container, but then maybe the medication for that is to actually use a sandbox. And but Runtime, such as givizer.

Daniele Polencic: or or something like cata containers, so that could mitigate that that part.

Daniele Polencic: So maybe you you're concerned that the applications that you deploy are very I/O intensive, and like Kafka, for example.

Daniele Polencic: and you don't want that to be sort of sort of running in the same same node as some other I/O-intensive I/O-intensive apps, like databases, for example.

Daniele Polencic: and a good option for that would be just having different set and different note pools, and using don affinity and tainted tolerations.

Daniele Polencic: And maybe you have all of these problems at once, right? And you can start the solution one on top of each other, until you get the sort of isolation that you really want.

Daniele Polencic: But of course, the more things you stuck on top of each other, the more you know the complexity and the costs will go up as well.

Daniele Polencic:  so so I think it's always. it's always, you know, going back to what is actually your requirement. What kind of problem are you trying to solve? Do I actually trust these users? What kind of things could go wrong? So you sort of have a profile for

Daniele Polencic: sort of risks that you're accepting. And then I guess these risks, then you can plan for the level the right level of multi-tenancy which is going to give you the right level of isolation and security. At least, this is how would approach the problem.

Michael Petersen: So like the answer basically is technically, yes, you can do this. You can use no groups and stuff like that. The the stipulation is that your security? You've got lockdown. And you understand, like, how you're gonna secure this and make sure that, like nothing escapes but in terms of like data, privacy and stuff like that, or making sure that people have their own clusters versus our their own actual physical nodes, or something like that versus having a shared environment. Yes, you can. You can use no groups, and you can specify. You know what V cluster is allowed to see and allowed to access, and where certain customers are allowed to deploy stuff.

Michael Petersen: And there are production. There are people using vcluster in production to do this thing that you're asking, which is like a multiple customer sass option. And they need to install this software to service in a certain location that only that customer has access to. So it is possible

Michael Petersen: technically, it might take some work on your side to figure out security, and then also, after you get it set up and installed, maybe do like a a dry run, and and make sure that it is secure, and you don't run into some some of the other issues. All right. So next one.

Michael Petersen: what do you recommend

Michael Petersen: for multiple tools to manage Kubernetes cluster or have a dedicated cluster for each team. Okay? So

Michael Petersen: alright. So yeah, I guess they're saying, what do you recommend between the the 2 that you're talking about having a single cluster or dedicated dedicated cluster for each team. But instead of being each user, we're talking about team based.

Daniele Polencic: if possible, I like to keep things as simple as possible. So I would. My first choice would be okay. Can I solve this with namespace namespaces?

Daniele Polencic: No, okay. Can I solve these, these. this problem with namespaces plus plus. So you know, this capsule or hierarchical namespace controller? No, okay, then I move up. They clustered. Then I move up dedicated clusters. So I think

Daniele Polencic: if I can, I just I just stay with the lowest complexities possible, because that is going to keep me also the lowest costs. But that's.

Daniele Polencic: of course, not always possible. So that's why I would basically just gradually increment

Daniele Polencic: and move on the ladder. At least, that's what I would do.

Michael Petersen: It's it's a loaded question, too, because what if these teams are one teams in Europe and one teams in the Us. Then? Yeah, you might have to do something like, give each one their own like dedicated cluster. And for data, privacy rules, yeah, like.

Daniele Polencic: yeah, it could be, it could be, ma. Maybe I don't have a choice. Maybe I don't have a choice, and just I could just have you know I'm not. I must have like one dedicated cluster per region.

Daniele Polencic: And that's very. And that's it.

Michael Petersen: We got a replied, your thing, too. Yeah. They like the idea of start it the easiest and then kind of increment up to see what kind of stuff you need to use. You need to do a presentation about just that, like, how how to think about that from the beginning, when you're setting up kubernetes like, what are your kind of what you showed us here? But do you have one that's just like completely dedicated to

Michael Petersen: every single time I started. But Kubernetes cluster, what are the exact steps I saw? You had the software, too, that you're talking about. But like. is there someone that can watch more information about that

Michael Petersen: like beyond this like, do you have like like, or do you? If someone comes in, asks you, do you have anything else beyond what you've presented here that you think about, cause the idea of starting at the easiest and then moving up namespace plus and beyond that like, where do you?

Michael Petersen: Where do you live? Done?

Daniele Polencic: I don't. I don't. To be honest, I don't. I don't know. I? Yeah, I don't think I've got.

Daniele Polencic: You know, I tend to

Daniele Polencic: just just focus on, you know, on on what I've got at the hand and just build on top. So I don't. I don't think I've got any abstract question. What do you think about software like Gardner or classics?

Daniele Polencic: So I haven't used gardening, and I'm not. I'm not familiar with it. So like, no, I cannot comment on plastic. So they've got 2 products, Komachi and Capsule. I think capture is part of these namespaces plus plus product. So this and

Daniele Polencic: this product, that augment namespaces.

Daniele Polencic: And I think it's actually part of this lightweight, too. So if you're looking for Lightwood, Tennessee, so soft multi-tenancy, I think it's actually a valid product. And maybe, you know, this is just personal preference. But maybe between the hierarchical namespace controller and capture. I would actually pick capsule

Daniele Polencic:  And I explained, you know some of

Daniele Polencic: I explained this. I back sort of proxy, which lists all the namespaces. So they sort of thought about the experience. So they understand what their users need. And I think the Iraqi controller still still is doing it, but not at the same scale, so I think I will. Probably

Daniele Polencic: I will properly look at that. And then they've got another product called kemagi, which is control plane as a service which is also part of what vcluster protons.

Daniele Polencic: And I think for that one is is a little bit different. I think the use case is is much narrower.

Daniele Polencic: so I think only if you, I would imagine that not many, not many companies can can attempt on stuff like that, at least in my experience, at least based on what I understood about a project.

Michael Petersen:  awesome. Yeah, I haven't used either, so I don't. I don't know. I can't add anything, but I'll go to the next one side of very best basic question. You know, what is vcluster? Is it industry approved? Or is it something by loft? So it's by loft.

Michael Petersen: We have other projects that are actually part of the Cnc F. Dev. Space. But

Daniele Polencic: but Josh, just wanna add, Mike, I think, is is important, at least for me. Is that all of the tools I presented are open source. So you do look any hierarchical namespace controller. You look at the cluster you look at command, you can just download Gpio.

Daniele Polencic: and you can get going.

Daniele Polencic: So while yes, it is a love product lost offers a premium version. Then.

Daniele Polencic: yeah, it. You can still just download it and use it. There is nothing stopping you from doing that.

Michael Petersen: I guess. Let me differentiate between that. Yeah. So we have V cluster, open source and then vcluster pro. So the open source one is actually something that's been presented at Mini kube cons, like, there are people out there that are using this in addition to other

Michael Petersen: industry approved. Or you want to call like stuff that's in the Cncf. Or in the What Ci CD. Or something like, or the Ci CD

Michael Petersen: group, whatever the other split-off one that's got like Argo or some other stuff anyways.

Michael Petersen: yes, kind of. I mean, it's something by loft. It's open source. You can use it. There are people using it in the Cncf.

Michael Petersen: yeah, I'd say it's industry approved. I mean, until until there's something that's very similar, which there's not. There's some similar things. But I think a lot of the presentations and stuff at kube Connor about this.

Michael Petersen: All right, let's see, how do we subscribe for the next presentation? So I think there were links in the chat, and if you go to one of those links for the series you should be able to sign up. So we got workshop series Part 2 on the seventh and then on the fourteenth

Michael Petersen: alright. And then I think maybe the follow up email, too, may have information about the next ones that are coming out all right. Should you define resource quotas per namespace in a multi-tenant environment where each tenant is on its own namespace. If so, what is the best approach to define quotas?

Michael Petersen: So I'll go to a good evening, and then I'll start. I'll start doing the the first one again next time.

Daniele Polencic: So which one? I answered the quotas, yeah, the quotas? Yeah. So this is the I already did the presentation one. So okay, okay.

Daniele Polencic: I got so so generally with quotas, I think the way the way I've seen quotas and the way I think quotas should be done. And this is not saying that this is the exact way, you know. You probably can take these and change it, or you have different ideas.

Daniele Polencic: But quotas are generally linked to costs.

Daniele Polencic: Right? So when you, when you, when you manage a multi-tenant cluster, then somehow you need to rip how you know you need to divide these costs and assign into teams and team will have some budgeting, you know, for how many resources they can run. So we're talking about real money right at the end of the day. These clusters and and notes isn't cheap, right? And we need to find a way to actually divide the the usage and and make sure that teams are paying proportionally right? Based on the usage that they have.

Daniele Polencic: I think the

Daniele Polencic: the idea is that if if you're gonna if you're gonna charge back your users for for the resources they're gonna use, then the simplest thing you can do is to actually define, you know, a small medium, large quotas.

Daniele Polencic: right? Assign a price to it. a price that is going to be paid by by the team.

Daniele Polencic: and then and then just give these quotas to

Daniele Polencic: to the teams. So in such a way that you know the the quota is linked to a monetary value, and

Daniele Polencic: and teams can make it bigger, but you also need to pay more. So there is an incentive to be efficient. That's that's basically how it would link money to quotas. at least in my experience.

Michael Petersen: There's tools out there, too. They're getting better with Chargeback, like kube costs, and all that, too. Right like. Say, if you're in a government contract, and you have to know exactly how much you're paying for everything because you have to do it at the end of the month, for, like your internal research and development, or something you have kube costs, which is gonna help you a little bit with actual usage. But yeah, I like the idea. I like that idea of actually just having small medium large, because then you're not monitoring it all the time and having to like, get all these metrics. And then you just say, Okay, this is. And then it's on them. But they're like, Hey, we're not even using medium anymore. We're just using small. Let's go ahead and switch over. Then you don't have to worry about anything on your team.

Daniele Polencic: Yeah.

Michael Petersen: okay? And then, so I mean, okay, so how do we just establish connectivity between Vcluster and the Kubernetes control plane. So I'm not sure on the question, for that, like the the sinker might be. What you're thinking of is like, how does Vcluster get a resource and actually create it on the base cluster? So it's gonna have the sinker. So

Michael Petersen: you're gonna create something in vcluster. And it's gonna actually still tell the base cluster, hey, create this thing for me! And then you're still using the same reconciliation loop. And the controllers of the

Michael Petersen: the base cluster.

Michael Petersen: I don't know if you want to have. If you have anything to add, and you're like I might move on to the next one. But I think that's what they're talking about the connectivity between the 2. Yeah, let me go. One more time. Let's talk about ingress. If you're talking about, how do I access my V cluster in?

Michael Petersen: You know the Kubernetes? I don't think so, because they said the control plane, but also ingress. You can access vcluster through an ingress resource, because it's just another thing that lives in Kubernetes.

Michael Petersen: Alright, how to isolate multiple tenants in the same cluster. Yeah, that's kind of what we're presenting on. Or Danielle was presenting on. you would use. You could do something like vcluster or some of the other options. You can isolate them based on nodes. Or you can just give them basic isolation by namespace. So I guess basically, the presentation was kind of about that so

Michael Petersen: hopefully. Maybe go back and re-watch the presentation unless you have something you want to share with recording, anyway. So yeah.

Michael Petersen: we're sure the recording sinker rewatch all right. So

Michael Petersen: which option is best for performance, matters having a shared ingress or date. How's it going? Having a shared ingress or dedicated ingress per team when using a load balancer in front of ingress like ha proxy. So what's

Daniele Polencic: yeah performance? That's really good question up for performance. That will basically be dedicated classes, but

Daniele Polencic: dedicated ingress controllers. But cost.

Daniele Polencic: you know, this is, gonna be interesting to say the less.

Daniele Polencic: So the idea being like you have little bouncers in front of each of these right like, unless you can.

Michael Petersen: You're managing your own load, Bouncer. For the most part you're gonna buy a load balancer. And you're gonna have a public IP address. And that's gonna be in front of every single ingress controller. Right?

Daniele Polencic: I mean that that's you know performance. But I

Daniele Polencic: but yeah, the cost is gonna be problematic.

Daniele Polencic: that's

Daniele Polencic: and and to be honest, sometimes you have to buy the bullet. There's nothing else you could do

Daniele Polencic: That's just the way it is.

Michael Petersen: You say you're not gonna one off, but then you're one off

Michael Petersen: 50 times, and then the left, because the norm.

Michael Petersen: So yeah, so I guess best basic answer is, depends on what you pay for. The performance is going to be dedicated. But that's going to come with the cost of a load balancer per ingress and having multiple ingress resources, ingress controllers that are running in your cluster. So now you're paying for that workload, too.

Michael Petersen: Instead of just having a single

Michael Petersen: alright. So let's go. Can you give an idea? Oh. can you give an idea? At what scale.

Michael Petersen: Would Kermata potentially make sense?

Daniele Polencic: To be honest? It's a good question. I asked myself the same question, and and the answer is.

Daniele Polencic: I really don't know. I really don't know. And and the reason is, I think I would use Kamata when I'm forced to use separate clusters.

Daniele Polencic: and I'm forced to use separate clusters, and I think the reason why I would use separate clusters is because one regulatory reasons.

Daniele Polencic: right and and and 2 is that

Daniele Polencic: you know some some sort of.

Daniele Polencic: you know, data protection. So it goes back to like requirements that are coming from the top that I'm not able to change right? I'm not able to

Daniele Polencic: work around, so that would be like my thinking

Michael Petersen: alright. And I haven't used it. So I'm not sure. All right. So the next question is, does this use case apply to openshift tanzu and rancher? And I would say I'd say, Yes, right openshift. I think I don't know how the licensing is now, so don't quote me on this. The last time I touched openshift, which is like 5 years ago, it was like a

Michael Petersen: I think it was like a license per per node. So if you can reduce yeah the amount of nodes, or in cluster, right? So you've got the cluster and the nodes. So if you can reduce how many you actually need to deploy, and how many nodes you need to deploy? I mean nodes being workload based. That's a little bit different. But if you can reduce the amount of open shift instances that you need to deploy and then share that among many different teams, yes, you can do that right. You're gonna give them access to it over

Michael Petersen: like coop. Ctl, I don't know how the Oc. Or Openshift Command line stuff integrates with that. You're just probably gonna interact with it like it's a Kubernetes cluster. Maybe you don't get all the stuff that's above Tanzu and then rancher, rancher. Same thing, rancher. You're just deploying the cluster, and then you wouldn't. You would install vcluster on top of the cluster itself. It's a helm install. So it's like you're deploying an application into Kubernetes. But then you just give your users access to that instead.

Michael Petersen: I think that kind of interests it, and I don't know if you have.

Michael Petersen: Alright, let's go. The next one is there slack channel? Can we get a link for the same? Yes, so slack channel. I've been posted in chat, slack dot loft. Sh.

Michael Petersen: You can join there and then go to the cluster channel and ask questions, or

Michael Petersen:  I mean, there's a bunch of other channels in the street. Alright. So next question, Hi, all, can we have

Michael Petersen: specific cnis like cilium?

Michael Petersen: On the tenet cluster running in a hub of cluster to be cluster so you can use cilium. I haven't.

Michael Petersen: So this might be a better question for you. I haven't installed clymium and used it with vcluster, right? So when you get to networking and stuff like that, too, it gets a little bit little bit interesting. I believe it works. I can't say that it works as in. I've tested it, but I believe there are people that are using it. And it it should be fine. Here's your your chances.

Daniele Polencic: Yeah, I think celium is installed in the notes right? So it can only run inside. A host cluster cannot run inside the tenant.

Daniele Polencic: And so I think.

Daniele Polencic: yeah, I think I think what you might be interested in is maybe running the

Daniele Polencic:  the network policies. And I think those can be created inside the the 10 control plane.

Daniele Polencic: But but then there is something called

Daniele Polencic: the senior network policies, which are a global network policy. And that's the Crd.

Daniele Polencic: and I don't know how that would work, because you would need to install the CRD. In the tenant.

Daniele Polencic: But you need to store celium in the host. Yeah.

Daniele Polencic: So I don't know how the installation will work. And then you need to basically configure a V cluster to copy that particular CID across. That will be the end-to-end solution.

Michael Petersen: So as as a tenant. No, probably not that. Easy as an admin. There's a lot of configuration right? Like I've we've had this question before, as I don't know I haven't installed it like you would use something like a generic Crd Sinker to share the resources up if you needed to. But then the administrator's still installing all the stuff in the base cluster, especially when it goes a little bit deeper than ingress or or something like that.

Michael Petersen: Alright, let's see. let's see.

Michael Petersen: what do you think about? Site 24 by 7. I haven't heard of that one?

Michael Petersen: Yeah, I'm not sure. Okay. So we'll say, we enter these live. I'm not positive about that. If you have more

Michael Petersen: to expand on that maybe put it in chat or ask question at expanding all the work question around application performance.

Michael Petersen: So I generally understand that running an application in Kubernetes has some level of performance degradation when compared to running on bare metal.

Michael Petersen: Correct me if I'm wrong enforcing V cluster, would it add another layer of performance degradation as it's running another kube inside kube. If I understand it correctly, does it have a performance? Degradation? So I'll answer it first, and then I'll send it over to you.

Michael Petersen: This is like, it's the cluster's abstraction. It's not virtualization, really. It's kind of abstracting your stuff still running in the base cluster. The performance that you degradation that that you would actually see would only be like

Michael Petersen: the pod that it takes to run vcluster. So the cluster resources that are running core dns and and stuff like that.

Michael Petersen: Then the application itself is actually running on the base cluster. When I create it, you're gonna be able to see as the as the admin, that it's running on the base cluster, and then other resources like ingress and stuff like that that I create

Michael Petersen: for the most part of running on the base cluster. So we're just using kubernetes

Michael Petersen: adding another Api, and then just deploying stuff still on the base cluster. So there's no, there's not a level up. It's just you're adding a couple of pods for for V cluster on top of what you would normally be running. So the workload stuff. Maybe if you've got like a hundred V clusters on a single single node, then you gotta be thinking about like, okay, I'm running like a hundred more pods or something like that. Then I then I would be running if I was just running on base cluster.

Michael Petersen: Daniel. I don't know if you want, if you have, that's a really good answer. That's a really good answer. Yes, exactly.

Michael Petersen: I'll say upsell on my side, so like something like V cluster. Pro takes the the V cluster workloads that are running on. The worker nodes and it removes those. And you basically have a control plane outside of the worker nodes, and what it does is it'll run. It'll run the workloads there instead. So then you have even less pods that are running for V cluster on the node. So really important, if you're doing stuff like Gpus, or something like that, and you want to use every bit that you can that you can use. And maybe you're running at 80% plus

Michael Petersen: optimization. I don't know.

Michael Petersen: Alright. So let's go to the next one. So we have 33 eks clusters across the globe for different entities. The Canmada manage them better than Argo, like upgrade and everything.

Daniele Polencic: If you have, if I have a quote, just stick with the heck. Yeah, I think there is. No.

Daniele Polencic: I think they are the same.

Daniele Polencic: You know that I think I would see. I go will probably be more feature rich, so we'll probably look into. I go. But I think Kirmada and I gotta, you know, fulfill

Daniele Polencic: for what I'm using them forward. There is a fair amount of overlap.

Michael Petersen: Okay, awesome. And looks like. But there's our last question, for now, unless someone queues up another one. So

Michael Petersen:  oh, this is just saying someone's just saying they need to run

Michael Petersen: alright. So if you have any other questions like post em up like I'll do. I'll take some information about like vcluster and vcluster, pro vcluster com. If you want to check out vcluster

Michael Petersen: lot of stuff in the open source that you can do. There's a lot of documentation and different things out there and use cases using vcluster open source.

Michael Petersen: recently we launched vcluster pro. It's the Enterprise version. It's got a Ui. It's got a bunch of other stuff built on it like embedded at CD and and other cool features that make it a little bit easier to get to day one self service integration with Argo, CD. To where, when you create a V cluster, it'll automatically get added to Argo. But if you're interested in vcluster pro, and you want to see just a demo just to see what it is, or if you're interested in vcluster Head over to the slack. Ask on there, go to vcluster.pro if you wanna

Michael Petersen: sign up for a demo and see what it looks like plus, we also have videos on our Youtube channel. I believe that this this video here is gonna upload it to our Youtube channels to be able to check out some of the V cluster pro stuff there as well.

Michael Petersen: But I think

Michael Petersen: I think we're good on questions. I mean, let me look in chat, just to make sure that something didn't like get get added over there. A lot of people having to drop off. They'll check it out later. Thanks for the presentation and all that. We'll see. Danielle. Thank you so much for the presentation. Thank you for having me very informative. Can't wait till the the next one, I think, is the next one

Michael Petersen: next week. More information next week. Alright. And then we're following up after that.

Daniele Polencic: Yeah, alright cool.

Michael Petersen: I don't think think we have anything else. Do you have anything you wanna say before? I guess we head out or

Daniele Polencic: no, I think that's pretty much it. I very much enjoyed. I spent a fair amount of time researching, and hopefully I hope it was useful.

Daniele Polencic: That's it.

Michael Petersen: Awesome. I think I think it was. We got so many questions. So I think people were engaged. Yeah, thanks. Thanks. So much for watching. Like I said, you can watch this again later. And then join us for the next one, too. I think the the next one's gonna go a little bit deeper into some of the questions we got here. Maybe, I answered, well, hope I mean some of them.

Michael Petersen: So yeah, if you have more questions beyond what what I answered, or you want a little bit more information on it, let us know. But thank you everyone for for joining

Michael Petersen: and thank you. I think we're good.

‍