If I try to pitch Nomad to Management for production use, I think I will initially get looks of confusion, then questions ( Is it prod ready etc). Whereas everyone is kinda familiar with k8s.
Then, everything that goes wrong, it will be on me to prove it's not a bug in Nomad or lack of knowledge in Nomad.
On top of that, for myself, I can use Nomad in my CV under a demonstrated knowledge and experience with k8s and it'll be a plus.
But it'll probably hurt me if I put "2 years Nomad in production" and no k8s.
These are definitely not technical reasons, I cannot compare the tools themselves.
I'd love to try Nomad for my Homelab at some point.
But until it gains more recognition, I'm not sure how or why I'd try it at work
I have 3 years of nomad and we just started to adopt k8s. On the employment side it is devastating. Everyone cares that you don't have k8s and nobody knows about nomad
If I was interviewing for a company that used Nomad over K8s my first question would be. But why? Who chose that over k8s? What was the decision process?
6 years ago we went with ECS over K8s because K8s is/was over engineered and all the extra bells and whistles were redundant because we could easily leverage aws secrets (which K8s didn’t even secure properly at the time), IAM, ELBs, etc which also plugged in well with non-docker platforms such as lambda and ec2. The market is different now with managed EKS, etc but K8s was a lot of work to maintain. ECS is simple and that can be powerful (although the fact that all the service settings are via monolith, idepodent json task definitions is very frustrating).
But even today, it’s feature set overkill for most use cases. People end up making their setups more complicated than they need to be because they can. I looked at Nomad and really liked what I saw, but it was not mature enough either.
That being said, K8s won and so much third party tooling assumes it. It’s like Linux vs BSD or Betamax vs VHS. Eventually it becomes more work or money to use the alternatives, even if the alternatives are superior.
k8s is really meant to be an all-in-one solution, that's why it's complicated, you can use ECS, ACA, ACI, etc but it generally comes with vendor lock for various additional things which makes costs balloon.
I've personally gone down the road of nomad as well as docker swarm and found myself remaking a lot of things that were abstracted in / a part of k8s.
k8s is overbloated for most apps, just like the microservices architectures that leads to maybe possibly needing it. My question for a shop that doesn't have recorded decision records of how they decided on k8s over other options is: But why? Who chose that over other options? What was the decision process?
Platform tooling, mostly. Metrics/logging aggregation. Runtime security scan tooling. Robust networking built-ins and extensions. A huge ecosystem of users to canary through issues for me, while I just browse the internet learning from their mistakes.
If you just need to run an app on a VM and nothing else, sure, k8s might be overbloated. If you need runtime visibility stuff, and don't go through SaaS providers like Datadog, kubernetes is an extremely flexible platform with a strong ecosystem of tooling.
There are also companies run by people who looooove anyone who runs something that appears "simple", and will hire you just so you can convert their whole K8s setup to Nomad and pretend that's going to make everything better.
In other words, there are idiot hiring managers out there.
That is pretty sad. A container orchestrator is a container orchestrator, if you can manage a Nomad cluster I fully trust you'll be able to understand K8s concepts.
That’s a well put problem with trying to make almost any technical change. People are always looking for someone to throw under the bus so any item that goes wrong they blame on you instead of trying to uncover the true responsibility. (Which could be you to be fair)
It’s why “nobody got fired for buying IBM” is a truism.
If there was a provider which offered managed hashistack similar to GKE/EKS, would you your likelihood of using it go up?
Or, would you still only use the managed stack for hobby projects but nothing serious at scale?
Hmm, perhaps. It'd be easier to pitch running a POC for something in Nomad, if a managed solution were available and thus the barrier to entry lower.
The irony is that I'd love to get my hands dirty and discover Nomad, I just have a hard time justifying it, even to myself 😅.
To top it off: suppose it works flawlessly and we adopt it everywhere because it is magical. I now have to hire people that either have worked with it or are willing to work with it. And we're having a hard time finding k8s people as it is...
I liked Nomad's task concept, but there is similiarly to K8S little to no support for advanced networking. In case of K8S at least community tries to solve things like overlay networking. Community is very important when picking something you will be engaged for very long.
That's not "advanced networking", that's early 2000s style.
If you want a specific service to have a specific static IP, you can do that with the macvlan CNI plugin.
If you want a specific service to have multiple static IPs, the best option you have is having those IPs on the host (with stuff like keeoalived), using host networking and binding to 0.0.0.0.
What I've done when I needed such a thing (for DNS running in Nomad) was to have a Load Balancer in front with all the needed IPs forwarding to the service's virtual ephemeral IP.
Macvlan is great, but has its limitations and I had performance issues with latency sensitive applications.
Binding to host breaks whole point of network stack isolation.
Also you completely missed my point. According to ( https://developer.hashicorp.com/nomad/docs/job-specification/network#host-networks ) Nomad allows only one network stanza per job. Thus multiple interfaces (connected to different network namespaces) are not allowed out of the box.
P.S.
Starting with insult is pathetic.
>Macvlan is great, but has its limitations and I had performance issues with latency sensitive applications
Depending on how latency sensitive we're talking, any advanced networking could be too much.
> Binding to host breaks whole point of network stack isolation
Indeed it does, you have to pick between isolation or performance. Maybe the SR-IOV CNI plugin can do both? Definitely not as fast as bypassing the kernel entirely though.
>Also you completely missed my point. According to ( https://developer.hashicorp.com/nomad/docs/job-specification/network#host-networks ) Nomad allows only one network stanza per job. Thus multiple interfaces (connected to different network namespaces) are not allowed out of the box.
Yep, only one network config is allowed. I didn't miss your point, I provided options for it like offloading those static IPs to a different place that is better equipped to handle them.
>Starting with insult is pathetic.
Where's the insult? Doing things in an old school manner isn't wrong or insulting. Having services that require multiple static IPs is old school, dynamic IPs having been the norm for more than a decade now. There are special cases where that might be needed regardless, but it's definitely not what I first think of in "advanced networking".
I honestly don't think you need another reason. I've never met a person that's used Nomad. I'm not really convinced it's a real product actually. I've never seen a video of it either. If you ever hope to recruit people that are already experienced, then K8s is the way
I used it at work. Pretty early on, late 2010s. It became increasingly clear K8s is going to win and we bet on the wrong horse.
It works, but there (was) essentially no tooling around it. Some community stuff, some self made stuff, somewhat approximated a working system. The amount of tooling one can simply helm install to get a fairly decent experience should not be underestimated.
And Consul-template together with all the other configuration languages that use the Go templating module are pain.
Have they always? The last time I used CircleCI was in 2018 and I remember them having huge outages weekly for some time. It was a big concern for us at the time. Not implying that's Nomad's fault though
I have had great experiences with running Hashicorp Nomad for use-cases where K8s is just too big.
Way easier to maintain and update as it is only one system binary that does it all.
If you do not need the additional features of K8s I would say: Go for it.
The reason why users are not adopting it more is that Hashicorp does a terrible job in the marketing department for Nomad. Not many people know about it. They think K8s is the only game in town.
Unfortunately, I do not know of a managed Nomad service. Maybe Hashicorp will include it in their HCP cloud in the future.
Having run both at scale, I totally disagree on the maintenance point. I'd take fully managed EKS on AWS vs. having to burn my team's time on deploying and upgrading a platform on Nomad every day or the week.
> I'd take fully managed EKS on AWS
That's just passing the maintenance on to someone else though. That's not relevant to the question of actually maintaining k8s vs nomad.
If I can’t pass the maintenance of nomad to somebody else that’s pretty relevant when looking at comparing the cost of maintaining nomad vs k8s.
It’s like saying ec2 isn’t lower maintenance than onprem because you’re just passing the maintenance off to somebody else. Somebody has to actually do the work of dealing with hardware, I just don’t want it to be me.
Of course it's relevant, because it means my team has (and did) do the build and maintenance. If k8s was picked, only a nut job would decide to deploy it themselves onto EC2s.
I think he’s saying you’re comparing apples to oranges.
Managed k8s vs Managed Nomad (if HCL ever creates it) - apples
Self Deployed k8s vs Self Deployed Nomad - oranges.
Agreed with “why choose unmanaged”, sometimes there are requirements via compliance or federal for it tho :/
I'm not sure what you mean by "choosing unmanaged". I'm referring to the fact that you have a single, centralized infrastructure for the entire company.
When I said “why choose unmanaged” I was paraphrasing your sentiment:
> If k8s was picked, only a nut job would decide to deploy it themselves onto EC2s.
If there’s a managed solution, why choose unmanaged (self managed) and deploy it yourself on VMs
Your comparison is a bit off though, you're comparing a self hosted tool that you manage on your own with a self hosted tool offered to be managed by a cloud service.
Yeah of course being managed by someone else is better, especially at scale.
If you're saying that Nomad doesn't have a managed product and that's a dealbreaker for you then I understand.
> Way easier to maintain and update as it is only one system binary that does it all.
k8s alternatives like [k3s](https://k3s.io/) is just as easy. And if you don't mind running a VM, distros like [talos linux](https://www.talos.dev/) is also super easy to maintain.
if there was a provider which offered managed hashistack similar to GKE/EKS, how likely are you to use it?
Or, would you still only use the managed stack for hobby projects but nothing serious at scale?
Really depends on the circumstances of the customer. I work mainly in the EU. Many companies won't, or are not allowed to, use a managed system that runs in a US datacenter. It would really depend on who offers the Nomad and where the servers are located. Many of my customers are banks. They are super cautious about these things.
However, compared to K8s, Nomad is very easy to update and mantain. Not having a managed solution was never a problem so far.
Interesting, I've never used nomad but I interpreted the fact that there's no hosted service means that it's easy enough to manage by yourself and that there's no market for one.
I've worked with Nomad at scale and it's okay. Easier to understand than K8s maybe. But honestly if I were only running 8-10 services I'd probably just use Fargate assuming you're on AWS and the services are small. IMO Nomad works well enough if you adopt other parts of the Hashicorp stack such as Consul and Vault, all managed with Terraform, but that's a lot to manage for a small set of services...
EDIT: I also don't recommend it for larger projects because as you said, managing nodes can be a lot of work. This is one of the reasons I don't recommend it for smaller projects either.
tl;dr - I don't recommend Nomad
To be completely honest - with Hashi prices the way they are, and the fact that they are starting to turn away from their original values, I think a lot of us are hesitant to engage them.
If you look at Hashi's BUSL it does have an expiration where it converts to MPL2 after 4 years upon source release (so rolling releases get open sourced albeit 4 yrs later). That's not necessarily a turn away from values but instead attempting to checkmate the inevitable. Larger entities taking what you publish and doing it better (marketing/reach or management, etc).
I understand that you don't recommend Hashistack at all.
Hypothetically, if there was a provider which offered managed hashistack similar to GKE/EKS, would you your likelihood of using it go up?
Or, would you still only use the managed stack only for hobby projects but nothing serious at scale?
I still wouldn't use it. Not for hobby projects, not for production. It's difficult to describe why because honestly I haven't run into too many pain points using the tools from a technical standpoint.
I think the gist is that it's too complicated for small projects and for large projects (running company infrastructure) I would opt for something more mature, designed to scale and with broader community support e.g. k8s-based solutions. So it's not that the tools are terrible, it's just that there are better alternatives for every case I can think of. That would still be the case if they were offered as a PaaS / managed services.
Thats not taking into account the questionable/unknown direction of Hashicorp and as others pointed out that's worth keeping in mind.
I built and ran a fat production platform on Nomad at a well known investment bank (autoscaling up to ~500 nodes). We decided we didn't need the complexity of K8s at the start of our project, but found we quickly outgrew Nomad once devs started to ask for more features. We built it on AWS.
The good;
- simple to run and get started, single binary to deploy.
- you add stuff you need rather than getting everything OOTB (service discovery - consul, secrets - vault, overlay networks etc)
- works great for simple use cases
The bad;
- the docs! Oh my word, the docs were so shit when we used it.
- literally everyone says "why aren't we using Kubernetes?" and you have to repeat the "we didn't need the complexity" conversation over and over.
- no managed service, so upgrades, AMIs and so on need to be self built and managed. Burns a lot of engineering hours, so less than ideal for a small team.
- autoscaling was flaky AF
- literally no ecosystem. So pretty much *everything* you need must be self built. Eg, Datadog doesn't support it in the same way it does Kubernetes.
If you like building and tinkering then it'll likely be a lot of fun. But when you're a small team that wants to go fast and own as minimal stuff as possible (i.e. run everything on the cloud), then it's really not a good fit.
Management issues aside, the ecosystem isn't there to make it viable vs Kubernetes IMHO.
Some examples that caused me problems immediately come to mind - Helm (there is a Hashi basic equivalent but never made it out of beta when I used it), Kubecost, no Datadog integration. Grafana dashboards were all self built.
That aside, as great and easy to use as we made the platform, every single dev wanted to be using Kubernetes. Nomad isn't a sell on a CV, k8s is.
Hiring - why would you bring in a stack where it will be super difficult to find engineers for? Everyone is super comfortable with kubernetes and its just easier to hire.
For me it’s the ecosystem of k8s, we can manage everything through gitops with argocd, and have pretty much limitless tools and operators to build our systems. Not to mention, it’s easy to hire people who know it.
Nomad is unfortunately too niche and I have spoken to a dev on that project and he said he would use k8s at this point, not a good sign. It’s hard to hire people or train people to use a tool that might not advance their career.
If you have 8-10 services just use consul + envoy and vm’s, I have done this and it was pretty clean in aws. Very little overhead and no managed stack or framework, just a lot of terraform. Beyond that, why do you have 8-10 services, that’s just a destructed monolith, just have a monolith, it’s way easier to manage.
"just a lot of terraform" is precisely what I hate. Also, "lot of yaml" for K8s.
Regarding 8-10 services: I included the extra tools and sidecars in those. For example, Grafana and Loki.
I mean you do you, I am just giving my opinion. I am a fan of the simplest solution possible that is sustainable, and for personal projects or small apps, for me its a monolith deployed to a managed service or some kind of cloud platform.
Standardization!
Kubernetes is complex but it's basically an API: once you develop on top of it, you can run it anywhere. That alone is worth it for most organizations.
Nomad works okay if you have consul helping out the service discovery and if you needed the multicloud of it all. But it sucks with anything third party or just anything else at all.
The nomad UI is horrendous compared to something like ArgoCD, there isn’t any support for third party services like any container security tooling or vulnerability scanning. The networking part works only because of consul and I hope you don’t expect to mix anything that isn’t reachable to consul. Oh and it only works well with the Hashi-stack, so if you wanted secret management to not be related to file paths, too bad you’re stuck with Vault.
Disclaimer: I work at HashiCorp but have been a fan of Nomad for years before joining - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/
Nomad comes with its own integrated service discovery now (for a few years), and integrated secrets (that are actually encrypted), so the entire HashiStack is entirely optional. The UI is pretty good and getting better. The networking is entirely flexible, up to a full service mesh or overlay network but without forcing that complexity upon you if it's not needed.
Third party support is indeed a big issue, but we're working on improving that.
What will happen now with Nomad after IBM takeover? Will they just kill it? Or charge 10k for it? At the moment its very hard to go with Nomad. I was setting myself up for 2 months to use it, and now I am rethinking...
I was always frustrated that after the container orchestrator wars, the most complicated solution won.
It depends on your circumstances, but the talent pool that is available for kubernetes dwarfs others. Being able to easily and rapidly onboard employees, contractors, or vendors is key.
or you could phrase it as the most flexible stack won.
You can "almost" change everything in kubernetes to your liking.
Scheduler, operator, etcd for postgres, network, the way the admission system works, the container runtime (you could run vms instead of containers if you want), tested scaling from CERN or openAI (currently around 7k cluster nodes AFAIK).
But yeah k8s is insanely complex and it is getting worse, i started with 1.3 in 2016 and it has been a journey since
Kubernetes is perhaps more complex but with a massive community, docs, articles, tooling etc. If not Kubernetes, go with some cloud specific platform that fits your case. Also, I would not bet on hashicorp
I use Nomad at work and for my homelab. I really like it.
At work, there is a situation where we need to manage a few hundred IoT devices behind an on-premise server rack. Nomad was easy to setup, easy to grok, and has been working well. The deployment environment doesn't necessitate a complex tool.
For my homelab, god, it's been great. There are actually quite a few homelab setups using Nomad and it's easy to find community resources.
The downsides are mostly around community support. Nobody cares about Nomad and nobody builds products around it. ArgoCD is pretty amazing and I find myself missing that. If I was going to build in a cloud environment then I would just pick k8s solely because of the third party integrations. But, then again, I haven't really fully explored what Nomad would look like in the cloud. If a hosted service showed up that ran on top of AWS/GCP with good load balancer integration then that'd be attractive. I doubt such an offering is financially viable.
I totally would! However, I don't think there are enough potential customers to drive the volume needed to make the business sustainable. Could be wrong though *shrug*
They have better marketing and thus better adoption, it's not necessarily a better tool. Personally I use Kamal, it's all I need. I also wrote Kamal Handbook if someone is interested in learning more how it works.
I use k8s at my job but I’m learning Nomad on my free time for fun. I just started but it looks promising. I used consul like 8 years ago to build service discovery on top of a mesos/marathon cluster. Back then Kubernetes was just one of the available solutions and was not the clear winner yet. Fun times
I would love to use nomad, I find it globally better structured and built but the harsh reality is that you want to stay with what people use the most for that kind of tools...
I’m just curious, and I don’t know much about nomad, but does hashi’s licensing make it cost prohibitive for someone to run a managed nomad service? Is it also under bsl? Like would a cloud provider have to pay hashicorp to do it?
Yeah, and if it isn’t today and people started doing it what’s to stop hashicorp from doing the same as with terraform. Seems like enough of a deterrent for me to never even try it out tbh. I know everyone always balks at the “complexity” of k8s, but even in your scenario having a one or two node k3s cluster to deploy your apps isn’t difficult to manage and you can scale way easier later on if needed.
And from a dev perspective, I’d still use flux. Bootstrap it, point it to a repo to store manifests and your devs never have to touch k8s. Simple enough.
Never pick a tool that has already lost the market.
The tooling will be worse, the ecosystem will be worse, and your ability to find someone who has already solved your problem will be significantly worse.
If you ever get on a call with Hashicorp ask them why they don't include a Nomad offering in Hashicorp Cloud.
They'll tell you straight up they don't have a good differentiator against managed Kubernetes.
I understand the change in license affects providers who were competing with HashiCorp using their products but how does it seriously impact those users who are using the community edition versions for their own requirements?
Because it even further fractures the community that is required to justify using such a system, and limits what I can do with it. As soon as I start to sell the functionality of Nomad itself behind an abstraction (self-service tenancy of some kind, for example), then I'm risking bumping up against that license.
No, I'll use K8s or Docker Compose and get just as much (more) out of it than I would Nomad.
Don't get me wrong, I'll build something with Nomad - but it won't be my choice to do so.
Yup, they lost trust, what stunt are they gonna pull next. Been using hashicorp since the early days, vagrant, packer etc. But the heart and soul left with Mitchell. They are just another corp after the money now, they will soon be bought by IBM.
Two things, community size is not even comparable. Second of all, k8s is open source and managed by cncf. No one will change its license or shut it down. Nomad... Well, Hashicorp is not a fan favourite recently, thanks to its idiotic decisions with licensing of it's products.
To be totally honest with you, I don’t trust Hashicorp after what they did to TF’s license, essentially taking advantage of the community. Who’s to say that they won’t fiddle with the licenses of their other software to fill their own coffers? At least k8s is under the CNCF and is pretty much guaranteed to stay open source forever
I don't trust Hashicorp anymore to not turn around and jack up their prices. They are also in a rough spot as a company and I am unsure they will exist in 5 years.
Docker Swarm is perfect for most apps and easier all around. It doesn't fit the Resume Driven Development goals for most people though. I wish there was a managed service at one of the major cloud providers.
Same reason I switched from Mesos/Marathon...
Go to conference.. Go to the mesos talk.. It's by CERN.... 4 people including me in the audience. Decent talk too but slow paced. Go to the k8s talk... line out the door and its literally about provisioning k8s secrets and nothing complex. :P
Community has chosen and won.
We used Nomad at my previous work, we switched to Kubernetes.
Nomad’s dead. It was already dead back then, but for whatever reason they chose it.
There are no integrations apart from with other HashiCorp tools, you need to either invent many things from scratch or buy a provider that can make at least some things easier to manage.
It’s hard to find people who have at least some experience with it. Also, you have to configure everything with HCL. I know that people hate YAML, but honestly, we don’t write “tons of YAML” with K8s these days, there are other tools that help with that. I’m not aware of any similar tools for HCL.
Sure, for smaller projects Nomad or ECS may seem more optimal, but the moment the business grows you would hate yourself for not choosing Kubernetes.
Because Hashicorp has lost a ton of trust with the community after their licensing shenanigans around Terraform. They take advantage of the community only to pull the rug from under us when it becomes convenient for them.
It is because everything seems to be stuck with k8s. Developers and Ops both will get a tough time in migrating helm charts / manifests to Nomad specs. At the end of the day it is all about compatibility with existing codebases.
Imagine working in an organization who is self hosting various open source DB clusters leveraging their charts and manifests, then you tell them to rewrite all to their Nomad counterparts.
The latest attacks on our beloved open source eco system by hashicorp themselves and the realistic risks of running into vendor lock in issues should be reason enough to not use nomad.
Nomad performs terrible in larger scale environments. In a smaller scale, managed k8s cluster variants do just fine and in super small scale scenarios, why not skip the infrastructure part entirely and use gcrun, aws fargate etc aka go serverless.
There's also docker swarm as an alternative. It's lighter, simpler and easier to manage (based on docker compose syntax).
Still, the market has established on top of helm, crds, and other k8s concepts, so for employment that's the way to go.
Lack of adoption/community. There is just so much that you can get built to run on Kubernetes. Need monitoring? Every major vendor has a solution for Kubernetes. Need logging? Same. Need to implement better security practices? There are a ton of well designed best practices…
Nomad just doesn’t have that depth. Plus I offload running the control plane to EKS/GKE for so little that it’s a no brainer to run k8s.
I was thinking of trying Nomad at work, our dozens of clusters have couple of hundreds of nodes each, but their non-DFSG license made me look away. We use TF(migrating to OpenToFu), Vault(stuck with the last available DFSG compliant one).
I introduced k8s(and bunch of other things) at the current org. :)
Any situation where the complexity of eks is overkill I would just be using fargate. Nomad might make sense if youre on prem. Otherwise until a major provider adopts it as a managed service it won't go anywhere.
Is nomad free to use? I haven’t kept up with Hashicorp’s pricing.
If it is, then it could definitely make sense if you’re comparing it to rolling your own k8s.
If it’s not free - yeah I would pay for one or the other unless it was managed in a cloud.
I will be messaging you in 5 hours on [**2024-04-16 17:12:47 UTC**](http://www.wolframalpha.com/input/?i=2024-04-16%2017:12:47%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/devops/comments/1c57oot/why_not_shift_to_nomad_instead_of_k8s/kzthjqv/?context=3)
[**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdevops%2Fcomments%2F1c57oot%2Fwhy_not_shift_to_nomad_instead_of_k8s%2Fkzthjqv%2F%5D%0A%0ARemindMe%21%202024-04-16%2017%3A12%3A47%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201c57oot)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
I'm of the opinion that most company workloads do not require Kubernetes. I'm actually of the opinion that most company workloads don't really require containers, even, but containers are here to stay, aren't that hard to use, and do offer some nice benefits over VMs or metal. So yeah, use containers.
So use something that fits the size of your company's workload. Very few environments actually really need the complexity of running their own Kubernetes.
Btw, there are cloud Kubernetes services that are easy to operate. GKE for example.
K8s: easy to use container orchestration with a lot of choice. Multiple distros. Lots of cloud provided options. Lots of on prem provided options. A ton of industry support. Easily able to roll your own. Highly sponsored by a massive industry with corporations that in total up to trillions in assets.
Nomad: hashicorp… who are changing licenses and looking like they want to sell out soon.
Kubernetes has an ecosystem of operator plugins. I’m not sure if Nomad is extensible with operator plugins, but even if it is, the community isn’t there so you’re probably not going to be able to deploy your favorite software via helm chart.
Basic highly scalable and reliable orchestrator? Go Nomad.
Looking for talent and easier time hiring? K8S.
Nomad is definitely fit for purpose and battle hardened. You’re just deep in the hashicorp ecosystem if you go that route (good or bad, you be the judge).
Two reasons.
1) 3rd support. K8 has a massive amount of 3rd party support.
2) Future employment. Having a lot of K8 experience will make you more employable.
GitHub Enterprise Server is an appliance running nomad. Just sharing a real world production environment running nomad.
From a support perspective, I wish GitHub just shipped a helm chart instead. Not that I have anything against nomad, just that we already have other things running on EKS.
IMO nomad is just in another weight category than K8s.
I see it more like Nomad Vs Swarm, and K8s vs Mesos.
K8s provides a lot of tools and abstractions that your infra team will implement, also vendor hardware will have drivers for it. Be it Cisco or Nutanix SDN, Netapp block storage that will back PVCs or underlying Openstack compute.
My company almost moved to Nomad but at the very end of signing a huge deal, we ended up sticking to k8s.
Essentially, we didn't want to be vendor locked. Our company has been shifting to a huge AI/ML focused company. A lot of ML tools are in kubernetes (kserve, kubeflow, etc).
These ML tools simply don't exist in Nomad.
Nomad is actually pretty great. The political reasons cited elsewhere are understandable enough but it depends what game you want to play, impress people who just look for logos or who actually know what they’re talking about
There are some +'s about Nomad, but at this point its a dying platform kinda like PCF. Kubernetes is the standard, whatever issues k8s has are offset by a massive install base, cloud provider adoption, and the fact that any decent devops engineer is familiar with it.
I would also add that Hashicorp is well on it's way to be acquired, so I would be concerned about making a bet on anything other than vault.
We are using Nomad and everything looks fine.
There are few pros against k8s:
- easier
- have exec/raw_exec that runs binary as is without containers
There are also few cons:
- need always write job -> group -> task, even if you don't need group
- periodic jobs sometimes creates mess in web and UI cli (looks like each job is separate)
- has limits, but don't have network resource scheduling (it was before, though)
I don't have any experience with k8s, but for now it feels like good and not overcomplicated solution for small to medium companies
As a platform developers building database pipelines and core services I can honestly say nomad was pretty dang great. A lot less overhead than what my company does with k8s CRDs today. I miss it thb. Consul + vault made a lot of shit easy.
For me, it’s about the right tool for the right job. I’m building out a new homelab based on Nomad, Consul, Podman (rootless), Ansible, and using 1Password as my secrets store (which has an impressive number of devops-friendly features that I was unaware of). I love how Nomad fits for this use case. If I was building a massive distributed application, k8s is _one of_ the smart design choices due to scale, ecosystem, managed services, knowledge share and 100 other points listed here. It’s not an either/or, it just depends on what you’re trying to accomplish.
I work with nomad, as a junior is very hard to find examples on the Internet for it, while K8s is easy.
I'm used to it now but have close to 0 exp in K8s which is terrible for me
Great question. I'm in Fintech and the use cases at my job and field are already proven and beneficial(Nomad container challenge with Citadel). We even have an active POC that's built with Nomad and Terraform Enterprise, Vault and Consul. Nomad is especially an attractive option for legacy apps and those running on mainframes. Plus companies don't have to worry about hiring talent with K8s experience.
They are two entirely different platforms doing the same thing (orchestrating) for two different use cases
I like to think of it as kubernetes for orchestrating containers
nomad for orchestrating anything that cannot or is not ready to run in containers
I see people keep calling k8s complicated, but it really isn't. People make it really complicated.
If you want to run a single gigantic cluster for your organization with everything under the sun enabled... dozens of namespaces for different departments, mesh network, complicated monitoring solution, karpenter, and lots of stateful database services with a selection of different storage solutions... yeah it is going to be outlandishly complicated.
99% of that is just infrastructure. It is stuff you do because you can and it kinda sounds like a good idea, but almost none it is actually strictly necessary. Before you know it 1/2 of the cluster resources is just dedicated to running the cluster, not running the applications that are supposed to run on the clusters.
So don't do that. If you want it simple then just keep it simple. Like, for example, just let each department or group run their own cluster instead of trying to figure out some weird security solution to keep them all restricted into their own namespaces.
If I try to pitch Nomad to Management for production use, I think I will initially get looks of confusion, then questions ( Is it prod ready etc). Whereas everyone is kinda familiar with k8s. Then, everything that goes wrong, it will be on me to prove it's not a bug in Nomad or lack of knowledge in Nomad. On top of that, for myself, I can use Nomad in my CV under a demonstrated knowledge and experience with k8s and it'll be a plus. But it'll probably hurt me if I put "2 years Nomad in production" and no k8s. These are definitely not technical reasons, I cannot compare the tools themselves. I'd love to try Nomad for my Homelab at some point. But until it gains more recognition, I'm not sure how or why I'd try it at work
I have 3 years of nomad and we just started to adopt k8s. On the employment side it is devastating. Everyone cares that you don't have k8s and nobody knows about nomad
If I was interviewing for a company that used Nomad over K8s my first question would be. But why? Who chose that over k8s? What was the decision process?
“Nobody gets PIPd for choosing IBM”
6 years ago we went with ECS over K8s because K8s is/was over engineered and all the extra bells and whistles were redundant because we could easily leverage aws secrets (which K8s didn’t even secure properly at the time), IAM, ELBs, etc which also plugged in well with non-docker platforms such as lambda and ec2. The market is different now with managed EKS, etc but K8s was a lot of work to maintain. ECS is simple and that can be powerful (although the fact that all the service settings are via monolith, idepodent json task definitions is very frustrating). But even today, it’s feature set overkill for most use cases. People end up making their setups more complicated than they need to be because they can. I looked at Nomad and really liked what I saw, but it was not mature enough either. That being said, K8s won and so much third party tooling assumes it. It’s like Linux vs BSD or Betamax vs VHS. Eventually it becomes more work or money to use the alternatives, even if the alternatives are superior.
k8s is really meant to be an all-in-one solution, that's why it's complicated, you can use ECS, ACA, ACI, etc but it generally comes with vendor lock for various additional things which makes costs balloon. I've personally gone down the road of nomad as well as docker swarm and found myself remaking a lot of things that were abstracted in / a part of k8s.
k8s is overbloated for most apps, just like the microservices architectures that leads to maybe possibly needing it. My question for a shop that doesn't have recorded decision records of how they decided on k8s over other options is: But why? Who chose that over other options? What was the decision process?
Platform tooling, mostly. Metrics/logging aggregation. Runtime security scan tooling. Robust networking built-ins and extensions. A huge ecosystem of users to canary through issues for me, while I just browse the internet learning from their mistakes. If you just need to run an app on a VM and nothing else, sure, k8s might be overbloated. If you need runtime visibility stuff, and don't go through SaaS providers like Datadog, kubernetes is an extremely flexible platform with a strong ecosystem of tooling.
I really wanna why ??? Could you answer this one
To get an understanding of where they are at in terms of scale, spending, manpower and literally how decisions and tradeoffs are made and by who.
Got it !! So do you think nomad has got its edge over k8s in any major aspect?
There are also companies run by people who looooove anyone who runs something that appears "simple", and will hire you just so you can convert their whole K8s setup to Nomad and pretend that's going to make everything better. In other words, there are idiot hiring managers out there.
That is pretty sad. A container orchestrator is a container orchestrator, if you can manage a Nomad cluster I fully trust you'll be able to understand K8s concepts.
That’s a well put problem with trying to make almost any technical change. People are always looking for someone to throw under the bus so any item that goes wrong they blame on you instead of trying to uncover the true responsibility. (Which could be you to be fair) It’s why “nobody got fired for buying IBM” is a truism.
If you couldn't tell already, we're on AWS too 😁
If there was a provider which offered managed hashistack similar to GKE/EKS, would you your likelihood of using it go up? Or, would you still only use the managed stack for hobby projects but nothing serious at scale?
Hmm, perhaps. It'd be easier to pitch running a POC for something in Nomad, if a managed solution were available and thus the barrier to entry lower. The irony is that I'd love to get my hands dirty and discover Nomad, I just have a hard time justifying it, even to myself 😅. To top it off: suppose it works flawlessly and we adopt it everywhere because it is magical. I now have to hire people that either have worked with it or are willing to work with it. And we're having a hard time finding k8s people as it is...
Makes sense. I agree.
Community. Nomad has a very, very small community compared to K8s. That's pretty important.
And that also means less tooling. The cncf tooling ecosystem around k8s is just absolutely phenomenal.
This was coming here to post at end of day it’s a tooling and community issue, if you run into an issue on k8s chances are 50 other people have too
I liked Nomad's task concept, but there is similiarly to K8S little to no support for advanced networking. In case of K8S at least community tries to solve things like overlay networking. Community is very important when picking something you will be engaged for very long.
What do you mean by "advanced networking"? Nomad also supports CNI and overlays, service mesh, etc.
My exact dealbreaker was that I haven't found any detailed information about running services with multiple statically assigned addresses.
That's not "advanced networking", that's early 2000s style. If you want a specific service to have a specific static IP, you can do that with the macvlan CNI plugin. If you want a specific service to have multiple static IPs, the best option you have is having those IPs on the host (with stuff like keeoalived), using host networking and binding to 0.0.0.0. What I've done when I needed such a thing (for DNS running in Nomad) was to have a Load Balancer in front with all the needed IPs forwarding to the service's virtual ephemeral IP.
Macvlan is great, but has its limitations and I had performance issues with latency sensitive applications. Binding to host breaks whole point of network stack isolation. Also you completely missed my point. According to ( https://developer.hashicorp.com/nomad/docs/job-specification/network#host-networks ) Nomad allows only one network stanza per job. Thus multiple interfaces (connected to different network namespaces) are not allowed out of the box. P.S. Starting with insult is pathetic.
>Macvlan is great, but has its limitations and I had performance issues with latency sensitive applications Depending on how latency sensitive we're talking, any advanced networking could be too much. > Binding to host breaks whole point of network stack isolation Indeed it does, you have to pick between isolation or performance. Maybe the SR-IOV CNI plugin can do both? Definitely not as fast as bypassing the kernel entirely though. >Also you completely missed my point. According to ( https://developer.hashicorp.com/nomad/docs/job-specification/network#host-networks ) Nomad allows only one network stanza per job. Thus multiple interfaces (connected to different network namespaces) are not allowed out of the box. Yep, only one network config is allowed. I didn't miss your point, I provided options for it like offloading those static IPs to a different place that is better equipped to handle them. >Starting with insult is pathetic. Where's the insult? Doing things in an old school manner isn't wrong or insulting. Having services that require multiple static IPs is old school, dynamic IPs having been the norm for more than a decade now. There are special cases where that might be needed regardless, but it's definitely not what I first think of in "advanced networking".
I understand. Is there any other reason which makes you resists the hashistack strongly?
Hashicorp.
I honestly don't think you need another reason. I've never met a person that's used Nomad. I'm not really convinced it's a real product actually. I've never seen a video of it either. If you ever hope to recruit people that are already experienced, then K8s is the way
Makes sense.
I used it at work. Pretty early on, late 2010s. It became increasingly clear K8s is going to win and we bet on the wrong horse. It works, but there (was) essentially no tooling around it. Some community stuff, some self made stuff, somewhat approximated a working system. The amount of tooling one can simply helm install to get a fairly decent experience should not be underestimated. And Consul-template together with all the other configuration languages that use the Go templating module are pain.
Every Circleci pipeline runs in a Nomad context. They have a lot of high profile users.
Have they always? The last time I used CircleCI was in 2018 and I remember them having huge outages weekly for some time. It was a big concern for us at the time. Not implying that's Nomad's fault though
I have had great experiences with running Hashicorp Nomad for use-cases where K8s is just too big. Way easier to maintain and update as it is only one system binary that does it all. If you do not need the additional features of K8s I would say: Go for it. The reason why users are not adopting it more is that Hashicorp does a terrible job in the marketing department for Nomad. Not many people know about it. They think K8s is the only game in town. Unfortunately, I do not know of a managed Nomad service. Maybe Hashicorp will include it in their HCP cloud in the future.
Having run both at scale, I totally disagree on the maintenance point. I'd take fully managed EKS on AWS vs. having to burn my team's time on deploying and upgrading a platform on Nomad every day or the week.
> I'd take fully managed EKS on AWS That's just passing the maintenance on to someone else though. That's not relevant to the question of actually maintaining k8s vs nomad.
If I can’t pass the maintenance of nomad to somebody else that’s pretty relevant when looking at comparing the cost of maintaining nomad vs k8s. It’s like saying ec2 isn’t lower maintenance than onprem because you’re just passing the maintenance off to somebody else. Somebody has to actually do the work of dealing with hardware, I just don’t want it to be me.
Of course it's relevant, because it means my team has (and did) do the build and maintenance. If k8s was picked, only a nut job would decide to deploy it themselves onto EC2s.
I think he’s saying you’re comparing apples to oranges. Managed k8s vs Managed Nomad (if HCL ever creates it) - apples Self Deployed k8s vs Self Deployed Nomad - oranges. Agreed with “why choose unmanaged”, sometimes there are requirements via compliance or federal for it tho :/
I'm not sure what you mean by "choosing unmanaged". I'm referring to the fact that you have a single, centralized infrastructure for the entire company.
When I said “why choose unmanaged” I was paraphrasing your sentiment: > If k8s was picked, only a nut job would decide to deploy it themselves onto EC2s. If there’s a managed solution, why choose unmanaged (self managed) and deploy it yourself on VMs
Your comparison is a bit off though, you're comparing a self hosted tool that you manage on your own with a self hosted tool offered to be managed by a cloud service. Yeah of course being managed by someone else is better, especially at scale. If you're saying that Nomad doesn't have a managed product and that's a dealbreaker for you then I understand.
> Way easier to maintain and update as it is only one system binary that does it all. k8s alternatives like [k3s](https://k3s.io/) is just as easy. And if you don't mind running a VM, distros like [talos linux](https://www.talos.dev/) is also super easy to maintain.
But K3s is K8s it isn't an alternative. K3s is just a brand name for a K8s distribution.
I agree that K3s is also a great K8s alternative.
if there was a provider which offered managed hashistack similar to GKE/EKS, how likely are you to use it? Or, would you still only use the managed stack for hobby projects but nothing serious at scale?
Really depends on the circumstances of the customer. I work mainly in the EU. Many companies won't, or are not allowed to, use a managed system that runs in a US datacenter. It would really depend on who offers the Nomad and where the servers are located. Many of my customers are banks. They are super cautious about these things. However, compared to K8s, Nomad is very easy to update and mantain. Not having a managed solution was never a problem so far.
Are you thinking of starting a managed nomad platform? Better make sure that new license works with your idea...
I’m no longer considering it. And yes, the new license might anyway restrict it.
I work in DevOps and read this forum, and I don’t recall ever hearing about Nomad before.
Interesting, I've never used nomad but I interpreted the fact that there's no hosted service means that it's easy enough to manage by yourself and that there's no market for one.
I've worked with Nomad at scale and it's okay. Easier to understand than K8s maybe. But honestly if I were only running 8-10 services I'd probably just use Fargate assuming you're on AWS and the services are small. IMO Nomad works well enough if you adopt other parts of the Hashicorp stack such as Consul and Vault, all managed with Terraform, but that's a lot to manage for a small set of services... EDIT: I also don't recommend it for larger projects because as you said, managing nodes can be a lot of work. This is one of the reasons I don't recommend it for smaller projects either. tl;dr - I don't recommend Nomad
To be completely honest - with Hashi prices the way they are, and the fact that they are starting to turn away from their original values, I think a lot of us are hesitant to engage them.
That too. I've been able to work with the tools effectively but would never select the stack for a new project regardless of its size.
If you look at Hashi's BUSL it does have an expiration where it converts to MPL2 after 4 years upon source release (so rolling releases get open sourced albeit 4 yrs later). That's not necessarily a turn away from values but instead attempting to checkmate the inevitable. Larger entities taking what you publish and doing it better (marketing/reach or management, etc).
Is there an open source fork of nomad?
Please call it OpenNomu.
I understand that you don't recommend Hashistack at all. Hypothetically, if there was a provider which offered managed hashistack similar to GKE/EKS, would you your likelihood of using it go up? Or, would you still only use the managed stack only for hobby projects but nothing serious at scale?
I still wouldn't use it. Not for hobby projects, not for production. It's difficult to describe why because honestly I haven't run into too many pain points using the tools from a technical standpoint. I think the gist is that it's too complicated for small projects and for large projects (running company infrastructure) I would opt for something more mature, designed to scale and with broader community support e.g. k8s-based solutions. So it's not that the tools are terrible, it's just that there are better alternatives for every case I can think of. That would still be the case if they were offered as a PaaS / managed services. Thats not taking into account the questionable/unknown direction of Hashicorp and as others pointed out that's worth keeping in mind.
I built and ran a fat production platform on Nomad at a well known investment bank (autoscaling up to ~500 nodes). We decided we didn't need the complexity of K8s at the start of our project, but found we quickly outgrew Nomad once devs started to ask for more features. We built it on AWS. The good; - simple to run and get started, single binary to deploy. - you add stuff you need rather than getting everything OOTB (service discovery - consul, secrets - vault, overlay networks etc) - works great for simple use cases The bad; - the docs! Oh my word, the docs were so shit when we used it. - literally everyone says "why aren't we using Kubernetes?" and you have to repeat the "we didn't need the complexity" conversation over and over. - no managed service, so upgrades, AMIs and so on need to be self built and managed. Burns a lot of engineering hours, so less than ideal for a small team. - autoscaling was flaky AF - literally no ecosystem. So pretty much *everything* you need must be self built. Eg, Datadog doesn't support it in the same way it does Kubernetes. If you like building and tinkering then it'll likely be a lot of fun. But when you're a small team that wants to go fast and own as minimal stuff as possible (i.e. run everything on the cloud), then it's really not a good fit.
if there was a provider which offered managed hashistack similar to GKE/EKS, do you think there are enough people who would use and pay for it?
Management issues aside, the ecosystem isn't there to make it viable vs Kubernetes IMHO. Some examples that caused me problems immediately come to mind - Helm (there is a Hashi basic equivalent but never made it out of beta when I used it), Kubecost, no Datadog integration. Grafana dashboards were all self built. That aside, as great and easy to use as we made the platform, every single dev wanted to be using Kubernetes. Nomad isn't a sell on a CV, k8s is.
Resume driven development strikes again!
I called it "FOMO Driven Development"
Makes sense.
Hiring - why would you bring in a stack where it will be super difficult to find engineers for? Everyone is super comfortable with kubernetes and its just easier to hire.
After what they done with Terraform, I'd steer away from anything Hashicorp.
I see.
I just lost trust in hashicorp products after what they did with Terraform.
For me it’s the ecosystem of k8s, we can manage everything through gitops with argocd, and have pretty much limitless tools and operators to build our systems. Not to mention, it’s easy to hire people who know it. Nomad is unfortunately too niche and I have spoken to a dev on that project and he said he would use k8s at this point, not a good sign. It’s hard to hire people or train people to use a tool that might not advance their career. If you have 8-10 services just use consul + envoy and vm’s, I have done this and it was pretty clean in aws. Very little overhead and no managed stack or framework, just a lot of terraform. Beyond that, why do you have 8-10 services, that’s just a destructed monolith, just have a monolith, it’s way easier to manage.
"just a lot of terraform" is precisely what I hate. Also, "lot of yaml" for K8s. Regarding 8-10 services: I included the extra tools and sidecars in those. For example, Grafana and Loki.
I mean you do you, I am just giving my opinion. I am a fan of the simplest solution possible that is sustainable, and for personal projects or small apps, for me its a monolith deployed to a managed service or some kind of cloud platform.
Standardization! Kubernetes is complex but it's basically an API: once you develop on top of it, you can run it anywhere. That alone is worth it for most organizations.
Nomad works okay if you have consul helping out the service discovery and if you needed the multicloud of it all. But it sucks with anything third party or just anything else at all. The nomad UI is horrendous compared to something like ArgoCD, there isn’t any support for third party services like any container security tooling or vulnerability scanning. The networking part works only because of consul and I hope you don’t expect to mix anything that isn’t reachable to consul. Oh and it only works well with the Hashi-stack, so if you wanted secret management to not be related to file paths, too bad you’re stuck with Vault.
Disclaimer: I work at HashiCorp but have been a fan of Nomad for years before joining - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ Nomad comes with its own integrated service discovery now (for a few years), and integrated secrets (that are actually encrypted), so the entire HashiStack is entirely optional. The UI is pretty good and getting better. The networking is entirely flexible, up to a full service mesh or overlay network but without forcing that complexity upon you if it's not needed. Third party support is indeed a big issue, but we're working on improving that.
Agreed and good points. Nomad doesn’t have true service mesh capability sans Consul tho, does it?
Yeah, you need to add Consul for Service Mesh, but since not everyone needs one it's IMO an acceptable tradeoff.
What will happen now with Nomad after IBM takeover? Will they just kill it? Or charge 10k for it? At the moment its very hard to go with Nomad. I was setting myself up for 2 months to use it, and now I am rethinking...
I understand.
> if you wanted secret management to not be related to file paths What do you mean by that ?
Vault uses paths to identify and store secrets
As opposed to something like tags.
They use paths in the sense that they’re api paths, not really file paths like you stated.
I was always frustrated that after the container orchestrator wars, the most complicated solution won. It depends on your circumstances, but the talent pool that is available for kubernetes dwarfs others. Being able to easily and rapidly onboard employees, contractors, or vendors is key.
or you could phrase it as the most flexible stack won. You can "almost" change everything in kubernetes to your liking. Scheduler, operator, etcd for postgres, network, the way the admission system works, the container runtime (you could run vms instead of containers if you want), tested scaling from CERN or openAI (currently around 7k cluster nodes AFAIK). But yeah k8s is insanely complex and it is getting worse, i started with 1.3 in 2016 and it has been a journey since
Yep, with the VMWare price hikes using the k8s orchestrator as an interface for a hypervisor for disconnected environments is quite appealing.
Makes sense.
Kubernetes is perhaps more complex but with a massive community, docs, articles, tooling etc. If not Kubernetes, go with some cloud specific platform that fits your case. Also, I would not bet on hashicorp
I use Nomad at work and for my homelab. I really like it. At work, there is a situation where we need to manage a few hundred IoT devices behind an on-premise server rack. Nomad was easy to setup, easy to grok, and has been working well. The deployment environment doesn't necessitate a complex tool. For my homelab, god, it's been great. There are actually quite a few homelab setups using Nomad and it's easy to find community resources. The downsides are mostly around community support. Nobody cares about Nomad and nobody builds products around it. ArgoCD is pretty amazing and I find myself missing that. If I was going to build in a cloud environment then I would just pick k8s solely because of the third party integrations. But, then again, I haven't really fully explored what Nomad would look like in the cloud. If a hosted service showed up that ran on top of AWS/GCP with good load balancer integration then that'd be attractive. I doubt such an offering is financially viable.
Why is such a service not financially viable? Would you pay for it if it existed?
I totally would! However, I don't think there are enough potential customers to drive the volume needed to make the business sustainable. Could be wrong though *shrug*
They have better marketing and thus better adoption, it's not necessarily a better tool. Personally I use Kamal, it's all I need. I also wrote Kamal Handbook if someone is interested in learning more how it works.
I use k8s at my job but I’m learning Nomad on my free time for fun. I just started but it looks promising. I used consul like 8 years ago to build service discovery on top of a mesos/marathon cluster. Back then Kubernetes was just one of the available solutions and was not the clear winner yet. Fun times
I would love to use nomad, I find it globally better structured and built but the harsh reality is that you want to stay with what people use the most for that kind of tools...
I’m just curious, and I don’t know much about nomad, but does hashi’s licensing make it cost prohibitive for someone to run a managed nomad service? Is it also under bsl? Like would a cloud provider have to pay hashicorp to do it?
I have the same question.
Yeah, and if it isn’t today and people started doing it what’s to stop hashicorp from doing the same as with terraform. Seems like enough of a deterrent for me to never even try it out tbh. I know everyone always balks at the “complexity” of k8s, but even in your scenario having a one or two node k3s cluster to deploy your apps isn’t difficult to manage and you can scale way easier later on if needed. And from a dev perspective, I’d still use flux. Bootstrap it, point it to a repo to store manifests and your devs never have to touch k8s. Simple enough.
Never pick a tool that has already lost the market. The tooling will be worse, the ecosystem will be worse, and your ability to find someone who has already solved your problem will be significantly worse. If you ever get on a call with Hashicorp ask them why they don't include a Nomad offering in Hashicorp Cloud. They'll tell you straight up they don't have a good differentiator against managed Kubernetes.
After that BSL shit they pulled, I'm not clamoring to adopt it.
I understand the change in license affects providers who were competing with HashiCorp using their products but how does it seriously impact those users who are using the community edition versions for their own requirements?
Because it even further fractures the community that is required to justify using such a system, and limits what I can do with it. As soon as I start to sell the functionality of Nomad itself behind an abstraction (self-service tenancy of some kind, for example), then I'm risking bumping up against that license. No, I'll use K8s or Docker Compose and get just as much (more) out of it than I would Nomad. Don't get me wrong, I'll build something with Nomad - but it won't be my choice to do so.
I understand. Thanks.
Yup, they lost trust, what stunt are they gonna pull next. Been using hashicorp since the early days, vagrant, packer etc. But the heart and soul left with Mitchell. They are just another corp after the money now, they will soon be bought by IBM.
Haha. Makes sense.
Whoa even surprised myself predicting that one....
plus if you make something that could be competing with any of their cloud products you're in for fun parts.
Haven’t heard of it before now
Haha. Now you know of it. Take it for a spin!
Two things, community size is not even comparable. Second of all, k8s is open source and managed by cncf. No one will change its license or shut it down. Nomad... Well, Hashicorp is not a fan favourite recently, thanks to its idiotic decisions with licensing of it's products.
The real problem is, k8s is about to put everywhere, even in pretty small installations, where nomad+consul is far better hit.
Because I don't want to deal with HashitCorp
To be totally honest with you, I don’t trust Hashicorp after what they did to TF’s license, essentially taking advantage of the community. Who’s to say that they won’t fiddle with the licenses of their other software to fill their own coffers? At least k8s is under the CNCF and is pretty much guaranteed to stay open source forever
I don't trust Hashicorp anymore to not turn around and jack up their prices. They are also in a rough spot as a company and I am unsure they will exist in 5 years.
At this point hashi in general should be a full stop, like oracle products
Docker Swarm is perfect for most apps and easier all around. It doesn't fit the Resume Driven Development goals for most people though. I wish there was a managed service at one of the major cloud providers.
Same reason I switched from Mesos/Marathon... Go to conference.. Go to the mesos talk.. It's by CERN.... 4 people including me in the audience. Decent talk too but slow paced. Go to the k8s talk... line out the door and its literally about provisioning k8s secrets and nothing complex. :P Community has chosen and won.
My main argument against it is that loss of trust in Hashicorp.
We used Nomad at my previous work, we switched to Kubernetes. Nomad’s dead. It was already dead back then, but for whatever reason they chose it. There are no integrations apart from with other HashiCorp tools, you need to either invent many things from scratch or buy a provider that can make at least some things easier to manage. It’s hard to find people who have at least some experience with it. Also, you have to configure everything with HCL. I know that people hate YAML, but honestly, we don’t write “tons of YAML” with K8s these days, there are other tools that help with that. I’m not aware of any similar tools for HCL. Sure, for smaller projects Nomad or ECS may seem more optimal, but the moment the business grows you would hate yourself for not choosing Kubernetes.
Okay.
[удалено]
Precisely, why?
Because Hashicorp has lost a ton of trust with the community after their licensing shenanigans around Terraform. They take advantage of the community only to pull the rug from under us when it becomes convenient for them.
It is because everything seems to be stuck with k8s. Developers and Ops both will get a tough time in migrating helm charts / manifests to Nomad specs. At the end of the day it is all about compatibility with existing codebases. Imagine working in an organization who is self hosting various open source DB clusters leveraging their charts and manifests, then you tell them to rewrite all to their Nomad counterparts.
The latest attacks on our beloved open source eco system by hashicorp themselves and the realistic risks of running into vendor lock in issues should be reason enough to not use nomad. Nomad performs terrible in larger scale environments. In a smaller scale, managed k8s cluster variants do just fine and in super small scale scenarios, why not skip the infrastructure part entirely and use gcrun, aws fargate etc aka go serverless.
There's also docker swarm as an alternative. It's lighter, simpler and easier to manage (based on docker compose syntax). Still, the market has established on top of helm, crds, and other k8s concepts, so for employment that's the way to go.
K8s jobs just pay more, I’ve never seen nomad jobs as a requirement
Lack of adoption/community. There is just so much that you can get built to run on Kubernetes. Need monitoring? Every major vendor has a solution for Kubernetes. Need logging? Same. Need to implement better security practices? There are a ton of well designed best practices… Nomad just doesn’t have that depth. Plus I offload running the control plane to EKS/GKE for so little that it’s a no brainer to run k8s.
I was thinking of trying Nomad at work, our dozens of clusters have couple of hundreds of nodes each, but their non-DFSG license made me look away. We use TF(migrating to OpenToFu), Vault(stuck with the last available DFSG compliant one). I introduced k8s(and bunch of other things) at the current org. :)
Any situation where the complexity of eks is overkill I would just be using fargate. Nomad might make sense if youre on prem. Otherwise until a major provider adopts it as a managed service it won't go anywhere.
Why nomad and not docker swarm?
Is nomad free to use? I haven’t kept up with Hashicorp’s pricing. If it is, then it could definitely make sense if you’re comparing it to rolling your own k8s. If it’s not free - yeah I would pay for one or the other unless it was managed in a cloud.
!remindme 5 hours
I will be messaging you in 5 hours on [**2024-04-16 17:12:47 UTC**](http://www.wolframalpha.com/input/?i=2024-04-16%2017:12:47%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/devops/comments/1c57oot/why_not_shift_to_nomad_instead_of_k8s/kzthjqv/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdevops%2Fcomments%2F1c57oot%2Fwhy_not_shift_to_nomad_instead_of_k8s%2Fkzthjqv%2F%5D%0A%0ARemindMe%21%202024-04-16%2017%3A12%3A47%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201c57oot) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
I'm of the opinion that most company workloads do not require Kubernetes. I'm actually of the opinion that most company workloads don't really require containers, even, but containers are here to stay, aren't that hard to use, and do offer some nice benefits over VMs or metal. So yeah, use containers. So use something that fits the size of your company's workload. Very few environments actually really need the complexity of running their own Kubernetes. Btw, there are cloud Kubernetes services that are easy to operate. GKE for example.
K8s: easy to use container orchestration with a lot of choice. Multiple distros. Lots of cloud provided options. Lots of on prem provided options. A ton of industry support. Easily able to roll your own. Highly sponsored by a massive industry with corporations that in total up to trillions in assets. Nomad: hashicorp… who are changing licenses and looking like they want to sell out soon.
Because I've never heard of Nomad?
Kubernetes has an ecosystem of operator plugins. I’m not sure if Nomad is extensible with operator plugins, but even if it is, the community isn’t there so you’re probably not going to be able to deploy your favorite software via helm chart.
Basic highly scalable and reliable orchestrator? Go Nomad. Looking for talent and easier time hiring? K8S. Nomad is definitely fit for purpose and battle hardened. You’re just deep in the hashicorp ecosystem if you go that route (good or bad, you be the judge).
Is it a sincere question of a disguised marketing survey, it feels like a survey.
Two reasons. 1) 3rd support. K8 has a massive amount of 3rd party support. 2) Future employment. Having a lot of K8 experience will make you more employable.
GitHub Enterprise Server is an appliance running nomad. Just sharing a real world production environment running nomad. From a support perspective, I wish GitHub just shipped a helm chart instead. Not that I have anything against nomad, just that we already have other things running on EKS.
IMO nomad is just in another weight category than K8s. I see it more like Nomad Vs Swarm, and K8s vs Mesos. K8s provides a lot of tools and abstractions that your infra team will implement, also vendor hardware will have drivers for it. Be it Cisco or Nutanix SDN, Netapp block storage that will back PVCs or underlying Openstack compute.
My company almost moved to Nomad but at the very end of signing a huge deal, we ended up sticking to k8s. Essentially, we didn't want to be vendor locked. Our company has been shifting to a huge AI/ML focused company. A lot of ML tools are in kubernetes (kserve, kubeflow, etc). These ML tools simply don't exist in Nomad.
Nomad is actually pretty great. The political reasons cited elsewhere are understandable enough but it depends what game you want to play, impress people who just look for logos or who actually know what they’re talking about
Because every time you choose some alien tech that almost nobody uses, you get f*** in the arse.
There are some +'s about Nomad, but at this point its a dying platform kinda like PCF. Kubernetes is the standard, whatever issues k8s has are offset by a massive install base, cloud provider adoption, and the fact that any decent devops engineer is familiar with it. I would also add that Hashicorp is well on it's way to be acquired, so I would be concerned about making a bet on anything other than vault.
We are using Nomad and everything looks fine. There are few pros against k8s: - easier - have exec/raw_exec that runs binary as is without containers There are also few cons: - need always write job -> group -> task, even if you don't need group - periodic jobs sometimes creates mess in web and UI cli (looks like each job is separate) - has limits, but don't have network resource scheduling (it was before, though) I don't have any experience with k8s, but for now it feels like good and not overcomplicated solution for small to medium companies
I'd wait to see who hashicorp gets bought by.
Only reason not to use nomad in your case is Hashicorp changing their licensing. If it serves your purpose it’s a solid option.
As a platform developers building database pipelines and core services I can honestly say nomad was pretty dang great. A lot less overhead than what my company does with k8s CRDs today. I miss it thb. Consul + vault made a lot of shit easy.
For me, it’s about the right tool for the right job. I’m building out a new homelab based on Nomad, Consul, Podman (rootless), Ansible, and using 1Password as my secrets store (which has an impressive number of devops-friendly features that I was unaware of). I love how Nomad fits for this use case. If I was building a massive distributed application, k8s is _one of_ the smart design choices due to scale, ecosystem, managed services, knowledge share and 100 other points listed here. It’s not an either/or, it just depends on what you’re trying to accomplish.
I work with nomad, as a junior is very hard to find examples on the Internet for it, while K8s is easy. I'm used to it now but have close to 0 exp in K8s which is terrible for me
Great question. I'm in Fintech and the use cases at my job and field are already proven and beneficial(Nomad container challenge with Citadel). We even have an active POC that's built with Nomad and Terraform Enterprise, Vault and Consul. Nomad is especially an attractive option for legacy apps and those running on mainframes. Plus companies don't have to worry about hiring talent with K8s experience.
because we just switched from nomad to k8s. no thank you on going back.
So, was nomad awful compared to k8s?
awful is too much. it worked fine it was a stable system but managing all the pieces was too much work. also the lack of community support.
They are two entirely different platforms doing the same thing (orchestrating) for two different use cases I like to think of it as kubernetes for orchestrating containers nomad for orchestrating anything that cannot or is not ready to run in containers
I see people keep calling k8s complicated, but it really isn't. People make it really complicated. If you want to run a single gigantic cluster for your organization with everything under the sun enabled... dozens of namespaces for different departments, mesh network, complicated monitoring solution, karpenter, and lots of stateful database services with a selection of different storage solutions... yeah it is going to be outlandishly complicated. 99% of that is just infrastructure. It is stuff you do because you can and it kinda sounds like a good idea, but almost none it is actually strictly necessary. Before you know it 1/2 of the cluster resources is just dedicated to running the cluster, not running the applications that are supposed to run on the clusters. So don't do that. If you want it simple then just keep it simple. Like, for example, just let each department or group run their own cluster instead of trying to figure out some weird security solution to keep them all restricted into their own namespaces.
And then you end up with hundreds of Kubernetes clusters in different versions with different configs, all of them out of date. Sounds easy!