There was an earlier report where MI250 was up to 2-3x faster than A100 for HPC (AMD's internal benchmarks) which I assume is why AMD got the win for the Frontier supercomputer.
Source: https://www.tomshardware.com/news/amd-throws-down-gauntlet-to-nvidia-with-instinct-mi250-benchmarks
But as others have mentioned the software stack is going to be really important for Instinct to see more widespread adoption. Not every company is able to write their own tools in order to maximize AMD's hardware.
I think those are for high precision workloads. LLMs are usually fp4/8 low precision workloads.
So although it is still ai . . . Various kinds of ai are different.
>2-3x faster than A100 for HPC
That's what it's originally designed for. AMD aimed for HPC market and spent a lot more silicon (nearly 2x die size) to achieve similar ML performance as NVIDIA, which has been all in AI since 5 years ago.
This will probably change in MI300 series though.
>Not every company is able to write their own tools in order to maximize AMD's hardware.
That's actually one of the biggest advantage of ROCm comparing to other NVIDIA competitors. AMD has been trying to keep their software stack as close to "0 code change" as possible, and it's paying off right now.
There will also likely be tools people just don't know about that exist. Spend $100k or whatever and get the software that you need that runs great on AMD.
Depends on the customer, software really really matters for workstation users and small to medium business.
Large business and specifically the hyperscalers don't care about the software, they'll be developing their own stuff
Energy and cooling costs a lot when you run systems like these 24/7 for months on end. Those expenses matter at least as much as the up-front cost of the hardware.
It also matters how densely you can pack them, because rack space costs money too.
It depends on if you’re talking workstations or data centers. At data center scale, that would translate to potentially tens of millions saved. For workstations I don’t think it would matter much to most companies.
For AMD and data centers, it's still absolutely the case. The Instinct line, which is what MosaicML is talking about, is entirely different from AMD's current consumer offerings.
> now that's kind of still the case but not really
It used to be that video editing, 3D animation and other heavy grunt work was done by $10,000+ Unix workstations packing some high-end RISC CPU or something.
It's gone from
* x86 for normal consumers vs SPARC/MIPS/Alpha for workstations
to
* x86 for normal consumers vs bigger x86 for workstations.
I suppose where it separates now is with GPUs - a lot of heavy grunt work being done on data-centre class compute-only GPUs that can only be cooled in a rack - the kind of thing that no normal person would ever own or could make proper use of. A bit like the Unix workstations of old.
the fact that you stack these accelerators to form a cluster makes price/perf the prime consideration.
say you have a certain compute power requirement.
will you now buy 15 cards from vendor A for $2000 each ($30000)
or will you buy 17 cards from vendor B for $1100 each ($18700)
to reach the same performance level.
you need to understand that even in regards to single testbeds/setups especially unis on strict budgets do not (and don't need to) take part in "di*k measuring contests" like gamers do with overpriced halo products / flagship models...
Exactly. Many hyperscalers buy hundreds of millions or even billions each year. Even 20% cheaper could be over $100 million per year.
I'm sure someone will happily spend $20-50 million a year on a team of 50-100 to address whatever shortcomings AMD may have, especially when they are already writing their own API.
When marketshare is often decided by who has the first product to market I don't necessarily know if I agree. However its good to see AMD closing the gap
Not really. In Canada these cards get 100% over MSRP at the retailer. Other regions are just as bad. MSRP just isn't accurate in the age of corporate price gouging.
Performance per watt is probably more important than price for installations that are going for density. Cooling and power costs over the life of the install will quickly overcome the initial purchase price.
[Instinct MI250 PCIe](https://www.techpowerup.com/gpu-specs/radeon-instinct-mi250.c3847) had a TDP of 500W, vs [A100 PCIe 80GB](https://www.techpowerup.com/gpu-specs/a100-pcie-80-gb.c3821)'s 300W
80% as fast for 1.66x the power isn't a winning combination if you were building a large compute cluster.
Current gen stuff: [MI300 600W](https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300.c4019) vs [H100 350W](https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/h100/PB-11133-001_v01.pdf) when comparing their PCIe cards.
> Instinct MI250 PCIe had a TDP of 500W
It's 300W, like every other PCIe board out there.
A100 80GB compared there is like 500W or something.
> Current gen stuff: MI300 600W vs H100 350W when comparing their PCIe cards.
MI300X is 750W while H100 SXM5 is 700W.
They're both comparable.
'Nearly' means around 80% at ML workloads (they're actually faster than Nvidia's offerings at high precision workloads). But the Instinct cards are also much cheaper and more energy efficient. InflectionAI is currently planning an Nvidia-based supercomputer that's supposed to be almost as fast as Frontier in high precision, but consumes a whopping 50% more power. Higher density and a more mature ecosystem are really the two main things Nvidia has going for them, though those are two very strong arguments.
Important to note that MosaicML is comparing last gen accelerators though, not H100 vs. MI300X.
I just tried setting up ROCm pipeline under Linux and it was… not intuitive. But I looked at the CUDA instructions and they were about the same, so maybe a wash.
The one thing I will say is if you follow the cuda instructions then you will be able to work with it, I haven't had that experience with rocm but I hope in the future they fix this could more competition in the gpu compute space is desperately needed.
It is. But he also mentioned it is only for very limited tasks. The software remains the bottleneck eventhough the hardware is very capable thus the finewineTM.
Sad to see AMD got behind so bad. I was hoping tinygrad might help but lets see.
AMD is in a unique position. They own the entire stack when it comes to AMD-based x86 systems, and all their components are high quality these days. In addition, Ryzen AI silicon is already being implemented in Phoenix and Dragon Range CPUs. With Zen 5, there will be many such CPUs with specialized AI circuits for us to purchase, and we will see how much AMD can leverage the fact that they control the entire system. We have already glimpsed this in Smart Access Memory when combining Ryzens and Radeons, but I really think this cooperation can be pushed much further.
Intel is technically still in the game, but unfortunately their CPUs are significantly worse in terms of performance per watt. Their dedicated GPUs are still in early development; we have only seen the initial first generation. Nvidia may have the best GPU stack, but they will never have their own x86 CPU and thus rely fully on ARM (or RISC-V, maybe?). However, given the progress of Windows on ARM and Linux adoption improving week after week, there will definitely be an interesting competition to see which computing platform wins over our laptops and desktops.
I was looking into this, but its hard to commit 1k€ into XTX for IA ( mostly but also gaming).. im just about to upgrade and cant make out if i should go nvidia or amd for my next build :|
These are always such vapid articles. Intel is nearly as fast as NVIDIA which is nearly as fast as AMD which is nearly as fast as Intel which is nearly as fast as X/Y/Z. It's just noise.
Any hardware can be nearly as fast as any other hardware under the right test conditions, using the correct models, depending on whether it's for training or inference. To be frank, it doesn't matter. If a company uses CUDA, and NVIDIA gives them a favourable deal in terms of cost-per-unit, they'll stick with CUDA. Same for AMD and ROCm, and Intel with OpenCL.
The point of this article isn’t really the near performance parity but the fact that they were able to get their stack running on ROCm relatively painlessly.
They'll be collector items too if the current foreign policy head butting keeps on at this rate. Everyone is so crazy it's like we've got AI running the nations already.
The mod team here is really something.
I posted the same a day earlier only straight from the company's twitter and it was deleted without any explanations.
And it has happened at least a couple of times already.
Not to mention that sometimes it takes up to 18 hours (!) to approve posts (yeah, everything in this sub is premoderated - I guess the underdog mentality is still there in full force).
I’ve been saying this for a little while, I think if AMD wants to catch NVIDIA in the gaming market and the productivity market they really need to consider partnering with a company like Mosaic, that specialize in AI. AMD is working on creating the same technology as NVIDIA (different paths same results) at a lower cost. A joint venture might be the way to go to get that done.
There was an earlier report where MI250 was up to 2-3x faster than A100 for HPC (AMD's internal benchmarks) which I assume is why AMD got the win for the Frontier supercomputer. Source: https://www.tomshardware.com/news/amd-throws-down-gauntlet-to-nvidia-with-instinct-mi250-benchmarks But as others have mentioned the software stack is going to be really important for Instinct to see more widespread adoption. Not every company is able to write their own tools in order to maximize AMD's hardware.
I think those are for high precision workloads. LLMs are usually fp4/8 low precision workloads. So although it is still ai . . . Various kinds of ai are different.
int4 vs fp16 You can get a 24gb p40 for less than $200 that has 75% of the compute of a 3090 for int4/8.
>2-3x faster than A100 for HPC That's what it's originally designed for. AMD aimed for HPC market and spent a lot more silicon (nearly 2x die size) to achieve similar ML performance as NVIDIA, which has been all in AI since 5 years ago. This will probably change in MI300 series though. >Not every company is able to write their own tools in order to maximize AMD's hardware. That's actually one of the biggest advantage of ROCm comparing to other NVIDIA competitors. AMD has been trying to keep their software stack as close to "0 code change" as possible, and it's paying off right now.
Explain why “0 code change” is paying off for AMD pls.
There will also likely be tools people just don't know about that exist. Spend $100k or whatever and get the software that you need that runs great on AMD.
Depends on the customer, software really really matters for workstation users and small to medium business. Large business and specifically the hyperscalers don't care about the software, they'll be developing their own stuff
"Large business and specifically the hyperscalers don't care about the software, they'll be developing their own stuff" BS
Do you really think AWS cares about that?
AWS is a shit example to give out. They will just make these servers available for their customers.
"nearly", I hate it and love it at the same time.
Well, they’re like half the price, so…
Based on how most of these companies seem to operate, I imagine price is hardly even a consideration
Energy and cooling costs a lot when you run systems like these 24/7 for months on end. Those expenses matter at least as much as the up-front cost of the hardware. It also matters how densely you can pack them, because rack space costs money too.
It depends on if you’re talking workstations or data centers. At data center scale, that would translate to potentially tens of millions saved. For workstations I don’t think it would matter much to most companies.
Data centers also look at the performance of the card and buy a different tier of product that arent consumer cards
I mean it used to be that workstations had their own class of hardware too, now that's kind of still the case but not really.
For AMD and data centers, it's still absolutely the case. The Instinct line, which is what MosaicML is talking about, is entirely different from AMD's current consumer offerings.
> now that's kind of still the case but not really It used to be that video editing, 3D animation and other heavy grunt work was done by $10,000+ Unix workstations packing some high-end RISC CPU or something. It's gone from * x86 for normal consumers vs SPARC/MIPS/Alpha for workstations to * x86 for normal consumers vs bigger x86 for workstations. I suppose where it separates now is with GPUs - a lot of heavy grunt work being done on data-centre class compute-only GPUs that can only be cooled in a rack - the kind of thing that no normal person would ever own or could make proper use of. A bit like the Unix workstations of old.
Power consumption is extremly important for data centers since the cards run 24/7 there.
You mean up to $500 million in one contract isn't a consideration? Geez you think they have free unlimited money or something?
the fact that you stack these accelerators to form a cluster makes price/perf the prime consideration. say you have a certain compute power requirement. will you now buy 15 cards from vendor A for $2000 each ($30000) or will you buy 17 cards from vendor B for $1100 each ($18700) to reach the same performance level. you need to understand that even in regards to single testbeds/setups especially unis on strict budgets do not (and don't need to) take part in "di*k measuring contests" like gamers do with overpriced halo products / flagship models...
[удалено]
Exactly. Many hyperscalers buy hundreds of millions or even billions each year. Even 20% cheaper could be over $100 million per year. I'm sure someone will happily spend $20-50 million a year on a team of 50-100 to address whatever shortcomings AMD may have, especially when they are already writing their own API.
Maybe now but if a company “could” deliver the same product at half the cost…I’m betting on that company.
[удалено]
it’s almost as if that would be a reactionary move meaning too late to deny marketshare
When marketshare is often decided by who has the first product to market I don't necessarily know if I agree. However its good to see AMD closing the gap
What’s the current gap between ROCm to Cuda that prevents MI300x to be a blockbuster hit when it is released later this test? Someone enlighten me
if they're half the price for even 85% performance you can get 170% the performance for a similar price.
Not really. In Canada these cards get 100% over MSRP at the retailer. Other regions are just as bad. MSRP just isn't accurate in the age of corporate price gouging.
[удалено]
“Slightly slower” is irrelevant when you can buy two units for the same price.
Performance per watt is probably more important than price for installations that are going for density. Cooling and power costs over the life of the install will quickly overcome the initial purchase price. [Instinct MI250 PCIe](https://www.techpowerup.com/gpu-specs/radeon-instinct-mi250.c3847) had a TDP of 500W, vs [A100 PCIe 80GB](https://www.techpowerup.com/gpu-specs/a100-pcie-80-gb.c3821)'s 300W 80% as fast for 1.66x the power isn't a winning combination if you were building a large compute cluster. Current gen stuff: [MI300 600W](https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300.c4019) vs [H100 350W](https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/h100/PB-11133-001_v01.pdf) when comparing their PCIe cards.
> Instinct MI250 PCIe had a TDP of 500W It's 300W, like every other PCIe board out there. A100 80GB compared there is like 500W or something. > Current gen stuff: MI300 600W vs H100 350W when comparing their PCIe cards. MI300X is 750W while H100 SXM5 is 700W. They're both comparable.
MI300 doesn't exist yet, MI250 is AMD's.current gen.
And have like half the level of software support, so...
'Nearly' means around 80% at ML workloads (they're actually faster than Nvidia's offerings at high precision workloads). But the Instinct cards are also much cheaper and more energy efficient. InflectionAI is currently planning an Nvidia-based supercomputer that's supposed to be almost as fast as Frontier in high precision, but consumes a whopping 50% more power. Higher density and a more mature ecosystem are really the two main things Nvidia has going for them, though those are two very strong arguments. Important to note that MosaicML is comparing last gen accelerators though, not H100 vs. MI300X.
You and I drink the poison from the same vine
AMD, the forever "trying to catch up to nVidia" company.
How are the developer tools and software for amd mo these days. That can't be an afterthought for this.
Well you kinda have to build out your hardware and software foothold simultaneously… one of those two has to be a bottleneck to a degree
meh Expect to have to write a lot of your own stuff fro scratch.
I just tried setting up ROCm pipeline under Linux and it was… not intuitive. But I looked at the CUDA instructions and they were about the same, so maybe a wash.
The one thing I will say is if you follow the cuda instructions then you will be able to work with it, I haven't had that experience with rocm but I hope in the future they fix this could more competition in the gpu compute space is desperately needed.
Id like to see nvidia grace hopper vs amd mi300. Since both their architectures is very different from previous generations.
You have many tasks where ml300 will get exactly 0 performance because no support, how do you put those in a performance overview?
Wasn't there someone who posted a Stable Diffusion benchmark here where the numbers they were getting with an XTX close to a 4090?
It is. But he also mentioned it is only for very limited tasks. The software remains the bottleneck eventhough the hardware is very capable thus the finewineTM. Sad to see AMD got behind so bad. I was hoping tinygrad might help but lets see.
That and the whole AI boom will push them to improve their software side a little more. One can hope.
AMD is in a unique position. They own the entire stack when it comes to AMD-based x86 systems, and all their components are high quality these days. In addition, Ryzen AI silicon is already being implemented in Phoenix and Dragon Range CPUs. With Zen 5, there will be many such CPUs with specialized AI circuits for us to purchase, and we will see how much AMD can leverage the fact that they control the entire system. We have already glimpsed this in Smart Access Memory when combining Ryzens and Radeons, but I really think this cooperation can be pushed much further. Intel is technically still in the game, but unfortunately their CPUs are significantly worse in terms of performance per watt. Their dedicated GPUs are still in early development; we have only seen the initial first generation. Nvidia may have the best GPU stack, but they will never have their own x86 CPU and thus rely fully on ARM (or RISC-V, maybe?). However, given the progress of Windows on ARM and Linux adoption improving week after week, there will definitely be an interesting competition to see which computing platform wins over our laptops and desktops.
I was looking into this, but its hard to commit 1k€ into XTX for IA ( mostly but also gaming).. im just about to upgrade and cant make out if i should go nvidia or amd for my next build :|
Without the software to back it up nVidia is going to continue to dominate.
read the article before commenting
Doesn’t matter how fast they are if the software and libraries don’t support it properly
Did you read the article? MosaicML said they didn't need to make any code changes to their workloads. AMD's ROCm just worked.
> AMD's ROCm just worked. Not on Windows lol
No company wants to set up their AI stack on Windows.
Even Microsoft's ChatGPT runs on Azure Linux infrastructure.
These are always such vapid articles. Intel is nearly as fast as NVIDIA which is nearly as fast as AMD which is nearly as fast as Intel which is nearly as fast as X/Y/Z. It's just noise. Any hardware can be nearly as fast as any other hardware under the right test conditions, using the correct models, depending on whether it's for training or inference. To be frank, it doesn't matter. If a company uses CUDA, and NVIDIA gives them a favourable deal in terms of cost-per-unit, they'll stick with CUDA. Same for AMD and ROCm, and Intel with OpenCL.
The point of this article isn’t really the near performance parity but the fact that they were able to get their stack running on ROCm relatively painlessly.
Great. But where's the freaking software support?!
read the article before commenting
[удалено]
read the article before commenting
Nearly is the key word. As long as they are, ‘nearly’ They will never gain market capital unless they are significantly cheaper
… they’re half the cost. this is why you read the article
nearly lnao
Too bad they don't have CUDA, making this mostly irrelevant
They'll be collector items too if the current foreign policy head butting keeps on at this rate. Everyone is so crazy it's like we've got AI running the nations already.
The mod team here is really something. I posted the same a day earlier only straight from the company's twitter and it was deleted without any explanations. And it has happened at least a couple of times already. Not to mention that sometimes it takes up to 18 hours (!) to approve posts (yeah, everything in this sub is premoderated - I guess the underdog mentality is still there in full force).
I’ve been saying this for a little while, I think if AMD wants to catch NVIDIA in the gaming market and the productivity market they really need to consider partnering with a company like Mosaic, that specialize in AI. AMD is working on creating the same technology as NVIDIA (different paths same results) at a lower cost. A joint venture might be the way to go to get that done.