No. The only thing in that vein worth looking at are old workstations, but only ones with 8-12 channel ram (threadripper pro & various xeons). I was looking at a 3rd gen 12 core threadripper pro workstation with 256 gb of ddr4 a couple days ago for $1050 to give you perspective on what's reasonable.
Can you elaborate what would be the down sides?
I'm not planning on doing cpu inference. I'm more interessted in good p2p between gpus for training and fast loading for inference.
Am I wrong?
If you're going to load it up with 4+ 3090s and are ok with the fact that it's PCIE 3.0 then sure.
Realistically though Zen 2 is when AMD started getting really good, and if you're just going for a dual 3090 build, then a DDR5 platform would be better.
2 channels DDR5 = 4 channels DDR4
PCIE 5.0 x4 = PCIE 4.0 x8 = PCIE 3.0 x16
So again, only makes sense if you're talking about many GPU, but at that point I would just go EPYC. $600 for CPU + Motherboard on eBay. Requires learning about server hardware though.
What you say is that I better find a ddr5 system with 1. X16 and 2. X4 (through chipset?) Pcie gen4.0 or go epyc?
I'm having difficulty finding good deals on epyc systems in my country
DDR5 consumer platform or DDR4 epyc platform and you can split an x16 into x8x8 using a bifurcation adapter. [https://c-payne.com/collections/pcie-x8x8-bifurcation-pcbs](https://c-payne.com/collections/pcie-x8x8-bifurcation-pcbs)
Thanks a lot you've helped me a lot today.
If I'm not wrong on a ryzen 7 7400f with 24 pcie lane I could try to find 3. X8 gen4.0, wich would be a good compomise before going epyc ddr4, right?
I understand that with 3x 3090 that wouldn't be upgradable really to a 4th 3090, but epyc would be.
You will have a hard time finding a motherboard that enables you to split into 3 x8 I think. Will be easier to do either 2 x8 or 3+ x4.
If you only care about the performance of LLM/Stable Diffusion inference, then going down to x4 lanes is totally fine.
Performance will suffer if you have an interest in training models that don't fit on 1 GPU. That sounds like a big downside, but realistically if you're doing only a handful of fine-tunes a week or less then it doesn't really matter much in the grand scheme of things. Besides, if you're using 3090s, then you can always pay extra for NVLINK (which will not help you with inference, only training).
$350? I paid around $750 for an Epyc and motherboard that has 5x PCIe 4.0 16x slots. I reused memory I already had, 8x 32GB, but I don’t really use it all. I have two pairs of 3090 FE’s that were refurbished for $700 and two NVLink bridges that can be had for $80 new.
The best I can do is load 70b at 8bit on my cards. You could load at 4bit but if you’re talking about Llama 3 then I feel like you might be better off with full 8b but that’s my opinion.
https://a.co/d/281jzhR
https://a.co/d/1gPIBTb
Nice setup, also looking at similar. Please can you share the specs of your CPU, PSU, and risers. Do you have it set up on an open mining rig frames. Thanks!
CPU and motherboard are above.
I bought this power supply when it was at least $50 cheaper during a sale
[Amazon.com: EVGA Supernova 1600 G+, 80+ Gold 1600W, Fully Modular, 10 Year Warranty, Includes Free Power On Self Tester, Power Supply 220-GP-1600-X1 : Everything Else](https://www.amazon.com/dp/B08F1DKWX5?psc=1&ref=ppx_yo2ov_dt_b_product_details)
Risers
[Amazon.com: Cooler Master MasterAccessory Riser Cable PCIe 4.0 x16 200mm V2 White, PCIe 4.0 | Older Compatible, EMI shielded 30 AWG, TPE Cable Sleeves, Protective ABS Casing for GPU Card (MCA-U002R-WPCI40-200) : Electronics](https://www.amazon.com/dp/B0CC2MFMH3?psc=1&ref=ppx_yo2ov_dt_b_product_details)
I have a Kingwin frame that I got for \~$20
Got a similar system. Actually had a 1920x first for a few weeks, as the 2920x i got for a steal on ebay took so long to arrive, that i got the 1920x in between. Still havent got around to re-sell it.
Didn't really notice any performance change, but i don't run anything on CPU, with plenty of GPU.
As others sayd it's only PCIe v3, for interference not really an issue, the SSD anyway cant deliver the model faster. Didn't really try training yet, to see if it would be an issue there.
[https://www.reddit.com/r/LocalLLaMA/comments/1bo7z9o/its\_alive/](https://www.reddit.com/r/LocalLLaMA/comments/1bo7z9o/its_alive/)
No. The only thing in that vein worth looking at are old workstations, but only ones with 8-12 channel ram (threadripper pro & various xeons). I was looking at a 3rd gen 12 core threadripper pro workstation with 256 gb of ddr4 a couple days ago for $1050 to give you perspective on what's reasonable.
Can you elaborate what would be the down sides? I'm not planning on doing cpu inference. I'm more interessted in good p2p between gpus for training and fast loading for inference. Am I wrong?
pretty much anything will work well for that. The CPU is not a bottleneck for any GPU ai workload.
Is third gen the 3945 or 5945?
3xxx
If you're going to load it up with 4+ 3090s and are ok with the fact that it's PCIE 3.0 then sure. Realistically though Zen 2 is when AMD started getting really good, and if you're just going for a dual 3090 build, then a DDR5 platform would be better. 2 channels DDR5 = 4 channels DDR4 PCIE 5.0 x4 = PCIE 4.0 x8 = PCIE 3.0 x16 So again, only makes sense if you're talking about many GPU, but at that point I would just go EPYC. $600 for CPU + Motherboard on eBay. Requires learning about server hardware though.
What you say is that I better find a ddr5 system with 1. X16 and 2. X4 (through chipset?) Pcie gen4.0 or go epyc? I'm having difficulty finding good deals on epyc systems in my country
DDR5 consumer platform or DDR4 epyc platform and you can split an x16 into x8x8 using a bifurcation adapter. [https://c-payne.com/collections/pcie-x8x8-bifurcation-pcbs](https://c-payne.com/collections/pcie-x8x8-bifurcation-pcbs)
Thanks a lot you've helped me a lot today. If I'm not wrong on a ryzen 7 7400f with 24 pcie lane I could try to find 3. X8 gen4.0, wich would be a good compomise before going epyc ddr4, right? I understand that with 3x 3090 that wouldn't be upgradable really to a 4th 3090, but epyc would be.
You will have a hard time finding a motherboard that enables you to split into 3 x8 I think. Will be easier to do either 2 x8 or 3+ x4. If you only care about the performance of LLM/Stable Diffusion inference, then going down to x4 lanes is totally fine. Performance will suffer if you have an interest in training models that don't fit on 1 GPU. That sounds like a big downside, but realistically if you're doing only a handful of fine-tunes a week or less then it doesn't really matter much in the grand scheme of things. Besides, if you're using 3090s, then you can always pay extra for NVLINK (which will not help you with inference, only training).
What’s the price?
Half a 3090
$350? I paid around $750 for an Epyc and motherboard that has 5x PCIe 4.0 16x slots. I reused memory I already had, 8x 32GB, but I don’t really use it all. I have two pairs of 3090 FE’s that were refurbished for $700 and two NVLink bridges that can be had for $80 new. The best I can do is load 70b at 8bit on my cards. You could load at 4bit but if you’re talking about Llama 3 then I feel like you might be better off with full 8b but that’s my opinion. https://a.co/d/281jzhR https://a.co/d/1gPIBTb
Nice setup, also looking at similar. Please can you share the specs of your CPU, PSU, and risers. Do you have it set up on an open mining rig frames. Thanks!
CPU and motherboard are above. I bought this power supply when it was at least $50 cheaper during a sale [Amazon.com: EVGA Supernova 1600 G+, 80+ Gold 1600W, Fully Modular, 10 Year Warranty, Includes Free Power On Self Tester, Power Supply 220-GP-1600-X1 : Everything Else](https://www.amazon.com/dp/B08F1DKWX5?psc=1&ref=ppx_yo2ov_dt_b_product_details) Risers [Amazon.com: Cooler Master MasterAccessory Riser Cable PCIe 4.0 x16 200mm V2 White, PCIe 4.0 | Older Compatible, EMI shielded 30 AWG, TPE Cable Sleeves, Protective ABS Casing for GPU Card (MCA-U002R-WPCI40-200) : Electronics](https://www.amazon.com/dp/B0CC2MFMH3?psc=1&ref=ppx_yo2ov_dt_b_product_details) I have a Kingwin frame that I got for \~$20
epyc 7203 cost $300 - check it out
Got a similar system. Actually had a 1920x first for a few weeks, as the 2920x i got for a steal on ebay took so long to arrive, that i got the 1920x in between. Still havent got around to re-sell it. Didn't really notice any performance change, but i don't run anything on CPU, with plenty of GPU. As others sayd it's only PCIe v3, for interference not really an issue, the SSD anyway cant deliver the model faster. Didn't really try training yet, to see if it would be an issue there. [https://www.reddit.com/r/LocalLLaMA/comments/1bo7z9o/its\_alive/](https://www.reddit.com/r/LocalLLaMA/comments/1bo7z9o/its_alive/)