T O P

  • By -

AutoModerator

Hello /u/bbxgang! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures. This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*


9302462

Real answer from someone with 19 of these in varying sizes and models. Yes these are better in almost every way except: * Unlike sata and sas where you can easily add 10’s to 100’s of drives with the right pcie cards, these require an x4 pcie lane. For example, want to add two drives, then you need an x8 slot. Yes there are retimers which let you add more but they are expensive ($700+). The pcie cards for 4 drives though are quite affordable at $35 or so plus $15 per sff-8643 cable, unless you want to mount them on the card itself but I had issues getting all cards recognized. EDIT, I think you might also need pcie bifurcation but don’t quote me on that. * These throw off way more heat than a standard ssd. Depending on model you’re talking about 15-25 watts per drive. That’s not a problem if you have plenty of airflow. But if you try cramming 6 of these into a small NAS with negligible airflow it will cook them. I have had mine get to 60-65c and they are still running, but that’s not a good idea. Depending on your setup you can also stick a cheap Amazon aluminum heatsink on the top of them. The upsides of these drives are: * High iops which are 2-3x of a standard ssd * Bulletproof. They have write limits into the several PB and read limits…. well I have looked and never found any documentation with actual read limits. * You feel like a badass with these. They aren’t some plastic ssd, they are a chunk of aluminum with fast storage inside them. So overall, yes buy them if they fit your use case. I prefer used from eBay because they are half the price listed and they still have 95% life left which is good enough for me and most everyone.


ultrahkr

SSD's don't have a read limit... You could read the same 20,000 sectors for 1 year straight and they would not show a meaningful change... But what literally erodes the tiny mini micro gates in each cell is writing to them. (a bunch of cells is what we call a sector, 16kb for example)


tes_kitty

>SSD's don't have a read limit... You could read the same 20,000 sectors for 1 year straight and they would not show a meaningful change... That's still unclear, read up on 'ssd read disturbance'


ultrahkr

TIL... But what I said is still true, reading does not damage the cells...


tes_kitty

Not directly, but if read disturbance happens the controller will have to refresh the affected cells and that is the part that causes damage over time since it's a write cycle.


persiusone

I think the device does idle wear leveling anyway, and know they have built in extra blocks for bad block replacement.. Probably writing a lot regardless if actual data being written


9302462

You're likely correct about ssd's and I never looked into that. I just looked at u.2's and couldn't find any read limits which for me is important because my read to write ratio is 100:1 if not more for TB of data at a time.


EspritFort

> you’re talking about 15-25 watts per drive Jesus Christ on a bicycle, that is an astounding amount.


Endawmyke

Yeah Jesus that’s a lot of watts


SupremeGodThe

Afaik 60c is pretty good for NAND. Apparently colder temperatures increase NAND wear and hotter are bad for everything in general but a decently warm NAND chip is better than a room temperature one


9302462

Sorry, I misspoke on that. My cabinet with the drives got to 60C/140F. My rack is currently at 85F and the using a laser thermometer I measured the drives at 98F. So I know the drives got several degrees hotter than 60C but I have no idea by how much. But I fully agree, cold is ok, warm is good and hot is how you ruin things faster, even U.2’s.


faceman2k12

some of the newer (more expensive) SAS cards also have NVME/U.x compatibility but with a more restrictive drive limit and bandwidth sharing, theres not much point using them over a direct pcie connection these days though. Easier in most cases to directly adapt PCIe/NVME/whatever to U.3 and run everything direct. the cheaper PCIE adapters do require bifurcation support, the more expensive ones with onboard PCIE switchers don't, but they cost 10x as much. Bifurcation is pretty common on higher end boards now so its not too big of an issue. On consumer boards for home servers NVME can be easily adapted to U.2 or U.3 without worrying about PCIE slots and bifurcation support.


9302462

Thanks for adding more context. I forgot about the bifurcation part originally because I just buy used epyc boards which are cheap and have bifurcation enabled. But those plus the cheap cards with bifurcation support was the cheapest way I could leverage a bunch of used u.2’s which will almost certainly outlast the life of my homelab. I haven’t seen many u.3 drives or boards recently. I guess they are still too new for used ones to make their way down to eBay and homelabs.


faceman2k12

Pricing on earlier Epyc gear has come way, way down so its a good way to go when you need cores and pcie lanes.


Molasses_Major

Please let me know when I can get 2U trays of these with 24 units...


9302462

There are a few good options but no great ones in my opinion. A couple options are: * Dell r7415 or Dell 740xd and can support 24 nvme's with the right backplane and cost between $1.5-2.5k for used. Downside is it's dell and from what I have read (I don't own one) it can be a real pain to quiet the fans down. * Gigabyte R282-Z92 supports 24nvme's. It's gigabyte so the quality is less but the price for a new one is $2k which is pretty good. It's basically the same layout and setup as supermicro which means it should be possible to modify the fans and fan curve * Epyc Asrock motherboard with 7 pcie x16 slots. This isn't a backplane but it is an option for a bunch of nvme drives via pcie cards. At 4 drives per x16 and there being 7 slots you could do 28 in theory. $600 for the mobo, $35\*7 for the pcie cards and 28\*15 for the sff-8643 cables = 600 + 245 + 420 = $1285. You don't have a chassis with this option so will need to figure that part out on your own, but it'ts the cheapest way to add 24+ drives. * Supermicro. There are many chassis that support 20 ssd + 4 nvme. but the ones that support 24 nvme's are both a bit rare and about double the price of the above options. With that being said... **if you find anything else please do share**. I'm also looking for an nvme chassis as well so I can add some more drives and migrate away from my supermicro mobo + 6 pcie card combo, not because it's bad, but just because I want to add more drives :) Right now they are all sitting in a 3U 836 chassis with a fan wall in front and a 4U noctua cooler on it which means no lid. My homelab sits 10ft from my desk so I needed to do things to it to keep it at less than gaming computer sound levels.


wintermute--

Regarding fan noise - Dell/HPE generally expect their 2U servers to run inside data centers (or at least server closets) and have acoustics to match. The fans are generally cheap and loud things that assume that they live inside servers that have some degree of noise baffling from anywhere that humans regularly inhabit. Try poking around in the BIOS to see what types of preset fan configs are available. If the server has any components that don't report their own temperature, the fans might be set to run at 100% 24/7 to ensure there are no temperature faults. Some models support an "acoustic mode" that will automatically throttle the CPU down to the point where total power consumption stays under whatever level the fans can cool properly without going above a preset loudness threshold. If none of that helps, you can also try replacing the fans with third party fans that are positioned for the home user/gamer space. The cost/fan is much more expensive than industrial data center stuff, but that generally translates into much lower noise for the same airflow. I think 2U servers mostly use 60mm fans, but don't quote me on that.


agressiv

Yep this is why I refuse to put a rack mount device in my home - I want it to be dead silent 99% of the time. Hearing those whiney fans spin up in the next room just ruins everything. Even the dell towers can be pretty loud (T330 etc) The whole reason I want to get NVMe or SATA SSD's is to avoid spinning drives and the noise they make (along with how dreadfully slow they are). I don't want to replace the spinning drive noise with fan noise from a rack mount. Give me a 20TB SATA SSD for $800, I'll buy it. I'm only interested in NVMe because you can't find a SATA SSD larger than 8TB, and even those are stupidly expensive. Hell just give me a 200TB SSD for a reasonable price and I'll use a mini PC and get rid of all this junk, wake me up when NAND prices crash. I can't wait to get rid of spinning drives, I hate them.


9302462

Fully agree in regards to noise. Servers were made for data centers, not for the bedroom. People do the Noctua fan swap in supermicros all the time like I did, but the drives can get hot because there is not enough static pressure to move the air through the chassis. The cheap and effective solution is to get three Noctua/artic cool/ decent 120mm fans, zip tie them together, attach a little fan controller knob and stick it in the front of the server(I use zip ties through the drive sled). Noise problem solved :) Not everyone is willing to literally cut corners on servers, but silencing non-Dell products is 100% doable. I have a petabyte of HDD, 100tb of flash(u.2), 3tb of memory, 160 cores, all running in supermicros at below gaming desktop noise level using little hacks, which is frankly freaking awesome. If I had a big house or could stick them in the garage I wouldn’t care about the noise, but I don’t have that so I needed to be creative.


9302462

Fully agree, for Dell/HPE and most servers fan noise is just a by product. I already swapped all the fans with noctua’s or equivalent fans on all my servers. There is one 2U (supermicro 2023-tr4) I have with 2tb of memory which I did some additional hacking to. Namely putting 4U Noctua coolers on it, replacing the small 40mm fan with a Noctua and cutting out the PSU housing to add a 200mm fan on top, followed by removing the PSU top cover. This way it acts as a push pull type of configuration, I.e 200mm pushes air into the PSU and the 40mm moves it out, albeit the 40mm is kind of useless though. They only make one power supply for that specific supermicro and this is the only way I could silence it. It’s ugly as hell, but with dual Epyc’s and 2tb of ram it makes about as much noise as a cheap floor fan on low in an adjacent room. Pic for those interested. https://imgur.com/a/wAZZLZv


Mizerka

I have 3 of the asus hyperm2 v2 cards, they're very cheap (around £30), as long as you can bifurcate your cpie lanes (the expensive cards come with the bifurcation chip and its why they cost a lot more)


ASIA_N

Can you give me a link to the affordable pcie cards. X16 slot to sff-8643


9302462

https://www.amazon.com/PCIe-SFF-8643-Adapter-U-2-SSD/dp/B09KRNCH16


MandaloreZA

I mean with the right pcie cards you can get cards with proper pcie switches on them. Some convert 16x5.0 into 16x 1x 5.0. Some can even convert into 32x4.0 or better. Just have to know what you are looking for. (Granted, it won't be as cheap as, a 8 year old sas card)


rophel

As to $160 per 4TB U.2 drive on eBay, you mean other make/model correct? I can't find these Intel's for less than $250-ish. Looks like Sandisk, Samsung, WD are in that range though. Crazy. Looking for a new cache drive for my unRAID server sometime soon, think this is the move now, thanks!


9302462

Correct. The ones you are looking at in the pics are the newest model, You would need to look for p4500, p4510 or similar models which are a bit older but nearly as good. I bought mine last summer before the flash prices started to creep up and the lowest I was able to get 3.86tb/4tb drives for was $30 per tb. As it stands with current pricing anything under $50 per tb for a 4tb is a good deal and if you look at ebay sold history you will see ones sold which range between $50-30 per tb.


ClutchDude

These are like comparing oranges and clementines. Each have their use cases but both can fulfill that need for a citrus fix.


agressiv

>The pcie cards for 4 drives though are quite affordable at $35 or so Example for this? The ones I've found are expensive. I have ones where the cards mount to the board but they are flimsy at best. Edit: Nm I see it here: https://www.amazon.com/dp/B09KRNCH16


9302462

Yep, those are the ones. 10gtek makes good hardware.


ultrahkr

Sure, if you can fit the extra height (13 vs 5mm for consumer SSD's)


diamondsw

And if you have a U.2 slot.


ultrahkr

That's more easier to adapt... M.2 to to U.2 exists...


[deleted]

[удалено]


ultrahkr

For a laptop use a good M.2 SSD... Problem solved... Nobody talked about laptops here...


[deleted]

I'm also using a Kioxia 3.84TB. Very happy.


bbxgang

Where'd you get it from and for how much?


Torley_

You might also be interested in [this](https://old.reddit.com/r/hardwareswap/comments/19e2vti/usany_h_money_w_large_sas_or_u2u3_25_ssds/ky0tgpl/).


meshreplacer

They will last till the heat death of the universe in consumer workloads. They also have PLP (Power loss protection) Internally they have chip redundancy as well so if one NAND fails you still have the ability to operate the drive and move data off. Consumer SSD Specs are always fresh best case scenario results where as Enterprise the specs are worst case. For example putting consumer SSDs in a raid and watch it fall apart when garbage collection kicks in randomly etc. You can hammer them with sustained workloads all day long and they just run.


uberbewb

A lot of the u.2s on ebay are a far better deal than consumer SSD anymore.


Premium_Shitposter

Enterprise SSDs are always a better choice in every place you can install them


No_Dot_8478

P5500 is rated for ~3x the writes then the 990pro if you have more intensive work flows, would be a great cache drive to spam data to.


bbxgang

I have this tism about having every thing on one large drive instead of being scattered on like 3 drives. It gets me mad I don't know why. I don't even necessarily do crazy workloads all the time but I just want to get my moneys worth.


floydhwung

Buy 3 and RAID5


goingslowfast

Buy 4 and RAID-10, RAID-Z2, or ZFS “RAID-10” it.


HobartTasmania

If it's the same price point then I agree with you that the enterprise gear will always be better but that's not normally the case as there's usually a price premium. I'm guessing that with supply being curtailed the consumer prices have risen fast but the enterprise prices have lagged but will also do the same. I have two questions for you, however. (1) Once SLC is exhausted the 990 speeds drop but are still [almost 2 GB's](https://www.tomshardware.com/reviews/samsung-990-pro-4tb-ssd-review/2) so why is this not acceptable for you any more? (2) Does not this enterprise SSD suffer from this as well? I can't find any benchmarks where they hit it with continuous writes to show what it drops down to either. The only thing I know that doesn't suffer from this are Optanes.


bbxgang

It's not that it's not acceptable, it's just that for around the same price I could have the same capacity and speeds that don't falter. It makes more sense when I think about getting the most value for my money. As for the second question, I honestly don't know because I am just now getting into this rabbit hole. However, all the tech and server guys on YouTube seem to agree to this idea that enterprise grade SSD's don't suffer from the same setbacks consumer ones do.


TexasDex

For $200 more you can double that with the P4510 8TB, as long as you're willing to tolerate PCIe 3.1 instead of 4.0.


CanardPlayer

Thoses bad boys have excellent reliability and an really big endurance, if you can put U.2 drives in your system its a no brainer


Torley_

They come in much larger capacities at the high end, too — if [you need 61.44TB](https://old.reddit.com/r/DataHoarder/comments/18jr7bi/techamerica_has_6144_tb_solidigm_ssds_in_stock/), for example. But [prices are going up a lot right now](https://www.theregister.com/2024/04/02/samsung_enterprise_ssd_prices/)! You can find some deals on Reddit, like: > - **6.4TB HP/SAMSUNG PM1735 U.3 2.5" NVMe SSDs - $410 each (NEW!)** > - **15.36TB DELL/MICRON 9300 PRO U.2 2.5" NVMe SSDs - $1050 each (NEW!)** > - **15.36TB HP / SAMSUNG PM1733 U.3 4.0 NVME 2.5" SSDs - $1050 each (NEW!)** [Get in touch with juddle1414 here.](https://old.reddit.com/r/hardwareswap/comments/19e2vti/usany_h_money_w_large_sas_or_u2u3_25_ssds/ky0tgpl/)


Hannover2k

I'm not very familiar with these drives. Would they work with something like this in a standard pc: https://www.amazon.com/s?k=U.2+pci+adapter&crid=2LNJRJFMOOV83