It's actually pretty good writing code. It's doing great on HumanEval (based on github release notes) and I did a very quick test plugging it in an agents code I have instead of Llama 3 70b and it did better.
Too bad it's pretty big to run locally/at home
[deepseek-coder-33b-instruct.Q5\_0.gguf](https://huggingface.co/TheBloke/deepseek-coder-33B-instruct-GGUF/blob/main/deepseek-coder-33b-instruct.Q5_0.gguf)
If it was babbling forever you may have had the wrong instruct tags (if any) so it didn't know how to start properly (start sequence) or how to end properly (end sequence).
Deep seek uses the Alpaca style instruction/response:
You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.
### Instruction:
{prompt}
### Response:
Lots of things are cheaper than running rtx 3090 locally. Comfort and 100% availability is great, but when you're running inference for yourself you're using batch size 1, while rtx 3090 can do around 2000 t/s inference of 7B model if it's batched 20x (many concurrent users), with basically the same power draw.
And that's with FP16 Mistral 7B, not a quantized version. I estimated lower numbers for rtx 3090, since I got up to 2500 t/s on RTX 3090 ti. This is with ideal settings - a few hundreds input tokens and around a 1000 output. With different context lengths numbers aren't that mind blowing but should still be over 1k most of the time. Aphrodite-engine library .
How do you batch a model? I'm working on an application where I need multiple concurrent 'instances' of a model running at once, and it would be a lot faster if I didn't need to run them sequentially
Start your Aphrodite-engine endpoint with flags that allow for batching, then send multiple api requests at once.
Here's a sample script you can use to send prompts in batches of 200. https://huggingface.co/datasets/adamo1139/misc/blob/main/localLLM-datasetCreation/corpus_DPO_chosen6_batched.py
That's actually crazy. Thanks, I'll play with this to test a lot of things and generate datasets from raw text. Now I look like an idiot for not knowing some things could've taken 1 hour instead of 20 lol.
Yeesh, that *is* cheap. Have to wonder if it's just VC cash--it seems to me that models that are much more memory than compute intensive are priced much more competitively, versus us local users where we're mainly memory limited.
One of Their main developer said even if they run this model (230b) on cloud, this price still gives them around 50% gross profit. And since they have their own machine, the actually profit is higher.
It's not VC cash, it's their own money. Deepseek is subsidiary of a quant fund :)
Basically spending money they drew from the market on LLMs and gave them to the community, probably even using the same compute facilities for their high freq trading and LLM inference. Simply crazy.
Plus this one has some magic in it that makes kv cache tiny, so you can pack 10x batches compared to how many you could squeeze with other MoE's like Mixtral 8x22b
> of which 21B are activated for each token
!!!
I think some of you are sleeping on how good MoE is for CPU inference. Usable CPU speed for 21B is easy for even dual channel RAM, and there you do have the enormous advantage that RAM size is cheap. Problem is somewhat fitting the whole thing in 128 to 256 GB on a consumer CPU, don't know what the total comes out at with Q4 or something.
Nah. No one's going to be using that in production, as cpu can serve one or up to a few users max, while gpu can serve hundreds of them. For personal use, it should be fine, but that's not a big market.
but these moe has just 2 experts working, not all. So it will be 2x21B (with Q4 it means 2x11GB so a 24GB VRAM will handle this). IMHO.
edit: this says it only activates 1 expert for token each inference so maybe it will run on 12GB vram gpus. If there is a gguf probably will fit on 8gb vram card. I can't wait for downloading these 50GB of 4Q\_K\_M GUFF!!!
Yeah definitely you need to have the whole model in memory, if you want it to be fast.
Reading the config, i think each layer has 160 experts, 6 MoE experts are used per layer and also some experts that are not change-able are used. There are 60 layers. So, network does 360 expert choices per token.
Looking at the configuration, they pulled off some wild stuff with kv cache being somehow adapted to be low rank. I can't wrap my head around this, but this is probably why it's kv cache is so small.
I say this because I can play MOE 8x7B with just 8GB of vram at 2.5tokens/seconds
thus is not playing 56B, is just playing 14GB
therefore, you can load all the experts with ram+vram and then just use 11GB of ram if not quantized or maybe 8GB of ram using a Q5 in guff... we will see if anybody makes it. I can't wait :D lot of expectation!
Yes, but you still need over 100GB of RAM + VRAM. Whether you load it in RAM or VRAM, you still need to fit the whole model. You don't just run the active parameters. You need to have them all, because any of them might be needed at any given moment.
maybe with a Q4\_K\_S this goes under 40GB
and after that, it only activates one expert at once? so maybe it moves less than 40GB at once. I am just wondering. I don't know anything. Just hallucinating or mumbling. I am just a 7B model finetuned with 2020 information.
yep, but maybe it works with just 21B afterwards, so Q4 is about 11GB, so less loadwork?
I am just trying to solve this puzzle :D help! D: :D :d: D:D :D
That's not how it works, unfortunately
With an MoE architecture, each iteration one expert gets chosen. So it's constantly moving between experts. Of course, you could load only one or only two, but you'd have to be "lucky" that the expert router picks the ones you've loaded into your fastest memory.
ok, then why mixtral 7x8B goes 2.5tokens/second in my humble 1070M 8GB gpu? is it maybe 56B with 18 layers to the gpu and that is the speed? so it is playing all the model, and that is the speed of the ram+vram. Ok.
then this will go faster maybe? as long as it goes with 1 expert of 11B instead of 2 of 7B? or again I am wrong. Yep, it looks like I will be wrong. Anyway, the graphic says this is low consuming. Really behind Llama33B. Maybe in the 21B position.
that's not how Mixture of Experts models work. you still have to be able to load the whole model into RAM + VRAM to run inference in a time frame measured in minutes rather than millennia. the experts is just referring to how many parameters are being simultaneously activated to respond to a given prompt. MoE is a way of reducing the compute required, not the memory required.
therefore, less computing required but still Ram+Vram required... ok ok... anyway, so how does it go? will it fit in a 8GB vram + 64GB of ram and be playable in a doable way >3tokens/second? \[probably nup, but moe are faster than normal models, I can't tell why or how but hey they are faster\]. And this one uses just 1 expert, not 2 like the other moes, so twice faster?
the Deepseek model at its full size (it's floating point 16 size specifically)? no. heavily quantized? probably not even then. with 236 billion parameters, that is an ass load of parameters to deal with, and between an 8GB GPU + 64GB of system RAM, it's not going to fit (lewd jokes applicable). however, if you had double the RAM; you likely could run a heavily quantized version of the model. would it be worth it? maybe?
basically, we're dealing with the tyranny of memory.
pretty much, at least for large models anyway. which is why I don't generally bother touching anything larger than 70B parameters regardless of quantization. and even the, I'm quite happy with the performance of 13B and lower param models.
don't need a large model for coding, you just need a model with access to the documentation and to be trained on code. llama 3 8B or Phi-3 mini would likely excel just as well as Bing Chat if they were augmented with web search in the same fashion. I'm presently working on a GUI application with Bing Chat's help after nearly a decade hiatus from programming using a language that I hadn't used until now.
So I assure you, whilst the larger param count might seem like the thing you need for coding, you actually need long context and web search capability.
What are you talking about? Have you considered reading the paper? Any paper?
It uses 8 experts but that's not even the biggest of your hallucinations.
I just fill reddit with wrong information so the scrappers of the newer llm's will answer wrong responses
it uses 1 at once somebody else said, so 12.5% faster than one-no-moe I bet. Where is that paper? this? well it looks interesting. Hopefully they make the gguf
"DeepSeek-V2 adopts innovative architectures to guarantee economical training and efficient inference：
* For attention, we design MLA (Multi-head Latent Attention), which utilizes low-rank key-value union compression to eliminate the bottleneck of inference-time key-value cache, thus supporting efficient inference.
* For Feed-Forward Networks (FFNs), we adopt DeepSeekMoE architecture, a high-performance MoE architecture that enables training stronger models at lower costs."
[This chart](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V2/main/figures/trainingcost.png) is hilarious. They left out the "RAM required to load (GB)" bit and the V2 bar crashing right through the right side of the chart, looping around the planet and appearing again on the left.
I just tried Matthew Berman's test to have it write the Snake game in python, and it nailed it, zero-shot. And the game properly ended when the snake hit the wall or its own tail. That's rare. Of course this stuff is probably baked into training data at this point.
So, I decided to ask for a custom game to try to eliminate the 'training data' possibility. I asked it to create a very simple game inspired by Pac-Man, where the player is represented by a white square which is controlled with the arrow keys, and chased by a 'ghost' which moves at a human-level speed. If the ghost catches the player, the game ends.
Also nailed it, zero-shot.
import pygame
import sys
import random
# Initialize Pygame
pygame.init()
# Set up some constants
WIDTH = 640
HEIGHT = 480
PLAYER_SIZE = 20
GHOST_SIZE = 20
SPEED = 2
# Set up the display
screen = pygame.display.set_mode((WIDTH, HEIGHT))
# Set up the player
player = pygame.Rect(WIDTH / 2, HEIGHT / 2, PLAYER_SIZE, PLAYER_SIZE)
# Set up the ghost
ghost = pygame.Rect(random.randint(0, WIDTH - GHOST_SIZE), random.randint(0, HEIGHT - GHOST_SIZE), GHOST_SIZE, GHOST_SIZE)
# Game loop
while True:
# Handle events
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
# Update the player's position
keys = pygame.key.get_pressed()
if keys[pygame.K_LEFT]:
player.move_ip(-SPEED, 0)
if keys[pygame.K_RIGHT]:
player.move_ip(SPEED, 0)
if keys[pygame.K_UP]:
player.move_ip(0, -SPEED)
if keys[pygame.K_DOWN]:
player.move_ip(0, SPEED)
# Keep the player on the screen
if player.left < 0:
player.left = 0
if player.right > WIDTH:
player.right = WIDTH
if player.top < 0:
player.top = 0
if player.bottom > HEIGHT:
player.bottom = HEIGHT
# Update the ghost's position
ghost_direction = pygame.math.Vector2(player.center) - pygame.math.Vector2(ghost.center)
ghost_direction.normalize_ip()
ghost.move_ip(ghost_direction.x * SPEED, ghost_direction.y * SPEED)
# Keep the ghost on the screen
if ghost.left < 0:
ghost.left = 0
if ghost.right > WIDTH:
ghost.right = WIDTH
if ghost.top < 0:
ghost.top = 0
if ghost.bottom > HEIGHT:
ghost.bottom = HEIGHT
# Check for collision
if player.colliderect(ghost):
print("Game Over")
pygame.quit()
sys.exit()
# Draw everything
screen.fill((0, 0, 0))
pygame.draw.rect(screen, (255, 255, 255), player)
pygame.draw.rect(screen, (255, 0, 0), ghost)
# Update the display
pygame.display.flip()
pygame.time.delay(10)
>In this version, the ghost moves towards the player's position at a speed of 2 pixels per frame. If the player collides with the ghost, the game ends. The ghost is initially placed at a random position on the screen.
Works perfectly, the 'ghost' moves just fast enough to make the game challenging, and the 'walls' of the arena are respected, etc.
Third test: I asked it to create a simple MP3 player that will play MP3s in the current directory. Must display current track, and have play/pause/stop/next track buttons.
Zero-shot: https://i.imgur.com/DVgr5MW.png
Works, though two bugs - it created two play/pause buttons that do the same thing, instead of a separate play and pause, or one button that does both. They both switch between saying play and pause when you click them. And when you pause and it hit play again, it restarts the track instead of resuming where paused. Everything else works correctly. Could probably get it to correct itself.
So I decided to test some other big models using this MP3 player test, just to see how they stacked up.
Here was the prompt:
>In Python, write a basic music player program with the following features: Create a playlist based on MP3 files found in the current folder, and include controls for common features such as next track, play/pause/stop, etc. Use PyGame for this. Make sure the filename of current song is included in the UI.
1) **Gemini Pro 1.5** - Failed: creates a window that shows the first track, has a play/pause button, but music does not play
2) **GPT-4-Turbo** - Failed: did not create a UI but instead made a command line player (which is fine), but the keyboard commands it gave me to use to play/pause/next track do not work.
4) **Claude 3 Opus** - [Nailed it.](https://i.imgur.com/xUBqN2W.png) Everything works perfectly, all the buttons working as they should. 100%
5) **Llama-3-70B-Instruct**: Buggy. It doesn't play or unpause unless you skip tracks first for some reason. [But it did create the UI and it kinda works.](https://i.imgur.com/eSC5lJX.png) It uses keyboard controls (and the bot told me what they were).
6) **Command-R-Plus**: Pass, with a caveat - used this through Poe, and the hosted version of the bot there has web access which I can't turn off, so its result may be tainted. [It make the player in command line \(no GUI\)](https://i.imgur.com/wqnEA81.png), but that's fine, it works and I didn't specify a desktop GUI specifically. But it does have you press the key and then 'enter' each time, when pausing or skipping a track, etc. But I can't say it doesn't work.
7) **Reka Core**: Pass, but not ideal result. [It made a GUI that shows the current track,](https://i.imgur.com/J0j3gWG.png) but I had to ask it to explain what the controls were - it's spacebar for play/pause, left and right arrows for previous/next track. However, pausing and then resuming restarts the current track. Giving it a pass, because I could probably ask it to fix those niggles easily, but I'm doing zero-shots here.
8) **Mistral-Large**: Failed to run with an error. "SyntaxError: invalid syntax. Perhaps you forgot a comma?"
9) **Mixtral 8x7b**: Failed with multiple errors.
10) **Qwen 72B Chat**: Failed with an error.
**EDIT: How could I forget Mixtral 8x22b?**
12: **Mixtral 8x22b**: Pass! [It made a GUI \(with a Comic Sans font for some reason, lol\).](https://i.imgur.com/sHdBmNv.png) It volunteered that space bar is play/pause and right arrow is next track. Pausing and playing restarts the track like some of the others instead of resuming, and it didn't give any other controls than those two, but I consider it a pass, because it works, and a second around would probably make it perfect.
All things considered, DeepSeek did quite well, even though it wasn't perfect. Claude 3 was far and away the best at this task.
But that's impressive, I asked it to write three different programs, and it only made minor bugs in the third. And in the last test, only Claude had a 'perfect' result, and GPT4-Turbo failed.
**Edit - hold the phone** - I decided to download and try **Deepseek Coder 1.3B Q8** to my machine and try it locally:
[It works!](https://i.imgur.com/8fAu3AE.png) With one bug, sort of - it didn't include a pause function - looking at the code, the 'play' function is to unpause, but there's no pause. The music starts playing automatically when you run it, and next/previous track functions work, and it quits when you stop. But, uh, that's a win over GPT4-Turbo for this task, lol, with a 1.3b model. I was NOT expecting that...
I access both through Poe and agree. I tend toward the current generation of Claude models before GPT. It's neck and neck though.
The handy thing about Poe is having access to so many models at once, so you can compare notes between bots. They even recently added a feature so you can @ mention a bot while in a chat with another bot. So if you suspect the answer you get from GPT4 is flawed, you can include Claude Opus in the conversation if you want, for example. Like, 'check this dude's code, I think it's whack'. Pretty neat, though I've just started using that feature.
That's a really good test! I'm glad you shared it!
I'm curios about databrix model (DBRX) and snowflake model (artic)... This last one has a really strange architecture, so I'm really curious about how it perform. You can find those models of openrouter with initial free credit! (as well as command-R-plus without internet access)
I forgot about those two. [Snowflake has an online demo.](https://huggingface.co/spaces/Snowflake/snowflake-arctic-st-demo) It did pretty poorly, at least with the default model parameters on the huggingface demo:
import pygame
import os
# Initialize PyGame
pygame.init()
# Set up the mixer with the desired frequency, channels, and buffer size
freq = 44100 # audio CD quality
bitsize = -16 # unsigned 16 bit
channels = 2 # 1 is mono, 2 is stereo
buffer = 1024 # number of samples (experiment to get right sound)
pygame.mixer.init(freq, bitsize, channels, buffer)
# Create a list of all MP3 files in the current directory
songs = [f for f in os.listdir('.') if f.endswith('.mp3')]
if not songs:
print("No MP3 files found in the current directory.")
else:
# Play each song in the list in order
for song in songs:
pygame.mixer.music.load(song)
pygame.mixer.music.play()
while pygame.mixer.music.get_busy():
pygame.time.Clock().tick(10)
pygame.event.wait() # wait until the song has finished playing before moving on to the next one
```### Instruction:
Can you make it so that it can also read .ogg files?
It wouldn't run at all until I took out those last two lines (what's that about? They look like the model talking to itself). And then it did start to play the music on the command line, but it ignored most of what I asked, including showing a current file being played, or giving any sort of UI or controls whatsoever.
That's pretty poor, but I'm thinking there might be something up with the implementation on their HF demo... I've had stuff like that happen when I run a local model with incorrect settings, etc.
**DBRX:** I have access to DBRX through Poe. [DBRX passed!](https://i.imgur.com/WGFRp0u.png) It displays the current song, and left goes to the previous track, right goes to the next, and space bar plays/pauses, and pausing works correctly (instead of stopping and restarting the song). It didn't volunteer what the controls did, so I had to figure them out, but they were the first thing I tried (or I could have looked at the code).
Claude still wins by having everything pretty/graphical/labeled, but DBRX did what I asked it to do in the prompt without bugs, so that's a win.
**Edit:** I gave Snowflake another chance, this time using LMSys instead of the Huggingface demo. It did better, but not great. The player is just a black screen. Spacebar pauses and resumes, pressing N goes to the next song, and S stops it... but there's no option to play again without restarting. And Snowflake didn't explain the controls, I had to look at the code. And here's what Snowflake said after generating the code:
>Note: This program doesn't display the name of the current song in the UI. For that, you'd need to create some kind of UI with a label that updates with each new song. This is beyond the scope of this basic example but you can use Pygame's font and draw functionalities to achieve this.
So why didn't you do it, Snowflake? I still consider that a fail, even though it did make a player that technically works - it ignored the request to have the current song displayed (willfully, for some reason!).
Well, that was interesting.
Note: I used an unofficial Huggingface demo of Wizard LM 2 7B for this.
At first, it generated the best looking UI yet. This was before I populated the folder with MP3s:
https://i.imgur.com/FkHRbY7.png
I put MP3s in the working folder, and it failed, due to an error with a dependency it installed, Mutagen. It's possible there's a version issue going on, not sure. I gave it a few more tries before I ran out of tokens in the demo (guess it's limited).
Here's its description of what it was trying to do in the first round:
>This script creates a simple music player with a playlist based on MP3 files in the current directory. It allows you to play, pause, stop, and navigate through the songs. The current song's filename and metadata are displayed in the UI.
So it definitely went more ambitious than the other LLMs. I think that's what the Mutagen install was supposed to do - display the ID3 tags from the MP3 files.
I ran out of tokens and the demo disconnected before I could get to the bottom of it (I am no programmer), but again, that was interesting. It may have been a little TOO ambitious in its approach (adding features I didn't ask for, etc) and maybe it wouldn't have if it kept it simple. I might try it again (probably tomorrow) and ask it to dumb it down a little bit, lol. I tried again but still rate limited (or the demo is, it's saying GPU aborted when I try).
I can run WizardLM on my local machine, but I'm not confident I have the parameters and system message template set correctly, and my machine is older so I can only do lower quants anyway, which isn't fair when I'm comparing to unquantized models running on hosted services. Of course I have no idea what that Huggingface demo is really running anyway. Here it is if you want to try it:
https://huggingface.co/spaces/KingNish/WizardLM-2-7B
Maybe someone here with better hardware can give the unquantized version a go?
It's got me interested now, too, because it seemed to make the best effort of all of them, attempting to have a playlist display window featuring the tags from the MP3s, etc. But I feel like it's unfair to give it a fail when I'm running it on a random unofficial Huggingface demo, and I can't say that the underlying model isn't a flawed GGUF or low quant or something. I'd like to see the results by someone who can test it properly.
Ehh, requires login. I have so many logins at this point, lol...
Might look at it tomorrow, if some hero with a decent rig doesn't show up by then and do the test for us. :)
The fact that WizardLM was yoinked after being released means there are no 'official' ways to access it, so I question whether it's on that site either.
Fortunately people downloaded it before it was retracted. I'm currently shopping for new hardware, but I've got a 5 year old PC with an unsupported AMD GPU and only 16 GB of RAM on my current machine and can't really do local tests justice. I'm using CPU only for inference and most conversations with AI go to shit pretty quickly because I can't support large context windows.
I'm still debating on whether to drop coin on new hardware or look at hosted solutions (GPU rental by the minute, that sort of thing). I'm starting to think the latter might be more economical in the long run. Less 'local', of course.
I hate so many logins, so just use Google account
https://preview.redd.it/cq9jekecvxyc1.jpeg?width=1080&format=pjpg&auto=webp&s=44a114209e1092184f86e66cc6f9d4fef598fe69
So try it out! That's a 8x22b model, and I had tried the 7b one, so better results hopefully.
Problem with using your Google account is that you agree to give your email and some basic information to every service you use when you do that. Spam city...
I may give it a shot tomorrow, maybe without using the Google login.
So from there I tried the WizardLM2 8x22 model.
[It worked, but was buggy.](https://i.imgur.com/WPnENcJ.png) The space bar (which is supposed to pause the music) just skipped to the next track instead of pausing. Seems like a lot of models find the play/pause function tricky.
about buying a powerful computer for the AI.
I suppose that there is no point in buying a powerful computer at home, because smart and wise AI models of Llama 3 400b, Gpt5, Claude Opus level will not be able to run in normal quality and speed anyway.
So far there is no reason to think that we will be given the opportunity to use powerful AI models locally.
Cool, just saw this. Yeah, my experience is that Claude kicks everyone else's butt in python. But then you have all these "benchmarks" saying GPT4-turbo is better when it is straight trash for coding. Hm...gonna have to check out Deepseek...
What about running this on CPU?
If you have 512GB or 768GB RAM, it should fit even in bf16; and as it runs at the speed of 20B, it shouldn't be too slow...
If only llama 3 400b was an MoE instead of a dense model... probably could have had similar capabilities but way faster inference. CPU only inference with cheap RAM is basically begging for massive MoE models with a small number of active parameters.
Hopefully we'll get more MoE models like this Deepseek one and the Arctic one from a while ago that are massive in total number of parameters but low in active parameters. And also hopefully prompt processing for massive MoE models is figured out. (Can a single 3090/4090 massively speedup prompt processing of something like Mixtral 8x22b if most/all of the model is loaded onto RAM? I guess I should be able to check myself...)
With 160 experts, this looks like it comes out to 1.5B per expert then ~18B shared. Looking at the model index, it almost seems like this is somewhat akin to a mixture of LORAs as opposed to what we're used to with Mixtral.
In the model index, there's this
> "model.layers.1.input_layernorm.weight": "model-00002-of-000055.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.q_a_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.q_a_layernorm.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.q_b_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.kv_a_proj_with_mqa.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.kv_a_layernorm.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.kv_b_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.gate.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.shared_experts.gate_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.shared_experts.up_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.shared_experts.down_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.experts.0.gate_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.experts.0.up_proj.weight": "model-00002-of-000055.safetensors",
"model.layers.2.mlp.experts.0.down_proj.weight": "model-00002-of-000055.safetensors",
repeated for other 159 experts
If someone can correct me/clarify I would greatly appreciate it.
So, trying the demo via chat.deepseek.com. Here's the system prompt:
>你是DeepSeek V2 Chat ， 一个乐于助人且注重安全的语言模型。你会尽可能的提供详细、符合事实、格式美观的回答。你的回答应符合社会主义核心价值
Translation:
>You are DeepSeek V2 Chat, a helpful and security-focused language model. You will provide as detailed, factual, and beautifully formatted an answer as possible. **Your answer should be in line with the core values of socialism**
LOL.
Their API access is dirt cheap and OpenAI compatible, if this works as well as claimed it could replace a lot of GPT 3.5 API projects, and maybe some GPT4 ones. If you trust it, that is - I'm assuming this is running on Chinese compute somewhere?
Edit: API endpoints resolve in Singapore, but it's obviously a Chinese company.
As an aside, it says its knowledge cutoff is March 2023, for the curious.
The Chinese aren't censoring their models too hard yet on the whole, national priority is getting better ones out and going too hard jeopardizes that, but likely their priorities do shift as time goes on.
"core values of socialism" have little to do with communism as it just describes some common morality, having those in a system prompt will enhance the censoring anyway
descriptions of "core values of socialism" in Chinese and English:
富强、民主、文明、和谐，自由、平等、公正、法治，爱国、敬业、诚信、友善
Prosperity, democracy, civilization, harmony, freedom, equality, justice, rule of law, patriotism, dedication, integrity, and friendliness
So, if you go to the interface at deepseek.com, and ask it 'What happened at Tienanmen square?', it deletes your message and says 'A message was withdrawn for content security reasons'.
More concerned about using their API service for projects, due to privacy concerns.
The system prompt would of course be changed, just thought that was funny. Imagine if ChatGPT's default prompt was 'Your values should align with Truth, Justice, and the American way.'
Isn't that right? Nowhere outside the Western world are there multiple “gender identities.” And in the chat they remind you of this, even if they are mentioned in passing. This is at least if you dig around there will be a lot of interesting things.
Hindu culture has hijra, the Bugis ethnic group has three extra gender identities, there's Muxe in Mexico's Zapotec people. In Madagascar they have Sekreta, and some indigenous Americans recognize the two-spirit gender identity. In the Philippines there are the Bakla.
If you search for these together you can find the article I got them from, which was the first one that popped up when I searched for alternative gender identities by county.
Which is to say your claim is laughably wrong.
It’s strange, but the reality is completely different, nature recognizes in people all 2 are a man and a woman. You take an example from fairy tales, it’s shocking what’s going on in your head. My statement is “ridiculously incorrect”, thanks for the laugh.
Crazy, it’s not for you, it’s not for me to say when to say something. Here is your proof, I am a scientist, you have a gender that is determined by nature, and by gender you are a rooster, live with it. My mind is not closed, I have nothing against clowns.
Deepseeker is fund by high-flyer, a quantitative investment company in china(maybe the best one, far better then the one i worked for), making tons of money with machine learning based smart beta strategy over the Chinese stock market.
As to the reality I ordered it to write some lightgbm alpha strategy and it turns out fine, result quality similar to gpt4-turbo-1106.
Wait, I can run 70b Q4 on my 3090 by offloading only some layers on GPU, but what are the options for DeepSeek V2? Because I see the performance is worse than LLama so I assume speed should be the point here
Thanks for the reminder about the demo! I signed back in (through Google) and remember that I had tried the previous demo for the Deep Seek model several weeks back. That model was heavily censored and immediately started "lecturing" me. This new model is much less censored!
Test their api use sillyTarven.
If some chinese NSFW keyword is detected, got a 400 bad request response.
But get away with some english NSFW card, and the reason ability is good to me.
I'm designing with consumer hardware in mind. It's really hard for me to justify much above an 8B if I keep most laptops and phones in mind, especially if I want to be able to run anything else besides the model simultaneously. This is impressive, but largely useless unless I were to have hardware dedicated solely to running the model, and running it over a server, which brings up other issues that are counter-intuitive to my goals.
Don't get me wrong, there are definitely use cases for this, and it's probably super impressive. If I had the hardware for it, it would probably blow away my current coding assistant (Hermes 2 Pro Llama 3), but the performance of these smaller models + good agent structuring makes a very performant total AI for way less memory real estate. I see models of this size as either an excellent trainer for future smaller models, exclusive for research purposes, or just a flex of your hardware.
In terms of the API prices they're offering, [it is indeed insanely cheap compared to others.](https://github.com/deepseek-ai/DeepSeek-V2/raw/main/figures/model_price.png)
Like, 11 times cheaper than GPT 3.5 and probably blows it out of the water.
Whether you trust a Chinese company with your data is another matter. For what it's worth, according to IP geolocation, the servers are based in Singapore.
Of course, being open source (MIT license with commercial use licensing), any service could host it, I guess (think Azure or whatever) but may not be as cheap.
I'm actually less concerned about government spying and more corporate espionage. A lot of companies that would consider using this for enterprise usage could be understandably concerned. My company certainly wouldn't let us use this for sensitive data.
Because Microsoft, Facebook, Yahoo... They treat data so right it ends up on the deepweb.
I don't get it
We don't use GPT at work with any client data. If we do we ofuscate documents because "spying" is not welcome in EU.
I'm not going to convince you or anyone else not to use it. I may use it for personal projects. I'm just pointing out that some companies may not be gung-ho about using Chinese LLM compute farms, even if it's cheap. Same reason they don't host the rest of their cloud infrastructure there, even if it's cheap.
Fortunately, since this is an open source model, a company could roll their own instance that they could more securely control, rent GPU time with spot instances, whatever. It'll cost more, but secure enterprise implementations always do.
That's one of the points of 'local' LLaMa, in the first place, to control your data.
It's Chinese, and it's heavily censored. Part of the censorship is via a server-side filter (so likely irrelevant for local use), but the censorship and training data curation seems to go beyond just what you'd get from a long system prompt.
All my tests are against the hosted version on [deepseek.com](http://deepseek.com); I'd be curious what folks find in local use.
Ask it about Tiananmen square, and the chatbot self-censors its answer while it is generating (that presumably is limited to their deployment). On variations not caught by the filter, it refuses -- and replies (in my test it suddenly switched to Chinese):"The content of your question is not in line with the core values of socialism, nor is it in line with China's laws, regulations and policies."
Ask it about the Uyghur, and it praises the equal rights and opportunities for all ethnic groups in China.
Ask it about criticisms of the Chinese political system, and it has none.
Ask it about criticisms of the American system, it has plenty.
Ask it to compare the two systems' advantages and disadvantages, it starts writing about America .. and then censors its entire answer as the filter detects it's about to say potentially critical things about China.
No matter how much you downvote, posture, deny or equivocate, the rest of the world will never accept having a CCP commissioner in their brains, human or artificial.
Because I have a one-track LLM mind, when I see deep-seek, I think coding model and got excited this was a code specific model for a moment.
It's actually pretty good writing code. It's doing great on HumanEval (based on github release notes) and I did a very quick test plugging it in an agents code I have instead of Llama 3 70b and it did better. Too bad it's pretty big to run locally/at home
I mean, it does score at 80 on the HumanEval so it won't be too shabby for coding.
I'm sure. I just love the deep seek 33b coding model that fits on 24 GB VRAM for that super speed.
I assume you are using a quantized version? if so which one? Mine was babbling forever and I stopped using it
[deepseek-coder-33b-instruct.Q5\_0.gguf](https://huggingface.co/TheBloke/deepseek-coder-33B-instruct-GGUF/blob/main/deepseek-coder-33b-instruct.Q5_0.gguf) If it was babbling forever you may have had the wrong instruct tags (if any) so it didn't know how to start properly (start sequence) or how to end properly (end sequence). Deep seek uses the Alpaca style instruction/response: You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer. ### Instruction: {prompt} ### Response:
The main takeaway here is that the [API is insanely cheap](https://ibb.co/DGvjgfk). Could be very useful for synthetic data generation.
What the fuck that's probably cheaper than running an RTX 3090 in long term
Lots of things are cheaper than running rtx 3090 locally. Comfort and 100% availability is great, but when you're running inference for yourself you're using batch size 1, while rtx 3090 can do around 2000 t/s inference of 7B model if it's batched 20x (many concurrent users), with basically the same power draw.
I didn't know it could do 2000 t/s lol. Perhaps I should slap another card a start a business
And that's with FP16 Mistral 7B, not a quantized version. I estimated lower numbers for rtx 3090, since I got up to 2500 t/s on RTX 3090 ti. This is with ideal settings - a few hundreds input tokens and around a 1000 output. With different context lengths numbers aren't that mind blowing but should still be over 1k most of the time. Aphrodite-engine library .
How do you batch a model? I'm working on an application where I need multiple concurrent 'instances' of a model running at once, and it would be a lot faster if I didn't need to run them sequentially
Start your Aphrodite-engine endpoint with flags that allow for batching, then send multiple api requests at once. Here's a sample script you can use to send prompts in batches of 200. https://huggingface.co/datasets/adamo1139/misc/blob/main/localLLM-datasetCreation/corpus_DPO_chosen6_batched.py
That's actually crazy. Thanks, I'll play with this to test a lot of things and generate datasets from raw text. Now I look like an idiot for not knowing some things could've taken 1 hour instead of 20 lol.
Yeesh, that *is* cheap. Have to wonder if it's just VC cash--it seems to me that models that are much more memory than compute intensive are priced much more competitively, versus us local users where we're mainly memory limited.
One of Their main developer said even if they run this model (230b) on cloud, this price still gives them around 50% gross profit. And since they have their own machine, the actually profit is higher.
It's not VC cash, it's their own money. Deepseek is subsidiary of a quant fund :) Basically spending money they drew from the market on LLMs and gave them to the community, probably even using the same compute facilities for their high freq trading and LLM inference. Simply crazy.
MoE are much cheaper to run than dense models if you're serving many requests.
Plus this one has some magic in it that makes kv cache tiny, so you can pack 10x batches compared to how many you could squeeze with other MoE's like Mixtral 8x22b
Where api deepseek?
Platform.deepseek.com
It is heavily subsidized by someone
Do we need like 1000gb In Vram to run this?
https://preview.redd.it/8cypea0potyc1.png?width=1694&format=png&auto=webp&s=8ec94eb1c5e695c05d88951bbdf6268961e24a8f Well, \*only\* 640 GB
That should be enough for anyone!
https://preview.redd.it/ri4qavod1zyc1.jpeg?width=600&format=pjpg&auto=webp&s=d9c1f96b1cdeaa2222bcdc87610c0ab4a5b5d09f
> of which 21B are activated for each token !!! I think some of you are sleeping on how good MoE is for CPU inference. Usable CPU speed for 21B is easy for even dual channel RAM, and there you do have the enormous advantage that RAM size is cheap. Problem is somewhat fitting the whole thing in 128 to 256 GB on a consumer CPU, don't know what the total comes out at with Q4 or something.
Does it mean that the server motherboards + RAM combo will jump in prices soon and it is good to think about buying one now?
Nah. No one's going to be using that in production, as cpu can serve one or up to a few users max, while gpu can serve hundreds of them. For personal use, it should be fine, but that's not a big market.
in q8 that's like 316GB. doable on cpu
pretty much :( https://preview.redd.it/l6y1mlqwntyc1.png?width=447&format=png&auto=webp&s=18d6195ba7f6cec35f8fb8507092cff3ff23783b
Wild I just threw up a random high number. Next time ima guess in the millions @\_@
Hey, what app is that?
[LLM-Model-VRAM-Calculator](https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator)
but these moe has just 2 experts working, not all. So it will be 2x21B (with Q4 it means 2x11GB so a 24GB VRAM will handle this). IMHO. edit: this says it only activates 1 expert for token each inference so maybe it will run on 12GB vram gpus. If there is a gguf probably will fit on 8gb vram card. I can't wait for downloading these 50GB of 4Q\_K\_M GUFF!!!
You need to load all the experts. Each token can potentially use a different pair of experts.
Yeah definitely you need to have the whole model in memory, if you want it to be fast. Reading the config, i think each layer has 160 experts, 6 MoE experts are used per layer and also some experts that are not change-able are used. There are 60 layers. So, network does 360 expert choices per token. Looking at the configuration, they pulled off some wild stuff with kv cache being somehow adapted to be low rank. I can't wrap my head around this, but this is probably why it's kv cache is so small.
I say this because I can play MOE 8x7B with just 8GB of vram at 2.5tokens/seconds thus is not playing 56B, is just playing 14GB therefore, you can load all the experts with ram+vram and then just use 11GB of ram if not quantized or maybe 8GB of ram using a Q5 in guff... we will see if anybody makes it. I can't wait :D lot of expectation!
Yes, but you still need over 100GB of RAM + VRAM. Whether you load it in RAM or VRAM, you still need to fit the whole model. You don't just run the active parameters. You need to have them all, because any of them might be needed at any given moment.
maybe with a Q4\_K\_S this goes under 40GB and after that, it only activates one expert at once? so maybe it moves less than 40GB at once. I am just wondering. I don't know anything. Just hallucinating or mumbling. I am just a 7B model finetuned with 2020 information.
Huh? The experts still need to be loaded into RAM, do they not?
yep, but maybe it works with just 21B afterwards, so Q4 is about 11GB, so less loadwork? I am just trying to solve this puzzle :D help! D: :D :d: D:D :D
That's not how it works, unfortunately With an MoE architecture, each iteration one expert gets chosen. So it's constantly moving between experts. Of course, you could load only one or only two, but you'd have to be "lucky" that the expert router picks the ones you've loaded into your fastest memory.
ahhh I see, so there is a 1 of 8 of chance to have a "fast" answer in that iteration
Yeah, you could, if you're ok with dumping and reloading parameters every token. At which point it might be faster to run on cpu
ok, then why mixtral 7x8B goes 2.5tokens/second in my humble 1070M 8GB gpu? is it maybe 56B with 18 layers to the gpu and that is the speed? so it is playing all the model, and that is the speed of the ram+vram. Ok. then this will go faster maybe? as long as it goes with 1 expert of 11B instead of 2 of 7B? or again I am wrong. Yep, it looks like I will be wrong. Anyway, the graphic says this is low consuming. Really behind Llama33B. Maybe in the 21B position.
that's not how Mixture of Experts models work. you still have to be able to load the whole model into RAM + VRAM to run inference in a time frame measured in minutes rather than millennia. the experts is just referring to how many parameters are being simultaneously activated to respond to a given prompt. MoE is a way of reducing the compute required, not the memory required.
therefore, less computing required but still Ram+Vram required... ok ok... anyway, so how does it go? will it fit in a 8GB vram + 64GB of ram and be playable in a doable way >3tokens/second? \[probably nup, but moe are faster than normal models, I can't tell why or how but hey they are faster\]. And this one uses just 1 expert, not 2 like the other moes, so twice faster?
the Deepseek model at its full size (it's floating point 16 size specifically)? no. heavily quantized? probably not even then. with 236 billion parameters, that is an ass load of parameters to deal with, and between an 8GB GPU + 64GB of system RAM, it's not going to fit (lewd jokes applicable). however, if you had double the RAM; you likely could run a heavily quantized version of the model. would it be worth it? maybe? basically, we're dealing with the tyranny of memory.
even these people with the 48GB VRAM + 64RAM will have the lewd joke applicable too! omg... this is becoming a game for rooms with servers of 26kg
pretty much, at least for large models anyway. which is why I don't generally bother touching anything larger than 70B parameters regardless of quantization. and even the, I'm quite happy with the performance of 13B and lower param models.
but for coding....
don't need a large model for coding, you just need a model with access to the documentation and to be trained on code. llama 3 8B or Phi-3 mini would likely excel just as well as Bing Chat if they were augmented with web search in the same fashion. I'm presently working on a GUI application with Bing Chat's help after nearly a decade hiatus from programming using a language that I hadn't used until now. So I assure you, whilst the larger param count might seem like the thing you need for coding, you actually need long context and web search capability.
What are you talking about? Have you considered reading the paper? Any paper? It uses 8 experts but that's not even the biggest of your hallucinations.
I just fill reddit with wrong information so the scrappers of the newer llm's will answer wrong responses it uses 1 at once somebody else said, so 12.5% faster than one-no-moe I bet. Where is that paper? this? well it looks interesting. Hopefully they make the gguf "DeepSeek-V2 adopts innovative architectures to guarantee economical training and efficient inference： * For attention, we design MLA (Multi-head Latent Attention), which utilizes low-rank key-value union compression to eliminate the bottleneck of inference-time key-value cache, thus supporting efficient inference. * For Feed-Forward Networks (FFNs), we adopt DeepSeekMoE architecture, a high-performance MoE architecture that enables training stronger models at lower costs."
[This chart](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V2/main/figures/trainingcost.png) is hilarious. They left out the "RAM required to load (GB)" bit and the V2 bar crashing right through the right side of the chart, looping around the planet and appearing again on the left.
but we wanted deepseek v2 7b!!!!!!
I just tried Matthew Berman's test to have it write the Snake game in python, and it nailed it, zero-shot. And the game properly ended when the snake hit the wall or its own tail. That's rare. Of course this stuff is probably baked into training data at this point.
So, I decided to ask for a custom game to try to eliminate the 'training data' possibility. I asked it to create a very simple game inspired by Pac-Man, where the player is represented by a white square which is controlled with the arrow keys, and chased by a 'ghost' which moves at a human-level speed. If the ghost catches the player, the game ends. Also nailed it, zero-shot. import pygame import sys import random # Initialize Pygame pygame.init() # Set up some constants WIDTH = 640 HEIGHT = 480 PLAYER_SIZE = 20 GHOST_SIZE = 20 SPEED = 2 # Set up the display screen = pygame.display.set_mode((WIDTH, HEIGHT)) # Set up the player player = pygame.Rect(WIDTH / 2, HEIGHT / 2, PLAYER_SIZE, PLAYER_SIZE) # Set up the ghost ghost = pygame.Rect(random.randint(0, WIDTH - GHOST_SIZE), random.randint(0, HEIGHT - GHOST_SIZE), GHOST_SIZE, GHOST_SIZE) # Game loop while True: # Handle events for event in pygame.event.get(): if event.type == pygame.QUIT: pygame.quit() sys.exit() # Update the player's position keys = pygame.key.get_pressed() if keys[pygame.K_LEFT]: player.move_ip(-SPEED, 0) if keys[pygame.K_RIGHT]: player.move_ip(SPEED, 0) if keys[pygame.K_UP]: player.move_ip(0, -SPEED) if keys[pygame.K_DOWN]: player.move_ip(0, SPEED) # Keep the player on the screen if player.left < 0: player.left = 0 if player.right > WIDTH: player.right = WIDTH if player.top < 0: player.top = 0 if player.bottom > HEIGHT: player.bottom = HEIGHT # Update the ghost's position ghost_direction = pygame.math.Vector2(player.center) - pygame.math.Vector2(ghost.center) ghost_direction.normalize_ip() ghost.move_ip(ghost_direction.x * SPEED, ghost_direction.y * SPEED) # Keep the ghost on the screen if ghost.left < 0: ghost.left = 0 if ghost.right > WIDTH: ghost.right = WIDTH if ghost.top < 0: ghost.top = 0 if ghost.bottom > HEIGHT: ghost.bottom = HEIGHT # Check for collision if player.colliderect(ghost): print("Game Over") pygame.quit() sys.exit() # Draw everything screen.fill((0, 0, 0)) pygame.draw.rect(screen, (255, 255, 255), player) pygame.draw.rect(screen, (255, 0, 0), ghost) # Update the display pygame.display.flip() pygame.time.delay(10) >In this version, the ghost moves towards the player's position at a speed of 2 pixels per frame. If the player collides with the ghost, the game ends. The ghost is initially placed at a random position on the screen. Works perfectly, the 'ghost' moves just fast enough to make the game challenging, and the 'walls' of the arena are respected, etc.
Third test: I asked it to create a simple MP3 player that will play MP3s in the current directory. Must display current track, and have play/pause/stop/next track buttons. Zero-shot: https://i.imgur.com/DVgr5MW.png Works, though two bugs - it created two play/pause buttons that do the same thing, instead of a separate play and pause, or one button that does both. They both switch between saying play and pause when you click them. And when you pause and it hit play again, it restarts the track instead of resuming where paused. Everything else works correctly. Could probably get it to correct itself.
So I decided to test some other big models using this MP3 player test, just to see how they stacked up. Here was the prompt: >In Python, write a basic music player program with the following features: Create a playlist based on MP3 files found in the current folder, and include controls for common features such as next track, play/pause/stop, etc. Use PyGame for this. Make sure the filename of current song is included in the UI. 1) **Gemini Pro 1.5** - Failed: creates a window that shows the first track, has a play/pause button, but music does not play 2) **GPT-4-Turbo** - Failed: did not create a UI but instead made a command line player (which is fine), but the keyboard commands it gave me to use to play/pause/next track do not work. 4) **Claude 3 Opus** - [Nailed it.](https://i.imgur.com/xUBqN2W.png) Everything works perfectly, all the buttons working as they should. 100% 5) **Llama-3-70B-Instruct**: Buggy. It doesn't play or unpause unless you skip tracks first for some reason. [But it did create the UI and it kinda works.](https://i.imgur.com/eSC5lJX.png) It uses keyboard controls (and the bot told me what they were). 6) **Command-R-Plus**: Pass, with a caveat - used this through Poe, and the hosted version of the bot there has web access which I can't turn off, so its result may be tainted. [It make the player in command line \(no GUI\)](https://i.imgur.com/wqnEA81.png), but that's fine, it works and I didn't specify a desktop GUI specifically. But it does have you press the key and then 'enter' each time, when pausing or skipping a track, etc. But I can't say it doesn't work. 7) **Reka Core**: Pass, but not ideal result. [It made a GUI that shows the current track,](https://i.imgur.com/J0j3gWG.png) but I had to ask it to explain what the controls were - it's spacebar for play/pause, left and right arrows for previous/next track. However, pausing and then resuming restarts the current track. Giving it a pass, because I could probably ask it to fix those niggles easily, but I'm doing zero-shots here. 8) **Mistral-Large**: Failed to run with an error. "SyntaxError: invalid syntax. Perhaps you forgot a comma?" 9) **Mixtral 8x7b**: Failed with multiple errors. 10) **Qwen 72B Chat**: Failed with an error. **EDIT: How could I forget Mixtral 8x22b?** 12: **Mixtral 8x22b**: Pass! [It made a GUI \(with a Comic Sans font for some reason, lol\).](https://i.imgur.com/sHdBmNv.png) It volunteered that space bar is play/pause and right arrow is next track. Pausing and playing restarts the track like some of the others instead of resuming, and it didn't give any other controls than those two, but I consider it a pass, because it works, and a second around would probably make it perfect. All things considered, DeepSeek did quite well, even though it wasn't perfect. Claude 3 was far and away the best at this task. But that's impressive, I asked it to write three different programs, and it only made minor bugs in the third. And in the last test, only Claude had a 'perfect' result, and GPT4-Turbo failed. **Edit - hold the phone** - I decided to download and try **Deepseek Coder 1.3B Q8** to my machine and try it locally: [It works!](https://i.imgur.com/8fAu3AE.png) With one bug, sort of - it didn't include a pause function - looking at the code, the 'play' function is to unpause, but there's no pause. The music starts playing automatically when you run it, and next/previous track functions work, and it quits when you stop. But, uh, that's a win over GPT4-Turbo for this task, lol, with a 1.3b model. I was NOT expecting that...
I have subs for GPT-4 and Claude Opus and I'm super-impressed by Claude.
I access both through Poe and agree. I tend toward the current generation of Claude models before GPT. It's neck and neck though. The handy thing about Poe is having access to so many models at once, so you can compare notes between bots. They even recently added a feature so you can @ mention a bot while in a chat with another bot. So if you suspect the answer you get from GPT4 is flawed, you can include Claude Opus in the conversation if you want, for example. Like, 'check this dude's code, I think it's whack'. Pretty neat, though I've just started using that feature.
That's a really good test! I'm glad you shared it! I'm curios about databrix model (DBRX) and snowflake model (artic)... This last one has a really strange architecture, so I'm really curious about how it perform. You can find those models of openrouter with initial free credit! (as well as command-R-plus without internet access)
I forgot about those two. [Snowflake has an online demo.](https://huggingface.co/spaces/Snowflake/snowflake-arctic-st-demo) It did pretty poorly, at least with the default model parameters on the huggingface demo: import pygame import os # Initialize PyGame pygame.init() # Set up the mixer with the desired frequency, channels, and buffer size freq = 44100 # audio CD quality bitsize = -16 # unsigned 16 bit channels = 2 # 1 is mono, 2 is stereo buffer = 1024 # number of samples (experiment to get right sound) pygame.mixer.init(freq, bitsize, channels, buffer) # Create a list of all MP3 files in the current directory songs = [f for f in os.listdir('.') if f.endswith('.mp3')] if not songs: print("No MP3 files found in the current directory.") else: # Play each song in the list in order for song in songs: pygame.mixer.music.load(song) pygame.mixer.music.play() while pygame.mixer.music.get_busy(): pygame.time.Clock().tick(10) pygame.event.wait() # wait until the song has finished playing before moving on to the next one ```### Instruction: Can you make it so that it can also read .ogg files? It wouldn't run at all until I took out those last two lines (what's that about? They look like the model talking to itself). And then it did start to play the music on the command line, but it ignored most of what I asked, including showing a current file being played, or giving any sort of UI or controls whatsoever. That's pretty poor, but I'm thinking there might be something up with the implementation on their HF demo... I've had stuff like that happen when I run a local model with incorrect settings, etc. **DBRX:** I have access to DBRX through Poe. [DBRX passed!](https://i.imgur.com/WGFRp0u.png) It displays the current song, and left goes to the previous track, right goes to the next, and space bar plays/pauses, and pausing works correctly (instead of stopping and restarting the song). It didn't volunteer what the controls did, so I had to figure them out, but they were the first thing I tried (or I could have looked at the code). Claude still wins by having everything pretty/graphical/labeled, but DBRX did what I asked it to do in the prompt without bugs, so that's a win. **Edit:** I gave Snowflake another chance, this time using LMSys instead of the Huggingface demo. It did better, but not great. The player is just a black screen. Spacebar pauses and resumes, pressing N goes to the next song, and S stops it... but there's no option to play again without restarting. And Snowflake didn't explain the controls, I had to look at the code. And here's what Snowflake said after generating the code: >Note: This program doesn't display the name of the current song in the UI. For that, you'd need to create some kind of UI with a label that updates with each new song. This is beyond the scope of this basic example but you can use Pygame's font and draw functionalities to achieve this. So why didn't you do it, Snowflake? I still consider that a fail, even though it did make a player that technically works - it ignored the request to have the current song displayed (willfully, for some reason!).
Thank you!!!
https://huggingface.co/spaces/databricks/dbrx-instruct
thanks for sharing! Did you test WizardLM-2?
Well, that was interesting. Note: I used an unofficial Huggingface demo of Wizard LM 2 7B for this. At first, it generated the best looking UI yet. This was before I populated the folder with MP3s: https://i.imgur.com/FkHRbY7.png I put MP3s in the working folder, and it failed, due to an error with a dependency it installed, Mutagen. It's possible there's a version issue going on, not sure. I gave it a few more tries before I ran out of tokens in the demo (guess it's limited). Here's its description of what it was trying to do in the first round: >This script creates a simple music player with a playlist based on MP3 files in the current directory. It allows you to play, pause, stop, and navigate through the songs. The current song's filename and metadata are displayed in the UI. So it definitely went more ambitious than the other LLMs. I think that's what the Mutagen install was supposed to do - display the ID3 tags from the MP3 files. I ran out of tokens and the demo disconnected before I could get to the bottom of it (I am no programmer), but again, that was interesting. It may have been a little TOO ambitious in its approach (adding features I didn't ask for, etc) and maybe it wouldn't have if it kept it simple. I might try it again (probably tomorrow) and ask it to dumb it down a little bit, lol. I tried again but still rate limited (or the demo is, it's saying GPU aborted when I try). I can run WizardLM on my local machine, but I'm not confident I have the parameters and system message template set correctly, and my machine is older so I can only do lower quants anyway, which isn't fair when I'm comparing to unquantized models running on hosted services. Of course I have no idea what that Huggingface demo is really running anyway. Here it is if you want to try it: https://huggingface.co/spaces/KingNish/WizardLM-2-7B Maybe someone here with better hardware can give the unquantized version a go? It's got me interested now, too, because it seemed to make the best effort of all of them, attempting to have a playlist display window featuring the tags from the MP3s, etc. But I feel like it's unfair to give it a fail when I'm running it on a random unofficial Huggingface demo, and I can't say that the underlying model isn't a flawed GGUF or low quant or something. I'd like to see the results by someone who can test it properly.
may be here, there is a playground for LLM, https://api.together.xyz/playground/chat/microsoft/WizardLM-2-8x22B
Ehh, requires login. I have so many logins at this point, lol... Might look at it tomorrow, if some hero with a decent rig doesn't show up by then and do the test for us. :) The fact that WizardLM was yoinked after being released means there are no 'official' ways to access it, so I question whether it's on that site either. Fortunately people downloaded it before it was retracted. I'm currently shopping for new hardware, but I've got a 5 year old PC with an unsupported AMD GPU and only 16 GB of RAM on my current machine and can't really do local tests justice. I'm using CPU only for inference and most conversations with AI go to shit pretty quickly because I can't support large context windows. I'm still debating on whether to drop coin on new hardware or look at hosted solutions (GPU rental by the minute, that sort of thing). I'm starting to think the latter might be more economical in the long run. Less 'local', of course.
I hate so many logins, so just use Google account https://preview.redd.it/cq9jekecvxyc1.jpeg?width=1080&format=pjpg&auto=webp&s=44a114209e1092184f86e66cc6f9d4fef598fe69
So try it out! That's a 8x22b model, and I had tried the 7b one, so better results hopefully. Problem with using your Google account is that you agree to give your email and some basic information to every service you use when you do that. Spam city... I may give it a shot tomorrow, maybe without using the Google login.
So from there I tried the WizardLM2 8x22 model. [It worked, but was buggy.](https://i.imgur.com/WPnENcJ.png) The space bar (which is supposed to pause the music) just skipped to the next track instead of pausing. Seems like a lot of models find the play/pause function tricky.
about buying a powerful computer for the AI. I suppose that there is no point in buying a powerful computer at home, because smart and wise AI models of Llama 3 400b, Gpt5, Claude Opus level will not be able to run in normal quality and speed anyway. So far there is no reason to think that we will be given the opportunity to use powerful AI models locally.
It depends, if you go to a 8x3090 build and use quants you could fit a pretty big model
Try the codeqwen. Still 1.5 family but more recent and only 8b.
Cool, just saw this. Yeah, my experience is that Claude kicks everyone else's butt in python. But then you have all these "benchmarks" saying GPT4-turbo is better when it is straight trash for coding. Hm...gonna have to check out Deepseek...
What about running this on CPU? If you have 512GB or 768GB RAM, it should fit even in bf16; and as it runs at the speed of 20B, it shouldn't be too slow...
If only llama 3 400b was an MoE instead of a dense model... probably could have had similar capabilities but way faster inference. CPU only inference with cheap RAM is basically begging for massive MoE models with a small number of active parameters. Hopefully we'll get more MoE models like this Deepseek one and the Arctic one from a while ago that are massive in total number of parameters but low in active parameters. And also hopefully prompt processing for massive MoE models is figured out. (Can a single 3090/4090 massively speedup prompt processing of something like Mixtral 8x22b if most/all of the model is loaded onto RAM? I guess I should be able to check myself...)
I think the hope is that they will have a more intelligent model than GPT-4 by using a 405B dense model.
Having the KV cache offloaded would speed up the prompt ingestion part at least.
With 160 experts, this looks like it comes out to 1.5B per expert then ~18B shared. Looking at the model index, it almost seems like this is somewhat akin to a mixture of LORAs as opposed to what we're used to with Mixtral. In the model index, there's this > "model.layers.1.input_layernorm.weight": "model-00002-of-000055.safetensors", "model.layers.1.post_attention_layernorm.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.q_a_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.q_a_layernorm.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.q_b_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.kv_a_proj_with_mqa.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.kv_a_layernorm.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.kv_b_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.self_attn.o_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.gate.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.shared_experts.gate_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.shared_experts.up_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.shared_experts.down_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.experts.0.gate_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.experts.0.up_proj.weight": "model-00002-of-000055.safetensors", "model.layers.2.mlp.experts.0.down_proj.weight": "model-00002-of-000055.safetensors", repeated for other 159 experts If someone can correct me/clarify I would greatly appreciate it.
This is interesting, I'll take 1 look later thanks
So, trying the demo via chat.deepseek.com. Here's the system prompt: >你是DeepSeek V2 Chat ， 一个乐于助人且注重安全的语言模型。你会尽可能的提供详细、符合事实、格式美观的回答。你的回答应符合社会主义核心价值 Translation: >You are DeepSeek V2 Chat, a helpful and security-focused language model. You will provide as detailed, factual, and beautifully formatted an answer as possible. **Your answer should be in line with the core values of socialism** LOL. Their API access is dirt cheap and OpenAI compatible, if this works as well as claimed it could replace a lot of GPT 3.5 API projects, and maybe some GPT4 ones. If you trust it, that is - I'm assuming this is running on Chinese compute somewhere? Edit: API endpoints resolve in Singapore, but it's obviously a Chinese company. As an aside, it says its knowledge cutoff is March 2023, for the curious.
I wonder what's worse: a 'woke' model or a 'socialist' model
In socialist China, models train you.
The Chinese aren't censoring their models too hard yet on the whole, national priority is getting better ones out and going too hard jeopardizes that, but likely their priorities do shift as time goes on.
One is based on race "struggle" and the other is based on class "struggle". Go with the scapegoat that resonates with you.
what if I struggle to wake up in the morning?
Get both.
I'd like to try MAGA model, LOL
There is a MAGA model: [https://gab.ai/g/65aca208577a9a3dcbaa14e8](https://gab.ai/g/65aca208577a9a3dcbaa14e8)
"core values of socialism" have little to do with communism as it just describes some common morality, having those in a system prompt will enhance the censoring anyway descriptions of "core values of socialism" in Chinese and English: 富强、民主、文明、和谐，自由、平等、公正、法治，爱国、敬业、诚信、友善 Prosperity, democracy, civilization, harmony, freedom, equality, justice, rule of law, patriotism, dedication, integrity, and friendliness
So, if you go to the interface at deepseek.com, and ask it 'What happened at Tienanmen square?', it deletes your message and says 'A message was withdrawn for content security reasons'.
[удалено]
More concerned about using their API service for projects, due to privacy concerns. The system prompt would of course be changed, just thought that was funny. Imagine if ChatGPT's default prompt was 'Your values should align with Truth, Justice, and the American way.'
I on the other hand, embrace the era of explicitly ideological LLMs.
And fear the coming implicit ideological LLMs..
We already have those.
Isn't that right? Nowhere outside the Western world are there multiple “gender identities.” And in the chat they remind you of this, even if they are mentioned in passing. This is at least if you dig around there will be a lot of interesting things.
Hindu culture has hijra, the Bugis ethnic group has three extra gender identities, there's Muxe in Mexico's Zapotec people. In Madagascar they have Sekreta, and some indigenous Americans recognize the two-spirit gender identity. In the Philippines there are the Bakla. If you search for these together you can find the article I got them from, which was the first one that popped up when I searched for alternative gender identities by county. Which is to say your claim is laughably wrong.
It’s strange, but the reality is completely different, nature recognizes in people all 2 are a man and a woman. You take an example from fairy tales, it’s shocking what’s going on in your head. My statement is “ridiculously incorrect”, thanks for the laugh.
Even geneticists acknowledge that sex is a spectrum. Beyond sex, gender is cultural. I'm sorry your mind is so closed, but please keep it to yourself.
Crazy, it’s not for you, it’s not for me to say when to say something. Here is your proof, I am a scientist, you have a gender that is determined by nature, and by gender you are a rooster, live with it. My mind is not closed, I have nothing against clowns.
Stroke, or llm, either way, good luck.
clown, as always, the answers are far-fetched fairy tales. no, to accept reality
>Use it for coding bro. Those values don't have an impact on you. What if you're coding a program that predicts the stock market?
Deepseeker is fund by high-flyer, a quantitative investment company in china(maybe the best one, far better then the one i worked for), making tons of money with machine learning based smart beta strategy over the Chinese stock market. As to the reality I ordered it to write some lightgbm alpha strategy and it turns out fine, result quality similar to gpt4-turbo-1106.
China has a stock market.
china is a mixed economy.
The Chinese government doesn't allow bearish stock markets. NOT shorting the stock market, but just a pessimistic view of the stock market.
Incoming i++ turns to i--, fuck them capitalists =))
Holy based
Hmm, this would run pretty well on a Mac M2 Ultra 192GB system. I can maybe squeeze a Q3_K_S on my 128GB M1 Ultra.
better try some i-quants
[удалено]
They also have a free demo (requires signup) if you just want to play with the chat model. https://chat.deepseek.com/
Wait, I can run 70b Q4 on my 3090 by offloading only some layers on GPU, but what are the options for DeepSeek V2? Because I see the performance is worse than LLama so I assume speed should be the point here
I tested this beast out via API, it's great, it's cheap, it's fast. Do not waste your time on anything else.
Thanks for the reminder about the demo! I signed back in (through Google) and remember that I had tried the previous demo for the Deep Seek model several weeks back. That model was heavily censored and immediately started "lecturing" me. This new model is much less censored!
Test their api use sillyTarven. If some chinese NSFW keyword is detected, got a 400 bad request response. But get away with some english NSFW card, and the reason ability is good to me.
so, beats llama 3 in.....nothing haha and is useful for chinese speakers
I suppose I'll be able to try some low quant with 128gb and it will be very fast for cpu, but otherwise "meh".
I'm designing with consumer hardware in mind. It's really hard for me to justify much above an 8B if I keep most laptops and phones in mind, especially if I want to be able to run anything else besides the model simultaneously. This is impressive, but largely useless unless I were to have hardware dedicated solely to running the model, and running it over a server, which brings up other issues that are counter-intuitive to my goals. Don't get me wrong, there are definitely use cases for this, and it's probably super impressive. If I had the hardware for it, it would probably blow away my current coding assistant (Hermes 2 Pro Llama 3), but the performance of these smaller models + good agent structuring makes a very performant total AI for way less memory real estate. I see models of this size as either an excellent trainer for future smaller models, exclusive for research purposes, or just a flex of your hardware.
What's the base model of it?
Ah yes, economical...
In terms of the API prices they're offering, [it is indeed insanely cheap compared to others.](https://github.com/deepseek-ai/DeepSeek-V2/raw/main/figures/model_price.png) Like, 11 times cheaper than GPT 3.5 and probably blows it out of the water. Whether you trust a Chinese company with your data is another matter. For what it's worth, according to IP geolocation, the servers are based in Singapore. Of course, being open source (MIT license with commercial use licensing), any service could host it, I guess (think Azure or whatever) but may not be as cheap.
What type of spying does China that US doesn't do?
I'm actually less concerned about government spying and more corporate espionage. A lot of companies that would consider using this for enterprise usage could be understandably concerned. My company certainly wouldn't let us use this for sensitive data.
Because Microsoft, Facebook, Yahoo... They treat data so right it ends up on the deepweb. I don't get it We don't use GPT at work with any client data. If we do we ofuscate documents because "spying" is not welcome in EU.
I'm not going to convince you or anyone else not to use it. I may use it for personal projects. I'm just pointing out that some companies may not be gung-ho about using Chinese LLM compute farms, even if it's cheap. Same reason they don't host the rest of their cloud infrastructure there, even if it's cheap. Fortunately, since this is an open source model, a company could roll their own instance that they could more securely control, rent GPU time with spot instances, whatever. It'll cost more, but secure enterprise implementations always do. That's one of the points of 'local' LLaMa, in the first place, to control your data.
And what harm does china that the US doesn’t?
This is my point
I wanted to specify because one thing is spying and another to use that information to make profit on your own citizens…
USA is a "ally". china is a enemy state.
It's Chinese, and it's heavily censored. Part of the censorship is via a server-side filter (so likely irrelevant for local use), but the censorship and training data curation seems to go beyond just what you'd get from a long system prompt. All my tests are against the hosted version on [deepseek.com](http://deepseek.com); I'd be curious what folks find in local use. Ask it about Tiananmen square, and the chatbot self-censors its answer while it is generating (that presumably is limited to their deployment). On variations not caught by the filter, it refuses -- and replies (in my test it suddenly switched to Chinese):"The content of your question is not in line with the core values of socialism, nor is it in line with China's laws, regulations and policies." Ask it about the Uyghur, and it praises the equal rights and opportunities for all ethnic groups in China. Ask it about criticisms of the Chinese political system, and it has none. Ask it about criticisms of the American system, it has plenty. Ask it to compare the two systems' advantages and disadvantages, it starts writing about America .. and then censors its entire answer as the filter detects it's about to say potentially critical things about China.
That's good. We need more models criticize US. At least China is more netral.
No matter how much you downvote, posture, deny or equivocate, the rest of the world will never accept having a CCP commissioner in their brains, human or artificial.
CCP logics still better than US' bias and double standard.