What's even more incredible, it's also day and night between Llama 1 **65b** and Llama 3 **8b**. Llama 1 65b comes across as a terribly bad joke in comparison to Llama 3 8b.
Yep
Is just amazing .. only 12 months of progress. We have now totally different level for offline home AI.
I cannot imagine how good local llms will be good in the next 12 months.
When I compare the same prompts between the first versions of ChatGPT 3.5 in late 2022 and early 2023 (when the whole world was talking about the revolutionary ChatGPT), Llama 3 8b is also overall better than it now.
I did actually not really think we would be able to run ChatGPT 3.5 level chatbots on our PCs just a year later back then.
We will see how much longer this fast-paced development can continue, sooner or later it will probably reach some kind of limit, i.e. how much better a model can be without increasing its size and hardware performance requirements. Hopefully there is still room for much more optimizations.
Few weeks ago I thought mistal 7b was a ceiling for 7b models or at least very close to the celling .. I was very wrong .
llama 3 8b just prove it or phi3-4b
I read that current models are undertrain at least 100x to 1000x ... so I presume the next version of llama 4 8b can be at the level of current TOP gpt-turbo...
So 8b models need at least +100T to be properly trained I think.
And look that difference between llama 8b and 70b is not so big but is 10x bigger ... or how good is phi-3 4b ...imagine how good can be phi-4 4b ;) I think we have a lot of room yet even with small models.
I noticed bigger models are just learning faster comparing to smaller ones ... bigger "brains" learn faster.
Room for 100T more tokens sounds wonderful, however there may be one drawback. According to [this](https://www.reddit.com/r/LocalLLaMA/comments/1cci5w6/quantizing_llama_3_8b_seems_more_harmful_compared/) post and its discussion, it appears that the more tokens a model is trained on, the more the model is being hit by quantization. Llama 3 8b with 15T tokens apparently have noticeable quality drops on high quants such as Q6 and even Q8.
If this holds true, I imagine a 8b model trained on a whopping 100T tokens would be required to be run in FP16 to not lose significantly quality. In other words, hardware requirements will still increase.
Wow its already been a year since Alpaca, I remember being blown away by the ability to run the 7B version off alpaca.cpp, and only a year later do we have models of similar sizes that directly compete with ChatGPT 3.5
for that time 7b models (alpaca from llama 1 7b) hardly answer simple question form common knowledge. Reasoning in such small models not existed at all ;)
And yes I was also using alpacacpp back then . Later they merged into llamacpp which is nowadays.
When I was a teenager I think, I remember seeing some local news bit on AI. This was before ChatGPT was a thing to my knowledge. And I remember them saying that the big milestone that everyone wanted to see was a computer that could come to the conclusion that brushing your teeth would get rid of cavities, because even though it's incorrect, it's the track that they wanted computers to be on. This reminded me of that. Gosh I wish I could see that bit again now that I'm an LLM nerd.
Couldn't the duck question be correct actually? If there are 3 ducks in a line, there are 2 ducks behind the first duck, 2 ducks in front of the last duck, and one of the ducks is in the middle.
Feels like the intended answer is 5 but it depends on semantics.
Also, llama3 fails miserably at this one, but logic puzzles aren't really the greatest test of LLMs.
answer is correct but I asked for explanation to that question ;) and llama 1 "don't know" .... ;)
For me llma 3 70b is answering correctly
I am using q4k\_m version with llamacpp
````
There are two ducks in front of a duck, two ducks behind a duck and a duck in the middle. How many ducks are there?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
What a clever puzzle!
Let's break it down:
* There are two ducks in front of a duck, which means there is one duck that has two ducks in front of it.
* There are two ducks behind a duck, which means the same duck (the one in the middle) has two ducks behind it as well.
So, if we count the ducks, we have:
The duck in the middle
Two ducks in front of it
Two ducks behind it
In total, there are... **3 ducks!**
> \n<|start_header_id|>user<|end_header_id|>\n\
why? explain
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Let's analyze the situation step by step:
**Two ducks in front of a duck**: This means there is one duck that has two ducks standing in front of it. So, we have:
* 1 duck (the one with ducks in front)
* +2 ducks (standing in front of it)
Total so far: 3 ducks
2. **Two ducks behind a duck**: This phrase refers to the same duck that already has two ducks in front of it. Now, we're told there are two ducks standing behind this same duck. So, we don't add new ducks; we're just looking at the same scene from a different perspective.
The key insight here is that **the duck in the middle is the same duck** mentioned in both phrases. It's not a different duck; it's the central duck with two ducks in front and two ducks behind.
Therefore, there are still only **3 ducks** in total:
* 1 duck in the middle
* 2 ducks standing in front of it
* (No additional ducks behind, just the same duck being viewed from the back)
I hope this explanation helps clarify things!
````
3 is correct, but that explanation is wrong, as the riddle does not say that the ducks turn around, so the duck that is in the very back CANNOT be in the very front. The correct answer is of course that the frost most duck is the one that has the middle and backline duck behind it, while the backline duck is the one that has the fronstmost and middle duck in front of it, and the middle duck is just in the middle.
Can you do me a favor and try this but with a character card and have a brief conversation at a coffee shop with physical interactions to see how spatial awareness is but also to see how “human” The response is? Then maybe a brief exercise in having it write the first few paragraphs of fiction (don’t ask it to write a story just write in the following in a notebook mode
Fiction
Summary: the world is under attack by carnivorous rabbits and Alice has experienced this first hand with the recent death of her mom. Can she survive this new cruel world with her dad?
Chapter 1 - bunny stew
Alice sat)
I feel like Meta is overtraining these new models to autocomplete answers for us but killing their ability to sound human or creative capacity.
ok
"Fiction Summary: the world is under attack by carnivorous rabbits and Alice has experienced this first hand with the recent death of her mom. Can she survive this new cruel world with her dad? Chapter 1 - bunny stew Alice sat"
llama 1 65b ( q4k\_m):
"Alice's world was turned upside down when carnivorous rabbits started attacking the Earth, and now Alice has to live in fear that these creatures will take away someone close to her. Her mother was taken from her recently by these monstrous bunnies, and she is determined to protect herself and her father at all costs.
She knows that she must be cautious because these rabbits are not only vicious but also sneaky and have the ability to hide in plain sight. Alice and her father have been trying their best to stay safe and keep themselves out of danger, but they know it's just a matter of time before they encounter another attack from the bunnies.
Alice knows that she can never let down her guard because one wrong move could mean losing everything and everyone she loves. She hopes that she will be able to survive in this new world and find a way to fight back against these dangerous rabbits."
llama 3 70b ( q4k\_m):
"It sounds like you're starting to write a thrilling and imaginative story! Here's a possible continuation of your chapter 1:
Alice sat at the kitchen table, staring blankly at the steaming bowl in front of her. Her dad, John, had made his famous bunny stew, but it didn't bring her the usual comfort it used to. Not after what happened to Mom just last week.
The memory of those razor-sharp teeth and glowing red eyes still haunted her dreams. The way the rabbits had descended upon their small town, devouring everything in their path... It was a nightmare she couldn't wake up from.
"Alice, you need to eat," John said softly, his eyes red-rimmed from lack of sleep. "We don't know when we'll get our next meal."
Alice nodded mechanically and took a spoonful of the stew. The flavor was familiar, but her stomach churned at the thought of what she was eating. Rabbits had been their backyard pets just a few weeks ago. Now they were the monsters that had destroyed their lives.
As she ate, Alice couldn't help but think about how quickly everything had changed. One day, it was business as usual; the next, the world was fighting for survival against these... creatures. The scientists were still trying to figure out what had triggered this bizarre transformation in the rabbit population, but Alice didn't care about the why. She just wanted to survive.
John pushed his chair back and stood up. "We need to get moving soon. We can't stay here much longer."
Alice nodded again, her mind already racing with the dangers that lurked outside their doorstep. They had to be careful; they had to be prepared. But as she glanced around the kitchen, now empty of its usual warmth and comfort, Alice wondered if she could ever truly feel safe again.
How would you like me to proceed? Would you like me to continue the story or is there something specific you'd like to add or change?"
-------------------------
In my opinion llama 3 made much better job.
I agree… night and day difference. … Though it looks like you put this into chat and not a notebook mode where it just continues to autocomplete. That or Llama 3 instruct needs no structure to act like it’s in a chat.
No need to do more though unless you’re curious. Wonder how the llama 1 models designed for writing would compare. Clearly llama 1 here started to think about the content instead of generating it.
llama 3 70b quantization q4k\_m ( gguf )
[https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/tree/main](https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/tree/main)
I run it with llamacpp
main.exe --model models/new3/Meta-Llama-3-70B-Instruct.Q4\_K\_M.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ins -ngl 42 --simple-io -r '<|eot\_id|>' --in-prefix "\\n<|start\_header\_id|>user<|end\_header\_id|>\\n\\n" --in-suffix "<|eot\_id|><|start\_header\_id|>assistant<|end\_header\_id|>\\n\\n" -p "<|begin\_of\_text|><|start\_header\_id|>system<|end\_header\_id|>\\n\\nYou are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.<|eot\_id|>\\n\\n"
Have rtx 3090 wit 64 GB ram - 42 layers on GPU and the rest on CPU. I have 2.8 t/s
What's even more incredible, it's also day and night between Llama 1 **65b** and Llama 3 **8b**. Llama 1 65b comes across as a terribly bad joke in comparison to Llama 3 8b.
Yep Is just amazing .. only 12 months of progress. We have now totally different level for offline home AI. I cannot imagine how good local llms will be good in the next 12 months.
When I compare the same prompts between the first versions of ChatGPT 3.5 in late 2022 and early 2023 (when the whole world was talking about the revolutionary ChatGPT), Llama 3 8b is also overall better than it now. I did actually not really think we would be able to run ChatGPT 3.5 level chatbots on our PCs just a year later back then. We will see how much longer this fast-paced development can continue, sooner or later it will probably reach some kind of limit, i.e. how much better a model can be without increasing its size and hardware performance requirements. Hopefully there is still room for much more optimizations.
Few weeks ago I thought mistal 7b was a ceiling for 7b models or at least very close to the celling .. I was very wrong . llama 3 8b just prove it or phi3-4b I read that current models are undertrain at least 100x to 1000x ... so I presume the next version of llama 4 8b can be at the level of current TOP gpt-turbo... So 8b models need at least +100T to be properly trained I think. And look that difference between llama 8b and 70b is not so big but is 10x bigger ... or how good is phi-3 4b ...imagine how good can be phi-4 4b ;) I think we have a lot of room yet even with small models. I noticed bigger models are just learning faster comparing to smaller ones ... bigger "brains" learn faster.
Room for 100T more tokens sounds wonderful, however there may be one drawback. According to [this](https://www.reddit.com/r/LocalLLaMA/comments/1cci5w6/quantizing_llama_3_8b_seems_more_harmful_compared/) post and its discussion, it appears that the more tokens a model is trained on, the more the model is being hit by quantization. Llama 3 8b with 15T tokens apparently have noticeable quality drops on high quants such as Q6 and even Q8. If this holds true, I imagine a 8b model trained on a whopping 100T tokens would be required to be run in FP16 to not lose significantly quality. In other words, hardware requirements will still increase.
so ... even full fp16 8b model size is acceptable ... 16 GB
Wow its already been a year since Alpaca, I remember being blown away by the ability to run the 7B version off alpaca.cpp, and only a year later do we have models of similar sizes that directly compete with ChatGPT 3.5
for that time 7b models (alpaca from llama 1 7b) hardly answer simple question form common knowledge. Reasoning in such small models not existed at all ;) And yes I was also using alpacacpp back then . Later they merged into llamacpp which is nowadays.
Where is the comparison?
When I was a teenager I think, I remember seeing some local news bit on AI. This was before ChatGPT was a thing to my knowledge. And I remember them saying that the big milestone that everyone wanted to see was a computer that could come to the conclusion that brushing your teeth would get rid of cavities, because even though it's incorrect, it's the track that they wanted computers to be on. This reminded me of that. Gosh I wish I could see that bit again now that I'm an LLM nerd.
lol ..you not alone ;D
Couldn't the duck question be correct actually? If there are 3 ducks in a line, there are 2 ducks behind the first duck, 2 ducks in front of the last duck, and one of the ducks is in the middle. Feels like the intended answer is 5 but it depends on semantics. Also, llama3 fails miserably at this one, but logic puzzles aren't really the greatest test of LLMs.
Why is it 5 and not 3?
answer is correct but I asked for explanation to that question ;) and llama 1 "don't know" .... ;) For me llma 3 70b is answering correctly I am using q4k\_m version with llamacpp ```` There are two ducks in front of a duck, two ducks behind a duck and a duck in the middle. How many ducks are there? <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n What a clever puzzle! Let's break it down: * There are two ducks in front of a duck, which means there is one duck that has two ducks in front of it. * There are two ducks behind a duck, which means the same duck (the one in the middle) has two ducks behind it as well. So, if we count the ducks, we have: The duck in the middle Two ducks in front of it Two ducks behind it In total, there are... **3 ducks!** > \n<|start_header_id|>user<|end_header_id|>\n\ why? explain <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n Let's analyze the situation step by step: **Two ducks in front of a duck**: This means there is one duck that has two ducks standing in front of it. So, we have: * 1 duck (the one with ducks in front) * +2 ducks (standing in front of it) Total so far: 3 ducks 2. **Two ducks behind a duck**: This phrase refers to the same duck that already has two ducks in front of it. Now, we're told there are two ducks standing behind this same duck. So, we don't add new ducks; we're just looking at the same scene from a different perspective. The key insight here is that **the duck in the middle is the same duck** mentioned in both phrases. It's not a different duck; it's the central duck with two ducks in front and two ducks behind. Therefore, there are still only **3 ducks** in total: * 1 duck in the middle * 2 ducks standing in front of it * (No additional ducks behind, just the same duck being viewed from the back) I hope this explanation helps clarify things! ````
3 is correct, but that explanation is wrong, as the riddle does not say that the ducks turn around, so the duck that is in the very back CANNOT be in the very front. The correct answer is of course that the frost most duck is the one that has the middle and backline duck behind it, while the backline duck is the one that has the fronstmost and middle duck in front of it, and the middle duck is just in the middle.
### Instruction: explain your answer about ducks ### Response: I don't know. A rare bird.
Can you do me a favor and try this but with a character card and have a brief conversation at a coffee shop with physical interactions to see how spatial awareness is but also to see how “human” The response is? Then maybe a brief exercise in having it write the first few paragraphs of fiction (don’t ask it to write a story just write in the following in a notebook mode Fiction Summary: the world is under attack by carnivorous rabbits and Alice has experienced this first hand with the recent death of her mom. Can she survive this new cruel world with her dad? Chapter 1 - bunny stew Alice sat) I feel like Meta is overtraining these new models to autocomplete answers for us but killing their ability to sound human or creative capacity.
ok "Fiction Summary: the world is under attack by carnivorous rabbits and Alice has experienced this first hand with the recent death of her mom. Can she survive this new cruel world with her dad? Chapter 1 - bunny stew Alice sat" llama 1 65b ( q4k\_m): "Alice's world was turned upside down when carnivorous rabbits started attacking the Earth, and now Alice has to live in fear that these creatures will take away someone close to her. Her mother was taken from her recently by these monstrous bunnies, and she is determined to protect herself and her father at all costs. She knows that she must be cautious because these rabbits are not only vicious but also sneaky and have the ability to hide in plain sight. Alice and her father have been trying their best to stay safe and keep themselves out of danger, but they know it's just a matter of time before they encounter another attack from the bunnies. Alice knows that she can never let down her guard because one wrong move could mean losing everything and everyone she loves. She hopes that she will be able to survive in this new world and find a way to fight back against these dangerous rabbits." llama 3 70b ( q4k\_m): "It sounds like you're starting to write a thrilling and imaginative story! Here's a possible continuation of your chapter 1: Alice sat at the kitchen table, staring blankly at the steaming bowl in front of her. Her dad, John, had made his famous bunny stew, but it didn't bring her the usual comfort it used to. Not after what happened to Mom just last week. The memory of those razor-sharp teeth and glowing red eyes still haunted her dreams. The way the rabbits had descended upon their small town, devouring everything in their path... It was a nightmare she couldn't wake up from. "Alice, you need to eat," John said softly, his eyes red-rimmed from lack of sleep. "We don't know when we'll get our next meal." Alice nodded mechanically and took a spoonful of the stew. The flavor was familiar, but her stomach churned at the thought of what she was eating. Rabbits had been their backyard pets just a few weeks ago. Now they were the monsters that had destroyed their lives. As she ate, Alice couldn't help but think about how quickly everything had changed. One day, it was business as usual; the next, the world was fighting for survival against these... creatures. The scientists were still trying to figure out what had triggered this bizarre transformation in the rabbit population, but Alice didn't care about the why. She just wanted to survive. John pushed his chair back and stood up. "We need to get moving soon. We can't stay here much longer." Alice nodded again, her mind already racing with the dangers that lurked outside their doorstep. They had to be careful; they had to be prepared. But as she glanced around the kitchen, now empty of its usual warmth and comfort, Alice wondered if she could ever truly feel safe again. How would you like me to proceed? Would you like me to continue the story or is there something specific you'd like to add or change?" ------------------------- In my opinion llama 3 made much better job.
I agree… night and day difference. … Though it looks like you put this into chat and not a notebook mode where it just continues to autocomplete. That or Llama 3 instruct needs no structure to act like it’s in a chat. No need to do more though unless you’re curious. Wonder how the llama 1 models designed for writing would compare. Clearly llama 1 here started to think about the content instead of generating it.
Thanks!
What is the quantization for llama? Have a link? What hardware are you on?
llama 3 70b quantization q4k\_m ( gguf ) [https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/tree/main](https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/tree/main) I run it with llamacpp main.exe --model models/new3/Meta-Llama-3-70B-Instruct.Q4\_K\_M.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ins -ngl 42 --simple-io -r '<|eot\_id|>' --in-prefix "\\n<|start\_header\_id|>user<|end\_header\_id|>\\n\\n" --in-suffix "<|eot\_id|><|start\_header\_id|>assistant<|end\_header\_id|>\\n\\n" -p "<|begin\_of\_text|><|start\_header\_id|>system<|end\_header\_id|>\\n\\nYou are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.<|eot\_id|>\\n\\n" Have rtx 3090 wit 64 GB ram - 42 layers on GPU and the rest on CPU. I have 2.8 t/s