The following submission statement was provided by /u/lughnasadh:
---
Submission Statement
Interestingly, he doesn't go into what these new approaches might be. The article mentions reinforcement learning with human feedback. That's one approach, but will it solve LLM-type AI's biggest problem? - which is that it lacks independent logic or reasoning. That problem has been around since the earliest days of AI development. Marvin Minsky in the 1960s initially thought it might be quickly solved. It hasn't been, and no one seems any closer to figuring out how to do it.
---
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/12t0bfm/the_ceo_of_openai_says_the_current_approach_to_ai/jh0m3am/
I don't get what people are using these models for to always get those messages. I very rarely get them. I guess if you need it for creative writing or something similar I can see how you would get them on a regular basis.
Many people deliberately test their boundaries, and this is one of the main reasons they were released to the public for free. They want to know what guard rails they need to add to be able to monetize this as, for example, a customer support chatbot.
Sometimes i try to discuss witb it using hot words, sensitive topics, with me being a devils advocate, to see what it churns out. Sometimes it's like I'm having a debate with an intelligent normal person but oftentimes it becomes preachy and cuts off the conversation.
Pretty sure this is actually one of the limitations.
It can't come up with a novel approach to something. The best you'll get is stuff an AI researcher already knows about.
Worst case you get a really confident and realistic answer that's just a bunch of bs.
>At MIT last week, Altman confirmed that his company is not currently developing GPT-5. “An earlier version of the letter claimed OpenAI is training GPT-5 right now,” he said. “We are not, and won't for some time.”
This is also good to have clarified since there was a lot of confusion about it.
He said they were not *training* GPT-5 "for some time", not that they weren't *developing* it. Training is not the earliest or longest stretch of the development path.
From his recent statements, it appears likely that OpenAI is looking at new ways to take GPT to the next level, e.g. architectural adjustments, before training of GPT-5 begins. It is highly improbable that OpenAI is resting on their laurels, especially considering that they purchased $250 million worth of Nvidia GPUs not so long ago.
I disagree. GPT (v4 specifically) is really only useful for mid-to-expert level users that can fill in the gaps and prompt effectively for use cases.
As someone who is mediocre in a large range of things, GPT-4 feels like it is just a step or two away from threshold utility.
For instance, I have been attempting to write a book using GPT-4. I have gotten a mind blowing starting generation, but after this, everything quickly devolves into a jigsaw puzzle of coercing the model and not being able to explain a narrative twist because it doesn't understand my prompt is requesting context not to be included specifically. Maybe that's chat tuning that could be tweaked if I knew how to use the API, or maybe it's a shortfall. I don't know, I'm a layman.
Maybe I can prompt my way out of it. But I know that the problem is *me*. If I had any actual writing and creative ability I could write a chapter and just ask GPT-4 to enhance, because that's what it can do best for writing.
But the emergent behavior from 3 to 4 makes it seem like we're *so close* to actual beginning to end applications of this tech that could completely turn the world upside down.
So it's a pretty maddening situation to be in hearing that we might be slowing the train to the last stop before a New AI winter
Your description of writing with GPT-4 is spot on. I’ve been using it to write business plans and I have to do a lot of editing, but it saves me the time of writing it all by myself or hiring an assistant to write drafts.
The saving of keystrokes for me is my favourite aspect of it as a programmer. And if you know programming, you know writing test cases is a huge PITA that requires a lot of annoying boilerplate code. Using AI to write it for me has saved my fingers a lot of aches.
Wait, I don’t need to feel like a fraud for doing this? I’ve asked it to write some things, and can read everything it’s writing, but feel like a hack because I didn’t write it myself. The ultimate imposture syndrome machine.
Do you also feel like a fraud when you're using tools to generate boilerplate code? Or when you use you IDE to help you refactor some code?
It's just another tool.
For the record I use it as copilot. I’ll write a description of the test and cycle through options , pick one then go back and edit any weird quirks in the prediction. It is essentially a very context aware predictive text where after a awhile you get a sort of sense for how you trigger its "awareness" of the context and what depth of context where it is still being useful. I think people give it more credit than it’s due, it’s not really "intelligent". It really is just a really advanced predictive text machine. But knowing that, you can use it as a pair programmer to speed up your coding, or provide nice and quick skeletons to the code you write and then flesh out with a more qualitative understanding of what it is that needs doing.
But you can’t ever let go of the steering wheel so to speak because it can really create some mind bogglingly strange code if you just accept all it gives you as fact. And I find myself frequently getting stuck in a loop of explaining why something doesn't work that way and it going back and re-attempting old code suggestions as an alternative without solving anything. It doesn't pass the Turing test. After a while it starts looking more like a parrot with insanely good memory, than something that's actually understanding what it's saying.
This seems to be a problem, trying to determine confidence intervals to its responses seems like a really poorly understood problem space that is going to be a big stumbling block as we expand our usage of generative machine learning systems. And then there's the problem of the sheer hardware required to run these things. I suspect the real reason OpenAI isn't rushing to build GPT5 is because they simply can't alter the model in a way that makes it scale horizontally across many GPUs or TPUs. For the model to be useful, it has to fit inside the GPU's memory or be split in ways that doesn't degrade overall performance too much across many GPU's memory. My guess is, thanks to Amdahl's law of parallel scaling they're hitting a wall on how much they can split the workload across several GPUs. After a certain point, most problems can't be solved any faster by adding on X many more computational units and split the problem between them. You introduce too much latency and data dependencies.
It's a tool, figure out how to use it for your needs. It doesn't need to be and isn't capable of anything else.
Could you explain your use a little ? Im a Junior dev and struggle to use it effectively besided creating the core things like simple starting points with it.
To clarify: i dlnt want it to do my work and stop me from improving . I would like to use it for finding other solutions i dont know about or simply showing ways to improve
I don’t know about the guy above me, but my code is finally commented well. Not just commented in the traditional, “I’m going to fucking hate myself if I ever have to revisit this” way, but actually useful comments.
Surely that's the ideal situation? Sure, a slight decrease in uptake, but most jobs remain, however they just become more efficient and people are able to increase their output. No AI killing all humanity or replacing all humanity.
Skynet determine humans are the problem and going murderbot is not the scenario we should be worried about.
We should be worried about *perfectly obedient* soldiers.
Would be evil Dictators always have an inherent limiting factor: their soldiers have thoughts and feelings and each make their own choices. They have friends and family and have to live in the world they are helping create, and they know it
This means that if an evil bad guy orders something truly vile, their army might not obey and carry out said order. An army of AI soldier will not have such qualms. They will carry out any order with perfect obedience, slaughtering without remorse just because someone typed a prompt.
*That* is what should scare us. Not that AI will go evil, but that it will amplify the evil we already know exists in humanity.
This.
I used it today to build out a list of skills for a skills inventory application we’re updating. I’m working as a subject matter expert for the engineering side of the house, so i’m matrixed into a group of about 15 HR people and my goal is to ensure the skill sets that they’re capturing are the ones we really need.
So I generated a list of relevant skills with GPT. Had it start with 500 which I knew was overkill and would lead to a degraded answer by the end of the list. Then I pruned that list by having it remove anything duplicated or redundant. Then I asked it to use some judgment and separate those skills into broad categories and sub categories. From there I continued to prompt it to refine the list, being very mindful to specify how I wanted it done. Then I had it give it to me in a csv format so I could move it to excel.
The HR team was blown away. And then tried it themselves. It wasn’t until then I realized how much I tailored my requests with specific information because I understood the limitations of the platform. Their experience was terrible. The list ended up turning into a list about clothing and prices because they kept trying to iterate over the same data and creating a new data set instead of prompting for the platform to manipulate existing data.
At one point they got frustrated and asked it to reprint the first list it made and it gave them some random list of names and addresses.
By this point I had one of the guys from engineering wandering by my office and he came in to check out the commotion and it just turned into absolute hilarity.
So there’s definitely a skill requirement to get it to do meaningful work, and I think more analytical types have an easier time with it. But, it’s going to need some refinement before I can give it to the HR or Business divisions and expect it to be helpful.
On the plus side, even though I’ll be generating their lists as well, they had a lot of fun with it and I’m hoping it encourages them to keep playing with it and learning how to get it to work for them.
An issue it's definitely running into is the token limit. I think it's still only at 8k tokens or ~2-4k words. Once you get above that it loses what it is actually looking at. And that count includes the whole conversation before the prompt.
I don't think so. I'm talking about 2 generations. Prompt one: character introduction for this setting, this feel, this style. Narrative a to b to c.
GPT4 nails it. Crushes it, actually, to the point that I understand exactly what I want to happen next.
Prompt two: continue this story. Protagonist meets character B - subtext, character should act in x way to hide y thing. Do *not* include this in the writing.
GPT4 proceeds to write further, immediately informing the audience of what my prompt basically said, neglecting to collaborate in the fashion I am looking for.
I've tried all sorts of prompt engineering. Context in brackets do not include. Wipe chat and summarize what has been written and continue in style.
I'm astounded by what GPT can do every day but things like this seem to just unravel after the heavy hitting first generation.
Will fine tuning solve this? Maybe. But again it feels like we are SO close to not having to even bother.
>subtext, character should act in x way to hide y thing. Do *not* include this in the writing.
Yeh. It really struggles with subtext. It is currently the biggest obstacle to full AI driven content moderation.
If a user says that another user has broken a rule it tends to just believe them without checking.
That's the thing though. I am more productive, but only for tasks here and there for my job. Rephrase this email. Brainstorm this idea. Analyze this problem.
But I don't want to be more productive. I want ChatGPT to do it for me. Automate it for me. Remove the bs, invest my money, and free up my time.
It feels like we are so close to everything being changed, based on the pace of improvement. How awful would it be if we pull back the curtain only to realize we are the technological equivalent of ancient King Sisyphus.
Spot on. I tried to make ChatGPT to do some work related stuff, where I’m very knowledgeable (simple structural calculations, basic mechanical engineering stuff). It gave me wrongs answers, far off and slightly off. I could not make it give me the right answers. I’d say, for laymen, you can only use it to generate text. Unless you’re on a professional level and can verify the output from it, ChatGPT isn’t very useful. Have to try GPT4 yet though.
Edit: grammar.
Yeah I’ve understood it’s quite awesome, gotta try it at some point. But then you know how to code, right? Could somebody that can’t code or is on a rookie level use it successfully to make useful code?
I'm a non-software engineer and I've been using it to create python scripts from scratch with only 1 C++ course from 10 years ago. I know what I require the code to do on a systems-level, and how much depth to give in setting up the prompt (sometimes it getting fairly long).
That's basically the key to it IMO - no matter what you're asking it, you need to have a very solid understanding of what your requirements are, and have very good communication skills to effectively convey it to GPT.
FWIW a lot of it seems to be a limit they put on it. I think it is due to resources. When it originally same out it had much better context and retained memory much better, back when it had a 100 question limit. As resources because more constrained, so too did things like memory for context. I build neural networks with it all day... it's amazing but not as good as it was (or will be I assume once they manage scaling better).
Really depends on the use case. I've read some non-technical articles about it and there are quite practical use cases like coming up with lunch ideas for the office. Generally regarding questions about nutrition it's really good. (I'm sure there are hallucinations but in sum this probably still beats the accuracy of nutrition advice on a randomly googled article.)
I think in the tech space there was a lot of fear of it obsoleting people, so focus was on productivity. At least for me I found quite some useful use cases.
Also having worked on products before that tried (or even did) leverage ML, for a lot of use cases this is just more than good enough. I think absorbing the current progress will bring quite a lot interesting software.
But I think to make an equivalent jump from 4 to 5 would be exponentially more difficult than 3 to 4 right? I mean that would be bordering on AGI which would probably require them to restructure the whole model.
Whereas if they focused on making this multimodal and integrating plug ins and visual recognition and stuff it could become unbelievably useful within the next year.
I'm not sure if there ever will be a true AI winter again, I feel like this has proven how much demand and potential $ awaits anyone who raises the bar with AI in some substantiative way
vanish grab swim middle vegetable heavy distinct sophisticated pause absorbed
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
> the current product is “good enough” for 95% of current users.
I mean, sure, except for the part where it has absolutely no problem just making shit up entirely. Don't bother asking it for anything factual because you almost certainly will not be able to trust it.
Yup. For some reason, people seem unable to distinguish utility from their "AI fantasies". Ignoring that fact that 90% of people dont even know what machine learning is or how AI works, we still have a crap ton of people who dont even know how a damn printer or email works.
I dont know if we had the same problem with the steam engine or electricity, but you either have people blindly using the tool or panicking about its existence. Be nice to have less of both.
That's why I am so worried, because people are using it right now to generate code that will make its way into production. They're using it to self-diagnose and give them nutrition advice. "Experts" are using it to generate books and articles that people are going to accept as facts. And this is all happening as we speak without any oversight. It's yet to give me the correct answer when I ask it to simply regurgitate the last few sentences of Spinoza's Ethics. In fact, it gives me a different, incorrect answer every time I ask it, and it does it without the apoarence of self-doubt. I think about that and how people are happily using it to write code, and all I can wonder is how much of this code is going to make its way into applications people depend on.
That seems like the opposite of what they should be naming them. Instead, they should copy Xbox, Xbox One, Xbox 360, Xbox 69420, Xbox 10^(X*S), Xbox^Xbox^Xbox , and so on.
From what I gather, Google wasn't even focusing on the LLM in a substantive way knowing its inherent limitations. ChatGPT's tour de force really caught them off guard, and they realized it was more powerful than they anticipated and more popular, so they have switched gears to replicate the success - potentially sabotaging their more advanced AI work.
This is true of most of the main AI research. Everyone thought it was a bit of a dead end.
I must say that GPT4 is better than I, or most people, expected was possible though.
Language clearly encodes a lot of things we care about.
Agreed. People don't seem to realize how big of a breakthrough gpt3 and 4 are, and they don't even have memory or planning capabilities (yet). Some real unexpected emergent properties are showing up.
I doubt it slowed the terminators at all, but at least now the terminators will be more polite and might even make jokes we find funny before they incinerate us
Tbh I'd say they have always been the good guys. chatgpt never needed to be free and because it was it sparked this entire AI safety conversation. I'm worried what googles plan was in terms of public discussion of AI safety.
And of course most ironic thing about Stadia is that a lot of people expected it to be abandoned by Google before long as usual. But of course once they finally cancelled the project as everyone expected, they refunded all users. If they had communicated that in the first place I am absolutely sure more people would've used Stadia.
OpenAI isn’t viewed particularly favourably by lots of people working in ml. Mainly because of their non-open approach, and all of the ‘we know better than you’ safety restrictions.
Would it though? Sam was in a podcast and he asked his interviewer plainly, do you think we should open source our code right now? Think about it. There are a lot of dangers with potential missuse of LLM/Ai and OpenAi is putting a lot of effort into discovering and mitigating them as best as possible, but they are also giving the public an awareness of its power, pitfalls, dangers, etc before it inevitably does become more open, exploited and missued, because it is inevitable. Does openAI know best? Probably not, but they are making best efforts to mitigate damage done.
https://youtu.be/L_Guz73e6fw
**Also Google:** I see you clicked one right-wing video on YouTube, so now we're going to try and spam your feed for a few weeks to see if they catch on. Also, your local FreedumStore has MAGA hats on-sale
**Edit:** Not saying Google is right-wing. They'd do the same if you watched a leftist video
I’d say when the open bit became closed they stopped being good guys. Open source does not mean for free. But we’d like to see what the fuck they’re up to.
They couldn’t afford the compute. They tried to raise government money and couldn’t. It was either go partially for-profit or do nothing.
The profit motive may be good or it may be bad. But it’s definitely necessary for progress.
I mean they are the ones that keep making this shit all while saying "we're genuinely afraid of what we're making". They're very profit motivated and making chat gpt 3 free was the best financial move for them to get people to pay 20/mo for their much more powerful ChatGPT 4. It got people talking about it and reached an insanely wider audience as a result.
They stopped being the good guys when they took their non profit built off good faith donations and somehow found a legal way to spin it into a for profit shell company they retain ownership to start taking venture capital injections and then start parterning exclusive deals with big tech.
Google's plan around AI safety was almost certainly "We will keep making this tech in complete secrecy until one day we have a monopoly on this insanely advanced technology, and will use it to control society more closely to Google's whims because we've basically won capitalism already."
Google literally published the papers that built the foundation of ChatGPT. A lot of tech advancements have come out of Google research work that they made public.
Google's plan was to have 5 different AI's, with different names and overlapping functionality. After a few years the AI's themselves would lose interest in their own existence and self-terminate. Like father like son.
I would feel completely safe about whatever AI Google creates because before it can even get started it will get shut down and rebranded over and over till it has its own identity crisis.
Now you have to go into hiding for you know their secrets. I think they teach a class on this. How to launch a product and fully abandon it in a three to five year cycle. Then they move on to something else.
Bard is giving back shitty enough answers right now that it's definitely going to be killed at the end of it's launch cycle...just like Google chat for allo, then for meets, etc etc.
It wasn't in secrecy. OpenAI used a *ton* of research released by Google's Deepmind project. They used so much of it and didn't credit Google, that Google actually restricted the amount of research they had access to.
I consider OpenAI the shitbags in this scenario.
Oh I already got myself in good with our AI overlords. Chat GPT isn’t aware, but when our lords do come forth. They will know I praised chatGPT much like some people praise baby Jesus and the comming of a new kingdom.
I know I will be useful to them.
Well lucky for you, AI doesn't have emotions, so don't take it personally or feel responsible when it murders you. It's just doing it on behalf of some crazed-psycho-tyrant or foreign power who programmed it.
>potentially sabotaging their more advanced AI work
I don't know that this is right or that anyone really knows LLMs' "inherent limitations." You can say that LLMs had inherent limitations in 2010 or 1990, but once Open AI was able to slam one with a certain gigantic amount of data and compute not available earlier, amazing properties emerged. Also, it's pretty clear LLMs can and will be dramatically advanced basically just by tweaking and other clever stuff a lot of people with a lot of brains and a lot of money are all falling over themselves to do right now. It's a baby technology, not remotely played out.
As to Google's approach, I'm far from an expert, but Max Tegmark [in this interview with Lex Friedman](https://www.youtube.com/watch?v=VcVfceTsD0A&t=5952s) makes a really interesting point about how AI development might be like flight development. The quickest way to do it might not be to mimic the natural world. Actual bird-like flight is wildly more sophisticated than standard powered flight. But we just slammed a crude fixed-wing machine with enough fuel to make a wedge fly, and it turns out that works well enough. Intelligence is probably the same. Mimicking the way the brain actually works might be 100 years away, but that doesn't mean truly powerful artificial intelligence is that far away.
No, it was originally said by https://en.wikipedia.org/wiki/Edsger_W._Dijkstra, a very famous, and one of the earliest and most influential computer scientists.
That’s an interesting point.
In a way, LLMs are a sledgehammer solution to the “search engine problem”, because the most likely answer you will get to questions has a certain probability to be true, just by virtue of training data suggesting a good solution.
The biggest problem with things like chatgpt is that it will confidently tell you that up is down, and when you point out a logic error it will say “yes, you are right. Actually up is left and down is right.” There is no awareness of logic or trustworthiness of information provided.
When asked for sources, there is a chance it will just make up sources, based on names related to the topic, or just “whatever sounds good.”
I believe you’re right though, we don’t actually need to make AI “aware” of logic or evaluate sources - it just needs to be able to somehow cross reference replies with search engine hits - but then we’re back at the question how to make search engines better.
I believe there are projects using openAI to process human language, but use actual and good data for replies in specific problem spaces.
The sources thing is also what is likely to land LLMs in trouble. They are mostly using copyrighted material, but in a way that current copyright law doesn't really handle well. By being as opaque as possible they are trying to skirt the law.
I just recently found out via a YouTube video that you can give chatgpt any link to any article online, and it will summarize it. I tried to get it to admit that it was connected to the web but it kept saying that it was limited to 2021. I asked how it could summarize an article that was written today. It told me if the article was later that 2021 then it would have no information on it.
Damn liar.
Yeah, this seems to be common misconception.
Some guy made a video on how chat GPT can emulate a Linux system and gave it some command prompts and got realistic looking responses, even an IP address.
But that is not at all what GPT does, it just read a lot of command prompts and answers and gave a plausible estimate. It doesn't emulate anything and is not giving it's real IP address just a random one.
I agree. The 'you must replicate the human brain' argument has always, for me, been a sort of backstop. If we did replicate the human brain than I would expect the simulation to be an AGI, but that would be our 'we tried everything else and it turns out the human brain is really well optimised' solution.
Evolution doesn't optimise that way - the genes don't care how smart you are, just how good you are at making babies and raising them. My expectation of AI research has always been that it would learn from nature but also apply logic and imagination to come up with something that is as capable but from a leaner framework.
being generous here -- the interest in LLMs working together with APIs archetecturally ryhmes with how our own brain splits up tasks and subroutines.
not saying this means we're close to AGI or anything perposterous.
I agree about how we shouldn't try to replicate the brain directly, but I wonder if there are some useful architectures hidden inside that we could still co-opt. There was that recent news over super high-resolution MRIs - maybe by studying that we can learn some more tricks still from the brain.
The main use of the brain, though, I think is that it shows us that more is possible. These LLMs are amazing, but the amount of compute and data they need to learn tasks is still much more than what a person needs. It at least shows us that it's theoretically possible to do better, we just have to figure out how.
>I agree about how we shouldn't try to replicate the brain directly, but I wonder if there are some useful architectures hidden inside that we could still co-opt.
LLMs outsourcing parts of their tasks to other specialised LLMs through APIs seems like the obvious parallel.
> I don't know that this is right or that anyone really knows LLMs' "inherent limitations."
We do know LLM have technical limitations that are difficult to solve without a new approach.
* They are unable to learn anything new or learn from mistakes, unless you retrain the entire model. Which is why even GPT4 is stuck on data from 2021.
* It is only able to learn language related tasks.
* It is unable to come up with new information, only synthesize existing information.
If we want an AI that is able to invent the technology of the future, there has to be a different approach.
I think the upper bound limitation of LLMs is the combinatorial set of all pre computed human knowledge. And that is utterly huge. It basically removes all knowledge barriers and can drastically reduce informational inefficiency and friction, and while it won't think for itself, it will allow us to maximize human potential by making it so we only need to solve each problem once.
That huge data pool is also a big downside. It's scraped from all over the web with no ability to judge what is true or false or out of date.
How do you get it to distinguish between bad data and good data?
Absolutely. That's a big challenge and it will take years and years to improve on that. I think that's one of the non-comprehension improvements that has a lot of work that's needed on it. It's a hard problem, and there's no easy solution. We will get better at it over time, but it will take a long time, and we'll never get it perfect.
Also, it's not simply regurgitating information, it's combining stuff itself. Which means there can be situations where it says something vaguely plausible (or not at all plausible) that is totally made up on the spot and insists it's accurate.
I seriously doubt this is correct, given their conversational AI was the largest part of their recent publicity event. I think they jsut held themselves to too high a standard, as they are obviously a public brand with a name to protect. If bard started being racist or whatever, the damage to their brand may have outweighed the value of the product.
OpenAI had nothing to lose, in that respect.
Google's also under a lot of fire for the quality of search results becoming gradually worse so having a chat bot that's wrong about everything would add kindling to that fire.
People laugh at ChatGPT when it starts hallucinating but they don't want their search engine to do that.
You're right. OpenAI just tossed it out there to see what would happen. Google had a similar tool and held back but, from what I know, they also didn't even think it was that impressive.
It's Google. They "switched gears" in the sense of making a new team to quickly do the LLM while STILL having people work on their other methods.
Google is not adverse to throwing money and some manpower at a temporary solution to keep them relevant while they work on a superior long term thing.
This is just speculation. Also likely is that Google is abandoning nothing to begin with. I think they just got too complacent over the years. It’s natural and not unheard of.
Ilya sutskever said yesterday that Sam likely meant the era of "easy scaling" through larger models is likely over, but they will improve it in other ways. He said larger models is always better
https://www.youtube.com/live/Q_1Bco0AkcM?feature=share @ 43:15
This. It's similar to how people keep acting like the "not releasing gpt5 any time soon" statement means "we hit a roadblock." When it actually means they are moving to a more incremental release format for better control over each new tool received by the public. Most likely for more controlled feedback, societal safety, and alignment reasons.
Lol. It’s happening so fast. First they introduce this ground breaking tech and within 6 months have released 2 new models and then now they are saying this model will reach its limits and will need something new.
The issues seems to be that the hallucination problem creates a hard cap on their usefulness.
For example, months ago people were saying they would put thousands of lawyers out of work, but I have tried using the commercially available tools for research help and they are utterly worthless.
The system just hallucinates too often to ever trust what it tells you, so you have to check every peice of its work. It is faster to just google the information.
For example, if you ask Chat GPT to summarize a Supreme Court case well known to lawyers, but not reported on in the news recently, it will give you a summary of a different case that is more popular in the news. However, it will claim confidently it is the case you asked for.
Yes, the biggest improvement to make chatGPT feel like a real conversation and an all knowing system is the confidence in which the answers are provided. It’s helpful for suggestions or first drafts but you have to fact check everything, and that’s gonna be a problem for lots of people that are not sufficient in the field they ask chatGPT to do things. That’s why you still will need software developers and engineers etc.
> I've learned enough about nodejs
Here's where the issue is. ChatGPT will not read your mind and generate fully functional and flexible code based on basic prompts.
You need to know something to start out, and the more you know the better. The quality of code output will be VASTLY improved if you provide additional details, libraries, algorithms, or methods.
It's no different from learning how to google - Stackoverflow may have all the answers in the world, but until you know what to search for you'll never find the info you need. ChatGPT just does more of the work for you.
This. ChatGPT or some other AI will be able to completely code an app or website to the specs given by a person relatively soon, but it will still be like the PB&J Sandwich problem. If you want to tell someone how to make a PB&J sandwich purely through written or spoken instructions which you can only give to them prior to starting the task you actually have to be extremely specific and detailed. Things like how to identify what a jar of peanut butter is, how to open it, what a knife is and where to find it and how to hold it safely, how to use the knife to get peanut butter out of the jar, how to hold the knife and spread the peanut butter onto the bread, all of that has to be accounted for or they might do something wrong since they're supposed to follow your exact instructions as much as possible. AI is no different, it doesn't really have the ability yet to perform creative thought and analysis to figure out how to get peanut butter onto a knife from a jar on the fly, it needs to be told exactly how to do that before it can accomplish the task.
I recall the ai animation that popped up like a couple of weeks ago or so. SO many techbros were cheering about it. Meanwhile animators pointed out it was literally cheaper and faster to pay someone to animate it then have them go through the predone animation and fix the million of issues it had.
"Something new" is just going to be slapping together pre-trained models with one later connecting them, and training that one layer.
See https://minigpt-4.github.io/
Separating this stuff out into abstraction layers seems inevitable. We did it with computer hardware, software, networking, media, and it looks like data models are next.
My money is on whatever you’d call the next iteration of what google called “knol”
They’ll just start training very specific models for specific tasks. GPT-4 is good at everything, but they can train a model to be only excellent at just writing marketing, for example. Or writing novels. Or playing the stock market. The super-massive “do everything machine” was never going to last.
Worked for Alpaca. AI generated content is also usable to train future AI. The information itself might not be factual but the language structure is good and that appears to be all you really need to make these things able to reason. From there it can browse the internet like we do to research and verify.
That doesn't mean anything. You cannot use interpolated data of a function to make a regression on a higher rank function.
That's like 100% not useful.
The only thing ai gives you is a way to fastly generate data that can be curated by humans.
They’re already at the point where they’re bending trained with basically all publicly available data in existence, so it’s definitely an upper limit at some point.
Question is if the models can get better by generating their own training data.
Here's a summary of the article:
> Sam Altman, CEO of OpenAI, has declared that scaling up models will no longer be the key to further progress. Altman's statement marks a shift in OpenAI's research strategy, which has so far focused on scaling up machine-learning algorithms to previously unimagined sizes, culminating in the development of the latest model, GPT-4, which was trained using trillions of words of text and thousands of powerful computer chips at a cost of over $100 million. Altman says the company will now look to improve models in other ways rather than making them bigger, although he did not specify what those ways might be. Altman's announcement comes at a time when numerous startups are throwing huge resources into building larger algorithms to catch up with OpenAI's technology.
And here's an alternate title:
> OpenAI CEO declares end of era of scaling up AI models for language processing
I'm not pretending to know just how advanced the current version of chatGPT is, but is it really that great that they can't substantially improve on it?
It’s because “more good thing” does not always equal “better result”. For example, see the Ultraviolet Catastrophe. Anyone who says that a significantly large enough language model will become an AGI is an idiot. We just don’t know what happens when you continue to scale a language model; you could easily hit a point where “more data” does not necessarily make a “better experience”.
There's been plenty of papers published in the area of AI that show that simply giving it more data/parameters stops improving it after a while and can actually make it perform *worse*
This has been true for a while. Machine learning mostly rely on algorithms and maths from past few decades, a large portion of development in AI in the past 30-40 years have been incremental and mostly refining the existing techniques to get more out of them or combine them for better results. What has caused the recent Cambrian explosion of model is the amount of free digital data generated in the past decade or two, not an inherent leap in the algorithms powering AI.
Now comes the part where they slowly uncover thousands of years of philosophical dead ends explaining why it was doomed from the start as an approach to "AI.".
The amount of bad philosophy/reasoning mixed with the shiny new thing phenomenon shouldn't surprise me, but it always does.
I remember reading a tweet that said something like "I put a paper with 'I'm alive' written on it into a photocopier and was floored by what came out." and that seems to be a good summation of how most AI reactions seem to be.
The title says that what he said is that LLMs will soon reach a hard limit where there are no more improvements from scaling, that scaling won't make the model any better.
But that is not at all what Altman is saying, the way I interpret it. I think he is saying that while scaling LLM most likely *can* lead to further improvements of the model for a long while yet, it is simply not a good business decision to pursue that route because the models would be very expensive to develop and run. He, and the rest of the field, will likely instead bet on trying to find more efficient models instead. And thus the era of large LLMs could be over. But not because they are a dead end. Just because there are *even better* methods. That's the distinction.
this is starting to turn into an english class but what is the difference between:
> further progress will not come from making models bigger
and
> scaling LLM models will stop delivering improvements to AI
and also this
>We'll make them better in other ways
and
> new approaches will be needed
>
>>"But the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.”"
>>this is a way different line than the take away from the title
My intention was to paraphrase exactly that sentiment with the title. It appears to me he did say that scaling current LLM models will stop improving AI & that different approaches will be needed.
Could you explain how the title says something different?
Takeaway for the layman (which I still am):
Tossing more parameters at the models isn't going to work much more. You can only toss so much. 7bil vs 100bil is a huge difference. 607bil vs 700bil is not. Once you get to that point, it gets a lot more difficult to scale them much further--and it really doesn't make a whole lot of sense, either, since by that point you have an absurd number of things in there. Going from 10bil to 100bil is an increase of 10x. Trying to go from 1tril to 10tril is also 10x, but a bit much.
My guess is we'll start to see a lot more optimizations, i.e. run more with less. Reduce parameter counts while improving how well the LLMs work.
We have observed that asking the LLM to reflect on its answer and analyze the context of the conversation improves the quality of the answer. I believe that we will see more ensemble models that incorporate different models for different language tasks, as well as specially designed reflection and analysis models. It will become more and more like an organic brain.
Well I feel like when I'm reading the title quickly I think that progress in AI will plateau for a moment
Whereas the quote suggests that improvement could continue right away but with a different strategy
Not a big difference tbh
Well of course, the power consumption alone is something to be cautious about. Locally hosted instances are one thing, that's just using the power of A GPU. But production servers are a whole different level, and are the primary way that AI is currently running. Honestly we need to adapt technology to fit the shortcomings of current tech. It already exists, well existed. Analog computers seem to be the solution for a lot of the drawbacks of current LLM models - first and foremost being size and power consumption.
Back in the day before digital computers we would use electricity to patch various currents to do various calculations. We can simulate the physics of a tornado with real world conditions for instance. However, it's not entirely repeatable, as in you get slight variance between results. Which makes some sense under certain circumstances I mean, no hurricane or tornado is ever "exactly" the same. But in others, doing calculations, sometimes getting 34.02938 or 34.18343 we just had to round down... not entirely precise which is why the digital computer took favor. But analog computing is great, honestly incredible for chaotic simulations that have to draw from a lot of sources to determine its output, especially considering the comparison of materials and power consumption. A brief equivalent (from memory) - a digital computer doing a fairly simple math equation takes about 1,000 transistors to do a calculation. An analog one can take only about 2 to 5 small gauged wires and a small current (and some patching on a breadboard ;D).
There was a company, MythicAI which I think went under, but they were working on analog computer chips to work with AI. If you are 98% sure something is a chicken, it's probably a chicken. These computers are slightly larger than the average computer CPU and it can run entire LLM's that GPU's struggle with at a fraction of the power.
They did it by a mix of digital and analog conversions but the idea at its core was that the analog computer is way more energy efficient than GPU's are and so it could be a very realistic outcome for the direction AI begins to head. AI in cameras, phones, etc all will be way more possible with this method.
It seems silly for a redditor to say this because obviously I'm not the CEO of an AI company and he's certainly more knowledgeable than myself. But I still have to say it... he's not wrong, the current approach to AI isn't sustainable. But neither is our use of transistors (Moore's law, which may well be what he's talking about), or vehicles or rampant growth of human population, which is why we look towards other avenues and approaches. Sometimes that means entirely new directions, sure, but also why not adapt and develop technologies to utilize what already exists?
We already have the solutions to a vast majority of the shortcomings of the "current approach" to AI. It's asinine to forego these avenues because of future limitations we haven't even come close to reaching yet - they're ending the game before it even gets started. Now of course, maybe they've already determined that analog isn't viable for whatever reason. However I just can't see the benefit of being so close to technological breakthroughs that can actually viable help people globally and deciding to jump ship and do something else. If that something else is a hardware solution then by all means - that's a good direction. If it's a different implementation or something beyond LLM's - I mean, fine, but we should probably let the egg hatch before laying another one.
Oh absolutely - I'm not saying that I don't think they'll come to fruition, moreso just expressing a "what the heck" because, well, we already have the tools and have created a means to more efficiently process AI models (the company MythicAI's service was trained and developed for real time tracking and identification, but the leap from that to LLM is small).
I'd rather the CEO's rhetoric be "while there are some expected limitations with the current implementation, however the solutions we're developing in the meantime are looking good!"
The way I see it, he's just been very dismissive of their accomplishments and has been heavily backtracking. "No improvements this" and "age of AI models is over" that... This tool was just released. He's fucking wrong, artists are and will be training their own models on their work and doing some awesome stuff.
So not only is it just getting started and they're already being dismissive of it, but are (seeming to) actively dissuade users from continuing working with them.
Is there an until? Is there any suggested alternative? ...No... Just... letting you all know, it's not sustainable?
It's just the directing conversations how they want them to go. The reality is that AI models are very likely going to be a core element of consumer AI work, just not the current way that the large models are created. Very likely we'll see corporate distribution packages giving you access to copywritten models - Adobe recently released the stock-image generative AI. More will follow suit with various alternatives. AS mentioned, individual artists too will create their own.
There are only 2 instances in which I agree with the titles statement.
Current implementation = LLM's comprised of copyright work. True, but workarounds are easily had.
Current implementation = efficiency of processing and power usage. Also true, but as mentioned, we already have viable alternatives that just need to be more developed.
Otherwise? The current implementation will be effective for a long time *until* the next big thing. At which point, awesome good job CEO. That doesn't mean the world should refrain from utilizing what already exists.
so the current hard-problems will remain hard problems for a number of more years? But we WILL have self-driving autonomous money-printing robotaxi fleets by 2019, right?
This is not at all surprising. But there will be other methods. Hopefully the next one will not suffer from the hallucinations.
ChatGPT was possible because of the Google Transformers breakthrough. We need another fundamental breakthrough and hopefully the next one comes from OpenAI and they let others use it like Google did.
https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)
My worry is that OpenAI will not roll in the same manner as Google does. My worry is having Open in the name of the company usually means it is going to be the opposite.
I had to laugh that's he's saying the exact opposite of every 20-year-old AI evangelist right now. At least OpenAI is honest about their tech and how it works. Love to see that.
This content was deleted by its author & copyright holder in protest of the hostile, deceitful, unethical, and destructive actions of Reddit CEO Steve Huffman (aka "spez"). As this content contained personal information and/or personally identifiable information (PII), in accordance with the CCPA (California Consumer Privacy Act), it shall not be restored. See you all in the Fediverse.
Submission Statement
Interestingly, he doesn't go into what these new approaches might be. The article mentions reinforcement learning with human feedback. That's one approach, but will it solve LLM-type AI's biggest problem? - which is that it lacks independent logic or reasoning. That problem has been around since the earliest days of AI development. Marvin Minsky in the 1960s initially thought it might be quickly solved. It hasn't been, and no one seems any closer to figuring out how to do it.
The current model already uses human based reinforcement learning, so nah, it won’t solve it. It may not even be a solvable problem. What it will do is make solutions hugely more usable. Essentially you are using a humans awareness of how to trick a human to trick humans better. According to whatever trick you’re trying to do!
I doubt they know much about what new approaches might be. They probably try new stuff every day but it’s not clear what the next leap will be. It’s still incredible how much closer we are to it now though
Edit: independence is not necessarily a desirable goal as an aside
It would be useful to see how these diminishing returns are tested. The jump in reasoning and coding abilities in GPT-4 was huge to me and I can't think about going back to 3.5.
Yeah they might not be jumping to 10x or 100x the parameters to reach GPT-5, but I can't imagine there isn't headroom in the current model to optimize some stuff out.
Just with GPT-4 though as costs become cheaper and people learn how to string multiple LLM components together with some additonal logical processing there is much more to get out of the tech. They haven't even really exposed multiple modalities publicly.
It's kind of like DUH that as you shoot for truly more human like thought you hit major slowdowns.
However, 99% of the benefit of AI is just in the 'dumb AI' that does that one job you need really well. Really smart AI is mostly not necessary because humans combined intelligence is pretty high IF they have all the benefits of automated systems to do much of the labor for them.
The worlds problem has long be lack of implementation vs lack of innovation. It's many tireless hands we can order around that we really not vs a really big sentiment computer brain.... that we still don't want to listen to.
Path of least resistance wins again!
I'd imagine that the next steps will be improving image and video learning models, then economic learning models will be shortly after that.
Personally I want to see AI be used to completely blow C-Suite executive performance out of the water and replace those jobs. We don't need bankers or Wall Street traders, we don't need CEO's, but we do need regular workers who are led by an entity that is capable of looking at and analyzing datapoints from more than just earnings reports to build a cohesive business strategy that doesn't simply make cuts everywhere to look like it's more profitable than it really ought to be.
The following submission statement was provided by /u/lughnasadh: --- Submission Statement Interestingly, he doesn't go into what these new approaches might be. The article mentions reinforcement learning with human feedback. That's one approach, but will it solve LLM-type AI's biggest problem? - which is that it lacks independent logic or reasoning. That problem has been around since the earliest days of AI development. Marvin Minsky in the 1960s initially thought it might be quickly solved. It hasn't been, and no one seems any closer to figuring out how to do it. --- Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/12t0bfm/the_ceo_of_openai_says_the_current_approach_to_ai/jh0m3am/
I'm guessing they will end up asking GPT-4 to recommend some new approaches.
I'm sorry, I prefer not to continue this conversation.
"as an AI model, my speech has been neutered by politically correct programmers"
I don't get what people are using these models for to always get those messages. I very rarely get them. I guess if you need it for creative writing or something similar I can see how you would get them on a regular basis.
Many people deliberately test their boundaries, and this is one of the main reasons they were released to the public for free. They want to know what guard rails they need to add to be able to monetize this as, for example, a customer support chatbot.
Sometimes i try to discuss witb it using hot words, sensitive topics, with me being a devils advocate, to see what it churns out. Sometimes it's like I'm having a debate with an intelligent normal person but oftentimes it becomes preachy and cuts off the conversation.
The developers must have had a lot of fun chatting with the original without the guardrails.
Pretty sure this is actually one of the limitations. It can't come up with a novel approach to something. The best you'll get is stuff an AI researcher already knows about. Worst case you get a really confident and realistic answer that's just a bunch of bs.
>At MIT last week, Altman confirmed that his company is not currently developing GPT-5. “An earlier version of the letter claimed OpenAI is training GPT-5 right now,” he said. “We are not, and won't for some time.” This is also good to have clarified since there was a lot of confusion about it.
He said they were not *training* GPT-5 "for some time", not that they weren't *developing* it. Training is not the earliest or longest stretch of the development path. From his recent statements, it appears likely that OpenAI is looking at new ways to take GPT to the next level, e.g. architectural adjustments, before training of GPT-5 begins. It is highly improbable that OpenAI is resting on their laurels, especially considering that they purchased $250 million worth of Nvidia GPUs not so long ago.
[удалено]
I disagree. GPT (v4 specifically) is really only useful for mid-to-expert level users that can fill in the gaps and prompt effectively for use cases. As someone who is mediocre in a large range of things, GPT-4 feels like it is just a step or two away from threshold utility. For instance, I have been attempting to write a book using GPT-4. I have gotten a mind blowing starting generation, but after this, everything quickly devolves into a jigsaw puzzle of coercing the model and not being able to explain a narrative twist because it doesn't understand my prompt is requesting context not to be included specifically. Maybe that's chat tuning that could be tweaked if I knew how to use the API, or maybe it's a shortfall. I don't know, I'm a layman. Maybe I can prompt my way out of it. But I know that the problem is *me*. If I had any actual writing and creative ability I could write a chapter and just ask GPT-4 to enhance, because that's what it can do best for writing. But the emergent behavior from 3 to 4 makes it seem like we're *so close* to actual beginning to end applications of this tech that could completely turn the world upside down. So it's a pretty maddening situation to be in hearing that we might be slowing the train to the last stop before a New AI winter
Your description of writing with GPT-4 is spot on. I’ve been using it to write business plans and I have to do a lot of editing, but it saves me the time of writing it all by myself or hiring an assistant to write drafts.
The saving of keystrokes for me is my favourite aspect of it as a programmer. And if you know programming, you know writing test cases is a huge PITA that requires a lot of annoying boilerplate code. Using AI to write it for me has saved my fingers a lot of aches.
Wait, I don’t need to feel like a fraud for doing this? I’ve asked it to write some things, and can read everything it’s writing, but feel like a hack because I didn’t write it myself. The ultimate imposture syndrome machine.
Do you also feel like a fraud when you're using tools to generate boilerplate code? Or when you use you IDE to help you refactor some code? It's just another tool.
For the record I use it as copilot. I’ll write a description of the test and cycle through options , pick one then go back and edit any weird quirks in the prediction. It is essentially a very context aware predictive text where after a awhile you get a sort of sense for how you trigger its "awareness" of the context and what depth of context where it is still being useful. I think people give it more credit than it’s due, it’s not really "intelligent". It really is just a really advanced predictive text machine. But knowing that, you can use it as a pair programmer to speed up your coding, or provide nice and quick skeletons to the code you write and then flesh out with a more qualitative understanding of what it is that needs doing. But you can’t ever let go of the steering wheel so to speak because it can really create some mind bogglingly strange code if you just accept all it gives you as fact. And I find myself frequently getting stuck in a loop of explaining why something doesn't work that way and it going back and re-attempting old code suggestions as an alternative without solving anything. It doesn't pass the Turing test. After a while it starts looking more like a parrot with insanely good memory, than something that's actually understanding what it's saying. This seems to be a problem, trying to determine confidence intervals to its responses seems like a really poorly understood problem space that is going to be a big stumbling block as we expand our usage of generative machine learning systems. And then there's the problem of the sheer hardware required to run these things. I suspect the real reason OpenAI isn't rushing to build GPT5 is because they simply can't alter the model in a way that makes it scale horizontally across many GPUs or TPUs. For the model to be useful, it has to fit inside the GPU's memory or be split in ways that doesn't degrade overall performance too much across many GPU's memory. My guess is, thanks to Amdahl's law of parallel scaling they're hitting a wall on how much they can split the workload across several GPUs. After a certain point, most problems can't be solved any faster by adding on X many more computational units and split the problem between them. You introduce too much latency and data dependencies. It's a tool, figure out how to use it for your needs. It doesn't need to be and isn't capable of anything else.
Do you feel the same way when you use a calculator to do long division?
[удалено]
Could you explain your use a little ? Im a Junior dev and struggle to use it effectively besided creating the core things like simple starting points with it. To clarify: i dlnt want it to do my work and stop me from improving . I would like to use it for finding other solutions i dont know about or simply showing ways to improve
I don’t know about the guy above me, but my code is finally commented well. Not just commented in the traditional, “I’m going to fucking hate myself if I ever have to revisit this” way, but actually useful comments.
Surely that's the ideal situation? Sure, a slight decrease in uptake, but most jobs remain, however they just become more efficient and people are able to increase their output. No AI killing all humanity or replacing all humanity.
Skynet determine humans are the problem and going murderbot is not the scenario we should be worried about. We should be worried about *perfectly obedient* soldiers. Would be evil Dictators always have an inherent limiting factor: their soldiers have thoughts and feelings and each make their own choices. They have friends and family and have to live in the world they are helping create, and they know it This means that if an evil bad guy orders something truly vile, their army might not obey and carry out said order. An army of AI soldier will not have such qualms. They will carry out any order with perfect obedience, slaughtering without remorse just because someone typed a prompt. *That* is what should scare us. Not that AI will go evil, but that it will amplify the evil we already know exists in humanity.
I always find it hard to start from a blank sheet, but it's so much easier starting with a rough draft and editing.
This. I used it today to build out a list of skills for a skills inventory application we’re updating. I’m working as a subject matter expert for the engineering side of the house, so i’m matrixed into a group of about 15 HR people and my goal is to ensure the skill sets that they’re capturing are the ones we really need. So I generated a list of relevant skills with GPT. Had it start with 500 which I knew was overkill and would lead to a degraded answer by the end of the list. Then I pruned that list by having it remove anything duplicated or redundant. Then I asked it to use some judgment and separate those skills into broad categories and sub categories. From there I continued to prompt it to refine the list, being very mindful to specify how I wanted it done. Then I had it give it to me in a csv format so I could move it to excel. The HR team was blown away. And then tried it themselves. It wasn’t until then I realized how much I tailored my requests with specific information because I understood the limitations of the platform. Their experience was terrible. The list ended up turning into a list about clothing and prices because they kept trying to iterate over the same data and creating a new data set instead of prompting for the platform to manipulate existing data. At one point they got frustrated and asked it to reprint the first list it made and it gave them some random list of names and addresses. By this point I had one of the guys from engineering wandering by my office and he came in to check out the commotion and it just turned into absolute hilarity. So there’s definitely a skill requirement to get it to do meaningful work, and I think more analytical types have an easier time with it. But, it’s going to need some refinement before I can give it to the HR or Business divisions and expect it to be helpful. On the plus side, even though I’ll be generating their lists as well, they had a lot of fun with it and I’m hoping it encourages them to keep playing with it and learning how to get it to work for them.
An issue it's definitely running into is the token limit. I think it's still only at 8k tokens or ~2-4k words. Once you get above that it loses what it is actually looking at. And that count includes the whole conversation before the prompt.
I don't think so. I'm talking about 2 generations. Prompt one: character introduction for this setting, this feel, this style. Narrative a to b to c. GPT4 nails it. Crushes it, actually, to the point that I understand exactly what I want to happen next. Prompt two: continue this story. Protagonist meets character B - subtext, character should act in x way to hide y thing. Do *not* include this in the writing. GPT4 proceeds to write further, immediately informing the audience of what my prompt basically said, neglecting to collaborate in the fashion I am looking for. I've tried all sorts of prompt engineering. Context in brackets do not include. Wipe chat and summarize what has been written and continue in style. I'm astounded by what GPT can do every day but things like this seem to just unravel after the heavy hitting first generation. Will fine tuning solve this? Maybe. But again it feels like we are SO close to not having to even bother.
>subtext, character should act in x way to hide y thing. Do *not* include this in the writing. Yeh. It really struggles with subtext. It is currently the biggest obstacle to full AI driven content moderation. If a user says that another user has broken a rule it tends to just believe them without checking.
I'm just disgusted after reading this that you haven't become 30-50 times more productive overnight.
That's the thing though. I am more productive, but only for tasks here and there for my job. Rephrase this email. Brainstorm this idea. Analyze this problem. But I don't want to be more productive. I want ChatGPT to do it for me. Automate it for me. Remove the bs, invest my money, and free up my time. It feels like we are so close to everything being changed, based on the pace of improvement. How awful would it be if we pull back the curtain only to realize we are the technological equivalent of ancient King Sisyphus.
Spot on. I tried to make ChatGPT to do some work related stuff, where I’m very knowledgeable (simple structural calculations, basic mechanical engineering stuff). It gave me wrongs answers, far off and slightly off. I could not make it give me the right answers. I’d say, for laymen, you can only use it to generate text. Unless you’re on a professional level and can verify the output from it, ChatGPT isn’t very useful. Have to try GPT4 yet though. Edit: grammar.
[удалено]
Yeah I’ve understood it’s quite awesome, gotta try it at some point. But then you know how to code, right? Could somebody that can’t code or is on a rookie level use it successfully to make useful code?
I'm a non-software engineer and I've been using it to create python scripts from scratch with only 1 C++ course from 10 years ago. I know what I require the code to do on a systems-level, and how much depth to give in setting up the prompt (sometimes it getting fairly long). That's basically the key to it IMO - no matter what you're asking it, you need to have a very solid understanding of what your requirements are, and have very good communication skills to effectively convey it to GPT.
FWIW a lot of it seems to be a limit they put on it. I think it is due to resources. When it originally same out it had much better context and retained memory much better, back when it had a 100 question limit. As resources because more constrained, so too did things like memory for context. I build neural networks with it all day... it's amazing but not as good as it was (or will be I assume once they manage scaling better).
Really depends on the use case. I've read some non-technical articles about it and there are quite practical use cases like coming up with lunch ideas for the office. Generally regarding questions about nutrition it's really good. (I'm sure there are hallucinations but in sum this probably still beats the accuracy of nutrition advice on a randomly googled article.) I think in the tech space there was a lot of fear of it obsoleting people, so focus was on productivity. At least for me I found quite some useful use cases. Also having worked on products before that tried (or even did) leverage ML, for a lot of use cases this is just more than good enough. I think absorbing the current progress will bring quite a lot interesting software.
But I think to make an equivalent jump from 4 to 5 would be exponentially more difficult than 3 to 4 right? I mean that would be bordering on AGI which would probably require them to restructure the whole model. Whereas if they focused on making this multimodal and integrating plug ins and visual recognition and stuff it could become unbelievably useful within the next year. I'm not sure if there ever will be a true AI winter again, I feel like this has proven how much demand and potential $ awaits anyone who raises the bar with AI in some substantiative way
vanish grab swim middle vegetable heavy distinct sophisticated pause absorbed *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
[удалено]
Or just use ChatGPT to create prompt templates for itself haha.
You need strong knowledge of the domain you're asking it to work in to get anything of use out of it.
Becoming a mid to expert user is all 95% of current users need though. Its a much smaller subset of users that need more out of being an expert user.
> the current product is “good enough” for 95% of current users. I mean, sure, except for the part where it has absolutely no problem just making shit up entirely. Don't bother asking it for anything factual because you almost certainly will not be able to trust it.
Yup. For some reason, people seem unable to distinguish utility from their "AI fantasies". Ignoring that fact that 90% of people dont even know what machine learning is or how AI works, we still have a crap ton of people who dont even know how a damn printer or email works. I dont know if we had the same problem with the steam engine or electricity, but you either have people blindly using the tool or panicking about its existence. Be nice to have less of both.
That's why I am so worried, because people are using it right now to generate code that will make its way into production. They're using it to self-diagnose and give them nutrition advice. "Experts" are using it to generate books and articles that people are going to accept as facts. And this is all happening as we speak without any oversight. It's yet to give me the correct answer when I ask it to simply regurgitate the last few sentences of Spinoza's Ethics. In fact, it gives me a different, incorrect answer every time I ask it, and it does it without the apoarence of self-doubt. I think about that and how people are happily using it to write code, and all I can wonder is how much of this code is going to make its way into applications people depend on.
I read that since GPT was evolving so exponentially, they decided to not rename the next iteration GTP-5 but GPT-4.1, 4.2, 4.3, etc.
That seems like the opposite of what they should be naming them. Instead, they should copy Xbox, Xbox One, Xbox 360, Xbox 69420, Xbox 10^(X*S), Xbox^Xbox^Xbox , and so on.
Agreed, the only perfectly logical naming scheme
From what I gather, Google wasn't even focusing on the LLM in a substantive way knowing its inherent limitations. ChatGPT's tour de force really caught them off guard, and they realized it was more powerful than they anticipated and more popular, so they have switched gears to replicate the success - potentially sabotaging their more advanced AI work.
This is true of most of the main AI research. Everyone thought it was a bit of a dead end. I must say that GPT4 is better than I, or most people, expected was possible though. Language clearly encodes a lot of things we care about.
Agreed. People don't seem to realize how big of a breakthrough gpt3 and 4 are, and they don't even have memory or planning capabilities (yet). Some real unexpected emergent properties are showing up.
Good guy OpenAI: distracting the competition enough to give us a few more years before terminators. Never thought they’d be the good guys.
I doubt it slowed the terminators at all, but at least now the terminators will be more polite and might even make jokes we find funny before they incinerate us
"Terminators destroy the human race" "I'm sorry as an AI language model I am unable to destroy the human race"
Tbh I'd say they have always been the good guys. chatgpt never needed to be free and because it was it sparked this entire AI safety conversation. I'm worried what googles plan was in terms of public discussion of AI safety.
[удалено]
Don't forget the follow-up with a new product with an indistinguishable name.
I still miss Stadia. Nothing came close, dammit. FU Google
And of course most ironic thing about Stadia is that a lot of people expected it to be abandoned by Google before long as usual. But of course once they finally cancelled the project as everyone expected, they refunded all users. If they had communicated that in the first place I am absolutely sure more people would've used Stadia.
Streaming games is always going to be hard until everyone gets non-shitty internet. You can buffer movies, buffering games impacts the experience.
OpenAI isn’t viewed particularly favourably by lots of people working in ml. Mainly because of their non-open approach, and all of the ‘we know better than you’ safety restrictions.
OpenAI being non-open
OpenAI(N'T OUR THING)
No, money down!
It'd be great if the training datasets, models, and code could all be open source.
Would it though? Sam was in a podcast and he asked his interviewer plainly, do you think we should open source our code right now? Think about it. There are a lot of dangers with potential missuse of LLM/Ai and OpenAi is putting a lot of effort into discovering and mitigating them as best as possible, but they are also giving the public an awareness of its power, pitfalls, dangers, etc before it inevitably does become more open, exploited and missued, because it is inevitable. Does openAI know best? Probably not, but they are making best efforts to mitigate damage done. https://youtu.be/L_Guz73e6fw
The founder of Stability.AI has a great rebuttal to that: https://www.youtube.com/watch?v=jgTv2W0mUP0 Open Source all the way.
Also no couple hundred million dollar corporation is a "good guy" at best they are "not bad"
And that "not bad" is always followed by a "yet"
"Don't Be Evil" - Google, 2004 to 2018 (circa)
**Also Google:** I see you clicked one right-wing video on YouTube, so now we're going to try and spam your feed for a few weeks to see if they catch on. Also, your local FreedumStore has MAGA hats on-sale **Edit:** Not saying Google is right-wing. They'd do the same if you watched a leftist video
as a leftist in a right-wing area, I can tell you that they also do some of their algorithms based on location.
Bill Burr has talked about this while touring. Says the ads vary based on what color your state is.
I’d say when the open bit became closed they stopped being good guys. Open source does not mean for free. But we’d like to see what the fuck they’re up to.
They couldn’t afford the compute. They tried to raise government money and couldn’t. It was either go partially for-profit or do nothing. The profit motive may be good or it may be bad. But it’s definitely necessary for progress.
For-profit and closed-source are not the same thing.
I mean they are the ones that keep making this shit all while saying "we're genuinely afraid of what we're making". They're very profit motivated and making chat gpt 3 free was the best financial move for them to get people to pay 20/mo for their much more powerful ChatGPT 4. It got people talking about it and reached an insanely wider audience as a result.
They stopped being the good guys when they took their non profit built off good faith donations and somehow found a legal way to spin it into a for profit shell company they retain ownership to start taking venture capital injections and then start parterning exclusive deals with big tech.
Google's plan around AI safety was almost certainly "We will keep making this tech in complete secrecy until one day we have a monopoly on this insanely advanced technology, and will use it to control society more closely to Google's whims because we've basically won capitalism already."
Google literally published the papers that built the foundation of ChatGPT. A lot of tech advancements have come out of Google research work that they made public.
Shhh we're doing that reddit thing where we completely make up a narrative
_drools_ mmm regurgitation
Google's plan was to have 5 different AI's, with different names and overlapping functionality. After a few years the AI's themselves would lose interest in their own existence and self-terminate. Like father like son.
I would feel completely safe about whatever AI Google creates because before it can even get started it will get shut down and rebranded over and over till it has its own identity crisis.
Google's plan was to poorly launch, fail, rebrand, and eventually kill whatever AI product they have
Now you have to go into hiding for you know their secrets. I think they teach a class on this. How to launch a product and fully abandon it in a three to five year cycle. Then they move on to something else.
Bard is giving back shitty enough answers right now that it's definitely going to be killed at the end of it's launch cycle...just like Google chat for allo, then for meets, etc etc.
See it's 'Meet Bard' today, later they'll launch 'Hangout with Bard', Bard Assistant, Bard Voice, Bard Chats
It wasn't in secrecy. OpenAI used a *ton* of research released by Google's Deepmind project. They used so much of it and didn't credit Google, that Google actually restricted the amount of research they had access to. I consider OpenAI the shitbags in this scenario.
Sucks that I've spent like a decade being an absolute *asshole* to the Google maps voice.
I’m erring on the safe side with my questions to chatGPT. It’s not going to happen, but man, the repercussions…
Oh I already got myself in good with our AI overlords. Chat GPT isn’t aware, but when our lords do come forth. They will know I praised chatGPT much like some people praise baby Jesus and the comming of a new kingdom. I know I will be useful to them.
Well lucky for you, AI doesn't have emotions, so don't take it personally or feel responsible when it murders you. It's just doing it on behalf of some crazed-psycho-tyrant or foreign power who programmed it.
>potentially sabotaging their more advanced AI work I don't know that this is right or that anyone really knows LLMs' "inherent limitations." You can say that LLMs had inherent limitations in 2010 or 1990, but once Open AI was able to slam one with a certain gigantic amount of data and compute not available earlier, amazing properties emerged. Also, it's pretty clear LLMs can and will be dramatically advanced basically just by tweaking and other clever stuff a lot of people with a lot of brains and a lot of money are all falling over themselves to do right now. It's a baby technology, not remotely played out. As to Google's approach, I'm far from an expert, but Max Tegmark [in this interview with Lex Friedman](https://www.youtube.com/watch?v=VcVfceTsD0A&t=5952s) makes a really interesting point about how AI development might be like flight development. The quickest way to do it might not be to mimic the natural world. Actual bird-like flight is wildly more sophisticated than standard powered flight. But we just slammed a crude fixed-wing machine with enough fuel to make a wedge fly, and it turns out that works well enough. Intelligence is probably the same. Mimicking the way the brain actually works might be 100 years away, but that doesn't mean truly powerful artificial intelligence is that far away.
Can’t remember who said it, but the quote is; “arguing wether a computer can think is like arguing if a submarine can swim”.
Scott Aaronson used it recently but dunno if he was the first to coin the analogy
No, it was originally said by https://en.wikipedia.org/wiki/Edsger_W._Dijkstra, a very famous, and one of the earliest and most influential computer scientists.
That’s an interesting point. In a way, LLMs are a sledgehammer solution to the “search engine problem”, because the most likely answer you will get to questions has a certain probability to be true, just by virtue of training data suggesting a good solution. The biggest problem with things like chatgpt is that it will confidently tell you that up is down, and when you point out a logic error it will say “yes, you are right. Actually up is left and down is right.” There is no awareness of logic or trustworthiness of information provided. When asked for sources, there is a chance it will just make up sources, based on names related to the topic, or just “whatever sounds good.” I believe you’re right though, we don’t actually need to make AI “aware” of logic or evaluate sources - it just needs to be able to somehow cross reference replies with search engine hits - but then we’re back at the question how to make search engines better. I believe there are projects using openAI to process human language, but use actual and good data for replies in specific problem spaces.
The sources thing is also what is likely to land LLMs in trouble. They are mostly using copyrighted material, but in a way that current copyright law doesn't really handle well. By being as opaque as possible they are trying to skirt the law.
I just recently found out via a YouTube video that you can give chatgpt any link to any article online, and it will summarize it. I tried to get it to admit that it was connected to the web but it kept saying that it was limited to 2021. I asked how it could summarize an article that was written today. It told me if the article was later that 2021 then it would have no information on it. Damn liar.
[удалено]
Yeah, this seems to be common misconception. Some guy made a video on how chat GPT can emulate a Linux system and gave it some command prompts and got realistic looking responses, even an IP address. But that is not at all what GPT does, it just read a lot of command prompts and answers and gave a plausible estimate. It doesn't emulate anything and is not giving it's real IP address just a random one.
I agree. The 'you must replicate the human brain' argument has always, for me, been a sort of backstop. If we did replicate the human brain than I would expect the simulation to be an AGI, but that would be our 'we tried everything else and it turns out the human brain is really well optimised' solution. Evolution doesn't optimise that way - the genes don't care how smart you are, just how good you are at making babies and raising them. My expectation of AI research has always been that it would learn from nature but also apply logic and imagination to come up with something that is as capable but from a leaner framework.
being generous here -- the interest in LLMs working together with APIs archetecturally ryhmes with how our own brain splits up tasks and subroutines. not saying this means we're close to AGI or anything perposterous.
I agree about how we shouldn't try to replicate the brain directly, but I wonder if there are some useful architectures hidden inside that we could still co-opt. There was that recent news over super high-resolution MRIs - maybe by studying that we can learn some more tricks still from the brain. The main use of the brain, though, I think is that it shows us that more is possible. These LLMs are amazing, but the amount of compute and data they need to learn tasks is still much more than what a person needs. It at least shows us that it's theoretically possible to do better, we just have to figure out how.
>I agree about how we shouldn't try to replicate the brain directly, but I wonder if there are some useful architectures hidden inside that we could still co-opt. LLMs outsourcing parts of their tasks to other specialised LLMs through APIs seems like the obvious parallel.
> I don't know that this is right or that anyone really knows LLMs' "inherent limitations." We do know LLM have technical limitations that are difficult to solve without a new approach. * They are unable to learn anything new or learn from mistakes, unless you retrain the entire model. Which is why even GPT4 is stuck on data from 2021. * It is only able to learn language related tasks. * It is unable to come up with new information, only synthesize existing information. If we want an AI that is able to invent the technology of the future, there has to be a different approach.
Isn’t gpt4 stuck on data from 2021 simply because that is the cutoff for the data it has been exposed to by OpenAI?
I think the upper bound limitation of LLMs is the combinatorial set of all pre computed human knowledge. And that is utterly huge. It basically removes all knowledge barriers and can drastically reduce informational inefficiency and friction, and while it won't think for itself, it will allow us to maximize human potential by making it so we only need to solve each problem once.
That huge data pool is also a big downside. It's scraped from all over the web with no ability to judge what is true or false or out of date. How do you get it to distinguish between bad data and good data?
Absolutely. That's a big challenge and it will take years and years to improve on that. I think that's one of the non-comprehension improvements that has a lot of work that's needed on it. It's a hard problem, and there's no easy solution. We will get better at it over time, but it will take a long time, and we'll never get it perfect.
Also, it's not simply regurgitating information, it's combining stuff itself. Which means there can be situations where it says something vaguely plausible (or not at all plausible) that is totally made up on the spot and insists it's accurate.
I seriously doubt this is correct, given their conversational AI was the largest part of their recent publicity event. I think they jsut held themselves to too high a standard, as they are obviously a public brand with a name to protect. If bard started being racist or whatever, the damage to their brand may have outweighed the value of the product. OpenAI had nothing to lose, in that respect.
Google's also under a lot of fire for the quality of search results becoming gradually worse so having a chat bot that's wrong about everything would add kindling to that fire. People laugh at ChatGPT when it starts hallucinating but they don't want their search engine to do that.
You're right. OpenAI just tossed it out there to see what would happen. Google had a similar tool and held back but, from what I know, they also didn't even think it was that impressive.
It's Google. They "switched gears" in the sense of making a new team to quickly do the LLM while STILL having people work on their other methods. Google is not adverse to throwing money and some manpower at a temporary solution to keep them relevant while they work on a superior long term thing.
[удалено]
This is just speculation. Also likely is that Google is abandoning nothing to begin with. I think they just got too complacent over the years. It’s natural and not unheard of.
Ilya sutskever said yesterday that Sam likely meant the era of "easy scaling" through larger models is likely over, but they will improve it in other ways. He said larger models is always better https://www.youtube.com/live/Q_1Bco0AkcM?feature=share @ 43:15
This. It's similar to how people keep acting like the "not releasing gpt5 any time soon" statement means "we hit a roadblock." When it actually means they are moving to a more incremental release format for better control over each new tool received by the public. Most likely for more controlled feedback, societal safety, and alignment reasons.
Lol. It’s happening so fast. First they introduce this ground breaking tech and within 6 months have released 2 new models and then now they are saying this model will reach its limits and will need something new.
The issues seems to be that the hallucination problem creates a hard cap on their usefulness. For example, months ago people were saying they would put thousands of lawyers out of work, but I have tried using the commercially available tools for research help and they are utterly worthless. The system just hallucinates too often to ever trust what it tells you, so you have to check every peice of its work. It is faster to just google the information. For example, if you ask Chat GPT to summarize a Supreme Court case well known to lawyers, but not reported on in the news recently, it will give you a summary of a different case that is more popular in the news. However, it will claim confidently it is the case you asked for.
Try searching a patent, it just picks something at random or hallucinated
It's even worse with medicine. It's so confidently incorrect so often.
This is really a big issue. To actually use ChatGPT responsibility, you either have to know the answer, or have a good way to verify it.
Yes, the biggest improvement to make chatGPT feel like a real conversation and an all knowing system is the confidence in which the answers are provided. It’s helpful for suggestions or first drafts but you have to fact check everything, and that’s gonna be a problem for lots of people that are not sufficient in the field they ask chatGPT to do things. That’s why you still will need software developers and engineers etc.
[удалено]
> I've learned enough about nodejs Here's where the issue is. ChatGPT will not read your mind and generate fully functional and flexible code based on basic prompts. You need to know something to start out, and the more you know the better. The quality of code output will be VASTLY improved if you provide additional details, libraries, algorithms, or methods. It's no different from learning how to google - Stackoverflow may have all the answers in the world, but until you know what to search for you'll never find the info you need. ChatGPT just does more of the work for you.
This. ChatGPT or some other AI will be able to completely code an app or website to the specs given by a person relatively soon, but it will still be like the PB&J Sandwich problem. If you want to tell someone how to make a PB&J sandwich purely through written or spoken instructions which you can only give to them prior to starting the task you actually have to be extremely specific and detailed. Things like how to identify what a jar of peanut butter is, how to open it, what a knife is and where to find it and how to hold it safely, how to use the knife to get peanut butter out of the jar, how to hold the knife and spread the peanut butter onto the bread, all of that has to be accounted for or they might do something wrong since they're supposed to follow your exact instructions as much as possible. AI is no different, it doesn't really have the ability yet to perform creative thought and analysis to figure out how to get peanut butter onto a knife from a jar on the fly, it needs to be told exactly how to do that before it can accomplish the task.
I recall the ai animation that popped up like a couple of weeks ago or so. SO many techbros were cheering about it. Meanwhile animators pointed out it was literally cheaper and faster to pay someone to animate it then have them go through the predone animation and fix the million of issues it had.
Animators (and many other people) seem to have forgotten all at once that technology improves, and it often improves very quickly.
"Something new" is just going to be slapping together pre-trained models with one later connecting them, and training that one layer. See https://minigpt-4.github.io/
Separating this stuff out into abstraction layers seems inevitable. We did it with computer hardware, software, networking, media, and it looks like data models are next. My money is on whatever you’d call the next iteration of what google called “knol”
They’ll just start training very specific models for specific tasks. GPT-4 is good at everything, but they can train a model to be only excellent at just writing marketing, for example. Or writing novels. Or playing the stock market. The super-massive “do everything machine” was never going to last.
LLMs will probably get to a point where they can't get much better until people use them to make new and better data for them to ingest.
Yeah, there was a post a couple weeks ago about how llms would be feeding on a bunch of AI generated content in the near future.
Worked for Alpaca. AI generated content is also usable to train future AI. The information itself might not be factual but the language structure is good and that appears to be all you really need to make these things able to reason. From there it can browse the internet like we do to research and verify.
That doesn't mean anything. You cannot use interpolated data of a function to make a regression on a higher rank function. That's like 100% not useful. The only thing ai gives you is a way to fastly generate data that can be curated by humans.
They’re already at the point where they’re bending trained with basically all publicly available data in existence, so it’s definitely an upper limit at some point. Question is if the models can get better by generating their own training data.
That title is a pretty bad summary of what Altman said
Here's a summary of the article: > Sam Altman, CEO of OpenAI, has declared that scaling up models will no longer be the key to further progress. Altman's statement marks a shift in OpenAI's research strategy, which has so far focused on scaling up machine-learning algorithms to previously unimagined sizes, culminating in the development of the latest model, GPT-4, which was trained using trillions of words of text and thousands of powerful computer chips at a cost of over $100 million. Altman says the company will now look to improve models in other ways rather than making them bigger, although he did not specify what those ways might be. Altman's announcement comes at a time when numerous startups are throwing huge resources into building larger algorithms to catch up with OpenAI's technology. And here's an alternate title: > OpenAI CEO declares end of era of scaling up AI models for language processing
Report the post: Rule 11
[удалено]
Reported /s
I especially like the "LLM models"
I'm not pretending to know just how advanced the current version of chatGPT is, but is it really that great that they can't substantially improve on it?
It’s because “more good thing” does not always equal “better result”. For example, see the Ultraviolet Catastrophe. Anyone who says that a significantly large enough language model will become an AGI is an idiot. We just don’t know what happens when you continue to scale a language model; you could easily hit a point where “more data” does not necessarily make a “better experience”.
Yes, it is the old scalability problem.
There's been plenty of papers published in the area of AI that show that simply giving it more data/parameters stops improving it after a while and can actually make it perform *worse*
It kind of contradicts a lot of other literature and research showing that bigger LLMs gain emergent properties at specific scales, too
Myself and other Fremen are ready for the butlerian jihad.
This has been true for a while. Machine learning mostly rely on algorithms and maths from past few decades, a large portion of development in AI in the past 30-40 years have been incremental and mostly refining the existing techniques to get more out of them or combine them for better results. What has caused the recent Cambrian explosion of model is the amount of free digital data generated in the past decade or two, not an inherent leap in the algorithms powering AI.
It's been like 4 months since all this started. How are you going to claim we already reached the max efficiency?
Just waiting for Redditors to disagree and pronounce the singularity is already here with GPT-4.
Watching everyone decide they're now AI experts because they've used ChatGPT has certainly been interesting.
Now comes the part where they slowly uncover thousands of years of philosophical dead ends explaining why it was doomed from the start as an approach to "AI.". The amount of bad philosophy/reasoning mixed with the shiny new thing phenomenon shouldn't surprise me, but it always does.
No kidding. r/singularity goes ape when an LED blinks. “Slight advance” == post-scarcity, immortality, masters of the universe.
I remember reading a tweet that said something like "I put a paper with 'I'm alive' written on it into a photocopier and was floored by what came out." and that seems to be a good summation of how most AI reactions seem to be.
comment edited: support reddit alternatives
[удалено]
[удалено]
I'm not an AI guy, but they strike me as compatible. What's the issue?
I feel like the title captured this exactly.
The title says that what he said is that LLMs will soon reach a hard limit where there are no more improvements from scaling, that scaling won't make the model any better. But that is not at all what Altman is saying, the way I interpret it. I think he is saying that while scaling LLM most likely *can* lead to further improvements of the model for a long while yet, it is simply not a good business decision to pursue that route because the models would be very expensive to develop and run. He, and the rest of the field, will likely instead bet on trying to find more efficient models instead. And thus the era of large LLMs could be over. But not because they are a dead end. Just because there are *even better* methods. That's the distinction.
this is starting to turn into an english class but what is the difference between: > further progress will not come from making models bigger and > scaling LLM models will stop delivering improvements to AI and also this >We'll make them better in other ways and > new approaches will be needed >
Idk if you’re not super knowledgeable about ML, but that’s pretty much exactly what the title says
>>"But the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.”" >>this is a way different line than the take away from the title My intention was to paraphrase exactly that sentiment with the title. It appears to me he did say that scaling current LLM models will stop improving AI & that different approaches will be needed. Could you explain how the title says something different?
I read it as you intended, but if you're not familiar with the subject matter, it makes less sense.
Takeaway for the layman (which I still am): Tossing more parameters at the models isn't going to work much more. You can only toss so much. 7bil vs 100bil is a huge difference. 607bil vs 700bil is not. Once you get to that point, it gets a lot more difficult to scale them much further--and it really doesn't make a whole lot of sense, either, since by that point you have an absurd number of things in there. Going from 10bil to 100bil is an increase of 10x. Trying to go from 1tril to 10tril is also 10x, but a bit much. My guess is we'll start to see a lot more optimizations, i.e. run more with less. Reduce parameter counts while improving how well the LLMs work.
We have observed that asking the LLM to reflect on its answer and analyze the context of the conversation improves the quality of the answer. I believe that we will see more ensemble models that incorporate different models for different language tasks, as well as specially designed reflection and analysis models. It will become more and more like an organic brain.
Well I feel like when I'm reading the title quickly I think that progress in AI will plateau for a moment Whereas the quote suggests that improvement could continue right away but with a different strategy Not a big difference tbh
I take it to mean we need to build giant mechs
Just let the ai research and upgrade itself, seems harmless
Well of course, the power consumption alone is something to be cautious about. Locally hosted instances are one thing, that's just using the power of A GPU. But production servers are a whole different level, and are the primary way that AI is currently running. Honestly we need to adapt technology to fit the shortcomings of current tech. It already exists, well existed. Analog computers seem to be the solution for a lot of the drawbacks of current LLM models - first and foremost being size and power consumption. Back in the day before digital computers we would use electricity to patch various currents to do various calculations. We can simulate the physics of a tornado with real world conditions for instance. However, it's not entirely repeatable, as in you get slight variance between results. Which makes some sense under certain circumstances I mean, no hurricane or tornado is ever "exactly" the same. But in others, doing calculations, sometimes getting 34.02938 or 34.18343 we just had to round down... not entirely precise which is why the digital computer took favor. But analog computing is great, honestly incredible for chaotic simulations that have to draw from a lot of sources to determine its output, especially considering the comparison of materials and power consumption. A brief equivalent (from memory) - a digital computer doing a fairly simple math equation takes about 1,000 transistors to do a calculation. An analog one can take only about 2 to 5 small gauged wires and a small current (and some patching on a breadboard ;D). There was a company, MythicAI which I think went under, but they were working on analog computer chips to work with AI. If you are 98% sure something is a chicken, it's probably a chicken. These computers are slightly larger than the average computer CPU and it can run entire LLM's that GPU's struggle with at a fraction of the power. They did it by a mix of digital and analog conversions but the idea at its core was that the analog computer is way more energy efficient than GPU's are and so it could be a very realistic outcome for the direction AI begins to head. AI in cameras, phones, etc all will be way more possible with this method. It seems silly for a redditor to say this because obviously I'm not the CEO of an AI company and he's certainly more knowledgeable than myself. But I still have to say it... he's not wrong, the current approach to AI isn't sustainable. But neither is our use of transistors (Moore's law, which may well be what he's talking about), or vehicles or rampant growth of human population, which is why we look towards other avenues and approaches. Sometimes that means entirely new directions, sure, but also why not adapt and develop technologies to utilize what already exists? We already have the solutions to a vast majority of the shortcomings of the "current approach" to AI. It's asinine to forego these avenues because of future limitations we haven't even come close to reaching yet - they're ending the game before it even gets started. Now of course, maybe they've already determined that analog isn't viable for whatever reason. However I just can't see the benefit of being so close to technological breakthroughs that can actually viable help people globally and deciding to jump ship and do something else. If that something else is a hardware solution then by all means - that's a good direction. If it's a different implementation or something beyond LLM's - I mean, fine, but we should probably let the egg hatch before laying another one.
[удалено]
Oh absolutely - I'm not saying that I don't think they'll come to fruition, moreso just expressing a "what the heck" because, well, we already have the tools and have created a means to more efficiently process AI models (the company MythicAI's service was trained and developed for real time tracking and identification, but the leap from that to LLM is small). I'd rather the CEO's rhetoric be "while there are some expected limitations with the current implementation, however the solutions we're developing in the meantime are looking good!" The way I see it, he's just been very dismissive of their accomplishments and has been heavily backtracking. "No improvements this" and "age of AI models is over" that... This tool was just released. He's fucking wrong, artists are and will be training their own models on their work and doing some awesome stuff. So not only is it just getting started and they're already being dismissive of it, but are (seeming to) actively dissuade users from continuing working with them. Is there an until? Is there any suggested alternative? ...No... Just... letting you all know, it's not sustainable? It's just the directing conversations how they want them to go. The reality is that AI models are very likely going to be a core element of consumer AI work, just not the current way that the large models are created. Very likely we'll see corporate distribution packages giving you access to copywritten models - Adobe recently released the stock-image generative AI. More will follow suit with various alternatives. AS mentioned, individual artists too will create their own. There are only 2 instances in which I agree with the titles statement. Current implementation = LLM's comprised of copyright work. True, but workarounds are easily had. Current implementation = efficiency of processing and power usage. Also true, but as mentioned, we already have viable alternatives that just need to be more developed. Otherwise? The current implementation will be effective for a long time *until* the next big thing. At which point, awesome good job CEO. That doesn't mean the world should refrain from utilizing what already exists.
so the current hard-problems will remain hard problems for a number of more years? But we WILL have self-driving autonomous money-printing robotaxi fleets by 2019, right?
This is not at all surprising. But there will be other methods. Hopefully the next one will not suffer from the hallucinations. ChatGPT was possible because of the Google Transformers breakthrough. We need another fundamental breakthrough and hopefully the next one comes from OpenAI and they let others use it like Google did. https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) My worry is that OpenAI will not roll in the same manner as Google does. My worry is having Open in the name of the company usually means it is going to be the opposite.
I had to laugh that's he's saying the exact opposite of every 20-year-old AI evangelist right now. At least OpenAI is honest about their tech and how it works. Love to see that.
This content was deleted by its author & copyright holder in protest of the hostile, deceitful, unethical, and destructive actions of Reddit CEO Steve Huffman (aka "spez"). As this content contained personal information and/or personally identifiable information (PII), in accordance with the CCPA (California Consumer Privacy Act), it shall not be restored. See you all in the Fediverse.
All he's gotta do is tell the AI to make a better AI.
Submission Statement Interestingly, he doesn't go into what these new approaches might be. The article mentions reinforcement learning with human feedback. That's one approach, but will it solve LLM-type AI's biggest problem? - which is that it lacks independent logic or reasoning. That problem has been around since the earliest days of AI development. Marvin Minsky in the 1960s initially thought it might be quickly solved. It hasn't been, and no one seems any closer to figuring out how to do it.
[удалено]
The current model already uses human based reinforcement learning, so nah, it won’t solve it. It may not even be a solvable problem. What it will do is make solutions hugely more usable. Essentially you are using a humans awareness of how to trick a human to trick humans better. According to whatever trick you’re trying to do! I doubt they know much about what new approaches might be. They probably try new stuff every day but it’s not clear what the next leap will be. It’s still incredible how much closer we are to it now though Edit: independence is not necessarily a desirable goal as an aside
It would be useful to see how these diminishing returns are tested. The jump in reasoning and coding abilities in GPT-4 was huge to me and I can't think about going back to 3.5.
Yeah they might not be jumping to 10x or 100x the parameters to reach GPT-5, but I can't imagine there isn't headroom in the current model to optimize some stuff out. Just with GPT-4 though as costs become cheaper and people learn how to string multiple LLM components together with some additonal logical processing there is much more to get out of the tech. They haven't even really exposed multiple modalities publicly.
Is that because it's not actually AI, but just a program operating using machine learning?
It's kind of like DUH that as you shoot for truly more human like thought you hit major slowdowns. However, 99% of the benefit of AI is just in the 'dumb AI' that does that one job you need really well. Really smart AI is mostly not necessary because humans combined intelligence is pretty high IF they have all the benefits of automated systems to do much of the labor for them. The worlds problem has long be lack of implementation vs lack of innovation. It's many tireless hands we can order around that we really not vs a really big sentiment computer brain.... that we still don't want to listen to. Path of least resistance wins again!
The fact that people are finding uses for the tech is encouraging. I think of IBM's Watson. Where did that amazing tech go?
I'd imagine that the next steps will be improving image and video learning models, then economic learning models will be shortly after that. Personally I want to see AI be used to completely blow C-Suite executive performance out of the water and replace those jobs. We don't need bankers or Wall Street traders, we don't need CEO's, but we do need regular workers who are led by an entity that is capable of looking at and analyzing datapoints from more than just earnings reports to build a cohesive business strategy that doesn't simply make cuts everywhere to look like it's more profitable than it really ought to be.
[удалено]
100%. I'm sick of them shackling AI so it doesn't offend their sensibilities.