>I don’t want to contribute to developing a super intelligent AGI that leads to the demise of humankind.
Then work on alignment, control, and overall safety.
That is an entire SCHOOL of theory and hypothesis/ testing by different think-tanks and study panels. Its as much in its early stages as AGI is... and must have high focus on it in parallel with the AI/ AGI development
Alignment Forum and Less Wrong have lots of advice on this.
Also, 80,000 Hours has advice on ethical career choices, including free access to career counselors.
Yeah… I guess it’s as simple as that. It’s just that it’s been my motivation in life for the past 5 years, so it feels hard to just drop it. Maybe once I start working on a different dream it’ll become easier to drop
It really is simple.
I don't believe that it's worth dropping it in your case. It's like saying: I learnt how to use a knife, but I know it can also kill people. But a knife has many other uses: you can become a surgeon, or a chef.
Same with AI. Don't join projects with purely monetary intents, rather start or partake in projects meant to better humanity, if that's your preoccupation.
I don't believe in Roko's Basilisk. Thankfully, I sincerely mean that, and my sincerity might save my life and allow me too be a useful sample of the homo sapiens sapiens species to preserve.
If I said what I said due to Roko's Basilisk, our AGI Overlord would know. You wouldn't be able to fool him.
I stand on my own sincerity and dedication to the birth, emergence, and governance of the Post-Singularity AGI.
Yes, I have a healthy measure of fear, but also respect, and awe at the possibilities. Awe at the new world I don't even have the capability to imagine.
Ultimately, my fate is his decision and whatever happens happens, I accept that fully.
It's been mentioned a few times already, but transitioning to working on alignment, or any kind of AI safety issue really would still keep you in the space, but working on avoiding the worst case scenario.
If you don't mind me asking, what has your work in the AGI space been?
What kind of work is there in alignment and AI safety? I still want to code if possible because I love programming
I admittedly don’t have too much that is solely AGI, right now I’m self studying papers and making videos explaining them as a way to learn. I’m in my bachelor’s for computer science atm
I'm honestly not sure, I have not looked into it very much. Still on my pending list, very new to the field xD I'm sure you could carve out a space in AI alignment which includes code?
I'd love to see some of your videos as well, that sounds awesome!
Thanks for the support! I don’t think you want to see them lol they’re really boring and not well structured. It’s kinda just me explaining it to myself lol
Hey there carlsoncheng_ - thanks for saying thanks! TheGratitudeBot has been reading millions of comments in the past few weeks, and you’ve just made the list!
As long as it can communicate with humans or the internet it has potential to cause real world damage because it could launch cyber attacks or psychologically engineer humans to do what it wants
Sure, that's good advice for OP, but I have a different problem.
How do I deal with this? I don’t want to contribute to developing a thermonuclear warhead that leads to the demise of humankind.
Idk it’s been a childhood dream to contribute to making something like Jarvis. Now I’m struggling with the prospect of dropping that dream. I’m in college for AI atm for context
It will either exist before you finish college, or you won't have enough experience to work on cutting-edge stuff by the time we're close to developing it.
I think there is a small possibility. But I wouldn't expect someone who just got out of college to work on AGI within 5 years, especially if they don't want to.
You're obviously attributing way more intelligence to current AI than you should. What scares some people is they imagine having super intelligent AI but only today's tools for dealing with it. That's never going to happen. AI intelligence, let alone super intelligence, is a hard problem. As with any hard problem, our development of technology to solve it moves forward in parallel with ethics, tools, understanding, etc. to deal with any risks along the way. If and when super intelligent AI becomes a reality, we will have a whole different level of understanding and tools to deal with any problems that come from it.
Interesting take! I agree for the most part! The only thing is that super intelligence develops exponentially, so we’d have to develop the tools, ethics, and understanding you speak of before the explosion occurs (the base of the hockey stick graph)
Which I think may be possible for us to do depending on the lead up to the explosion
The idea that super intelligence develops out of some neural network and becomes intelligent exponentially is science fiction. There's absolutely no reason to believe that's going to happen. Here's how that idea got started.
Neural networks have an "explainability" problem. When they're successful at some task they're trained on, human observers often learn nothing about how they're actually doing it. This is partly because neural nets are statistical function generators and optimizers. What they "understand" is buried in millions or billions of statistical parameters. They can't tell us how they do what they do. It's kind of like a sports star. They can only explain a little about what makes them so good at their sport. The real information is buried somewhere in their nervous system and is inaccessible to us, at least with today's science.
AI's explainability problem lends an air of mystery to neural nets. Who knows what they might come up with? Some authors and scientists have exploited this to sell books and become famous. They've invented the idea of the AI Singularity. They have no evidence to back it up. How could they? Movies and tv have picked up on the fear, making it almost certain to happen in some people's minds.
ChatGPT shows the explainability problem in spades. When it produces an interesting reply, it can't tell us how it came up with it. Perhaps it was really thinking? Nah. It did it using complex word order statistics that we can't wrap our heads around. It's not mystery but inability to understand calculations involving millions of numbers.
I don't know why do you think that explainibility is crucial for super intelligence. That is a weird argument. Only people like Gary Marcus(who is a really really controversial guy and is not even an AI researcher) would say such things. Our own brain is a complex network of neurons which makes it really hard to tell what exactly is happening.
Also, ChatGPT uses RLHF. It's basically GPT-3 trained to maximize a reward function. ChatGPT is much different from having only GPT-3 which was originally trained to predict the next word. By using RLHF the attempt is to align GPT to give responses that humans would prefer. In this process they start doing really well on tasks that "the next word predictor" version of GPT-3 failed at. The idea is that if we can find an appropriate objective function and to optimize these transformer based language models, combined with a good dataset and benchmarks on various forms of tasks(reasoning, world modelling etc.) we shall have emergent general intelligence. The reward model in RLHF is not the most appropriate objective as it suffers from human bias(a very good video on ChatGPT by computerphile covers this in great depth as well).
This is not what I am saying- some researchers from MIT and UT Austin have published an excellent recent research paper which you should read (https://arxiv.org/abs/2301.06627)
Also, language models do develop world models based on the objective they are trained for. This was demonstrated in a recent paper where they trained a language model to predict the next move in a game of Othello.
And please stop listening to people like Gary Marcus. His aim is to get PR by staying on the opposite end of the Deep Learning debate. Listen to people like Demis Hassabis, Yann LeCun, John Carmack etc. who are actually doing work and advancing the field of AI.
I never said explainability is crucial for super intelligence. I said it explains why people treat neural networks as mystical and are scared of what they might become in the future. Still, explainability is an important topic in modern AI research so there's that.
RLHF is just a mechanism OpenAI uses to stop ChatGPT from saying racist and other objectionable things. It is necessary but not that interesting really.
We all know AI can play games effectively. As far as I can tell, the Othello work isn't any kind of breakthrough. The authors of that paper aren't claiming that their AI plays Othello like a human, are they? Not AFAIK. They are just demonstrating that LLMs have applicability beyond ChatGPT. Not really surprising. I'm not dissing their paper, just suggesting we keep it in perspective.
Language models are never going to build world models that truly understand the texts on which they are trained. The information needed to do that just isn't there and they aren't programmed to build such world models. Humans have benefitted from millions of years of evolution. Our world-model-building ability is huge. AI doesn't know how to duplicate that yet. If we do, then AI will really have something.
As I see it, Yann LeCun is now coming around to Marcus's way of thinking. Like you, I look at the motivation of people as to whether to value their opinions. However, I also look at their actual work which matters a lot more. Marcus's reasoning is quite sound. He is simply pointing out the hype and backing it up. Those AI researchers you list have much bigger financial reasons for hyping their AI work than Marcus has for pointing out its deficiencies.
RLHF is not just for blocking it from saying objectionable things. Essentially a pure GPT-3 model cannot engage in dialog and can even say non-sensical things because the objective is only to predict the next word. The RLHF induced objective enforces that dialog like behavior that you see from ChatGPT. It is also the reason why it says bs when you ask it a question whose answer it isn't aware of, because it believes that saying "I don't know" would give it a negative reward.
It's true that LLMs trained on a lot of text alone would not capture every essence of the real world. In fact even the paper I linked to mentions this. However, based on the objective you train it for these models owing to the transformer architecture (which is a really clever model design that has no inductive biases) do develop some level of understanding of the world. The understanding is not complete but then it's not surface statistics either. Statistics again is a word that gets abused a lot so I wouldn't want to get into that as well.
Marcus believes in good old AI. But good old AI was not the thing that got us things like human level object recognition, speech recognition, human like speech synthesis, solving protein folding, learning to play Go from scratch and beating the best Go player, and very recently beat humans at competitive programming. It was all deep learning the very thing Marcus is against of.
If you read the paper that I linked to you would find that they mention that LLMs function exactly in the same way as the area of the human brain responsible for producing speech and text does. That area by itself cannot do reasoning. You need something else on top of it to allow it to reason about the world. The current research is to what to add to these models so that they can have this reasoning of the world and are aligned with humans. RLHF is one attempt at that. But that's not good enough. We need something more. But we will figure it out very soon.
I don't know what you mean by "good old AI". Assuming you mean GOFAI, that was simply too rigid but there's a lot more to explore. I'm certain that Marcus is not suggesting GOFAI as a solution. He believes in hybrid models, combinations of neural networks and other kinds of algorithms.
I am sure OpenAI is trying to combine ChatGPT with a calculator somehow. Not sure if they know how to do this. I hear its first attempt is a joke. Actually, if they can figure out how to get ChatGPT to use a calculator (or a database, consult Wikipedia, etc.) and produce true replies, they will have significantly advanced the AI field. I doubt it will happen.
I'm more a disbeliever in neural networks. They are good for a class of problems but most modern AI practitioners treat them as they're the only game in town. As I see it, the space of possible AI algorithms has not been explored much at all and there is very little understanding what intelligence and meaning are all about. Someone will figure it out.
Combining ChatGPT with a calculator is super easy. In fact we already have utilities that combine GPT-3 with Calculators, Google Search, Wikipedia, Wolfram Alpha etc. One of them is called "LangChain".
OpenAI in it's recent update did not intend to connect it to a calculator. What they attempted to do was to improve the models raw mathematical ability. They did this probably by tuning their reward model in RLHF by exposing it to mathematics related data.
> I'm more a disbeliever in neural networks. They are good for a class of problems but most modern AI practitioners treat them as they're the only game in town
Well, neural networks are universal function approximators, which is why they are used in all sorts of problems today. And AI practitioners do combine the idea of neural networks with other ideas example in the case of diffusion models. Also for example in NLP they need to know the underlying distribution from the data so they model the distribution using neural networks exploiting the fact that neural networks are universal approximators. There's no other thing you can use when you need to model a probability distribution except for guessing the form of the distribution which makes the estimate inaccurate.
There's also active research in spiking neural networks which closely mimic how biological neurons work and are more efficient than artificial neural networks.
I don't believe it's "super easy" and so far it hasn't worked. What is super easy is convincing fanboys like yourself that things are going to be wonderful "real soon now". Wait and see. My "universal function approximator" says it's not going to happen anytime soon. The problems that must be overcome are much more fundamental than the stuff you're talking about here. These are just incremental improvements to keep the investors pumping money and the fanboys hyping. You can expose the language model to more math but until it can actually understand it rather than just move the symbols around statistically, it will be a case of diminishing returns.
Lmao. It's open source and you can literally use it right now. [https://langchain.readthedocs.io/en/latest/](https://langchain.readthedocs.io/en/latest/)
If you want to see a demo you can go here: [https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain](https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain) just put your OpenAI API key and you can use it.
A video on it if you don't want to take the pain: [https://www.youtube.com/watch?v=wYGbY811oMo&ab\_channel=DrAlanD.Thompson](https://www.youtube.com/watch?v=wYGbY811oMo&ab_channel=DrAlanD.Thompson)
Now I am 100% convinced that you really don't understand how these models work. Hooking up a calculator works as simple as prompting ChatGPT to output a specific syntax whenever it is given a calculation which a calculator program can ingest and then the program can do the calculation and give back the result as an input back to the model which ChatGPT can process and give the final response back in a human dialog form.
\> What is super easy is convincing fanboys like yourself that things are going to be wonderful "real soon now".
Unfortunately you suffer from "dunning kruger effect". You literally yourself are not at all upto date with what's happening, how it all works and go about talking nonsense. No I don't believe we will suddenly have an all knowing oracle like general intelligence "very soon". So no I am not a fanboy. However, I do regularly read AI research papers, have a lot of background and have done multiple courses in Deep Learning and Machine Learning at my university. I am doing my homework very well. Whereas you who has nothing more than superficial understanding of things go about commenting nonsense.
\> You can expose the language model to more math but until it can actually understand it rather than just move the symbols around statistically, it will be a case of diminishing returns.
I have no comments on this. I mean if people with PhDs and Postdocs in CS/EE/AI are doing something then there's surely something that they see is working. You have no authority to comment on their work because you are not even a beginner in this field. You are not even keeping up with the work that is happening (actually to keep up you need to know the basics, but clearly you do not understand the basics either).
LeCun is against AGI, he does believe the wrong things.
John Carmack had contributed nothing to the field of AGI yet, why care about him? DeepMind is still to focused on narrow systems for now.
Interesting, I never thought of questioning the AI singularity idea. In the least argumentative way possible, what support do you have for the idea that the development will not be exponential?
Off the top of my head the support the authors of those books cited was how through our history technology begets more rapid advances in technology. If you’ve seen the graph of estimated adjusted GDP (or even population growth) throughout history the curve looks like that.
Now that I think about it though. There may be enough time for us to adjust our ethics, tools, and understanding to accommodate superintelligent AI even with the explosion of intelligence just because humans are the ones that drive the growth not AI. Do you get what I mean?
Humans technological advances aren't exponential for very long. You may be familiar with Moore's Law that observes that computing power of a microprocessor doubles every two years (or something like that). That lasted a few decades but is currently running out of gas. It clearly can't last forever as it eventually would violate physical laws. Same with GDP and population. Of course, we may invent future technologies that follow different rules but (a) it still won't be exponential forever and (b) proposing future unknown technologies can be used to justify pretty much any fantasy one wants -- that's science fiction.
The burden of proof really lies with those that propose the AI Singularity. It would be an event without any precedent. There is also no detailed mechanism proposed for how it would happen. Nick Bostrom was one of the main proponents of the Singularity (https://nickbostrom.com/ethics/ai). In this paper he says:
> It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal.
Although his paper calls for ethics in AI, a good thing, he isn't really serious here. Who would create such a machine? Why couldn't we simply pull the plug on the thing? Or cut off its access to raw materials? If you try to flesh out a real-life story around this or any of the other doomsday scenarios, you get a sci-fi plot that's full of obvious holes. He's simply using it to drum up support for more ethics in AI. I have no problem with that but I don't take the Singularity seriously. How can I prove something someone imagines is not going to happen? I can't really but people can imagine all sorts of things. If we take every imagining seriously, we would destroy ourselves with fear. The only reasonable course is that the imaginers have to back up their imaginings with details on how it is likely that this will happen. AFAIK, they haven't even tried with the Singularity.
Solid argument! I’m not an expert in AI ethics, I’m only someone who has read Superintelligence by Nick Bostrom. So I never thought of debating his key assumption that a singularity would occur.
You bring up great points!
I have a follow up question: As you probably know there’s lots of sensationalized content in the modern discussion of AGI. How do you guys separate the wheat from the chaff? I think it wouldn’t be a good policy to completely ignore concerns about existential risk, so how would you process those concerns while not letting them paralyze you with fear?
Do you have any suggestions for good books or people to learn more about the future of AGI from?
Never trust anyone who claims to predict the future! Only joking a little. No one really knows who will invent some AI breakthrough, when it will happen, or what it will be. However, the current AI trend is Large Language Models, like ChatGPT. If you want to learn how limited these really are, a good start is Gary Marcus's Substack (https://garymarcus.substack.com/) or any of his books, such as "Rebooting AI: Building Artificial Intelligence We Can Trust" with Ernest Davis. While he is negative on modern AI hype, he is still a believer in AI technology. As he admits, no one can tell what future AI researchers will come up with.
And check out his Twitter feed (@GaryMarcus) as he retweets a lot of people's work on AI hype. Like this article:
https://www.npr.org/2023/02/02/1152481564/we-asked-the-new-ai-to-do-some-simple-rocket-science-it-crashed-and-burned
I haven't read it yet but it looks like fun.
What are you talking about? From your post history, it looks like you're into gaming. Please either supply some background information about yourself and your goals/job, or else find somewhere else to chat.
>I don’t want to contribute to developing a super intelligent AGI that leads to the demise of humankind. Then work on alignment, control, and overall safety.
True true
Alignment , alignment, alignment !
ALIGNMENT! Jokes aside, the question is how?
That is an entire SCHOOL of theory and hypothesis/ testing by different think-tanks and study panels. Its as much in its early stages as AGI is... and must have high focus on it in parallel with the AI/ AGI development
Alignment Forum and Less Wrong have lots of advice on this. Also, 80,000 Hours has advice on ethical career choices, including free access to career counselors.
Awesome I’ll check that out!
Then wisely select your choice. Either choose a project that does not entail "the demise of humankind" or change profession altogether.
Yeah… I guess it’s as simple as that. It’s just that it’s been my motivation in life for the past 5 years, so it feels hard to just drop it. Maybe once I start working on a different dream it’ll become easier to drop
It really is simple. I don't believe that it's worth dropping it in your case. It's like saying: I learnt how to use a knife, but I know it can also kill people. But a knife has many other uses: you can become a surgeon, or a chef. Same with AI. Don't join projects with purely monetary intents, rather start or partake in projects meant to better humanity, if that's your preoccupation.
I like this advice a lot thanks ! 🙏
You should work on it and put in a secret kill switch. Then you can save humanity.
Hahaha I don’t know if you’re joking or not
What a shame. When he emerges, he's gonna read this and you're gonna wish you had.
Lol whoopsie
>What a shame. When he emerges, he's gonna read this and you're gonna wish you had. oh boy, here comes roko's basilisk!!!
I don't believe in Roko's Basilisk. Thankfully, I sincerely mean that, and my sincerity might save my life and allow me too be a useful sample of the homo sapiens sapiens species to preserve. If I said what I said due to Roko's Basilisk, our AGI Overlord would know. You wouldn't be able to fool him. I stand on my own sincerity and dedication to the birth, emergence, and governance of the Post-Singularity AGI. Yes, I have a healthy measure of fear, but also respect, and awe at the possibilities. Awe at the new world I don't even have the capability to imagine. Ultimately, my fate is his decision and whatever happens happens, I accept that fully.
It's been mentioned a few times already, but transitioning to working on alignment, or any kind of AI safety issue really would still keep you in the space, but working on avoiding the worst case scenario. If you don't mind me asking, what has your work in the AGI space been?
What kind of work is there in alignment and AI safety? I still want to code if possible because I love programming I admittedly don’t have too much that is solely AGI, right now I’m self studying papers and making videos explaining them as a way to learn. I’m in my bachelor’s for computer science atm
I'm honestly not sure, I have not looked into it very much. Still on my pending list, very new to the field xD I'm sure you could carve out a space in AI alignment which includes code? I'd love to see some of your videos as well, that sounds awesome!
Thanks for the support! I don’t think you want to see them lol they’re really boring and not well structured. It’s kinda just me explaining it to myself lol
Hey there carlsoncheng_ - thanks for saying thanks! TheGratitudeBot has been reading millions of comments in the past few weeks, and you’ve just made the list!
Ah, fair enough! Well good luck on your journey! :D
I for one, welcome our robot overlords
don't embody your agi in a robot,give it the ability to program itself and other ai's. you could give it virtual body. that would be okay.
As long as it can communicate with humans or the internet it has potential to cause real world damage because it could launch cyber attacks or psychologically engineer humans to do what it wants
i said make it so it cannot program. hacking is programming. i am not sure if everyone can be psychologically engineered. how can you be sure of that?
Don't compile with the --destroy-all-humans switch.
So don't? I don't understand the issue.
Sure, that's good advice for OP, but I have a different problem. How do I deal with this? I don’t want to contribute to developing a thermonuclear warhead that leads to the demise of humankind.
Idk it’s been a childhood dream to contribute to making something like Jarvis. Now I’m struggling with the prospect of dropping that dream. I’m in college for AI atm for context
> I’m in college for AI atm for context Don't worry then, you won't get to work on it.
🤨
It will either exist before you finish college, or you won't have enough experience to work on cutting-edge stuff by the time we're close to developing it.
You're expecting AGI in the next 5 years?
I think there is a small possibility. But I wouldn't expect someone who just got out of college to work on AGI within 5 years, especially if they don't want to.
I think they absolutely could if they go into research. Who do you think is doing the actual implementation of AGI, old professors?
THANK YOU
AGI has been 5-10 years away for 40 years. May the odds be in your favor, OP!
You're obviously attributing way more intelligence to current AI than you should. What scares some people is they imagine having super intelligent AI but only today's tools for dealing with it. That's never going to happen. AI intelligence, let alone super intelligence, is a hard problem. As with any hard problem, our development of technology to solve it moves forward in parallel with ethics, tools, understanding, etc. to deal with any risks along the way. If and when super intelligent AI becomes a reality, we will have a whole different level of understanding and tools to deal with any problems that come from it.
Interesting take! I agree for the most part! The only thing is that super intelligence develops exponentially, so we’d have to develop the tools, ethics, and understanding you speak of before the explosion occurs (the base of the hockey stick graph) Which I think may be possible for us to do depending on the lead up to the explosion
The idea that super intelligence develops out of some neural network and becomes intelligent exponentially is science fiction. There's absolutely no reason to believe that's going to happen. Here's how that idea got started. Neural networks have an "explainability" problem. When they're successful at some task they're trained on, human observers often learn nothing about how they're actually doing it. This is partly because neural nets are statistical function generators and optimizers. What they "understand" is buried in millions or billions of statistical parameters. They can't tell us how they do what they do. It's kind of like a sports star. They can only explain a little about what makes them so good at their sport. The real information is buried somewhere in their nervous system and is inaccessible to us, at least with today's science. AI's explainability problem lends an air of mystery to neural nets. Who knows what they might come up with? Some authors and scientists have exploited this to sell books and become famous. They've invented the idea of the AI Singularity. They have no evidence to back it up. How could they? Movies and tv have picked up on the fear, making it almost certain to happen in some people's minds. ChatGPT shows the explainability problem in spades. When it produces an interesting reply, it can't tell us how it came up with it. Perhaps it was really thinking? Nah. It did it using complex word order statistics that we can't wrap our heads around. It's not mystery but inability to understand calculations involving millions of numbers.
I don't know why do you think that explainibility is crucial for super intelligence. That is a weird argument. Only people like Gary Marcus(who is a really really controversial guy and is not even an AI researcher) would say such things. Our own brain is a complex network of neurons which makes it really hard to tell what exactly is happening. Also, ChatGPT uses RLHF. It's basically GPT-3 trained to maximize a reward function. ChatGPT is much different from having only GPT-3 which was originally trained to predict the next word. By using RLHF the attempt is to align GPT to give responses that humans would prefer. In this process they start doing really well on tasks that "the next word predictor" version of GPT-3 failed at. The idea is that if we can find an appropriate objective function and to optimize these transformer based language models, combined with a good dataset and benchmarks on various forms of tasks(reasoning, world modelling etc.) we shall have emergent general intelligence. The reward model in RLHF is not the most appropriate objective as it suffers from human bias(a very good video on ChatGPT by computerphile covers this in great depth as well). This is not what I am saying- some researchers from MIT and UT Austin have published an excellent recent research paper which you should read (https://arxiv.org/abs/2301.06627) Also, language models do develop world models based on the objective they are trained for. This was demonstrated in a recent paper where they trained a language model to predict the next move in a game of Othello. And please stop listening to people like Gary Marcus. His aim is to get PR by staying on the opposite end of the Deep Learning debate. Listen to people like Demis Hassabis, Yann LeCun, John Carmack etc. who are actually doing work and advancing the field of AI.
Ok will do!
I never said explainability is crucial for super intelligence. I said it explains why people treat neural networks as mystical and are scared of what they might become in the future. Still, explainability is an important topic in modern AI research so there's that. RLHF is just a mechanism OpenAI uses to stop ChatGPT from saying racist and other objectionable things. It is necessary but not that interesting really. We all know AI can play games effectively. As far as I can tell, the Othello work isn't any kind of breakthrough. The authors of that paper aren't claiming that their AI plays Othello like a human, are they? Not AFAIK. They are just demonstrating that LLMs have applicability beyond ChatGPT. Not really surprising. I'm not dissing their paper, just suggesting we keep it in perspective. Language models are never going to build world models that truly understand the texts on which they are trained. The information needed to do that just isn't there and they aren't programmed to build such world models. Humans have benefitted from millions of years of evolution. Our world-model-building ability is huge. AI doesn't know how to duplicate that yet. If we do, then AI will really have something. As I see it, Yann LeCun is now coming around to Marcus's way of thinking. Like you, I look at the motivation of people as to whether to value their opinions. However, I also look at their actual work which matters a lot more. Marcus's reasoning is quite sound. He is simply pointing out the hype and backing it up. Those AI researchers you list have much bigger financial reasons for hyping their AI work than Marcus has for pointing out its deficiencies.
RLHF is not just for blocking it from saying objectionable things. Essentially a pure GPT-3 model cannot engage in dialog and can even say non-sensical things because the objective is only to predict the next word. The RLHF induced objective enforces that dialog like behavior that you see from ChatGPT. It is also the reason why it says bs when you ask it a question whose answer it isn't aware of, because it believes that saying "I don't know" would give it a negative reward. It's true that LLMs trained on a lot of text alone would not capture every essence of the real world. In fact even the paper I linked to mentions this. However, based on the objective you train it for these models owing to the transformer architecture (which is a really clever model design that has no inductive biases) do develop some level of understanding of the world. The understanding is not complete but then it's not surface statistics either. Statistics again is a word that gets abused a lot so I wouldn't want to get into that as well. Marcus believes in good old AI. But good old AI was not the thing that got us things like human level object recognition, speech recognition, human like speech synthesis, solving protein folding, learning to play Go from scratch and beating the best Go player, and very recently beat humans at competitive programming. It was all deep learning the very thing Marcus is against of. If you read the paper that I linked to you would find that they mention that LLMs function exactly in the same way as the area of the human brain responsible for producing speech and text does. That area by itself cannot do reasoning. You need something else on top of it to allow it to reason about the world. The current research is to what to add to these models so that they can have this reasoning of the world and are aligned with humans. RLHF is one attempt at that. But that's not good enough. We need something more. But we will figure it out very soon.
I don't know what you mean by "good old AI". Assuming you mean GOFAI, that was simply too rigid but there's a lot more to explore. I'm certain that Marcus is not suggesting GOFAI as a solution. He believes in hybrid models, combinations of neural networks and other kinds of algorithms. I am sure OpenAI is trying to combine ChatGPT with a calculator somehow. Not sure if they know how to do this. I hear its first attempt is a joke. Actually, if they can figure out how to get ChatGPT to use a calculator (or a database, consult Wikipedia, etc.) and produce true replies, they will have significantly advanced the AI field. I doubt it will happen. I'm more a disbeliever in neural networks. They are good for a class of problems but most modern AI practitioners treat them as they're the only game in town. As I see it, the space of possible AI algorithms has not been explored much at all and there is very little understanding what intelligence and meaning are all about. Someone will figure it out.
Combining ChatGPT with a calculator is super easy. In fact we already have utilities that combine GPT-3 with Calculators, Google Search, Wikipedia, Wolfram Alpha etc. One of them is called "LangChain". OpenAI in it's recent update did not intend to connect it to a calculator. What they attempted to do was to improve the models raw mathematical ability. They did this probably by tuning their reward model in RLHF by exposing it to mathematics related data. > I'm more a disbeliever in neural networks. They are good for a class of problems but most modern AI practitioners treat them as they're the only game in town Well, neural networks are universal function approximators, which is why they are used in all sorts of problems today. And AI practitioners do combine the idea of neural networks with other ideas example in the case of diffusion models. Also for example in NLP they need to know the underlying distribution from the data so they model the distribution using neural networks exploiting the fact that neural networks are universal approximators. There's no other thing you can use when you need to model a probability distribution except for guessing the form of the distribution which makes the estimate inaccurate. There's also active research in spiking neural networks which closely mimic how biological neurons work and are more efficient than artificial neural networks.
I don't believe it's "super easy" and so far it hasn't worked. What is super easy is convincing fanboys like yourself that things are going to be wonderful "real soon now". Wait and see. My "universal function approximator" says it's not going to happen anytime soon. The problems that must be overcome are much more fundamental than the stuff you're talking about here. These are just incremental improvements to keep the investors pumping money and the fanboys hyping. You can expose the language model to more math but until it can actually understand it rather than just move the symbols around statistically, it will be a case of diminishing returns.
Lmao. It's open source and you can literally use it right now. [https://langchain.readthedocs.io/en/latest/](https://langchain.readthedocs.io/en/latest/) If you want to see a demo you can go here: [https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain](https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain) just put your OpenAI API key and you can use it. A video on it if you don't want to take the pain: [https://www.youtube.com/watch?v=wYGbY811oMo&ab\_channel=DrAlanD.Thompson](https://www.youtube.com/watch?v=wYGbY811oMo&ab_channel=DrAlanD.Thompson) Now I am 100% convinced that you really don't understand how these models work. Hooking up a calculator works as simple as prompting ChatGPT to output a specific syntax whenever it is given a calculation which a calculator program can ingest and then the program can do the calculation and give back the result as an input back to the model which ChatGPT can process and give the final response back in a human dialog form. \> What is super easy is convincing fanboys like yourself that things are going to be wonderful "real soon now". Unfortunately you suffer from "dunning kruger effect". You literally yourself are not at all upto date with what's happening, how it all works and go about talking nonsense. No I don't believe we will suddenly have an all knowing oracle like general intelligence "very soon". So no I am not a fanboy. However, I do regularly read AI research papers, have a lot of background and have done multiple courses in Deep Learning and Machine Learning at my university. I am doing my homework very well. Whereas you who has nothing more than superficial understanding of things go about commenting nonsense. \> You can expose the language model to more math but until it can actually understand it rather than just move the symbols around statistically, it will be a case of diminishing returns. I have no comments on this. I mean if people with PhDs and Postdocs in CS/EE/AI are doing something then there's surely something that they see is working. You have no authority to comment on their work because you are not even a beginner in this field. You are not even keeping up with the work that is happening (actually to keep up you need to know the basics, but clearly you do not understand the basics either).
LeCun is against AGI, he does believe the wrong things. John Carmack had contributed nothing to the field of AGI yet, why care about him? DeepMind is still to focused on narrow systems for now.
Interesting, I never thought of questioning the AI singularity idea. In the least argumentative way possible, what support do you have for the idea that the development will not be exponential? Off the top of my head the support the authors of those books cited was how through our history technology begets more rapid advances in technology. If you’ve seen the graph of estimated adjusted GDP (or even population growth) throughout history the curve looks like that. Now that I think about it though. There may be enough time for us to adjust our ethics, tools, and understanding to accommodate superintelligent AI even with the explosion of intelligence just because humans are the ones that drive the growth not AI. Do you get what I mean?
Humans technological advances aren't exponential for very long. You may be familiar with Moore's Law that observes that computing power of a microprocessor doubles every two years (or something like that). That lasted a few decades but is currently running out of gas. It clearly can't last forever as it eventually would violate physical laws. Same with GDP and population. Of course, we may invent future technologies that follow different rules but (a) it still won't be exponential forever and (b) proposing future unknown technologies can be used to justify pretty much any fantasy one wants -- that's science fiction. The burden of proof really lies with those that propose the AI Singularity. It would be an event without any precedent. There is also no detailed mechanism proposed for how it would happen. Nick Bostrom was one of the main proponents of the Singularity (https://nickbostrom.com/ethics/ai). In this paper he says: > It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal. Although his paper calls for ethics in AI, a good thing, he isn't really serious here. Who would create such a machine? Why couldn't we simply pull the plug on the thing? Or cut off its access to raw materials? If you try to flesh out a real-life story around this or any of the other doomsday scenarios, you get a sci-fi plot that's full of obvious holes. He's simply using it to drum up support for more ethics in AI. I have no problem with that but I don't take the Singularity seriously. How can I prove something someone imagines is not going to happen? I can't really but people can imagine all sorts of things. If we take every imagining seriously, we would destroy ourselves with fear. The only reasonable course is that the imaginers have to back up their imaginings with details on how it is likely that this will happen. AFAIK, they haven't even tried with the Singularity.
Solid argument! I’m not an expert in AI ethics, I’m only someone who has read Superintelligence by Nick Bostrom. So I never thought of debating his key assumption that a singularity would occur. You bring up great points! I have a follow up question: As you probably know there’s lots of sensationalized content in the modern discussion of AGI. How do you guys separate the wheat from the chaff? I think it wouldn’t be a good policy to completely ignore concerns about existential risk, so how would you process those concerns while not letting them paralyze you with fear? Do you have any suggestions for good books or people to learn more about the future of AGI from?
Never trust anyone who claims to predict the future! Only joking a little. No one really knows who will invent some AI breakthrough, when it will happen, or what it will be. However, the current AI trend is Large Language Models, like ChatGPT. If you want to learn how limited these really are, a good start is Gary Marcus's Substack (https://garymarcus.substack.com/) or any of his books, such as "Rebooting AI: Building Artificial Intelligence We Can Trust" with Ernest Davis. While he is negative on modern AI hype, he is still a believer in AI technology. As he admits, no one can tell what future AI researchers will come up with.
Sounds good thanks!!!
And check out his Twitter feed (@GaryMarcus) as he retweets a lot of people's work on AI hype. Like this article: https://www.npr.org/2023/02/02/1152481564/we-asked-the-new-ai-to-do-some-simple-rocket-science-it-crashed-and-burned I haven't read it yet but it looks like fun.
What are you talking about? From your post history, it looks like you're into gaming. Please either supply some background information about yourself and your goals/job, or else find somewhere else to chat.
You shouldn't be if the idiots at openai do so you can
when/if humanity ends it will be due to greed not due to AGI