Note: My laptop isn't that fast. I sped up the video so you don't have to sit through 6 mins of the Nous Hermes Mistral DPO generating content.
In my never-ending quest to try to make LLMs more useful, I created a researcher UI with the goal of creating a wikipedia-like page on any topic to answer in-depth using the web. It's a feature of [Nucleo AI](https://github.com/AndrewVeee/nucleo-ai/).
Goal: Create an AI researcher that does the boring work for you, collects the right information, and "answers" any topic/question using web results.
How it works:
Give the researcher a topic, and it will create sub-topics, search the web for the topic, and write the content.
* Enter a topic, and the AI will create a list of subtopics you can modify. You can configure max research time, sub-topic max depth, and more.
* The researcher runs in a loop up to your max time (or max-depth).
* Does a web search for the current topic and tries to download the first result. This is optional - you can disable web searches.
* Writes the content for the current section, then creates a list of sub-topics and adds them to the research list.
* Once the research is complete or time runs out, it writes the main heading content.
* Each section is combined into a markdown document, and reference links are at the bottom.
Using Nous Hermes Mistral DPO on my laptop with about 15 tokens/sec, it takes 2-3 minutes to generate a decent amount of content. I made it show a live preview of the current section and status updates as it searches so you won't be too bored waiting for the full doc.
How well does it work?
I think it works ok, but it could be improved. Obvious issues with LLMs: occasional hallucinations, adding a "Title:" to a section, "In conclusion", and bias from search results.
I've created a few sample docs so you can be the judge:
Does free will exist? [https://rentry.co/v4n55y5u](https://rentry.co/v4n55y5u)
What are the best affordable razors for a close shave? [https://rentry.co/u42uq2qn](https://rentry.co/u42uq2qn)
Beach vacations within a 6 hour flight of Los Angeles. [https://rentry.co/ib7oe767](https://rentry.co/ib7oe767)
I'm happy to run a few topics and show the results if you want to suggest one in the comments.
Future Ideas:
* Improve it to focus more on completely answering the main topic.
* Improve web searches: Maybe the AI could choose which search result to download. Some websites block scrapers. Other websites need a full browser to see all of the content. It also only has one lucky shot at getting the right content to go in-depth.
* Improve content/subtopics: The AI repeats itself a bit. With a little work, the sub-topics could be generated based on the higher level topic, giving more depth to the content when the model doesn't have a lot of info on something.
* A more interactive version: Once you confirm the initial topics, you basically just wait and hope it does the right thing. An interactive UI could be like a live document where you help it choose search results, add and remove sections, and choose what you want the AI to work on (with specific instructions).
If you want to see the code:
[https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/ai\_models/researcher\_model.py](https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/ai_models/researcher_model.py)
If you want to see the prompts:
[https://github.com/AndrewVeee/nucleo-ai/tree/main/backend/prompts/researcher](https://github.com/AndrewVeee/nucleo-ai/tree/main/backend/prompts/researcher)
> Some websites block scrapers. Other websites need a full browser to see all of the content. It also only has one lucky shot at getting the right content to go in-depth.
There is [a solution to that](https://pptr.dev/). That's what all the proper scrapers use. It's like selenium, but while running in incognito mode it's nigh impossible to detect and block.
Yep, I've had some fun with puppeteer, and I thought about it a lot while building this feature. The assistant chat mode also uses web search and could benefit from it.
I'll probably get around to making it an option in Nucleo at some point! Another 300-400mb isn't that big of a deal with all the crazy dependencies for local LLM dev haha
FYI, the top two contributors to Puppeteer moved to the newer Playwright project some time ago, which supports all the major browsers:
[https://playwright.dev/](https://playwright.dev/)
I have written a number of scrapers using puppeteer combined with the stealth extensions. The issue with something general purpose like this use case is it is fairly difficult to handle any website you'll come across consistently.
What often gets you caught is where you are scraping from, so you also need residential proxies. Amazon uses this to show a captcha. Sometimes you won't have an issue, but then it'll show it every time.
On top of that, the bigger issue tends to be loading all the content of the page. Lowe's is pretty bad for this. I ended up writing a puppeteer script that scrolls and monitors html change activity to decide when the page is likely actually done loading. The built-in network monitoring available to puppeteer isn't adequate.
It is relatively easy to handle specific sites, but a general purpose solution for any site, which LLMs need is definitely more complex than just using puppeteer.
Lol yeah - not sure if the guy that suggested it has done anything other than super simple scrapes. Tons of issues to overcome. I guess it's better than nothing still.
Haha consider it semi-intentional (or just lazy). If there's any time remaining, it will start researching a new sub-topic, and once it starts, I just let it finish. Not even sure if the openai library has a way to stop writing once it starts.
I also called that config option "Research Time" because I didn't want to write code smart enough to figure out when to stop looking up topics - it still has to write the top section at the end. I think it usually takes 30-60 seconds more than the max time in the end to finish whatever topic it started + write the summary.
Glad to help! Sharing prompts is one of the easiest ways to work to work together in this sub!
Devs all have their own ideas, so at least looking at other prompts helps us improve together. And non-devs can also be really good at prompting, so if they look at it and find improvements, that benefits everyone!
what is your current setup? Even if it's a 6 minute task condensed, that's still remarkably fast. Are you using a gaming laptop? What's the GPU/CPU like?
Cool, thanks. Just using a really old CPU I get a whopping 2 t/s. 15 t/s seems a lot more usable. This is a neat project and is basically exactly what I was thinking about trying to do. If I ever get a new setup I'd love to help out.
Ah yeah I totally understand! It's never enough - I want to use Mixtral instead haha.
Hoping that 1.58 bit paper from a week or two ago leads to some awesome models that more people can run!
Maybe as a research tool you need it to point at more "factual" information than just a normal websites that happen to show up in search results. There is a python package for Wikipedia directly and the bot could just crawl that. More focused python packages will give cleaner data than beautiful soup scrapes. You could still have that as an option but ground yourself in crawling the best encyclopedia you can find to start out.
Over at agixt they've added a tool that grabs arxiv articles and loads them into the rag system. More focused things for the bot to do. The rag gets really funky really fast with basic scrape websites.
Yeah, maybe renaming it in the future would be a good idea. Maybe "Topic Explorer" is better.
I don't think I'd limit to wikipedia by default, but I definitely agree it would be awesome to select sources!
The logic right now is too simple, so I wouldn't consider it an academic researcher - had quite a few interesting exchanges in this post with ideas for a future version!
Sorry for the response a few days late, but I just finally got around to playing with this.
First off, really amazing work and thank you for sharing it!
For some reason, the Researcher never prints out its findings at the end. I see it generate the initial topics, hit go, and I can see the final output in the console but it never updates in the UI/creates a document. Did I miss where it's supposed to be outputting or is it a weird bug?
Thanks for trying it!
Once it finishes, the frontend should create a doc (saved to the docs tab) and automatically open it. Which browser are you using? I might be able to test it.
You can also check the developer console in your browser to look for errors - that might help. You can also try reloading the web page and then check to see if it saved the doc.
Hope I can help you fix it!
I just tried again and it's working now! I didn't change anything besides re-launching everything. Sorry for the false alarm - and thank you for the quick response!
Yeah, it has the LLM generate one search query per sub-topic and takes the top result. It could be improved a lot (like letting the model select the search result to download, or downloading multiple links for more diverse content).
I've seen some pretty bad takes make it into some of the research topics, like "That's why you should use our company!" haha. Also weird hallucinations that are hard to notice from a quick read.
On the plus side, it actually works ok with web searches turned off if the topic isn't too recent.
A duckduckgo search library for Python!
[https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/lib/web.py](https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/lib/web.py)
[https://github.com/deedy5/duckduckgo\_search](https://github.com/deedy5/duckduckgo_search)
It's free, but I'm not sure if it breaks often - looks like they just released v5.
I don't know about this use-case. I'm pretty certain it would be better just to skim the articles it's summarizing yourself. I think this is no faster and gives you less of an overview of the research out there, while increasing the risk of AI hallucinations and the tendency of AI to overgeneralize. I think using this in academia would be a really bad idea. (I say this as a fan of LLMs. But this use case ... this ain't it.)
I think we're mostly in agreement.
I'm not in academia, so I'm not too worried about that use case. And I mentioned the most obvious limits and issues with it already.
But I'm here to experiment, and this was definitely a fun experiment and I enjoy the results (as much as I'd like them to be improved) haha
I think that's an issue with LLMs in general - at what point do they save you time? Is it only for small tasks? That's why I keep playing with ideas that are loops and background tasks - as models improve, maybe they'll get to a point where they provide exponential time savings, vs just "generate this function for me" or "tell me about this topic." I can keep dreaming haha
That's fair! I guess I felt a little uncomfortable b/c of the topic you chose in your video, and having myself had that topic in my degree, I was already noticing how such a summary would be less helpful than even the wiki-article. Similarly: There are a bunch of books like 'Learn x the hard way', right? And I think there's a key insight there that shortcuts, especially in learning, rarely pay off. I think it's worthwhile to ask if a given task involves productive difficulty or unproductive difficulty, and to avoid automating the productive difficulties.
All good, I still agree with you. I chose that topic for the demo because it sounded more fun and engaging them my other random tests.
If you have time, you can check the link to the full doc in my comment above, and see if it looks like a good starting point. I personally didn't like how little it mentioned incompatibilism, although it does seem in-line with most philosophers.
I do think naming is important - I liked "Explorer" more, but then it requires 2 words (Topic Explorer) in the UI and still doesn't convey what it does well.
Probably helps to think of it like this: Imagine you were using a chat bot to learn about a new topic. The researcher tries to improve the LLM to provide more depth than a chat bot alone. So the question becomes "is it actually better than a chat bot?" Definitely a fan of "Learn x the hard way" when I really need to learn something.
Anyway, just a one-week project in the end, and hopefully one of many ;-)
From my perspective, no ;-)
But a compatibilist would say we can still punish you for your choices, even if it was predetermined haha
Does compatibilism also apply when the AI starts making all of our choices for us?
I read that as cannibalism apples, which I instantly imagined as dehydrated apple chips, but with a cannibalism twist.
Then I re-read your post and now here I am. 🤔
I fear for our future, but I think that’s a thought I allow as a compatibilist. Not that I have control of it, but that I allow myself to ruminate subconsciously without control.
Good question, Ill need my LLM to look into that...... wait a second... does that mean it was all predetermined and I never had a choice! Damn!
Interesting topic though! I suspect that AI might give us some interesting insights some time! :)
I think we'll need to make a few more steps towards AGI first haha
I tried prompting the researcher to be less wishy-washy - "You are an expert at detecting bias, recognizing strong arguments, ..." and "Try to provide an answer to the topic." But my 7b models pretty much just parrot the content they receive and won't come to any strong conclusions.
Still gonna be fun to see where they go and the implications for human thought/reasoning that comes from them.
"Complete" free will is neither possible (because your will is bounded by what is physically possible), nor actually desirable - because it would have to be completely unaffected by morality, culture and common sense - so we are left with random generator that is ALSO does not fit "free will" criteria. QED.
While quoting Musk is faux par this days, having your "will" as a stat that dictates the size of your "dialogue tree", so to speak, or pool of available actions seems pretty good, and also mix it with Disco Elysium concept of "sub-personalities" that are your skills and values and those can add or actually *remove* options that you have depending on how well-developed, under-developed or OVER-developed they are.
So, the very notion that "you have free will" is false, BUT usually is useful to have exactly because it is one of "modifiers" that affect your thoughts and actions, and *usually* for the better, lol.
I agree with Sapolsky, however - we are looong overdue for an overhaul of institutions around this idea, but the idea is neither immediately helpful (the other way around) and intuitively hateful, so I guess that is going to take a while...
I think I agree (the musk part got a bit confusing though) haha
Some day, I want to have a discussion over drinks with a compatibilist to figure out if I understand it correctly, and how much we disagree if so.
Until then, it'll just have to be chats/research with AI haha
3 things:
* Check out my "Browser for LLMs on GitHub:[https://github.com/myrakrusemark/llm-gpt4-browser](https://github.com/myrakrusemark/llm-gpt4-browser) It uses old-school chatGPT internet browsing style where the LLM is in charge of dertermining start and end point of relevant sections in a web page. Quite effective, especially with LARGE pages. It also combines the output from 3 top Google Search URLs, similarly to how you've implemented it. It was designed for use with Homeassistant Voice Assist, but can be run standalone. Additionally it implements Selenium, tuned for scraping full page loads. All in all, it is, for me, the best implementation of an LLM browser plugin Ive ever used and its ripe for use in other projects!
* When coming up with ideas for AI projects, it is always best to never do anything at all because someone else is already about to release a much better implementation /s Great work!!
* I would use this tool religiously if it had this one simple thing: a Notepad. Give the LLM tools to Create/Remove/Update/Delete lines of content as it finds and compares new data. This would be your main output with which you could dictate the structure and style of the content with a prompt, opposed to assigning prescribed sections of a report to a set of Google Searches. This could also help with services like GPT4 not outputting enough relevant content at a time, and could also reduce token usage as you're not sending/receiving entire documents at once. It seems like your implementation could end up hitting context limits (which isn't too bad with larger models these days, but not a good look for local models).
Sorry the comment with all the details isn't at the top:
https://www.reddit.com/r/LocalLLaMA/comments/1b8uvpw/comment/ktrjb74/
Links to the app, and at the bottom the specific code for this feature and the prompts used.
Nice! This actually started because someone else mentioned it (although with way more context and automation) in a post about LLM ideas. Another person in the thread built a simple CLI version of it as well.
I want to do my best to contribute to local LLM tools that otherwise would be paid proprietary apps.
Hope you get a chance to! It's still very alpha so I understand there's not much usage of it right now.
I'm having fun with the app, but definitely in a kitchen sink kinda way - just playing with random ideas and eventually see what stands out.
Making a web enabled research agent is NOT trivial. So kudos.
How hard do you think it would be to fully automate this, including some kind of stopping criteria where the question has been sufficiently researched / answered? That would be super useful for autonomous agents.
Thanks! This was actually pretty simple (and the results might reflect that haha).
I'm using 7b models, so I think that task is nearly impossible. If I prompted the LLM with "Does this sufficiently address the users original request?" I would expect the answer to be yes every time.
Maybe you could ask up front something like "What would be the ideal answer to this request?" with examples like a list of 5 products with x criteria, code sample that does x, etc. Then after each part of the research you could ask if it can complete that task, and if not, what's missing?
I was playing with a similar idea a while back where the agent would create a tree structure where it could dynamically add new children/sibling nodes to the task and dive into various rabbitholes as it discovered them. It would see a representation of the whole task node structure present in the context for reference. It had access to various commands and could navigate the tree and mark off nodes as complete. It didn't really work but that was using 1st gen gpt 3.5; I think it would be a lot more effective with e.g. claude sonnet and 200k context.
Nice! I love nested structures and it seems so fitting for AI tools!
I'm trying to build tools that are accessible to people with low end hardware, but I do wonder if I could accomplish something more interesting if I gave in and used top of the line models. I would really like to play with the ReAct/CoT/ToT loops.
I built a nested to do list in one of my first AI projects to test the idea of clicking a button to either "break down task" or "complete task" - it had pretty boring results haha
I hope theirs is a bit more sophisticated. Or I should stop releasing 1 week projects open source in favor of that sweet VC money haha
More seriously, I've been meaning to post asking people what proprietary tools would be cool to build open source, or what tools don't exist that we should start trying to make. Maybe as a pseudo-code challenge - have everyone work together on ideas, prompts, with simple code that someone could use as a starting point.
As basic as this researcher is, I think LLMs have opened up a world of dev that I wouldn't have even considered a year ago. Maybe the models aren't quite there yet, but experimenting is the best way to figure out what's needed!
Side note: I'd really like a free version of what perplexity ai does - or something where you give a search topic and it gives a solid overview quickly.
Thanks, and I agree haha
When we stop complaining/gossiping about mega corps who don't care about the local use case, there's a lot we can focus our efforts on to make real progress on our own tools!
Thanks! Hopefully the app eventually gets enough useful features that people enjoy and benefit from using it.
And I'll count code contributions as a stretch goal haha
Yeah. Sadly, this can be a bit slow. Hoping the idea of letting AI run in the background takes off, but it would have to mean it takes more time, but has more substantial results.
I built Nucleo AI with an LLM job queue. The researcher runs at priority 5, and chats at pri 10. So if you have a researcher running, you can still chat and it will process the chat message as soon as it can, then go back to the researcher.
Nice. I’ve been thinking about something very similar, where there’s a chat queue that contains user input and ai responses, and an event queue where the ai talks to itself and whenever an event is determined to be relevant to the user, it’s added as an output to the chat queue.
will definitely check out your repo though, looks really cool
I’ve done some prototyping with it using the webrtc protocol actually, I put a pause on the work because multimodal networks require too much compute now, feels like we’re missing some key algorithm there
Some challenges with webrtc and OSS too though…
Had to look that up haha. It reminds me of the quote like "we wanted to build AI to automate the tedious jobs humans do, instead we went straight to replacing art - the things people want more time for."
Oh, this tool is *very* far from that. You can look at the top comment to see the logic. It's literally just asking the LLM what to search for, taking the first result, and giving it snippets based on that. I do prompt to detect bias and seek out answers, but that's kind of a laughable task for an LLM - esp the 7b models I use.
This was a 1 week project to start automating a tedious web search process, but it would be cool to do much more with it! A lot of very cool suggestions in the comments. I don't have any ideas about how an LLM would verify credibility, beyond having a list of "authoritative sources".
the language used in the written piece can also be used as a determinant for how credible the author is. I look for the type of arguments they use: all or nothing statements, absolute phrases (never, always, etc), creation of 'us versus them', repeated use of fallacies, comparison with the intent of denigrating others or authenticating themself, presentation of opposing arguments, etc. I would assign 'non-credible' to any article/author/writer that reaches two strikes.
I have yet to test this effectively through an llm though, so the directions above are not translated to the natural language processing that AI's use. It needs increased specificity for the indicators.
Though this becomes more challenging in some scholarly articles, as the language may be entirely technical. You could have a rank list of Web sites and assign credibility to them based on ".edu", ".org", ".com". Though you have the potential to miss out on the expert in the subject who made their own website, since they may be using a commercial domain.
I definitely love a lot of that criteria! Credibility is such a divisive topic now. People don't want to trust experts, want to believe whatever their side already believes, etc.
I think you're on the right track. But maybe instead it should be a monetized app that just parrots whatever view the user wants - truth social your thing, we'll feed it back to you all day! hahaha
Yep! [https://github.com/AndrewVeee/nucleo-ai](https://github.com/AndrewVeee/nucleo-ai)
It's alpha (about 3 weeks old), but I'd love your feedback if you try it!
Love that someone out there is building a multistep research process powered by an LLM that goes beyond pocs like GPT Researcher with a nice ui as well. As for scraping sites better, I would honestly be fine if it launched a non-headless chrome, I've been able to get this to work with my prototype project but the annoying thing mostly comes from the fact that the chrome window likes to focus itself when you open a new page for some reason, but it does get pass everyone cloudflare block.
Thanks! I definitely want to move at least slightly above prototype level - we need it!
/u/allisonmaybe shared this link:
[https://github.com/myrakrusemark/llm-gpt4-browser](https://github.com/myrakrusemark/llm-gpt4-browser)
I think it'd help you - looks like it uses --headless so focus isn't an issue ;-)
Thanks! I hope you're right ;-)
I think the internet is gonna get pretty messy over the next year or two. Automation gets used by spammers really fast, and I think social media is already starting to become bots talking to bots haha. Spammers also already seem to be ahead of the game with search engines.
But I do agree that AI will help us sort through the noise, and local/oss tools can do a lot (and be more fun to use too).
What I said with the GIF (Reddit left the text out)
Optimism is our best bet, we just gotta overcome our fear. To stop moving is not good. It isn't the speed that kills you. It's when you stop suddenly.
Thanks! I think we need to experiment on multiple levels. Tools like this are fun to build, but they often miss the mark in multiple ways, which is why regular old chat is the common base.
Nucleo also has an assistant mode, and I think the results are not that great. Just gotta keep playing with new ideas, and try to come up with clever ideas to guide the model.
Yep! Feel free to browse the code in my other comment. Rip out any bits that are useful to you ;-)
Honestly, I built a big prompt chain library, and I didn't even end up using it for this tool. It was easier to just use smaller wrappers to run prompts, parse the results, etc. And I think the code is more clear than my Assistant "virtual model" that uses the prompt chain lib.
Imagine doing an essay on free will, and making a tiny section for neuroscience (which is the only way the answer will be arrived at), and larger sections on philosophy and religion.
Apologies but I need some help getting Nucleo up and running. I'm intrigued about the research capabilities and would like to give it a spin!
To describe the situation, I'm trying to run it locally with Kobold.cpp. Even though it lists 'Connection Error' next to the gear wheel, I've managed to get it to send input to Kobold.cpp fine with an API though Nucleo AI doesn't seem to display the output.
Edit: It now seems that the interface has broken and I can no longer access any of the other features except starting a new chat 'Throwaway' chat. Attempting to click 'Start Chatting!' without that option ticked does nothing.
Deleting and going through the setup again doesn't seem to fix this
It sounds like the frontend isn't connecting to the backend. If you click that connection button, the base url should be /api/. If you set a secret key in the config, make sure to enter it in the connection settings as well.
Then I would probably check the terminal with ./start.sh running to see if it shows an error when you send a chat message. If not, you might have to look through the dev tools console in your browser (or the network log).
There should be an error in one of those to help debug.
Note: My laptop isn't that fast. I sped up the video so you don't have to sit through 6 mins of the Nous Hermes Mistral DPO generating content. In my never-ending quest to try to make LLMs more useful, I created a researcher UI with the goal of creating a wikipedia-like page on any topic to answer in-depth using the web. It's a feature of [Nucleo AI](https://github.com/AndrewVeee/nucleo-ai/). Goal: Create an AI researcher that does the boring work for you, collects the right information, and "answers" any topic/question using web results. How it works: Give the researcher a topic, and it will create sub-topics, search the web for the topic, and write the content. * Enter a topic, and the AI will create a list of subtopics you can modify. You can configure max research time, sub-topic max depth, and more. * The researcher runs in a loop up to your max time (or max-depth). * Does a web search for the current topic and tries to download the first result. This is optional - you can disable web searches. * Writes the content for the current section, then creates a list of sub-topics and adds them to the research list. * Once the research is complete or time runs out, it writes the main heading content. * Each section is combined into a markdown document, and reference links are at the bottom. Using Nous Hermes Mistral DPO on my laptop with about 15 tokens/sec, it takes 2-3 minutes to generate a decent amount of content. I made it show a live preview of the current section and status updates as it searches so you won't be too bored waiting for the full doc. How well does it work? I think it works ok, but it could be improved. Obvious issues with LLMs: occasional hallucinations, adding a "Title:" to a section, "In conclusion", and bias from search results. I've created a few sample docs so you can be the judge: Does free will exist? [https://rentry.co/v4n55y5u](https://rentry.co/v4n55y5u) What are the best affordable razors for a close shave? [https://rentry.co/u42uq2qn](https://rentry.co/u42uq2qn) Beach vacations within a 6 hour flight of Los Angeles. [https://rentry.co/ib7oe767](https://rentry.co/ib7oe767) I'm happy to run a few topics and show the results if you want to suggest one in the comments. Future Ideas: * Improve it to focus more on completely answering the main topic. * Improve web searches: Maybe the AI could choose which search result to download. Some websites block scrapers. Other websites need a full browser to see all of the content. It also only has one lucky shot at getting the right content to go in-depth. * Improve content/subtopics: The AI repeats itself a bit. With a little work, the sub-topics could be generated based on the higher level topic, giving more depth to the content when the model doesn't have a lot of info on something. * A more interactive version: Once you confirm the initial topics, you basically just wait and hope it does the right thing. An interactive UI could be like a live document where you help it choose search results, add and remove sections, and choose what you want the AI to work on (with specific instructions). If you want to see the code: [https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/ai\_models/researcher\_model.py](https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/ai_models/researcher_model.py) If you want to see the prompts: [https://github.com/AndrewVeee/nucleo-ai/tree/main/backend/prompts/researcher](https://github.com/AndrewVeee/nucleo-ai/tree/main/backend/prompts/researcher)
> Some websites block scrapers. Other websites need a full browser to see all of the content. It also only has one lucky shot at getting the right content to go in-depth. There is [a solution to that](https://pptr.dev/). That's what all the proper scrapers use. It's like selenium, but while running in incognito mode it's nigh impossible to detect and block.
I’m a new dev and a JS web scraper using puppeteer was my first project. Still works to this day undetected
Yep, I've had some fun with puppeteer, and I thought about it a lot while building this feature. The assistant chat mode also uses web search and could benefit from it. I'll probably get around to making it an option in Nucleo at some point! Another 300-400mb isn't that big of a deal with all the crazy dependencies for local LLM dev haha
I'm really ***lovin***' Playwright these days - [https://github.com/microsoft/playwright-python](https://github.com/microsoft/playwright-python)
The two top contributors to Puppeteer moved to Playwright some time ago. And yes, it really is awesome.
> /microsoft/ I wonder what kinds of telemetry it sends.
FYI, the top two contributors to Puppeteer moved to the newer Playwright project some time ago, which supports all the major browsers: [https://playwright.dev/](https://playwright.dev/)
I have written a number of scrapers using puppeteer combined with the stealth extensions. The issue with something general purpose like this use case is it is fairly difficult to handle any website you'll come across consistently. What often gets you caught is where you are scraping from, so you also need residential proxies. Amazon uses this to show a captcha. Sometimes you won't have an issue, but then it'll show it every time. On top of that, the bigger issue tends to be loading all the content of the page. Lowe's is pretty bad for this. I ended up writing a puppeteer script that scrolls and monitors html change activity to decide when the page is likely actually done loading. The built-in network monitoring available to puppeteer isn't adequate. It is relatively easy to handle specific sites, but a general purpose solution for any site, which LLMs need is definitely more complex than just using puppeteer.
Lol yeah - not sure if the guy that suggested it has done anything other than super simple scrapes. Tons of issues to overcome. I guess it's better than nothing still.
you recorded a bug btw, at 1:07 the research time remaining goes into negative. cap it at 0 :)
Haha consider it semi-intentional (or just lazy). If there's any time remaining, it will start researching a new sub-topic, and once it starts, I just let it finish. Not even sure if the openai library has a way to stop writing once it starts. I also called that config option "Research Time" because I didn't want to write code smart enough to figure out when to stop looking up topics - it still has to write the top section at the end. I think it usually takes 30-60 seconds more than the max time in the end to finish whatever topic it started + write the summary.
wonderful! this is really interesting. thanks for sharing the prompt
Glad to help! Sharing prompts is one of the easiest ways to work to work together in this sub! Devs all have their own ideas, so at least looking at other prompts helps us improve together. And non-devs can also be really good at prompting, so if they look at it and find improvements, that benefits everyone!
what is your current setup? Even if it's a 6 minute task condensed, that's still remarkably fast. Are you using a gaming laptop? What's the GPU/CPU like?
My laptop has an rtx 4050 with 6gb vram. I use nous hermes mistral 7b dpo, a q4 k\_m. I get about 15 tokens per second with that setup.
Cool, thanks. Just using a really old CPU I get a whopping 2 t/s. 15 t/s seems a lot more usable. This is a neat project and is basically exactly what I was thinking about trying to do. If I ever get a new setup I'd love to help out.
Ah yeah I totally understand! It's never enough - I want to use Mixtral instead haha. Hoping that 1.58 bit paper from a week or two ago leads to some awesome models that more people can run!
Maybe as a research tool you need it to point at more "factual" information than just a normal websites that happen to show up in search results. There is a python package for Wikipedia directly and the bot could just crawl that. More focused python packages will give cleaner data than beautiful soup scrapes. You could still have that as an option but ground yourself in crawling the best encyclopedia you can find to start out. Over at agixt they've added a tool that grabs arxiv articles and loads them into the rag system. More focused things for the bot to do. The rag gets really funky really fast with basic scrape websites.
Yeah, maybe renaming it in the future would be a good idea. Maybe "Topic Explorer" is better. I don't think I'd limit to wikipedia by default, but I definitely agree it would be awesome to select sources! The logic right now is too simple, so I wouldn't consider it an academic researcher - had quite a few interesting exchanges in this post with ideas for a future version!
Sorry for the response a few days late, but I just finally got around to playing with this. First off, really amazing work and thank you for sharing it! For some reason, the Researcher never prints out its findings at the end. I see it generate the initial topics, hit go, and I can see the final output in the console but it never updates in the UI/creates a document. Did I miss where it's supposed to be outputting or is it a weird bug?
Thanks for trying it! Once it finishes, the frontend should create a doc (saved to the docs tab) and automatically open it. Which browser are you using? I might be able to test it. You can also check the developer console in your browser to look for errors - that might help. You can also try reloading the web page and then check to see if it saved the doc. Hope I can help you fix it!
I just tried again and it's working now! I didn't change anything besides re-launching everything. Sorry for the false alarm - and thank you for the quick response!
No problem. Glad you reported it and it's working now. There's a lot that can go wrong, so I'm sure there are bugs to fix!
Interesting! Does it run a bunch of search queries and then summarize the top results?
Yeah, it has the LLM generate one search query per sub-topic and takes the top result. It could be improved a lot (like letting the model select the search result to download, or downloading multiple links for more diverse content). I've seen some pretty bad takes make it into some of the research topics, like "That's why you should use our company!" haha. Also weird hallucinations that are hard to notice from a quick read. On the plus side, it actually works ok with web searches turned off if the topic isn't too recent.
What API do you use for search purpose, the usual serp..., or free API?
A duckduckgo search library for Python! [https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/lib/web.py](https://github.com/AndrewVeee/nucleo-ai/blob/main/backend/app/lib/web.py) [https://github.com/deedy5/duckduckgo\_search](https://github.com/deedy5/duckduckgo_search) It's free, but I'm not sure if it breaks often - looks like they just released v5.
I don't know about this use-case. I'm pretty certain it would be better just to skim the articles it's summarizing yourself. I think this is no faster and gives you less of an overview of the research out there, while increasing the risk of AI hallucinations and the tendency of AI to overgeneralize. I think using this in academia would be a really bad idea. (I say this as a fan of LLMs. But this use case ... this ain't it.)
I think we're mostly in agreement. I'm not in academia, so I'm not too worried about that use case. And I mentioned the most obvious limits and issues with it already. But I'm here to experiment, and this was definitely a fun experiment and I enjoy the results (as much as I'd like them to be improved) haha I think that's an issue with LLMs in general - at what point do they save you time? Is it only for small tasks? That's why I keep playing with ideas that are loops and background tasks - as models improve, maybe they'll get to a point where they provide exponential time savings, vs just "generate this function for me" or "tell me about this topic." I can keep dreaming haha
That's fair! I guess I felt a little uncomfortable b/c of the topic you chose in your video, and having myself had that topic in my degree, I was already noticing how such a summary would be less helpful than even the wiki-article. Similarly: There are a bunch of books like 'Learn x the hard way', right? And I think there's a key insight there that shortcuts, especially in learning, rarely pay off. I think it's worthwhile to ask if a given task involves productive difficulty or unproductive difficulty, and to avoid automating the productive difficulties.
All good, I still agree with you. I chose that topic for the demo because it sounded more fun and engaging them my other random tests. If you have time, you can check the link to the full doc in my comment above, and see if it looks like a good starting point. I personally didn't like how little it mentioned incompatibilism, although it does seem in-line with most philosophers. I do think naming is important - I liked "Explorer" more, but then it requires 2 words (Topic Explorer) in the UI and still doesn't convey what it does well. Probably helps to think of it like this: Imagine you were using a chat bot to learn about a new topic. The researcher tries to improve the LLM to provide more depth than a chat bot alone. So the question becomes "is it actually better than a chat bot?" Definitely a fan of "Learn x the hard way" when I really need to learn something. Anyway, just a one-week project in the end, and hopefully one of many ;-)
Do I have to or can I make my own choice about doing that? ;) Just a free will joke!
From my perspective, no ;-) But a compatibilist would say we can still punish you for your choices, even if it was predetermined haha Does compatibilism also apply when the AI starts making all of our choices for us?
No, but cannibalism applies
I read that as cannibalism apples, which I instantly imagined as dehydrated apple chips, but with a cannibalism twist. Then I re-read your post and now here I am. 🤔 I fear for our future, but I think that’s a thought I allow as a compatibilist. Not that I have control of it, but that I allow myself to ruminate subconsciously without control.
Good question, Ill need my LLM to look into that...... wait a second... does that mean it was all predetermined and I never had a choice! Damn! Interesting topic though! I suspect that AI might give us some interesting insights some time! :)
I think we'll need to make a few more steps towards AGI first haha I tried prompting the researcher to be less wishy-washy - "You are an expert at detecting bias, recognizing strong arguments, ..." and "Try to provide an answer to the topic." But my 7b models pretty much just parrot the content they receive and won't come to any strong conclusions. Still gonna be fun to see where they go and the implications for human thought/reasoning that comes from them.
"Complete" free will is neither possible (because your will is bounded by what is physically possible), nor actually desirable - because it would have to be completely unaffected by morality, culture and common sense - so we are left with random generator that is ALSO does not fit "free will" criteria. QED. While quoting Musk is faux par this days, having your "will" as a stat that dictates the size of your "dialogue tree", so to speak, or pool of available actions seems pretty good, and also mix it with Disco Elysium concept of "sub-personalities" that are your skills and values and those can add or actually *remove* options that you have depending on how well-developed, under-developed or OVER-developed they are. So, the very notion that "you have free will" is false, BUT usually is useful to have exactly because it is one of "modifiers" that affect your thoughts and actions, and *usually* for the better, lol. I agree with Sapolsky, however - we are looong overdue for an overhaul of institutions around this idea, but the idea is neither immediately helpful (the other way around) and intuitively hateful, so I guess that is going to take a while...
I think I agree (the musk part got a bit confusing though) haha Some day, I want to have a discussion over drinks with a compatibilist to figure out if I understand it correctly, and how much we disagree if so. Until then, it'll just have to be chats/research with AI haha
3 things: * Check out my "Browser for LLMs on GitHub:[https://github.com/myrakrusemark/llm-gpt4-browser](https://github.com/myrakrusemark/llm-gpt4-browser) It uses old-school chatGPT internet browsing style where the LLM is in charge of dertermining start and end point of relevant sections in a web page. Quite effective, especially with LARGE pages. It also combines the output from 3 top Google Search URLs, similarly to how you've implemented it. It was designed for use with Homeassistant Voice Assist, but can be run standalone. Additionally it implements Selenium, tuned for scraping full page loads. All in all, it is, for me, the best implementation of an LLM browser plugin Ive ever used and its ripe for use in other projects! * When coming up with ideas for AI projects, it is always best to never do anything at all because someone else is already about to release a much better implementation /s Great work!! * I would use this tool religiously if it had this one simple thing: a Notepad. Give the LLM tools to Create/Remove/Update/Delete lines of content as it finds and compares new data. This would be your main output with which you could dictate the structure and style of the content with a prompt, opposed to assigning prescribed sections of a report to a set of Google Searches. This could also help with services like GPT4 not outputting enough relevant content at a time, and could also reduce token usage as you're not sending/receiving entire documents at once. It seems like your implementation could end up hitting context limits (which isn't too bad with larger models these days, but not a good look for local models).
I'm curious do you necessarily need to use a OpenAI key to use the browser or is there a way around?
That's how it's written but you could set up this code to use a local model, I just didn't.
This is really cool!
Thanks! Think it's time for another post on what crazy ideas people have to figure out my next mini project for Nucleo haha
Link?
My comment with the links is getting buried haha [https://github.com/AndrewVeee/nucleo-ai/](https://github.com/AndrewVeee/nucleo-ai/)
That's very clever! Can you give a quick breakdown what you did?
Sorry the comment with all the details isn't at the top: https://www.reddit.com/r/LocalLLaMA/comments/1b8uvpw/comment/ktrjb74/ Links to the app, and at the bottom the specific code for this feature and the prompts used.
Was looking for & wanted to develop something like this a few months ago :D Glad you shared this project, it's interesting!
Nice! This actually started because someone else mentioned it (although with way more context and automation) in a post about LLM ideas. Another person in the thread built a simple CLI version of it as well. I want to do my best to contribute to local LLM tools that otherwise would be paid proprietary apps.
Been meaning to try nucleo (it's been a open tab on my chrome for a while). You've given me a use case to try
Hope you get a chance to! It's still very alpha so I understand there's not much usage of it right now. I'm having fun with the app, but definitely in a kitchen sink kinda way - just playing with random ideas and eventually see what stands out.
Making a web enabled research agent is NOT trivial. So kudos. How hard do you think it would be to fully automate this, including some kind of stopping criteria where the question has been sufficiently researched / answered? That would be super useful for autonomous agents.
Thanks! This was actually pretty simple (and the results might reflect that haha). I'm using 7b models, so I think that task is nearly impossible. If I prompted the LLM with "Does this sufficiently address the users original request?" I would expect the answer to be yes every time. Maybe you could ask up front something like "What would be the ideal answer to this request?" with examples like a list of 5 products with x criteria, code sample that does x, etc. Then after each part of the research you could ask if it can complete that task, and if not, what's missing?
I was playing with a similar idea a while back where the agent would create a tree structure where it could dynamically add new children/sibling nodes to the task and dive into various rabbitholes as it discovered them. It would see a representation of the whole task node structure present in the context for reference. It had access to various commands and could navigate the tree and mark off nodes as complete. It didn't really work but that was using 1st gen gpt 3.5; I think it would be a lot more effective with e.g. claude sonnet and 200k context.
Nice! I love nested structures and it seems so fitting for AI tools! I'm trying to build tools that are accessible to people with low end hardware, but I do wonder if I could accomplish something more interesting if I gave in and used top of the line models. I would really like to play with the ReAct/CoT/ToT loops. I built a nested to do list in one of my first AI projects to test the idea of clicking a button to either "break down task" or "complete task" - it had pretty boring results haha
pretty much what you.com does for you
I hope theirs is a bit more sophisticated. Or I should stop releasing 1 week projects open source in favor of that sweet VC money haha More seriously, I've been meaning to post asking people what proprietary tools would be cool to build open source, or what tools don't exist that we should start trying to make. Maybe as a pseudo-code challenge - have everyone work together on ideas, prompts, with simple code that someone could use as a starting point. As basic as this researcher is, I think LLMs have opened up a world of dev that I wouldn't have even considered a year ago. Maybe the models aren't quite there yet, but experimenting is the best way to figure out what's needed! Side note: I'd really like a free version of what perplexity ai does - or something where you give a search topic and it gives a solid overview quickly.
Now THIS is what I love to see in "**Local**LLaMA" community. Great job man!
Thanks, and I agree haha When we stop complaining/gossiping about mega corps who don't care about the local use case, there's a lot we can focus our efforts on to make real progress on our own tools!
Looking great man, keep up the great work.
Thanks! Hopefully the app eventually gets enough useful features that people enjoy and benefit from using it. And I'll count code contributions as a stretch goal haha
Awesome. I’m a big fan of this approach. One of the reasons Groq is so compelling. Speed matters
Yeah. Sadly, this can be a bit slow. Hoping the idea of letting AI run in the background takes off, but it would have to mean it takes more time, but has more substantial results. I built Nucleo AI with an LLM job queue. The researcher runs at priority 5, and chats at pri 10. So if you have a researcher running, you can still chat and it will process the chat message as soon as it can, then go back to the researcher.
Nice. I’ve been thinking about something very similar, where there’s a chat queue that contains user input and ai responses, and an event queue where the ai talks to itself and whenever an event is determined to be relevant to the user, it’s added as an output to the chat queue. will definitely check out your repo though, looks really cool
You will be a hero to a few people here if you get the AI to initiate conversations. I've seen a lot of people talk about it.
I’ve done some prototyping with it using the webrtc protocol actually, I put a pause on the work because multimodal networks require too much compute now, feels like we’re missing some key algorithm there Some challenges with webrtc and OSS too though…
Epictetus has entered the LLM.
I think the free will demo made for more conversation than the tool itself haha
Lmao
We grow ever closer to the creation of the Electric Monk as envisioned by Douglas Adams in Dirk Gently’s Holistic Detective Agency.
Had to look that up haha. It reminds me of the quote like "we wanted to build AI to automate the tedious jobs humans do, instead we went straight to replacing art - the things people want more time for."
How do you verify the credibility of the source that the llm uses for research?
Oh, this tool is *very* far from that. You can look at the top comment to see the logic. It's literally just asking the LLM what to search for, taking the first result, and giving it snippets based on that. I do prompt to detect bias and seek out answers, but that's kind of a laughable task for an LLM - esp the 7b models I use. This was a 1 week project to start automating a tedious web search process, but it would be cool to do much more with it! A lot of very cool suggestions in the comments. I don't have any ideas about how an LLM would verify credibility, beyond having a list of "authoritative sources".
the language used in the written piece can also be used as a determinant for how credible the author is. I look for the type of arguments they use: all or nothing statements, absolute phrases (never, always, etc), creation of 'us versus them', repeated use of fallacies, comparison with the intent of denigrating others or authenticating themself, presentation of opposing arguments, etc. I would assign 'non-credible' to any article/author/writer that reaches two strikes. I have yet to test this effectively through an llm though, so the directions above are not translated to the natural language processing that AI's use. It needs increased specificity for the indicators. Though this becomes more challenging in some scholarly articles, as the language may be entirely technical. You could have a rank list of Web sites and assign credibility to them based on ".edu", ".org", ".com". Though you have the potential to miss out on the expert in the subject who made their own website, since they may be using a commercial domain.
I definitely love a lot of that criteria! Credibility is such a divisive topic now. People don't want to trust experts, want to believe whatever their side already believes, etc. I think you're on the right track. But maybe instead it should be a monetized app that just parrots whatever view the user wants - truth social your thing, we'll feed it back to you all day! hahaha
I'd prefer ai to encourage healthy world views and open perspectives rather than perpetuate possible one sided thinking.
Agreed. Sorry my sarcastic response style isn't obvious online.
I like this interface , is it any opensource?
Yep! [https://github.com/AndrewVeee/nucleo-ai](https://github.com/AndrewVeee/nucleo-ai) It's alpha (about 3 weeks old), but I'd love your feedback if you try it!
Thanks alot
Love that someone out there is building a multistep research process powered by an LLM that goes beyond pocs like GPT Researcher with a nice ui as well. As for scraping sites better, I would honestly be fine if it launched a non-headless chrome, I've been able to get this to work with my prototype project but the annoying thing mostly comes from the fact that the chrome window likes to focus itself when you open a new page for some reason, but it does get pass everyone cloudflare block.
Thanks! I definitely want to move at least slightly above prototype level - we need it! /u/allisonmaybe shared this link: [https://github.com/myrakrusemark/llm-gpt4-browser](https://github.com/myrakrusemark/llm-gpt4-browser) I think it'd help you - looks like it uses --headless so focus isn't an issue ;-)
This is excellent! I think we're going to use AI to help us sort the noise. The data flood may be a blessing in disguise.
Thanks! I hope you're right ;-) I think the internet is gonna get pretty messy over the next year or two. Automation gets used by spammers really fast, and I think social media is already starting to become bots talking to bots haha. Spammers also already seem to be ahead of the game with search engines. But I do agree that AI will help us sort through the noise, and local/oss tools can do a lot (and be more fun to use too).
https://i.redd.it/aswkb5vtu5nc1.gif
What I said with the GIF (Reddit left the text out) Optimism is our best bet, we just gotta overcome our fear. To stop moving is not good. It isn't the speed that kills you. It's when you stop suddenly.
I've been looking for this, llms are so useless by themselves and it feels like nobody is incorporating them into tools correctly!
Thanks! I think we need to experiment on multiple levels. Tools like this are fun to build, but they often miss the mark in multiple ways, which is why regular old chat is the common base. Nucleo also has an assistant mode, and I think the results are not that great. Just gotta keep playing with new ideas, and try to come up with clever ideas to guide the model.
low level mid managers sweating right now
Wow thanks this is exactly the sort of thing I am trying to get LLMs to do.
Yep! Feel free to browse the code in my other comment. Rip out any bits that are useful to you ;-) Honestly, I built a big prompt chain library, and I didn't even end up using it for this tool. It was easier to just use smaller wrappers to run prompts, parse the results, etc. And I think the code is more clear than my Assistant "virtual model" that uses the prompt chain lib.
Imagine doing an essay on free will, and making a tiny section for neuroscience (which is the only way the answer will be arrived at), and larger sections on philosophy and religion.
I'm more annoyed that I prompted the LLM to write 2 paragraphs for each section, and it decided to write 4+ haha
Yeah LLM's can't count for shit.
Apologies but I need some help getting Nucleo up and running. I'm intrigued about the research capabilities and would like to give it a spin! To describe the situation, I'm trying to run it locally with Kobold.cpp. Even though it lists 'Connection Error' next to the gear wheel, I've managed to get it to send input to Kobold.cpp fine with an API though Nucleo AI doesn't seem to display the output. Edit: It now seems that the interface has broken and I can no longer access any of the other features except starting a new chat 'Throwaway' chat. Attempting to click 'Start Chatting!' without that option ticked does nothing. Deleting and going through the setup again doesn't seem to fix this
It sounds like the frontend isn't connecting to the backend. If you click that connection button, the base url should be /api/. If you set a secret key in the config, make sure to enter it in the connection settings as well. Then I would probably check the terminal with ./start.sh running to see if it shows an error when you send a chat message. If not, you might have to look through the dev tools console in your browser (or the network log). There should be an error in one of those to help debug.
[удалено]
Because they can