I always love your experiments, Andrew. I really want to try this on works that I know better. I find that reading stuff like this is so difficult to parse when you both don't know what is accurate and don't know the field well. At least with peer-reviewed works I can (hopefully..) assume that someone smarter and more knowledgeable than me read through the draft and picked it apart.
With AI - there is such a keen semblance of authority I find it hard to know when it is full of... well, when it is hallucinating.
I don't think we are close to these systems replacing experts - but I also wonder if those without expertise can spot the uncanny or fakeness of the text if they DONT know it is AI?
For folks reviewing it, will field expertise be enough? Will they need the AI expertise to spot the "tells"? How long will those tells last before a future iteration allows them to be removed?
Experts might be able to navigate these - but I worry most for entry level folks, students, and the public... many already have a hard time parsing through it all. Now a semi-realistic dissertation can be generated that is somewhat passable. As usual with your experiments - both excited to try it out and a bit terrified by the possibilities.
Also - love the Shower Thought origin of this project!
Thanks Leonard -- and I agree, trust and evaluation are incredibly important and getting increasingly challenging here! A substantial concern is that the investment needed in checking everything -- even for an expert -- when the prose are so honey-smooth, is so large that eventually most people will cave and just trust what they read. I'd like to say that a critical skill everyone needs is to ask questions about what they are reading/hearing and use critical thinking to assess the plausibility and validity of stuff -- but this is hard. Maybe everyone needs their own personal AI critical thinking advisor ...
The one thing that is clear is that the genie isn't going back in the bottle
On a far more limited basis... what I've found in a PhD process, where I decided to test various AI tools but in the main not use them (and declared what I did use) is that AI is very poor at contextual understanding.
Assuming the mechanisms by which it decides what is important are algorithmic, one images that it uses frequency citations and weight even including factors like the font size of headlines. This leads to situations where critical but understated facts or assumptions skews the AI product. This is not hallucination or fallibility, not relating to gaps in the underlying data (qualitative studies are very poorly represented in some LLMs databases) and/or gaps in the training datasets for the LLMs, or deliberately curated content that excludes 'unsavoury' facts on behalf of the LLM owner!
You still need a human to read through it and decide what's important and insightful. IMHO!
Yes, agree that the human in the loop is still absolutely essential. But these new reasoning models are fascinating to watch as they seek out sources, draw inferences, then start looking for other sources to support, challenge, or extend on what they originally surmised. They are getting close to simulating good critical scholarship.
As a thought experiment, let's imagine that AI develops to the point where it can mass produce valid, credible, useful PhD level scientific research papers. Knowledge development explodes. How should we regard that possibility?
It's typically assumed that more knowledge is automatically a good thing. Is that true? Is there any limit to that concept? Can society at large successfully manage ANY amount of knowledge driven change, delivered at ANY rate?
This is certainly an interesting experiment, and it's exciting to think how much further we can take things with AI pushing the boundaries everywhere we look. The big caveat to me (as a science PhD holder), is that it doesn't involve any original data generation, at least from what I could see - I haven't read the whole thing. We obviously have to spend most of our time in the lab (or field), gathering data to test our hypotheses, rather than just taking evidence from other written sources, and that's the definition of a PhD in my mind. Which AI will be able to assist with too at some point, I'm sure!
Yep - this is more in line with a philosophy/humanities/theory PhD which was useful as a proof of concept, and as I say it is not a PhD dissertation. But imagine a time when AI agents can conduct experiments in the lab through access to instruments (we're getting there) or develop and use surveys, or employ other methods that involve interacting with physical systems. This would open the door to AI agents that can ideate, develop research strategies, generate new data, and analyze it. I still think we're a long way from AI only research, but imagine you have a neat idea in the morning, tell your AI agents to research it, and have a draft study on your desk for review in a few days ...
Absolutely, and in the time between now and then, it could provide help along many of those steps. So far, I haven't had much luck getting it to do more than provide useful feedback and very general information. My experience is that, like humans, it can suffer from blind spots. Missing potentially important information because it doesn't think to go down that avenue. (In other words, it doesn't know what it doesn't know). This may also be caused by the prompting and the questioner having those same issues, so they're not asking the right questions.
A recent example using the 'Research' mode in you.com: I asked it to give me any and all info about making food in a traveling space colony and it said it went through 120+ refs. Then it spoke only about how to grow plants (not mentioning vertically I might add) - lighting, GMO, seed banks, recycling excrement etc. I had to prompt it myself to go into meat production, fermentation practices, fungi, and bacteria, and I didn't even mention insect farms or fish yet, and neither did the AI.
Fascinating, and a bit frightening. I agree that this is a fantastic opportunity, but only if we move carefully. Do we lose the important "process" if we rely too much on AI? We need to think very carefully about how we build this into our curriculum, not only with PhD theses.
Yep -- I think the most important takeaway here is that the academic community needs to be thinking critically and fast about what it does and values, why, how AI potentially threatens or enhances this, and how it needs to respond
Whoa. This is amazing and frightening, considering where OpenAI was in 2023. We need to rethink the concept of research in this new world, or it might lead us to ruin.
I think it is a fantastic opportunity for humanity, but our educators need to embrace, not shun the possibilities.
I always love your experiments, Andrew. I really want to try this on works that I know better. I find that reading stuff like this is so difficult to parse when you both don't know what is accurate and don't know the field well. At least with peer-reviewed works I can (hopefully..) assume that someone smarter and more knowledgeable than me read through the draft and picked it apart.
With AI - there is such a keen semblance of authority I find it hard to know when it is full of... well, when it is hallucinating.
I don't think we are close to these systems replacing experts - but I also wonder if those without expertise can spot the uncanny or fakeness of the text if they DONT know it is AI?
For folks reviewing it, will field expertise be enough? Will they need the AI expertise to spot the "tells"? How long will those tells last before a future iteration allows them to be removed?
Experts might be able to navigate these - but I worry most for entry level folks, students, and the public... many already have a hard time parsing through it all. Now a semi-realistic dissertation can be generated that is somewhat passable. As usual with your experiments - both excited to try it out and a bit terrified by the possibilities.
Also - love the Shower Thought origin of this project!
Thanks Leonard -- and I agree, trust and evaluation are incredibly important and getting increasingly challenging here! A substantial concern is that the investment needed in checking everything -- even for an expert -- when the prose are so honey-smooth, is so large that eventually most people will cave and just trust what they read. I'd like to say that a critical skill everyone needs is to ask questions about what they are reading/hearing and use critical thinking to assess the plausibility and validity of stuff -- but this is hard. Maybe everyone needs their own personal AI critical thinking advisor ...
The one thing that is clear is that the genie isn't going back in the bottle
Hi Andrew - really cool experiment. I'm very keen to try out Deep Research. Guess I just need to bite the bullet and subscribe to Pro ;)
I'm an ex-textbook publisher. How well do you think it might research and then write content for textbooks, say high-school physics?
Better than you would imagine -- but still needs a human eye and human editing :)
Hi Andrew
On a far more limited basis... what I've found in a PhD process, where I decided to test various AI tools but in the main not use them (and declared what I did use) is that AI is very poor at contextual understanding.
Assuming the mechanisms by which it decides what is important are algorithmic, one images that it uses frequency citations and weight even including factors like the font size of headlines. This leads to situations where critical but understated facts or assumptions skews the AI product. This is not hallucination or fallibility, not relating to gaps in the underlying data (qualitative studies are very poorly represented in some LLMs databases) and/or gaps in the training datasets for the LLMs, or deliberately curated content that excludes 'unsavoury' facts on behalf of the LLM owner!
You still need a human to read through it and decide what's important and insightful. IMHO!
Yes, agree that the human in the loop is still absolutely essential. But these new reasoning models are fascinating to watch as they seek out sources, draw inferences, then start looking for other sources to support, challenge, or extend on what they originally surmised. They are getting close to simulating good critical scholarship.
As a thought experiment, let's imagine that AI develops to the point where it can mass produce valid, credible, useful PhD level scientific research papers. Knowledge development explodes. How should we regard that possibility?
It's typically assumed that more knowledge is automatically a good thing. Is that true? Is there any limit to that concept? Can society at large successfully manage ANY amount of knowledge driven change, delivered at ANY rate?
This is certainly an interesting experiment, and it's exciting to think how much further we can take things with AI pushing the boundaries everywhere we look. The big caveat to me (as a science PhD holder), is that it doesn't involve any original data generation, at least from what I could see - I haven't read the whole thing. We obviously have to spend most of our time in the lab (or field), gathering data to test our hypotheses, rather than just taking evidence from other written sources, and that's the definition of a PhD in my mind. Which AI will be able to assist with too at some point, I'm sure!
Yep - this is more in line with a philosophy/humanities/theory PhD which was useful as a proof of concept, and as I say it is not a PhD dissertation. But imagine a time when AI agents can conduct experiments in the lab through access to instruments (we're getting there) or develop and use surveys, or employ other methods that involve interacting with physical systems. This would open the door to AI agents that can ideate, develop research strategies, generate new data, and analyze it. I still think we're a long way from AI only research, but imagine you have a neat idea in the morning, tell your AI agents to research it, and have a draft study on your desk for review in a few days ...
Absolutely, and in the time between now and then, it could provide help along many of those steps. So far, I haven't had much luck getting it to do more than provide useful feedback and very general information. My experience is that, like humans, it can suffer from blind spots. Missing potentially important information because it doesn't think to go down that avenue. (In other words, it doesn't know what it doesn't know). This may also be caused by the prompting and the questioner having those same issues, so they're not asking the right questions.
A recent example using the 'Research' mode in you.com: I asked it to give me any and all info about making food in a traveling space colony and it said it went through 120+ refs. Then it spoke only about how to grow plants (not mentioning vertically I might add) - lighting, GMO, seed banks, recycling excrement etc. I had to prompt it myself to go into meat production, fermentation practices, fungi, and bacteria, and I didn't even mention insect farms or fish yet, and neither did the AI.
Fascinating, and a bit frightening. I agree that this is a fantastic opportunity, but only if we move carefully. Do we lose the important "process" if we rely too much on AI? We need to think very carefully about how we build this into our curriculum, not only with PhD theses.
Yep -- I think the most important takeaway here is that the academic community needs to be thinking critically and fast about what it does and values, why, how AI potentially threatens or enhances this, and how it needs to respond
Is it the ultimate irony that AI alone can fix it?
Ha!
Whoa. This is amazing and frightening, considering where OpenAI was in 2023. We need to rethink the concept of research in this new world, or it might lead us to ruin.
I think it is a fantastic opportunity for humanity, but our educators need to embrace, not shun the possibilities.
I agree!
Had to stop something to read this with attention:)
2025 would be fun*😅