One big flag for me is that social science research is already rife with fraud and p-hacking. It's a 'soft' science with a notorious history of false conclusions.
Maybe, instead of training on that and removing the humans, AI can help structure better research for real humans to conduct on real humans?
It's probably important to make a strong distinction between good scholarship in the social sciences -- of which there is a lot, and much important and relevant research to boot -- and cases that grab headlines but aren't necessarily indicative of the whole. Plus, fraud and p-hacking are not solely the domain of the social sciences.
That said, there are intriguing possibilities around extending robust knowledge generation around research focused on people and society using AI.
“Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents: computational software agents that simulate believable human behavior.”
And:
“A society full of generative agents is marked by emergent social dynamics where new relationships are formed, information diffuses, and coordination arises across agents.”
I would add a critical note to that: AI agents DO NOT in any shape or form replicate the complexities of real human behavior, emotions and thought. So, while the concept and application of AI in the social sciences could be valuable, caution is advised.
A concern that I have is the data that’s being used to power these models could lead to results that amplify predictions of prejudicial behavior — as described in Weapons of Math Destruction by Cathy O’Neil. Garbage in, garbage out, as they say. If all, much, or even some of the historical data a model has access to is based on flawed/prejudicial/disproven social theory and evidence, then the predictions of the models will necessarily be skewed.
A colleague described Chat GOT as “the world’s most powerful graphing calculator” and that seems to me the most appropriate use case thus far. I understand that LLMs are exciting new toys and researchers are trying to find the boundaries of how they might be used, but this application for simulating social science experiments strikes me as inappropriate. Social science experiments are hard to run because they are hard to do well and easy to screw up. I reject the silicon valley bro-science thinking that any arduous task can be done easier if a “disruptive” engineering approach is taken. Some things are difficult! Some data is hard won and cannot be cheated! There is no royal road to geometry!
One big flag for me is that social science research is already rife with fraud and p-hacking. It's a 'soft' science with a notorious history of false conclusions.
Maybe, instead of training on that and removing the humans, AI can help structure better research for real humans to conduct on real humans?
How to win friends and influence people :)
It's probably important to make a strong distinction between good scholarship in the social sciences -- of which there is a lot, and much important and relevant research to boot -- and cases that grab headlines but aren't necessarily indicative of the whole. Plus, fraud and p-hacking are not solely the domain of the social sciences.
That said, there are intriguing possibilities around extending robust knowledge generation around research focused on people and society using AI.
It reminds me of this paper: https://arxiv.org/pdf/2304.03442.pdf
“Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents: computational software agents that simulate believable human behavior.”
And:
“A society full of generative agents is marked by emergent social dynamics where new relationships are formed, information diffuses, and coordination arises across agents.”
I would add a critical note to that: AI agents DO NOT in any shape or form replicate the complexities of real human behavior, emotions and thought. So, while the concept and application of AI in the social sciences could be valuable, caution is advised.
A concern that I have is the data that’s being used to power these models could lead to results that amplify predictions of prejudicial behavior — as described in Weapons of Math Destruction by Cathy O’Neil. Garbage in, garbage out, as they say. If all, much, or even some of the historical data a model has access to is based on flawed/prejudicial/disproven social theory and evidence, then the predictions of the models will necessarily be skewed.
A colleague described Chat GOT as “the world’s most powerful graphing calculator” and that seems to me the most appropriate use case thus far. I understand that LLMs are exciting new toys and researchers are trying to find the boundaries of how they might be used, but this application for simulating social science experiments strikes me as inappropriate. Social science experiments are hard to run because they are hard to do well and easy to screw up. I reject the silicon valley bro-science thinking that any arduous task can be done easier if a “disruptive” engineering approach is taken. Some things are difficult! Some data is hard won and cannot be cheated! There is no royal road to geometry!