Really enjoyed this article. I get that you're trying to stay at a high level, but I think there are at least a couple of areas that could be explored further. First is in tech revolutions that had massive, society-wide impacts, thinking industrialization, electrification, computing, & networks. The second is the massive extent of engineering & application developments predicated on scientific breakthroughs. There's a parallel but in many ways different story about how medical/health sciences have developed vis a vis industrialization/elec/comp/networking.
Thanks !! Both interesting thoughts, and both are perfect examples of the necessity of bringing multiple fields/ technologies/ innovations together to bring about progress.
I have a post in the works to look ahead, but a couple examples here include the varied combinations of our ability to process information beter (AI etc), better ability to manipulate matter (nanotech of varied forms), ability to affect biology (crispr etc) and seeing what the impacts of that's likely to be.
The big question is: What's the next massive scientific breakthrough that will fundamentally alter how society operates? & how close is it to happening? I don't have the answer - I just hope it happens in my lifetime. It might also be the massive spread of renewables & related technologies, which are necessary to just keep civilization going while also opening massive prospects for economic activity.
This is great. Brings together a lot of research. I think a paper I read on the "Step and Wait Model" would also fit here.
I would like the S-curve to be a bit more defined - it's originally (diffusion of innovations) a population prevalence curve for an innovation in equipment or practice in the technologies that are being reproduced (still used, not obsolescent). There is a related 'phenotype performance' curve that is made up of increments of improvement from combination of innovations, but it seems to me that fitting an S-curve to that is not so obvious, although no doubt there are good examples. But niches and the qualities needed in equipment for them are conceptual handles and there are a lot of valid levels of description.
Thanks Andrew! The S curve here is a bit of a fudge, to stand in for any function that has a gradual takeoff in the beginning and a plateau at the end - the exact function can be changed if needed. Phenotype performance curve is an interesting one in that it can also reduce for larger values of the X axis, but practically I think of that as the equivalent of a technology S curve becoming obsolete.
Which means there's definitely a lot further to take it, but the idea is to expound if the theoretical argument holds, and later to specify it with particular functional forms..
So what I got from this is that ideas and technology need space to have lots of sex in order to produce progress, so it's natural that intellectually prudish societies don't have much progress.
Just wanted to say that this is a great post. I might appreciate it more than others because it aligns with my intuitions, but I found the argumentation very compelling.
Relatedly: I always thought a good share of the progress study people were going in the wrong direction, with a very strong focus on the industrial revolution and a "make innovation great again" outlook. Although there some good things have come out of that thought approach, I feel they are looking for a silver bullet that is just not there (or at least, not where they are looking for it). Do you agree? Or I am misinterpreting you?
I do like the theory quite a lot, and hope to continue to expand it. And yes, I do feel the focus is almost entirely to reduce the negative forces (change institutions or funding models) which as a theory makes me feel the basic assumption is that "if it wasn't for all these bureaucracy and red tape etc, we'd be innovating so fast!". It's also reductionist in the search for a silver bullet for sure - hence the screed on anti-regulation, anti-govt, anti-bureaucracy etc, as if the natural state re most innovations as de novo emanations of a great man theory.
I would like however to also figure out the why or how regarding how innovations actually do come about, rather that relying on just removing obstacles and letting the market do its work. If you know that you could do stuff that actually could increase innovation directly eg enable more interdisciplinary communication or encourage experimentation.
I think the culture of an institution matters much more than you give it credit. Entire societies have cultures that either cause or retard scientific progress. To me it is basically true China, India are basically paper mills, and US is also on it's way there. No intellectual output of any worth comes from these countries (even though they are far more populous), sometimes you find brilliant people but they would succeed anywhere. The country if anything kills intellectual growth. On the other hand if you witnessed Japan a few decades ago, they were a hotbed of Scientific Progress (Video Games, Robotics, Cameras) Though it seems everything went downhill after 1990s recession. Now even US builds better robots than Japan. I don't doubt your analysis, it is reasonable, in my opinion it is ignoring some of the most important factors that it has no predictive power. I predict Quantum Computing will go the way of Nano Technology, vapourware. The recent Quantum Supremacy articles are positively mendacious, while. At least Peter Shors original papers were honest but a bit too speculative. Self driving cars are easily more than 10+ years away. Andrew Ng recently admitted AI won't take away Radiologist jobs, we should be pretty close to an AI winter if even vision is not as good as it was marketed to be. GPT is a useless toy and is definitely not in any sense a path to AGI and if their internal stories are to be believed, OpenAI seems to be in a lot of turmoil. The thread is same, a larger focus on marketing (mendacity), lesser on doing. Gregory Perelman may have too exacting standards for even 25years ago, but 50 years ago he would have accepted Fields or whatever medal they gave him. The fact that Mathematics, the most rigorous science is facing mendacity is a bad sign. I do think mendacity in Academia has grown over the last century, though Maths is relatively more immune to it than other fields. Physics has suffered the most as demonstrated by Vladen Koltun (Stanford prof) gave a talk on CMU AI Seminar (available on youtube) basically indicating how metrics like H-Index are being gamed out of existence at the expense of science. He suggested new metrics but he has missed the deeper issues. I don't have a strong position on the Woit-Motl debate but I see more and more people coming around to the fact that String Theory is useless and a dead end. How did so much time get wasted in it (if true), it's almost like Nanotechnology. I highly recommend reading this article :
I think what you're saying is def rings true, but I'm not sure that's the core reason behind said stagnation. Despite the insanities there has been progress in pockets, AI being one, or bio w vaccines and Crispr being another. The trick it seems is to find an actual society wide S curve that you could glom multiple ones onto. But that's just much harder without a) having enough sub fields that see strong growths, and b) the ability to pursue multiple combinatorial search patterns.
None of this is to even suggest that university systems work properly and is actively a hindrance. We can and should fix it and it will help us with the problem. I'm just showing, modeling rather, that what we should expect in terms of growth or progress itself needs to be understood in terms of its components, and that trajectory shouldn't be thought of as an exponential curve, with deviations due to our bad management.
True we've had some progress but doesn't your theory predict / assume progress should be more randomly distributed or weighted to population? Ignoring clusters of geniuses, shouldn't we expect China close to US while in reality US is well ahead of China. A large proportion of modern technologies came out of DARPA/ Bell Labs than rest of the institutions combined. Doesn't this show management is the most predictive criteria for progress
It's a good question, and will get to the geographic clusters and catch-up growth in a future article. Short version - not really. My combinatorial theory predicts how progress/ innovation happens at a macro level where different innovations can cross-fertilise, and this will happen in clusters (of geniuses and otherwise like geographically). However, now that information/ knowledge has started to spread more freely, and the world is coming together, we *do* see China catching up - first via catchup growth, and later via innovation (in AI etc.) ...
I think the reason is not lack of talent. I think the reason is that the Kafkaesque bureaucracy, untrusting or even hostile attitude of administrators (who are also other faculty, often from a deprived generation) and lack of systemic incentives and individual recognition for excellence, humiliates, depresses and eventually ruins the zest for research of all but a few thick-skinned faculty.
You can see evidence of this on the ground. IITs have been hiring people with great potential since about 2008 (since the 6th pay commission - roughly a year before this FAQ post started). Now would be a good time to take stock. How many of them have achieved their potential? And how many have degenerated into pade-shadows of their earlier selves, still talented but drained and defeated? I think a sadly large number are in the latter category.
Senior IISc Prof says
The anon posters above..please do not forget that things that Prof. Giri has said before here and Prof. Balaram put it eloquently "IIX has benign neglect for people who perform."
IIX is a government job. Nobody cares a damn whether you work or not work. They may be some minimum requirements for promotion like 5 papers in 5 years, 1 ph.d student etc. So, people work towards working for these requirements. Beyond that, do not expect to be appreciated if you publish 50 papers in 5 years. You will be treated the same.
Same number of students, same number of space, same money, same salary for everyone. We practice socialism but expect capitalism in output.
So, you might wonder why people talk about increasing research output and rankings etc. Then, please read this.
"One of my colleague once said that we talk about PhD program because it is in fashion to talk about research, and we need to justify not doing research. But we really don't want to admit more PhD students because we are afraid we will have to work harder. "
Take examples everywhere. Hard working professors in IIX work hard because they are self motivated. They will not get 1 rupee more either in research or personal money. They will not be given even one research student extra that prescribed maximum in the institute. No one (Expect their friends) will appreciate what they are doing.
This is reality. IF you do not know what is the reality of a government job, do not join. And repeat after me: IIX is a government job.
Senior prof in IISc.
"
This is nothing like the situation in US and I suspect very different from China, which is why my claim that management is key. Case in point Chinese Institutions had similar ranking to Indian Institutions in 2000, now all of them are in top 100, top 50 etc trying to compete with US. Not a single uni from India in top 100
Thanks for sharing! I feel I'm coming across as somehow eliding the fact that management doesn't matter as much, which is *not* what I'm saying at all. It is of course critical and we do have low hanging fruit from when we actively hinder our own efforts at innovating.
What I'm suggesting is that this institutional morass is not unique to this moment in time, and has been true throughout history, so cannot be the only rate limiting factor. If we fix it it will very definitely help however. I am also trying to change the frame from "if we get out of our own way through reducing bureaucracy etc we'll get exponential growth through innovation" to "to get exponential growth through innovation we need to be a little more mindful of how different sub-fields interact".
Your China/ India point is interesting, but there is the added factor there of catch-up growth. And it's less on institutional prestige, or even patents/papers, but actual breakthrough innovation - i.e., something equivalent to microchips. AI might get there, but the impact is tbd still.
Really enjoyed this article. I get that you're trying to stay at a high level, but I think there are at least a couple of areas that could be explored further. First is in tech revolutions that had massive, society-wide impacts, thinking industrialization, electrification, computing, & networks. The second is the massive extent of engineering & application developments predicated on scientific breakthroughs. There's a parallel but in many ways different story about how medical/health sciences have developed vis a vis industrialization/elec/comp/networking.
Thanks !! Both interesting thoughts, and both are perfect examples of the necessity of bringing multiple fields/ technologies/ innovations together to bring about progress.
I have a post in the works to look ahead, but a couple examples here include the varied combinations of our ability to process information beter (AI etc), better ability to manipulate matter (nanotech of varied forms), ability to affect biology (crispr etc) and seeing what the impacts of that's likely to be.
The big question is: What's the next massive scientific breakthrough that will fundamentally alter how society operates? & how close is it to happening? I don't have the answer - I just hope it happens in my lifetime. It might also be the massive spread of renewables & related technologies, which are necessary to just keep civilization going while also opening massive prospects for economic activity.
This is great. Brings together a lot of research. I think a paper I read on the "Step and Wait Model" would also fit here.
I would like the S-curve to be a bit more defined - it's originally (diffusion of innovations) a population prevalence curve for an innovation in equipment or practice in the technologies that are being reproduced (still used, not obsolescent). There is a related 'phenotype performance' curve that is made up of increments of improvement from combination of innovations, but it seems to me that fitting an S-curve to that is not so obvious, although no doubt there are good examples. But niches and the qualities needed in equipment for them are conceptual handles and there are a lot of valid levels of description.
Thanks Andrew! The S curve here is a bit of a fudge, to stand in for any function that has a gradual takeoff in the beginning and a plateau at the end - the exact function can be changed if needed. Phenotype performance curve is an interesting one in that it can also reduce for larger values of the X axis, but practically I think of that as the equivalent of a technology S curve becoming obsolete.
Which means there's definitely a lot further to take it, but the idea is to expound if the theoretical argument holds, and later to specify it with particular functional forms..
So what I got from this is that ideas and technology need space to have lots of sex in order to produce progress, so it's natural that intellectually prudish societies don't have much progress.
I'm left speechless at this synthesis!
Just wanted to say that this is a great post. I might appreciate it more than others because it aligns with my intuitions, but I found the argumentation very compelling.
Relatedly: I always thought a good share of the progress study people were going in the wrong direction, with a very strong focus on the industrial revolution and a "make innovation great again" outlook. Although there some good things have come out of that thought approach, I feel they are looking for a silver bullet that is just not there (or at least, not where they are looking for it). Do you agree? Or I am misinterpreting you?
Thank you very very much!!
I do like the theory quite a lot, and hope to continue to expand it. And yes, I do feel the focus is almost entirely to reduce the negative forces (change institutions or funding models) which as a theory makes me feel the basic assumption is that "if it wasn't for all these bureaucracy and red tape etc, we'd be innovating so fast!". It's also reductionist in the search for a silver bullet for sure - hence the screed on anti-regulation, anti-govt, anti-bureaucracy etc, as if the natural state re most innovations as de novo emanations of a great man theory.
I would like however to also figure out the why or how regarding how innovations actually do come about, rather that relying on just removing obstacles and letting the market do its work. If you know that you could do stuff that actually could increase innovation directly eg enable more interdisciplinary communication or encourage experimentation.
I think the culture of an institution matters much more than you give it credit. Entire societies have cultures that either cause or retard scientific progress. To me it is basically true China, India are basically paper mills, and US is also on it's way there. No intellectual output of any worth comes from these countries (even though they are far more populous), sometimes you find brilliant people but they would succeed anywhere. The country if anything kills intellectual growth. On the other hand if you witnessed Japan a few decades ago, they were a hotbed of Scientific Progress (Video Games, Robotics, Cameras) Though it seems everything went downhill after 1990s recession. Now even US builds better robots than Japan. I don't doubt your analysis, it is reasonable, in my opinion it is ignoring some of the most important factors that it has no predictive power. I predict Quantum Computing will go the way of Nano Technology, vapourware. The recent Quantum Supremacy articles are positively mendacious, while. At least Peter Shors original papers were honest but a bit too speculative. Self driving cars are easily more than 10+ years away. Andrew Ng recently admitted AI won't take away Radiologist jobs, we should be pretty close to an AI winter if even vision is not as good as it was marketed to be. GPT is a useless toy and is definitely not in any sense a path to AGI and if their internal stories are to be believed, OpenAI seems to be in a lot of turmoil. The thread is same, a larger focus on marketing (mendacity), lesser on doing. Gregory Perelman may have too exacting standards for even 25years ago, but 50 years ago he would have accepted Fields or whatever medal they gave him. The fact that Mathematics, the most rigorous science is facing mendacity is a bad sign. I do think mendacity in Academia has grown over the last century, though Maths is relatively more immune to it than other fields. Physics has suffered the most as demonstrated by Vladen Koltun (Stanford prof) gave a talk on CMU AI Seminar (available on youtube) basically indicating how metrics like H-Index are being gamed out of existence at the expense of science. He suggested new metrics but he has missed the deeper issues. I don't have a strong position on the Woit-Motl debate but I see more and more people coming around to the fact that String Theory is useless and a dead end. How did so much time get wasted in it (if true), it's almost like Nanotechnology. I highly recommend reading this article :
https://www.unqualified-reservations.org/2007/07/my-navrozov-moments/
The author in my opinion is basically right with the problem and the solution
Fascinating article, thanks for sharing!
I think what you're saying is def rings true, but I'm not sure that's the core reason behind said stagnation. Despite the insanities there has been progress in pockets, AI being one, or bio w vaccines and Crispr being another. The trick it seems is to find an actual society wide S curve that you could glom multiple ones onto. But that's just much harder without a) having enough sub fields that see strong growths, and b) the ability to pursue multiple combinatorial search patterns.
None of this is to even suggest that university systems work properly and is actively a hindrance. We can and should fix it and it will help us with the problem. I'm just showing, modeling rather, that what we should expect in terms of growth or progress itself needs to be understood in terms of its components, and that trajectory shouldn't be thought of as an exponential curve, with deviations due to our bad management.
True we've had some progress but doesn't your theory predict / assume progress should be more randomly distributed or weighted to population? Ignoring clusters of geniuses, shouldn't we expect China close to US while in reality US is well ahead of China. A large proportion of modern technologies came out of DARPA/ Bell Labs than rest of the institutions combined. Doesn't this show management is the most predictive criteria for progress
It's a good question, and will get to the geographic clusters and catch-up growth in a future article. Short version - not really. My combinatorial theory predicts how progress/ innovation happens at a macro level where different innovations can cross-fertilise, and this will happen in clusters (of geniuses and otherwise like geographically). However, now that information/ knowledge has started to spread more freely, and the world is coming together, we *do* see China catching up - first via catchup growth, and later via innovation (in AI etc.) ...
Blog quote you might find interesting regarding Indian Scientific Institutions Management from https://giridharmadras.blogspot.com/search?updated-max=2014-12-24T05:30:00%2B05:30&max-results=7&start=7&by-date=false:
"Ankur Kulkarni says
I think the reason is not lack of talent. I think the reason is that the Kafkaesque bureaucracy, untrusting or even hostile attitude of administrators (who are also other faculty, often from a deprived generation) and lack of systemic incentives and individual recognition for excellence, humiliates, depresses and eventually ruins the zest for research of all but a few thick-skinned faculty.
You can see evidence of this on the ground. IITs have been hiring people with great potential since about 2008 (since the 6th pay commission - roughly a year before this FAQ post started). Now would be a good time to take stock. How many of them have achieved their potential? And how many have degenerated into pade-shadows of their earlier selves, still talented but drained and defeated? I think a sadly large number are in the latter category.
Senior IISc Prof says
The anon posters above..please do not forget that things that Prof. Giri has said before here and Prof. Balaram put it eloquently "IIX has benign neglect for people who perform."
IIX is a government job. Nobody cares a damn whether you work or not work. They may be some minimum requirements for promotion like 5 papers in 5 years, 1 ph.d student etc. So, people work towards working for these requirements. Beyond that, do not expect to be appreciated if you publish 50 papers in 5 years. You will be treated the same.
Same number of students, same number of space, same money, same salary for everyone. We practice socialism but expect capitalism in output.
So, you might wonder why people talk about increasing research output and rankings etc. Then, please read this.
http://dsanghi.blogspot.in/2011_02_01_archive.html
"One of my colleague once said that we talk about PhD program because it is in fashion to talk about research, and we need to justify not doing research. But we really don't want to admit more PhD students because we are afraid we will have to work harder. "
Take examples everywhere. Hard working professors in IIX work hard because they are self motivated. They will not get 1 rupee more either in research or personal money. They will not be given even one research student extra that prescribed maximum in the institute. No one (Expect their friends) will appreciate what they are doing.
This is reality. IF you do not know what is the reality of a government job, do not join. And repeat after me: IIX is a government job.
Senior prof in IISc.
"
This is nothing like the situation in US and I suspect very different from China, which is why my claim that management is key. Case in point Chinese Institutions had similar ranking to Indian Institutions in 2000, now all of them are in top 100, top 50 etc trying to compete with US. Not a single uni from India in top 100
Thanks for sharing! I feel I'm coming across as somehow eliding the fact that management doesn't matter as much, which is *not* what I'm saying at all. It is of course critical and we do have low hanging fruit from when we actively hinder our own efforts at innovating.
What I'm suggesting is that this institutional morass is not unique to this moment in time, and has been true throughout history, so cannot be the only rate limiting factor. If we fix it it will very definitely help however. I am also trying to change the frame from "if we get out of our own way through reducing bureaucracy etc we'll get exponential growth through innovation" to "to get exponential growth through innovation we need to be a little more mindful of how different sub-fields interact".
Your China/ India point is interesting, but there is the added factor there of catch-up growth. And it's less on institutional prestige, or even patents/papers, but actual breakthrough innovation - i.e., something equivalent to microchips. AI might get there, but the impact is tbd still.
FYI, Bacon isn't from the 1200's
Great catch- thanks fixed!