This article is structured as follows: First, I will explain what I mean by “estimating tech maturity”, and why I think you might care. Then, I will discuss the difficulties that are associated with estimating tech maturity, or technology readiness level. After that, I will explain why and how data can help. Finally, I will show you a back-of-the-envelope, data-based method that you can use for estimating tech maturity.
What do you mean by “tech maturity”, and why should I care?
By “tech maturity”, I mean whether a technology is still in an early research stage, or if it has become a commodity product already. Or if it’s anything in between. For example, I think that asteroid mining is closer to the “research” stage. By contrast, smartwatches are something you can buy in a store.
There are many different reasons why you might want to estimate technology maturity levels. Let’s take product design. For instance, let’s say you want to make a smartwatch with the computing power of a powerful gaming laptop. Compared to current smartwatches (it’s 2020), this would be quite something. But could you do this? Can you get small, low-energy processors that give you this kind of computing power? In other words, what’s the maturity level of such processors? If the answer to this question is “research stage”, you might want to reconsider your smartwatch plans.
The Technology Readiness Level scale
In the 1970s, NASA was making plans for space stations and other things (the space shuttle was well under way by then already). In order to evaluate whether a given technology was ready to be used in spaceflight, NASA developed a scale called Technology Readiness Level, or TRL ( here is a detailed description of the scale).
In the TRL scale, “TRL 9” means “yes, has been flown in space already”. Here, I will use the terms “TRL” and “tech maturity level” to mean the same thing.
By now, many other organizations have adopted the TRL scale, or some version of it. These organizations include the European Space Agency, the European Union’s Horizon 2020 R&D program, Google X, and SpaceX, for example.
So, we’ve got a scale. But we have to handle the scale with care. Doing tech maturity assessments is not easy.
Why estimating tech maturity is difficult
There are a number of things that make estimating TRLs difficult. Here are some of them.
TRL estimates can seem subjective
Consider the quote below from a paper by Alison Olechowski, Steven Eppinger, Nitin Joglekar, and Katharina Tomaschek. In the paper, they investigated how TRLs are being used in practice.
Some TRL users stated that the scale can sometimes seem subjective rather than objective, and that power and influence bias the ultimate assignment of TRLs.
From Olechowski, AL, Eppinger,SD, Joglekar, N & Tomaschek, K (2020). Technology readiness levels: Shortcomings and improvement opportunities. Systems Engineering, 23(4), 395–408.
Clearly, if the outcomes of TRL estimates are determined by the most expensive suit in the room, this is a problem.
The scope of the TRL assessment can be difficult to define
Here is what I mean by “scope”: Let’s say you want to estimate the technology maturity level of hydrogen-powered cars. Of course, a hydrogen-powered car is not one technology, but many. For example, you need an engine, hydrogen storage, and perhaps you might even include the refilling infrastructure, or hydrogen production. And each of these technologies in turn consists of many other technologies. Where do you draw the line? What do you include in your TRL estimate? And do you pack all of this into one maturity level score? Or would it be better to estimate some type of compound TRL score? This is what I mean by “scope”.
Tech maturity level depends on the context
Let’s say you want to estimate the TRL of a LiDAR sensor. You can’t do this without considering your context. For example, if you want to use the LiDAR sensor in an iPad, it would not be nice if the sensor didn’t work with perfect reliability. But it would probably not be dangerous. By contrast, this is different if you want to put the LiDAR sensor into a driver assistance system, for example. In this case, the sensor would have to be reliable under all kinds of conditions (weather, etc.). So the same sensor may get a “TRL 9” for the iPad, but a lower TRL for the driver assistance system.
Let’s make things even more complicated. What if we wanted to use an iPad with a LiDAR sensor to scan the pressure chamber of a nuclear reactor? We’d really, really want those pressure chamber measurements to be correct. Perhaps even more so than for the driver assistance system.
So, “context” is not really another technology or product. Instead, “context” is a user story. By the way, at Mergeflow we think that user stories are essential to tech discovery and innovation activities generally. We talk about this in Point 3 of our article, “5 ways to escape from innovation theater”.
When you do tech maturity estimates, try to do so in the context of a user story.
So there is a lot to consider before pinning a TRL number to a technology.
“But management wants a number!”
Yes, I know. But consider this: Think of a car cockpit. It has all kinds of indicators. For speed, oil temperature, perhaps wheel air pressure, whether the high beam lights are on, and several more. Would you want to replace all these indicators by one single “carness indicator”? Probably not. We’ll just have to live with some degree of complexity, ambiguity, and uncertainty.
(Just to be clear, the “carness indicator” was not my idea. It was Andrew McAfee ‘s, who used it in an interview that I heard, in a different context.)
But this does not mean that we should throw up our hands. Next, let’s see why and how using data can help us estimate technology maturity. Even if we can’t get a simple TRL number, using data makes it easier to at least define a corridor for reasonable numbers with some confidence.
Advantages of using data for tech maturity estimates
Here are some reasons why data help with assessing tech maturity.
Data make TRL estimates more objective
Above I pointed out the subjectivity problem. If you use data, you can make the TRL discussion more fact-based. Notice that this does not mean that you should have no discussions. It simply means that the anchor point for the TRL discussion can move away from power structures to something more rational.
The TRL estimation process becomes more scalable
Discussions are not scalable. You can’t modify some parameters, re-run a discussion a thousand times, and check how the parameter changes affect the outcomes. With data, you can do this. For example, you can use techniques such as Monte Carlo experiments.
Data give you global reach
Unlike discussion participants, data have no time zones. They don’t sleep, and they are available from anywhere. As long as you have an internet connection.
What kinds of data can we use for tech maturity assessments?
Let’s re-phrase the question: Depending on the maturity level of a technology, in what kinds of data would we expect the technology to show up? In other words, what are typical kinds of data that we’d associate with a mature technology vs. a research-stage technology?
For a mature technology, we’d expect to see mainstream news, (venture capital) investments, and market analyses, for example. By contrast, for a research-stage technology, we’d expect to see scientific publications but not much else. Perhaps some blogs that speculate on future scenarios.
There’s more data we can use. Patents, for example, or research fundings, technology licensing, and clinical trials. Let’s look at what each of these kinds of data could tell us about technology readiness level. In addition to using these data for maturity assessments of known technologies, you can also use them for discovering new, potentially game-changing innovations.
But first, let’s simplify the TRL scale for our purposes here. And let’s add another stage, the “Hype” stage. By “hype”, I mean something that receives lots of attention, and perhaps even investment, but that has no science to back it up. The “Hype” stage is important, I think. After all, you really don’t want to bet on the next Theranos. Theranos was a biotech company that got lots of attention and some serious investments but that turned out to be a scientific fraud. Admittedly, an extreme case of hype.
Now, let’s map different kinds of data to tech maturity level.
Venture capital investments
Venture capitalists almost never invest in basic research technologies. Generally, if a technology shows up in venture capital investments, this is probably at least an Intermediate technology. In some cases it may also be a Mature technology. Think of venture investors who basically copy a business model from one geography to another, for example.
Market analyses
If a technology makes it into a market analysis, this technology is usually Mature, or at least Intermediate.
Mainstream industry news
Of course, mainstream industry news report on technologies at all kinds of maturity stages. But in most cases, if there is more or less consistent reporting on a technology, and if this reporting is in the context of big public corporations, the technology tends to be Mature or at least Intermediate.
Tech blogs
Tech blogs report on technologies at various levels. But if a technology shows up in tech blogs and nowhere else, or perhaps only in venture capital investments and tech blogs, this can indicate Hype stage.
Scientific publications
While there is R&D on technologies at all levels, if a technology is featured in scientific publications but nowhere else, this usually indicates a Research-Stage or perhaps Intermediate technology.
Patents
The reason for patenting a technology is to secure freedom to operate, i.e. freedom to commercialize the technology. This usually only makes sense once a technology has at least Intermediate maturity.
Technology licensing
“Technology licensing” refers to technologies that universities and R&D organizations make available for commercialization. This typically makes most sense for Intermediate technologies.
Funded research projects
If publicly funded research projects went into Mature technologies, it would not be research funding but subsidizing. So, I’d argue that funded research projects typically indicate a Research-Stage or Intermediate maturity stage.
Clinical trials
Clinical trials are quite convenient for tech maturity assessments. As we describe in the Technical FAQ for our tech discovery software, clinical trials are labeled with phases. I’d argue that the “Preclinical” phase and Phases 1–3 correspond to Intermediate maturity stage, and that Phase 4 corresponds to Mature.
Data-supported tech maturity assessments are a first useful approximation, not mathematical axioms
Now, these proposed alignments of “kinds of data” and “tech maturity level” are approximations, and they depend on the context of an individual situation. But there is research that aims for a more rigorous understanding of these relationships between data signals and tech maturity levels. For example, Safa Faidi and Alison Olechowski (I mentioned Alison above already) from the University of Toronto, “Identifying gaps in automating the assessment of technology readiness levels”. For their research, Safa and Alison used data from Mergeflow’s API to test how various kinds of data could support tech maturity estimates.
So, yes, there are some caveats here. But as we already said elsewhere, tech discovery is more about getting a first useful approximation. Keeping this in mind, next, I’d like to introduce to you a simple, data-supported method for tech maturity estimation.
A back-of-the-envelope, data-supported method for estimating tech maturity levels
The idea behind back-of-the-envelope estimates is to use relatively simple methods to get plausible, approximate answers quickly. In another article, we used this method for checking the plausibility of market estimates.
Below is how I would do a quick tech maturity level estimate. Just a quick note on what I mean by “is present” below: this is not easy to quantify. But I’d argue that “1 item” in a category is not enough. Still, going “from zero to one” in a category is interesting. For example, when you see the first venture investment in a topic, you should probably start paying attention. This may signal the beginning of a “phase shift” into a higher maturity level.
Mature
If your topic is present in at least two of...
- Venture capital investments
- Market estimates
- Mainstream industry news
- Clinical trials, Phase 4
If your topic is in none of the data sets above, move to Intermediate.Intermediate
If your topic is present in at least one of...
- Venture capital investments
- Technology licensing
- Patents
If your topic is in none of these data sets, move on to Research-Stage.Research-Stage
If your topic is present in one or both of...
- Scientific publications
- Funded research projects
...but nowhere else.Hype
(we added this level)
If your topic is present in one or both of...
- Venture capital investments
- Technology blogs
...but nowhere else (particularly not in scientific publications).
That’s it, really. For the method above, we have also made a checklist. You can see the checklist in our Technical FAQ. And in our article on “green hydrogen”, you can see how we apply the method to various technologies for making green hydrogen.
Would you like to try out Mergeflow and do a tech maturity assessment of your topics? Sign up for a free trial by clicking here.
Featured image by SpaceX. Landing rockets retropulsively on Earth was long thought to be impossible. Within a few years, SpaceX pushed the technology to TRL 9.
Originally published at https://scope.mergeflow.com on September 15, 2020.
