Trend forecasting with Quant Semiotics

Quant Semiotics uses three network topology metrics to identify repeating network patterns within large volumes of web and social data to forecast real-world innovation trends.

Trend forecasting with Quant Semiotics


Quant Semiotics has some really useful analytical features that appear to have a degree of predictive value. Through experiment we've found that Quant Semiotics can identify repeating network patterns within large volumes of text data that appear to anticipate real-world change and innovation trends with a fair degree of accuracy over one year, two year and sometimes longer time horizons.

To run our trend forecasting experiments we used a huge database of web articles that cover 17 years of news reports from the Media and Market Research Industry. You can read the headline report for this study here.

We set out to develop a way to forecast the upward and downward movements of innovation trends within the Media and Market Research industry. We wanted to predict if an idea or concept, represented by a keyword, was going to get more important over time or whether it was going to fade from view. This is the kind of thing you'd want to do in any market sector or field of cultural interest.

We decided to use the frequency of web article appearances for a keyword within annual time periods as our 'dependent variable'. This means we used the number of times in a 12 month period an idea, concept or innovation is mentioned within a web article as the trend outcome we wanted to explain. This seemed like a reasonable way to define the 'success' or 'failure' of a trend over time because if people ain't talking about a thing, it just ain't a thing.

The network analysis component of Quant Semiotics employs a variety of algorithms to compute the topological significance of keywords within a dataset. After a lot of experimental trial-and-error we identified three network topology metrics that, in combination, appear to have a fair degree of predictive value. They are called Keyword Connectivity, Strategic Significance and Concept Centrality.


Keyword Connectivity

Keyword Connectivity measures the number of distinct keywords a specific keyword is connected to, which is defined as at least one co-occurrence within a web article (known technically as a link or edge). 

For instance, if 'Big Data' and 'Business Intelligence' co-occur within the same article a connection or link is established between the two keywords. When this happens at scale across thousands of news articles, co-occurrences become statistically significant and meaningful. 

Keyword Connectivity tells us important things about the status of a keyword within market discourse. The number of connections a keyword develops tends to rise simply by appearing in a growing number of articles. The more web articles a keyword features in the broader the range of other keywords it's likely to co-occur with. 

However, Keyword Connectivity does far more than just mirror frequency of article appearances. Things get more interesting when we find keywords co-occurring more often than the average would allow for. Quant Semiotics uses these statistical outliers to establish a shortlist of structurally significant keywords that characterise a market sector or field of interest.

Like a young plant pushing out stronger roots into the soil, trending keywords expand their connections to other significant keywords that already shape market debate and conversation. In many cases a growing Keyword Connectivity score precedes future increases in web article appearances for a new idea or innovation.


Strategic Significance

The Strategic Significance metric identifies strategically connected keywords that have important 'acquaintances'. This doesn't just mean keywords that are well connected or closely connected to lots of other keywords. It also means keywords that are connected to other terms that are strategically important. In many respects Strategic Significance gives us a ‘market prestige' score for a keyword. 

When a keyword (perhaps representing a new concept or market innovation) begins to co-occur in news articles with high-prestige keywords its Strategic Significance score will rise. This tends to indicate the growing usefulness or importance of the keyword and the ideas it represents. Keywords with the highest Strategic Significance tend to represents concepts that either define the structure or influence the development of market discourse.

As with Keyword Connectivity, when we see a rising Strategic Significance score it can foreshadow a rise in web article mentions for a specific concept or innovation.


Concept Centrality

Concept Centrality is a metric that tells us how indispensable a keyword is (and therefore the concept or idea it represents) to the market discourse it forms a part of. 

From a technical point of view it measures how often a specific keyword lies on the shortest network path between two keywords as compared to all possible shortest paths in the total network of keywords. 

In plainer language that means Concept Centrality identifies important 'junctions' or 'bottlenecks' within language network structures that play an essential role in shaping and defining the core characteristics of a market topic or theme.

When a keyword's Concept Centrality scores rises it tends to mean the idea represented is becoming more central to debate and that other ideas or themes in the market are realigning themselves around it. 

Conversely, when Concept Centrality scores fall it tends to signify decline in a keyword's market salience or relevance. This makes it a very useful predictor of when downward trends in popularity are likely to set in for specific ideas.

To talk about applying Quant Semiotics to your brand or market contact us now.

Mark Uttley
Group Strategy Director, AKQA, San Francisco, US

"Colourtext always take a fresh approach to discovering hidden patterns in data. Whether you want to conduct segmentation or mine your CRM for deep insights, they will help you see your customers , and what they are seeing , in new ways."

Joaquim Bretcha
International Director of Netquest and President of ESOMAR, Barcelona, Catalonia, Spain

"Colourtext and Netquest have partnered on different occasions to showcase the power of the digital behavioural data extracted from Netquest’s consumer panel. Colourtext has not only the expertise of dealing with big datasets but also has brilliant creativity to make the most sense out of them. Working with Colourtext is a rich, agreeable and easy going experience"

Mark Barber MBE
Planning Director, RadioCentre, London, UK

"More than a decade of success for Radiog auge across the world is a testament to Colourtext’s experience and perseverance. There are many obstacles to creating a global currency for measuring radio advertising effectiveness - Colourtext has managed to tackle all of them."

"It’s rare to find research companies that really understand the unique radio / listener relationship and what makes radio advertising work. With experience of researching the medium on both sides of the fence, Colourtext is one of those rare companies."

Simone Lawrence
Director of Insights, Canadian Broadcast Sales, Toronto, Canada

Colourtext has an agile approach to research that ensures a successful outcome to every challenge CBS gives them. The depth of experience and understanding the Colourtext team brings to each study demonstrates their value in the audio space. We have a very positive nine year working relationship behind us that has created significant value for CBS and our clients. We remain committed to working with Colourtext in the coming years.

Jon Puleston
Vice President of Innovation, Kantar Profiles, London, UK

It's rare to find a partner with the mix of analytical, technological and creative skills offered by Colourtext. Jason and his team seem to roam around freely in data, find the stories that matter and bring a distinctive aesthetic vision to the presentation of information.