Stock and Futures Sentiment from Social Media and News Analysis

By Bill Dennis, CEO of EOTPRO Developments Inc.


This article will discuss social media and news analytics sentiment for short term intraday trading.  We will outline our experience surrounding both data sets and highlight some common challenges that users may experience when incorporating this data into their workflow. Finally, we will explain how we have solved some of the challenges and used these analytics in our own applications.



In an effort to find an informational edge, increasing resources are being devoted by investors and traders to analyse non-traditional sources of data as an investment decision support tool. To help create this edge we (EOTPRO) build intraday trading strategies and analytical techniques targeted towards prop shops and institutional customers interested in short term, intraday trading opportunities. Having analyzed data from several other sentiment firms, we have had our best outcomes using news analytics from Thomson Reuters and social media sentiment from Social Market Analytics (SMA). We have concluded that there is alpha in both data sets.

There are advantages to both sources. On the news side, the stories benefit from vetting by professional reporters. Social media are a primary source for journalists themselves, so that using Twitter may pick up news items/sentiment quicker. A second advantage in the use of social media analytics is that, as the newer set of data, Twitter feeds and the like are only just beginning to get the attention of large institutions.


Where is the edge now?

EOTPRO, over the last few years, has built several hundred strategies based on each analytic set.

Right now, in our opinion, social media sentiment seems to have an edge over news analytics in terms of profitability. The tricky part when looking at social media is how to filter out the irrelevant tweets and only look at financially relevant tweets. As a consequence, most companies in this space use complicated algorithms to identify relevant tweets. The general experience is that these complicated filtering algorithms  don’t work.

Not only is effective filtering difficult, but we have found that some data can actually worsen the performance of a basic strategy, instead of improving it. Therefore, in working with social media as a data source it is a key task to identify which part(s) of the available data to use in building models to capture sentiment change.

Most of the work EOTPRO has done in this area has been applied to trading models for futures and common shares. We will now look at each in turn.



We have found that trading signals generated from using social media sentiment indices can provide an effective directional forecast for the most liquid e-mini futures.

We started off by building a market cap-weighted index aggregation of sentiment. Creating an aggregation for an index is typically known as a roll-up. The stock weighting in the roll-up is equal to the stock’s market cap weighting in the index. For example, one of the largest cap stocks like AAPL carries one of the largest weights in the roll-up.  In use we have found that looking at market cap weighted roll-ups of the indices can be misleading.

The problem we experienced when looking at the Russell 2000 or the SP500 for example, is that the roll up then becomes a non-responsive moving average of all sentiment. This works if you are looking for an overall bias for a high time frame chart, however, this is not an effective technique when looking to trade intraday or on short time frames. The problem is that market cap weighted roll-ups act like a moving average, thereby  smoothing out the full market impact of events that move markets in shorter time-frames.

When a big news/social media event hits an individual stock it will typically not be seen in an index roll-up indicator. This will tend to apply to most of the models developed by news and social media analytics companies, as it is common to use moving averages to smooth sentiment out. As ever, the models work best with specific purposes in mind, and models developed for investment time-frames will be sub-optimal when applied to trader time-frames.

This has been challenging and has led us to develop techniques to address these issues. We have found that the techniques we have developed have been more responsive and a much better way to trade index futures and ETF’s than a traditional market cap based roll-up approach..



Looking at social media sentiment on each stock can also give a statistically significant edge to trading. The typical approach has been to look for the highest sentiment score and hope for a big move to follow, however, we have found alternative ways to look at sentiment rather than just looking at extremes.

We have found that delta changes in sentiment are much more effective than extremes or moving averages of sentiment for timing trades. It is important to note that we are looking at intraday and not at multi-day strategies. It is very interesting to look at large changes in sentiment for example, when an instrument displays negative sentiment then abruptly changes to positive sentiment in one big move, as can be seen in the AAPL and Gold (GC) examples below:

In one case (AAPL) the change in social media sentiment allowed a trade to be put on and held through a subsequent price test. In the case of gold the intraday low in price was confirmed by a washout in social media sentiment.


Hedge funds and social media sentiment

American hedge funds with large and/or diverse trading teams (like Millennium Capital Partners) have been using news analytics as a trading input for some years. They have recently adopted social media sentiment into their workflow and have started to compare the results of using the two data sources. Consequently, some have decided to discard news analytics in preference for social media sentiment tools.

The reasons for working with one source over another will vary with the specific experience of the hedge funds. What is becoming clear is that it is not just the potential for alpha generation that is important – the  risk control around the alpha source is a significant factor in how widely these analytical tools are applied.  It would seem that avoiding news and social media whipsaw is on the top of the list.

Risk control analytics has led us to seek out new partners in the field and we are now working with the newest machine learning techniques based loosely on the teachings of Michael Ng at Stanford University. The purpose of this research is to accurately predict the amount of price movement an equity is likely to experience when a certain level of sentiment is generated, which will ultimately feed into money management algorithms.


Challenging data

Sentiment data can be very challenging to understand. It has been our experience that some of the most intuitive ways to look at data usually do not work and adding context to sentiment indicators can be extremely challenging. For this reason we have developed applications to allow those who are new to using sentiment analytics to get quickly up the learning curve.

EOTPRO has solved the issue of context by building a copyrighted HTML5 application targeted toward firms interested in looking at sentiment analytics but that may not have the resources or the time to integrate via an API. The second group of target clients are firms wishing to use a “turn-key” application for risk control and alpha gen. The customer simply logs into a web portal with a username and password.

The EOTPRO web portal capabilities include:

1. Real time data to visualize social media sentiment and tweet volume on 1000’s of positions

2. Real time portfolio risk control alerts on unlimited number of positions

3. Real time charting with live equity data with sentiment indicators built in on over 2700 US stocks

5  Real time news headlines on each stock

6  Real time StockTwits data on each stock

7  Real time fundamental data built in on each stock

8. Over 60 standard technical analysis indicators built into charts, plus the ability to overlay symbols for pairs trading.


These capabilities bring intuitive visualizations and the ability to allow the user to confirm social media  sentiment prediction by displaying headlines, social media tweets and indicators all on one page.

In this example, the webportal shows a high sentiment positive event occurring on FCX (shown by the green bubble). The software makes it easy to spot where attention should be focused. The user can then put the sentiment-based trigger into context by looking at live charts, news and headlines, and stocktwits data on the equity. The analytical capabilities of the software need not be free standing – the HTML5 coding  means the sentiment insights can be viewed embedded in the trader’s core systems, or even on mobile devices. That is the sentiment input can be a component of whatever set-up (like Bloomberg) the trader has habitually used.


Scalability and risk control

The system is very scalable. Whether you are a quant with a 5,000 instrument global universe or a large cap manager looking at the Russell 1000. Each user can store up to 10 portfolios specific to them with unlimited stocks in each portfolio. It is as easy as dragging and dropping a CSV file into the bubble chart to upload a portfolio.

After a portfolio name has been entered, each stock in the portfolio is represented as a bubble in the chart. Each bubble is the current sentiment of the equity, and each bubble is sized by tweet volume, whilst the color reflects the type sentiment.

If a risk event is detected the bubble will move up to the upper quadrants indicating that a sentiment high has just occurred. As the delta can be just as insightful as the extreme movements, the event will also be reflected in the delta change line below the bubbles. The user can be alerted visually, by audio and by email as they prefer. This feature is attracting a lot of attention from managers with quant processes who look to exclude stocks with event risk from their active universe.

In this example, our web portal detected a negative event on the shares of Lorillard Inc.(LO), an SP500 index constituent. The user was alerted to this event and could have taken appropriate action to hedge or short the position for alpha generation.

LO after the risk event was detected: