COMPOSITE MACRO ETF WEEKLY ANALYTICS (9/28/2015)

/

COMPOSITE ETF CUMULATIVE RETURN MOMENTUM

These charts show the sum (cumulative) of the daily returns of each composite ETF over the specified period. The daily return is calculated as the log of the percent change between daily adjusted close prices.

These charts help determine asset class return momentum. This is important because momentum is arguably the strongest and most persistent market anomaly. Poorly performing asset classes are likely to continue under performing while outperforming asset classes are likely to continue their relative strength. 

LAST 63 TRADING DAYS

composite_ETF_barplot_cumlrets_L63 Days_[Wednesday, 07.01.2015 - Monday, 09.28.2015].png

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

COMPOSITE ETF CUMULATIVE RETURN (BEST VS WORST VS BENCHMARK)

These charts visualize the cumulative return performance of the best and worst performing asset classes over the specified period. These best and worst asset classes are then compared to a benchmark ETF composite represented by the Large Cap category. 

These charts help give investors an idea of how an actual investment in the represented asset classes would have performed over the period in percentage terms. This also helps visualize the relative strength or weakness of various asset classes as compared to the most common Large Cap benchmarks.

LAST 63 TRADING DAYS

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

COMPOSITE ETF Z-SCORES OF AVERAGE ROLLING RISK ADJUSTED RETURNS

These charts show the z-scored average of the composite ETF's rolling risk adjusted returns.  Risk adjusted returns are used to improve the robustness of the chart and the information presented. 

By examining the standardized values we can see how each asset class performed relative to the group. This adds further clarity to the relative strength (weakness) of asset class return performance. 

LAST 63 TRADING DAYS; ROLLING PERIOD = 10 DAYS

LAST 21 TRADING DAYS; ROLLING PERIOD = 5 DAYS

LAST 10 TRADING DAYS; ROLLING PERIOD = 5 DAYS

COMPOSITE ETF RISK-ADJUSTED RETURN CORRELATIONS HEATMAP CLUSTERPLOTS

These charts visualize the correlations of the asset class returns over the specified period. Red indicates highly correlated returns while blue indicates negatively correlated returns. 

By examining a clustered heatmap investors are able to quickly determine the intensity and grouping of asset return correlations. Generally speaking, investors should seek to diversify their portfolios by holding uncorrelated assets. Better diversification among asset classes helps to lower overall portfolio volatility with the implication of improving a portfolio's long term performance. 

LAST 63 TRADING DAYS

Composite ETF Correlation ClusterMap L63Days.png

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

All data sourced from Yahoo Finance API

How to get Free Intraday Stock Data with Python and BarCharts OnDemand API

/

To this day the most popular article I have ever written on this blog was "How to get Free Intraday Stock Data with Netfonds". Unfortunately the Netfonds API has really declined in terms of usability, with too many popular stocks missing, and irregular trade and price quotes. Simply put, as the API went down, so did the code.

However, all hope is not lost. The wonderful people at BarChart.com have created a well documented, easily accessible API for intraday stock data and even near real-time quote access. The only caveat is that you must request access to get a personal API key. Again this is FREE, and the process is extremely simple and straightforward. I think I received my API key within the same day, max 24 hours. 

Step 1: Go to http://www.barchartondemand.com/api.php and request an API key. 

Step 2: Use or modify my code to get FREE intraday stock data. 

Something to note, in this example I use the SP500 components as my list of stock symbols. I covered how to get fresh SPY holdings data directly from the provider in a previous post titled "GET FREE FINANCIAL DATA W/ PYTHON (STATE STREET ETF HOLDINGS - SPY)".  Now onto the code...

First I import the necessary modules.

# -*- coding: utf-8 -*-
import time
t0 = time.clock()

import pandas as pd
from pandas.tseries.offsets import BDay
import numpy as np
import datetime as dt
from copy import copy
import warnings
warnings.filterwarnings('ignore',category=pd.io.pytables.PerformanceWarning)

Next I set up what I refer to as a 'datetime management' section of my code. I do this for ALL my time series analysis as a convenient way to standardize my code across projects. Sometimes I only use one of the variables, as I do in this case, but it's so convenient when doing any sort of exploratory analysis with time series. I also do the same for my filepaths.

# ================================================================== #
# datetime management

d = dt.date.today()
# ---------- Days ---------- 
l10 = d - 10 * BDay()
l21 = d - 21 * BDay()
l63 = d - 63 * BDay()
l252 = d - 252 * BDay()
# ---------- Years ---------- 
l252_x2 = d - 252 * 2 * BDay() 
l252_x3 = d - 252 * 3 * BDay() 
l252_x5 = d - 252 * 5 * BDay()
l252_x7 = d - 252 * 7 * BDay() 
l252_x10 = d - 252 * 10 * BDay() 
l252_x20 = d - 252 * 20 * BDay() 
l252_x25 = d - 252 * 25 * BDay()

# ================================================================== #
# filepath management

project_dir = r'D:\\' 
price_path = project_dir + r'Stock_Price_Data\\'

Next I set up a convenience function for creating the BarChart url to access the API. 

# ================================================================== #
apikey = 'insert_your_api_key'
def construct_barChart_url(sym, start_date, freq, api_key=apikey):
    '''Function to construct barchart api url'''
    
    url = 'http://marketdata.websol.barchart.com/getHistory.csv?' +\
            'key={}&symbol={}&type={}&startDate={}'.format(api_key, sym, freq, start_date)
    return url

Now for the fun part. I create a function that does the following:

  1. initializes an empty dictionary and the minimum required API variables,
  2. iterates through my list of SP500 stocks,
  3. constructs the proper API url,
  4. reads the data returned by the db as a csv file conveniently making use of Pandas read_csv function.
  5. adds the price dataframe to the dictionary dynamically
  6. converts the python dictionary into a Pandas Panel and returns the Panel
def get_minute_data():
    '''Function to Retrieve <= 3 months of minute data for SP500 components'''
    
    # This is the required format for datetimes to access the API
    # You could make a function to translate datetime to this format
    start = '20150831000000'
    #end = d
    freq = 'minutes'    
    prices = {}
    symbol_count = len(syms)
    N = copy(symbol_count)
    try:
        for i, sym in enumerate(syms, start=1):
            api_url = construct_barChart_url(sym, start, freq, api_key=apikey)
            try:
                csvfile = pd.read_csv(api_url, parse_dates=['timestamp'])
                csvfile.set_index('timestamp', inplace=True)
                prices[sym] = csvfile
            except:
                continue
            N -= 1
            pct_total_left = (N/symbol_count)
            print('{}..[done] | {} of {} symbols collected | percent remaining: {:>.2%}'.format(\
                                                                sym, i, symbol_count, pct_total_left)) 
    except Exception as e: 
        print(e)
    finally:
        pass
    px = pd.Panel.from_dict(prices)

    return px

Now I import our list of stock symbols, make some minor formatting edits and run the code.

# ================================================================== #

# header=3 to skip unnecesary file metadata included by State Street    
spy_components = pd.read_excel(project_dir +\
                             '_SPDR_holdings/holdings-spy.xls', header=3)
syms = spy_components.Identifier.dropna()
syms = syms.drop(syms.index[-1]).order()

pxx = get_minute_data()

This script takes roughly 40 minutes to run, longer if you try to get the full 3 months they provide, less if you need less data. 

Now let's test our output to make sure we got what we expected. 

print(pxx)
print(pxx['AAL'].tail())
print(pxx['ZTS'].tail())

The code ran correctly it appears, and the output is what we expected. One thing you may have noticed is that time stamps are not 'EST'. If you want to convert them use the following one liner. 

# convert timestamps to EST
pxx.major_axis = pxx.major_axis.tz_localize('utc').tz_convert('US/Eastern')

There is one last consideration that is easy to overlook if you're unfamiliar with some of the technical challenges of 'big data'. When you first run a script like this it is tempting to use the usual storage techniques that pandas provides such as 'pd.to_csv()' or 'pd.to_excel()'. However, consider the volume of data we just collected: 502 (items) x 5866 (major_axis) x 7 (minor_axis) = 20,613,124. 

Look at it again and consider this simple code collected over 20 million data points! I ran into trouble with Python/Excel I/O with only 3.5 million data points in the past. Meaning importing and exporting the data took minutes. That's a serious hangup for any type of exploratory research, especially if you plan on sharing and/or collaborating using this dataset. 

Pandas HDF5 file storage format to the rescue! Feel free to investigate the power, speed and scalability of HDF5 via the Pandas docs or any of the numerous quality blogs out there accessible by a google search. Needless to say, I/O was reduced from several minutes both ways to seconds. Here is the code I used to store the panel.  

try:
    store = pd.HDFStore(price_path + 'Minute_Symbol_Data.h5')
    store['minute_prices'] = pxx
    store.close()
except Exception as e:
    print(e)
finally:
    pass

Here's a sample plot with the intraday data. 

get_free_intraday_barcharts_api_AAL.png

The entire code is posted below using Gist.

Guest Post Previously Featured on RectitudeMarket.com (09/02/2015)

/

**Note: This post already appeared as a guest post on rectitudemarket.com. The reason I'm posting this article when it is 'outdated', is twofold. 1) I think it's beneficial to review previous works especially when one has the benefit of hindsight. This helps us determine the accuracy and bias of the research presented. 2) I further introduce the concept of conditional cumulative returns, which adds insight to what happens to our securities' returns given some other event occurring. In this case, the event is simply whether our benchmark ETF's cumulative returns are rising or falling during some period. 

Are There Any Equity Industry Groups Outperforming During this Recent Market Volatility?

Stock market volatility has picked up in a big way. It has been some time since market participants have experienced consecutive 2-4% up/down moves in the major market averages. The selling has been broad based and market sentiment has deteriorated as well.

With this backdrop I wondered if any industry groups were doing well. By doing well I mean 1) relative outperformance when compared to other industries 2) absolute performance greater than zero.

To answer this question I collected all the NYSE and Nasdaq symbols (~3300 symbols), filtered them for firms with a market cap greater than $300 million then I grouped them by industry. I dropped any industries with less than 10 stock symbols. With the filtered dataset I ran the analysis.

First I examined the cumulative return momentum over various lookback periods. Then I examined which industry groups had the best cumulative return performance given a benchmark ETF’s returns are rising (declining).

The benchmark ETF’s I selected for comparison are (SPY, QQQ, VTI). 

Unfortunately there are no industry groups with absolute cumulative return performance greater than zero over the last quarterly period (Last 63 Trading Days).

In fact you'd have to back out to view industry group performance Year-to-Date, to find any industry groups with positive cumulative returns. Those industries are: Biotechnology: Commercial Physical & Biological Research, Medical/Nursing Services, Forest Products,  and Movies/Entertainment.

To examine relative conditional performance, I selected the top 5% of industry group cumulative return performance given increasing benchmark returns (Last 63 Trading Days)

Focusing on SPY, the strongest performers given increasing returns have been: Finance: Consumer Services, Investment Bankers/Brokers/Service, Investment Managers, and Clothing/Shoe/Accessory Stores. Clearly financial firms’ returns are highly sensitive to the performance of SPY.  Somewhat surprisingly Clothing/Shoe/Accessory Stores have outperformed given increasing returns in all 3 benchmark ETFs.

For long only investors, unfortunately there are no industries which have recorded positive absolute cumulative return performance given a decline in benchmark returns. The best one can hope for is to find industries that decline less than peers during periods of negative market returns. 

During the last quarter (Last 63 Trading Days) the best industry performance given declining returns in SPY are: Life Insurance, Auto Parts 0.E.M., Computer Software: Prepackaged Software, and Business Services. I find it surprising that Business Services would be a relative outperformer for all 3 benchmark ETFs.

Another item of note is precious metals being a relative outperformer if VTI returns are declining. I think this makes sense given that VTI is representative of the entire domestic equity market, therefore negative returns in this index is more likely to induce catastrophe hedges and/or ‘panic trades’.

If your portfolio has been hit during this period of heightened volatility take solace in knowing you are not alone. No industry is safe. However, all is not lost, by creating a shopping list of your favorite quality names now on discount, you can be ready to strike with conviction when the proper opportunity presents itself.

COMPOSITE MACRO ETF WEEKLY ANALYTICS (9/20/2015)

/

COMPOSITE ETF CUMULATIVE RETURN MOMENTUM

These charts show the sum (cumulative) of the daily returns of each composite ETF over the specified period. The daily return is calculated as the log of the percent change between daily adjusted close prices.

These charts help determine asset class return momentum. This is important because momentum is arguably the strongest and most persistent market anomaly. Poorly performing asset classes are likely to continue under performing while outperforming asset classes are likely to continue their relative strength. 

LAST 63 TRADING DAYS

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

COMPOSITE ETF CUMULATIVE RETURN (BEST VS WORST VS BENCHMARK)

These charts visualize the cumulative return performance of the best and worst performing asset classes over the specified period. These best and worst asset classes are then compared to a benchmark ETF composite represented by the Large Cap category. 

These charts help give investors an idea of how an actual investment in the represented asset classes would have performed over the period in percentage terms. This also helps visualize the relative strength or weakness of various asset classes as compared to the most common Large Cap benchmarks.

LAST 63 TRADING DAYS

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

COMPOSITE ETF Z-SCORES OF AVERAGE ROLLING RISK ADJUSTED RETURNS

These charts show the z-scored average of the composite ETF's rolling risk adjusted returns.  Risk adjusted returns are used to improve the robustness of the chart and the information presented. 

By examining the standardized values we can see how each asset class performed relative to the group. This adds further clarity to the relative strength (weakness) of asset class return performance. 

LAST 63 TRADING DAYS; ROLLING PERIOD = 10 DAYS

LAST 21 TRADING DAYS; ROLLING PERIOD = 5 DAYS

LAST 10 TRADING DAYS; ROLLING PERIOD = 5 DAYS

COMPOSITE ETF RISK-ADJUSTED RETURN CORRELATIONS HEATMAP CLUSTERPLOTS

These charts visualize the correlations of the asset class returns over the specified period. Red indicates highly correlated returns while blue indicates negatively correlated returns. 

By examining a clustered heatmap investors are able to quickly determine the intensity and grouping of asset return correlations. Generally speaking, investors should seek to diversify their portfolios by holding uncorrelated assets. Better diversification among asset classes helps to lower overall portfolio volatility with the implication of improving a portfolio's long term performance. 

LAST 63 TRADING DAYS

LAST 21 TRADING DAYS

LAST 10 TRADING DAYS

All data sourced from Yahoo Finance API

COMPOSITE MACRO ETF WEEKLY ANALYTICS (9/14/2015)

/

COMPOSITE ETF CUMULATIVE RETURN MOMENTUM

These charts show the sum (cumulative) of the daily returns of each composite ETF over the specified period. The daily return is calculated as the log of the percent change between daily adjusted close prices.

These charts help determine asset class return momentum. This is important because momentum is arguably the strongest and most persistent market anomaly. Poorly performing asset classes are likely to continue under performing while outperforming asset classes are likely to continue their relative strength. 

LAST 63 TRADING DAYS

Data Source: Yahoo Finance

LAST 21 TRADING DAYS

Data Source: Yahoo Finance

LAST 10 TRADING DAYS

Data Source: Yahoo Finance

LAST 5 TRADING DAYS

Data Source: Yahoo Finance

COMPOSITE ETF CUMULATIVE RETURN (BEST VS WORST VS BENCHMARK)

These charts visualize the cumulative return performance of the best and worst performing asset classes over the specified period. These best and worst asset classes are then compared to a benchmark ETF composite represented by the Large Cap category. 

These charts help give investors an idea of how an actual investment in the represented asset classes would have performed over the period in percentage terms. This also helps visualize the relative strength or weakness of various asset classes as compared to the most common Large Cap benchmarks.

LAST 63 TRADING DAYS

Data Source: Yahoo Finance

LAST 21 TRADING DAYS

Data Source: Yahoo Finance

LAST 10 TRADING DAYS

Data Source: Yahoo Finance

COMPOSITE ETF Z-SCORES OF AVERAGE ROLLING RISK ADJUSTED RETURNS

These charts show the z-scored average of the composite ETF's rolling risk adjusted returns.  Risk adjusted returns are used to improve the robustness of the chart and the information presented. 

By examining the standardized values we can see how each asset class performed relative to the group. This adds further clarity to the relative strength (weakness) of asset class return performance. 

LAST 63 TRADING DAYS; ROLLING PERIOD = 10 DAYS

Data Source: Yahoo Finance

LAST 21 TRADING DAYS; ROLLING PERIOD = 5 DAYS

Data Source: Yahoo Finance

LAST 10 TRADING DAYS; ROLLING PERIOD = 5 DAYS

Data Source: Yahoo Finance

COMPOSITE ETF RISK-ADJUSTED RETURN CORRELATIONS HEATMAP CLUSTERPLOTS

These charts visualize the correlations of the asset class returns over the specified period. Red indicates highly correlated returns while blue indicates negatively correlated returns. 

By examining a clustered heatmap investors are able to quickly determine the intensity and grouping of asset return correlations. Generally speaking, investors should seek to diversify their portfolios by holding uncorrelated assets. Better diversification among asset classes helps to lower overall portfolio volatility with the implication of improving a portfolio's long term performance. 

LAST 63 TRADING DAYS

Data Source: Yahoo Finance

LAST 21 TRADING DAYS

Data Source: Yahoo Finance

LAST 10 TRADING DAYS

Data Source: Yahoo Finance