<< . .

. 29
( : 30)



. . >>

ables. By way of demonstration, such a portfolio will be compiled and run with
the standard exit strategy.

THE BIG PICTURE
For this perspective, each class of entry model (e.g., all trend-following moving
average models, all breakouts, all small neural networks) was examined in its
entirety. All the tests for each model type were averaged. Out-of-sample and
in-sample performances were separately evaluated.
By far the best out-of-sample performer was the genetic model: Out of all the
models, it was the only one that showed a substantial profit when averaged over
all the different tests. The profit per trade was $3,271.
Next best, in terms of out-of-sample behavior over all tests, were the small
neural networks. The neural network models were broken down into those for
small and large nets because curve-fitting appeared to be a serious issue, especial-
ly affecting large nets. The breakdown was a natural and easy one to accomplish
because, in the tests, each model was tested with one small and one large network.
Out-of-sample, all the small neural networks taken together averaged a loss of
$860 per trade. This indicates a significantly better than chance entry in that ran-
dom entries produced an average loss of over $2,000, with a standard deviation of
slightly under $400.
Going down in quality, the next best overall approach involved seasonal&y.
Altogether, all tests of seasonality models showed an average loss of $966 per trade.
Three of the moving average models (crossover, slope, and supportIre&
tance) followed the performance of seasonality. These models, when averaged
across tests, lost around $1,500 per trade, which is close to the $2,100.per-trade
loss expected when using random entries. In other words, the moving average
models were only marginally better than random.
All the remaining models tested provided entries that were very close to ran-
dom. Cycles were actually worse than random.
In-sample, the genetic models ($12,533 per trade), all the neural network
models (small, $8,940, and large, $13,082), and the breakouts ($1,537) traded
profitably. Out-of-sample, the genetic models continued to be profitable, the nets
were better than chance (although there was significant shrinkage in their perfor-
mance due to curve-fitting), and the breakouts deteriorated to chance (optimiza-
tion cannot be a factor in this case).
The next best performers in-sample were the support/resistance moving average
models ($300 loss per trade) and the seasonality models ($671 loss per trade).
Further down the ladder of best-to-worst performers were the lunar and solar
models, which lost $1,076 and $1,067, respectively. Losses in the $1,300 to
$1,700 range were observed for the moving average models. The oscillator and
cycle models exhibited losses of over $2,000 per trade; when taken as a whole,
these models were no better than chance.
It is interesting that the genetic model and the small neural network models
were the ones that held up out of sample. Such models offer great opportunities
for curve-fitting and tend to fail in out-of-sample tests and real trading.
Seasonality, which is only rarely the topic of articles, also exhibited potential. On
the other hand, the most popular methods (e.g., moving average, oscillator, and
cycle models) performed the worst, trading badly both in- and out-of-sample. The
breakout models are noteworthy in the sense that, taken as a group, they worked
well in the past; but, due to increased market efficiency, have since deteriorated to
the point where they currently perform no better than chance.
Table C-l contains the annualized return-on-account (the first line of each model-
order combination) and average dollars-per-trade (the second line of each model-order
combination) data for all the tests (all model and order combinations) conducted for entry
models using the standard exit strategy. The data presented arc for portfolio performance
as a whole. The model descriptions (leftmost column) are the same as used elsewhere in
this book. The last six lines of the table contain data that can be used as a baseline against
which the various entry models can be compared. The baseline data are derived from
using the random entry strategy with the unmoditied standard exit strategy. Mean ROA%
is the average return on account over several sets of random entries; StdLh ROA% is the
standard deviation of the return-on-account. Mean $TRD is the average dollars-per-trade
over several sets of random entries; and StdDev $TRD is the standard deviation of the
dollars-per-trade.
Breakout models had the unique characteristic of being consistently profitable
in-sample across almost every model-order combination tested. Except for the
volatility breakouts, these models performed much better than chance, albeit not
profitably, out-of-sample; i.e., the loss per trade was under $1,000, sometimes under
$300 (the average loss to be expected was around $2,000 with random entries). In
other words, the breakouts, taken together, were better than random. However, out-
of-sample, the volatility breakouts did much worse than chance: With at-open and
on-stop orders, the model lost more than $5,000 per trade, as though the market™s
current behavior is precisely designed to make these systems costly to trade.
The trend-following moving average models (crossover and slope) all per-
formed slightly better than chance in-sample: They all lost rather heavily, but the
losses were almost always less than $2,000. None of the systems, however, did
very well. Out-of-sample, the picture was somewhat more variable, but had the
same flavor: Most of the models were somewhat better than chance, and one or
two were much better than chance (but still not profitable).
356



TABLE C-l

Summary of Portfolio for All Entry Models Tested with All Order Types
Crossoverwith -10 -7 6 -14 -21 20 -4 -5
-1512 -3408 1877 -364 - 1 0 8 1
conflmation -1185 -032 846

-20 -23 1 -7 -14
Crossover with -10 -3 -2
TABLE C-l

(Continued)
The countertrend moving average models were more variable than the trend-
following ones. Many of them showed much smaller losses or even small profits,
in-sample. A similar picture was seen out-of-sample, especially with the simple
moving average support/resistance model.
Except for the MACD divergence model, which behaved differently from the
others, oscillators performed very poorly. There was a lot of variability, but on the
whole, these models gave per-trade profits that were worse than expected by
chance both in-sample and out-of-sample. The RSI overbought/oversold model
was the worst of them all. In both samples, it provided staggering losses that were
(statistically) significantly worse than those that would have been achieved with a
random entry.
The seasonal models, on the whole, were clearly better than chance. Although
only one of these models actually provided profits in both samples, two of them
were profitable out-of-sample, and several had only very small losses (much less
than would be expected by chance using random entries) across samples.
The basic lunar model had mixed findings. Most of the in-sample results
were slightly positive when compared with chance (the random entry), but not
profitable. The basic crossover model, however, was decidedly biased above
chance in both samples.
Although the solar models performed slightly better than chance in-sample,
they were mixed and variable out-of-sample. This was also true for the cycle models.
However, the cycle models, when using entry at open or on limit, actually performed
significantly worse in recent years than a random entry. As with breakouts, the find-
ings are not due to optimization; significant curve-fitting was only detected with the
genetic and neural network models. Because of the tremendous sample involved in
the portfolio, the optimization of one or two parameters, necessary for most models
(other than the genetic and neural ones), had minimal curve-fitting effect.
Surprisingly, the neural network models showed a fairly consistent bias to per-
form better than chance out-of-sample. In-sample, of course, performance was stel-
lar across all tests. There was shrinkage (evidence of curve-fitting), but the shrinkage
was not complete, leaving some predictive utility in the out-of-sample data.
The results for the genetically evolved rules were the best. In-sample, per-
formance was excellent, Out-of-sample, performance was exceptional for models
involving long positions.

Summary
Many of the models described as significantly better than chance (i.e., better than
what would be produced by a random entry) would likely become profitable if cou-
pled with a better exit strategy. In Part III, it was evident that when tested with ran
dom entries, the use of a good exit could bolster profits (or cut losses) by about $1,000
per trade. This means that, with a good exit, some of the entry models that had loss-
es of several hundred dollars could be brought into positive, profitable territory.
CO”Cl”Si””




As mentioned above, the journey was a long one, sometimes tedious and dis-
couraging. However, this birds-eye view revealed that a lot of potentially profitable
entry models were indeed discovered. There were also a number of surprises: Despite
terrible reputations and dangerous tendencies toward curve-fitting, the neural network
and genetic models were the best performers when tested with data that was not used
in training or evolving. Another surprise was that some of the most popular trading
approaches, e.g., moving-average crossovers and oscillator-based strategies, turned
out to be among the worst, with few exceptions. The results of the cycle models were
also revealing: Because of their theoretical elegance, better-if not ideal-perfor-
mance was expected. However, perhaps due to their popularity, poor performance
was observed even though the implementation was a solid, mathematical one.


POINTS OF LIGHT
The portfolio performance of each model was examined for each of the three order
types (at open, on limit, and on stop). Out-of-sample and in-sample performances
were separately evaluated.
The out-of-sample performance was, by far, the best for the long-side genet-
ic models. Entry at open was especially noteworthy, with a 64.2% in-sample return
and 41.0% out-of-sample. The same model was also profitable with entry on limit
and on stop, yielding very high dollars-per-trade profits, although small numbers
of trades (easily increased with more elaborate models of this kind).
In terms of out-of-sample performance, the next best specific model-order
combination was the seasonal crossover with confirmation using entry on stop. Like
the long genetic models, this one was significantly profitable in both sampling peri-
ods: in-sample, $846 per trade, with a 7.4% return; out-of-sample, $1,677 per trade,
with a 9.5% return. Other seasonal models also did okay, with out-of-sample profits
being made using the simple seasonal crossover model,
Next down the list was the short turning-point model that used the small 16-
10-1 network. This model was profitable in both samples across all orders: out-of-
sample, at open, a 9.3% annualized return, with $580 per trade; in-sample, a
35.2% return and $8,448 per trade profit.
While still on the subject of neural networks, the reverse Slow %K model
was a profitable performer, especially when a stop order was used. Out-of-sample,
the annualized return was 6.1%, with $362 per trade. In-sample, there was a 22.5%
return, with a $6,764per-trade profit. Note the large shrinkage from in-sample to
out-of-sample for these models: While this is evidence of curve-fitting, enough
real curves were caught for profits to be made in the verification sample.
Another model that produced profits in both samples was the MACD diver-
gence model, especially with entry on limit. This model had a 6.1% annualized
return, out-of-sample, and a $9X5-per-trade profit. In-sample, the figures were a
6.7% return and a $1,250 profit per trade.
361
Conclusion




Finally, among the models that were profitable in both samples was the sim-
ple moving-average suppotVresistance model with entry on stop: It took 6.4% out
of the market, with $482 per trade, out-of-sample; and $5.8%, with $227 per trade,
in-sample.
Almost all other models lost money out-of-sample, and often in-sample, as
well, The only exception was the volatility breakout model restricted to the curren-
cies, which performed fairly well. Out-of-sample, it had an 8.5% return and made
$2,106 per trade. In sample, it had a 12.4% return, with $3,977 profit per trade.

Summary
Even though most of the other model-order combinations lost out-of-sample, in
many cases, the losses were much less than would be expected with a totally ran-
dom entry. In a number of instances, however, they were worse.
It seems evident that there are a number of models that, although not ideal
and in need of further development, do yield profitable trading that holds up in a
verification sample and yields reasonable statistics.

LOOKING INTO THE LIGHT
Until this point, only tests and models that operate on the whole portfolio have
been discussed. In the course of the tests, many observations were made regarding
the performance of specific models when trading individual markets. A recurrent
observation was that certain models seem to trade certain markets well, while
other models trade them poorly. Some markets just seem to be difficult to trade,
regardless of model. There is no doubt that by selecting a number of the better
models, and then selecting markets that these models trade well, a portfolio of sys-
tems to trade a portfolio of markets could be assembled. Therefore, good system-
market combinations were selected from among the tests conducted in this book.
No optimization of model parameters was performed.
A portfolio was assembled on the basis of in-sample statistical significance.
The intention was to find one good model-order combination for each of the mar-
kets in the portfolio. If there were several potential models for a given market, the
additional ones were discarded based on such things as model complexity (the
more complex the model, the less it was trusted), mediocre portfolio performance,
and other similar factors. The specific model-order combinations spanned the
entire spectrum of models and orders tested, with various oscillators, moving aver-
ages, lunar and solar models, and seasonal and neural network models being rep-
resented; genetic models, however, were not included. In the current tests, the
particular genetic models that were evolved only traded rare events. For those in-
sample markets that performed well, there were generally no out-of-sample trades.
The profitable out-of-sample behavior was achieved on almost a totally different
362



FIGURE C-l

Equity Growth for Multiple-System and Market Portfolio




set of markets than the in-sample behavior of the model. This does not mean that
the out-of-sample performance was bad while the in-sample performance was
good, but rather that most markets simply did not trade in one sample if they did
in the other. The low number of trades observed with the genetic models was due
to the specific namre of the particular rule templates and the ways in which the
individual rules were combined to obtain buy and sell signals. With some changes
in the rule templates, especially in the number of rules used and in how they are
combined, the pattern of rare event trading can be entirely altered.
There were times when the preferred kind of model was not available for a
given market. In such cases, models were examined that performed poorly on a
whole-portfolio basis, but that did trade one or two difficult markets acceptably.
For example, the RSI overbought/oversold model with entry on limit was a poor
performer on a portfolio-wide basis. However, this model traded Gold and Silver
reasonably well. It pulled returns of 27.3 and 3.9%, annualized, on the in-sample
data, with average trades of $9,446 and $4,164, respectively. Out-of-sample, the
system pulled 23.6% out of the Gold and 51.7% out of the Silver markets, with
average trades yielding $12,194 and $24,890, respectively.
One of the large neural networks that appeared to be highly over-optimized
was used for the three wheat markets-markets that did not trade at a statistically
significant level with any of the other models. The large, long-side, turning-point
network with entry on limit, however, had high statistical significance when trad-
ing each of the wheats, pulling more than 40% annually from each, and more than
$15,000 per trade. The amazing thing is that, out-of-sample, despite the size of the
net and the degree of curve-fitting seen on its portfolio performance, the model
pulled in no less than 24%, with $5,000 per trade, from each of the wheats.
The cycle model, which worked well on hardly any market, did trade the
S&P 500 profitably-returning 15.3%, with an average in-sample trade of $4,613,
and 21.4% with $4,698-per-trade profit out-of-sample. It should be noted that a
cycle model was found to trade the S&P-500 successfully in the tests reported in
our earlier study (Katz and McCormick, May 1997).
Once each market was paired with a good model-order combination, the per-
formance data were analyzed, both in- and out-of-sample, for each of the markets.
An equity curve was prepared that covered both periods (see Figure C-l). Returns
and statistical significance were calculated for the multiple-model portfolio, both
in-sample and out-of-sample. It was surprising to discover that the out-of-sample
performance data revealed a return-on-account of 625% annualized! A manifesta-
tion of the Holy Grail? Because model-market combinations were selected on the
basis of their in-sample statistical significance, the 544% annualized in-sample
return was not unexpected. The probability of obtaining an in-sample profit as
large as that is less than 1 in 3,000,000,000,000,000,000 (i.e., 3 X lo™*). Even if
massive amounts of optimization, with tests of tens of thousands of combinations,
took place, the results would still be extremely significant, in a statistical sense.
Out-of-sample, the probability of finding a risk-to-reward ratio or annualized
return as good as that observed is less than 1 in 40 million. Again, even corrected
for extensive optimization, the results would still be of extreme statistical signifi-
cance. In fact, no out-of-sample optimization took place. In-sample, the systems
were only optimized on the entire portfolio. Model parameters were never adjust-
ed for the selected markets on which the models were to be traded. And only the
minimal standard exit strategy was used. Performance could be very substantially
improved using the best of the exits found in Part III.
These findings demonstrate that while most systems do not work and most
tests show losses, a sufficiently extensive search (as conducted in this book) can
discover enough that do work to put together a portfolio trading strategy capable
of producing nothing less than stellar results.
Conclusion
364




COMPANION SOFTWARE AVAILABLE
We invite all readers to visit our website at: www.wientiLic-consultants.com or
to e-mail us at kat.z@scientl6c-consultants.com.
Those who wish to replicate and expand on our research may obtain afree
copy of the C-Trader Toolkit (the software. required to run the code presented in
this book) from our website at www.s&nGfic-eonsultants.com. A CD-ROM is
also available for the nominal cost of $59.00. It contains the following:
m Complete code for every method tested in this book
. Commodities data from Pinnacle
. Spreadsheets containing all optimization data, market-by-market analy-
ses, equity cunw, figures, and tables
m The C-Trader Toolkit, which includes the C+ + Trading Simulator,
OptEvolve (the genetic optimizer), the Portfolio Simulation Shell, and
related manuals
NtlllE Company
Address
City state ˜ ZP
C”U”try Country code
Phone: home (-) office (-)
E-mail
Fax (-1

The companion CD-ROM: $59.00 X ˜ copies $
Numerical Recipes in C:
994 page book $54.95 X ___ copies
software IBM disks $39.95 X ˜ copies

SHLPPlNG g. HANDLING:
for CD only: add $3.50 per copy US, $7.50 outside US $
for Numericul Recipes: add $12 US, $35 outside US $
SALES TAX
(NYS residents add -% for your county 1s
TOTAL $
CHECK ONE:
_ Enclosed is my check “I money order (U.S. only)
_ charge my ldkuMas˜ard/AmEx acwunt (ml in inf”nnati”” below)
acc”““t # expiration
signature
E-mail your order (katz@sclent&-consultants.com), “I mail, phone, “ I fax it to:
SCIENTIFIC CONSULTANT SERVICES, INC. Phone & fax:
20 Stagecoach Road, Selden, New York 11784 631-696-3333
APPENDIX


References and Suggested
Reading




Alexander, Colin (June 1993). “Trade with Moving Averages.” Technical Anolysir of Stockr a n d
Commodities, pp. 61-7 I.
Appel, Gerald (1990). The Advanced Moving Average Convergence-Divergence Trading Method.
Videotape and manual distributed by Signalert Cotportion, New York (516-829-6444).
Bartie, Scott (September 1996). “The COT Index.” Technical Analysis of Stocks and Commodities,
pp. 16-36.
Barr&, Scott (October 1996). “Pork Bellies and the CtX Index.” Technical Analysis ofStocks und
Commodities, pp. 79-92.
Bernstein, Jake (1995). Trade Your Way fo Riches. MBH Commodity Advisors, Inc. (l-800-457-
0X25), 1995.
Blau, William (January 1993). “Stochastic Momentum:™ Technical Analysis of Stocks and
Commodities. pp. 26-35.
Burke, Gibbons (May 1993). “Good Trading a Matter of Breeding?” Futures Magazine, pp. 26329.
Center for Solar and Space Research, Yale University (1997). Sunspot Predictions. Release distib-
uted by Virtual Publishing Company.
Chande. Tushar S. (March 1992). ˜Adapting Moving Averages to Market Volatility.” Technical
Analysis of Stockr and Commodities, pp. 4653.
Davies, D. W. (June 1993). “Cyclical Channel Analysis and the Commodity Channel Index.”
Technical Analysis of Stocks and Commodities, pp. 3845.
Davis, Lawrence (Ed.) (1991). Handbook of Genetic Algorithms. New York: Van Nostrand Reinhold.
Ehlers, John (March 1989). “Moving Averages and Smoothing Filters,” Technical Analysis of Stocks
and Commodities, pp. 4246.
Gauquelin, H., Gauquelin, R., and Eysenck, S. B. G. (1979). “Personality and Position of the Planets
at Birth: An Empirical Study.” British Journal of Social and Clinical Psychology, Vol. 18, pp.
71-75.
Goedde, Richard (March 1997). “Timing a Stock Using the Regression Oscillator.” Technical
Analysis of Stocks and Commodities, pp. 54-60.
Hannula. Hans (November 1991). “The Seasonal Cycle.” Technical Analysis of Stocks and
Commodities, pp. 6548.
Hoel, Paul G. (1966). Elemental Statiufics, 2d ed.. New York: John Wiley & Sons.
365
366



Holland, John (1975). A d a p t a t i o n i n Narural a n d Arfificial Systems. Ann Arbor: The University of
Michigan Press.
Jurik, Mark (1999). “Finding the Best Data.” Computerized Trading. Mark Jurik (Ed.). New York:
New York Institute of Finance/Prentice Hall, pp. 355-382.
Katz, Jeffrey Owen (April 1992). “Developing Neural Network Forecasters for Trading.” Technicul
Analysis of Stocks and Commodities, pp. 5848.
Katz, Jeffrey Owen, and McCormick, Donna L. (1990). Calendar Effects Chart. New York:
Scientific Consultant Services.
Katz. Jeffrey Owen, and McCormick, Donna L. (March/April 1993). “Vendor™s Forum: The
Evolution of N-TRAIN.” PCAI, pp. 4&46.
Katz, Jeffrey Owen, and McCormick, Donna L. (1994). “Neural Networks: Some Advice to
Beginners.” Trader™s Cat&g and Resource Guide, Vol. II, No. 4, p. 36.
Katz, Jeffrey Owen. and McCormick, Donna L. (July/August 1994). “Neurogenetics and Its Use in
Trading System Development.” NeumVe$t Joumnl, pp. X-1 1 .
Katz, Jeffrey Owen, and McCormick, Donna L. (1995a). “Introduction to Artificial Intelligence:
Basics of Expert Systems, Fuzzy Logic, Neural Networks, and Genetic Algorithms.” Virlual
Trading, I. Lederman and R. A. Klein (Eds.). Chicago: Probus P u b l i s h i n g , pp, 3 - 3 4 .
Katz, Jeffrey Owen, and McCormick, Donna L. (1995b). “Neural Networks in Trading.” Vinual
Trading, J. Lederman and R. A. Klein (Ed%). Chicago: Pmbus Publishing, pp. 3544.
Katz, Jeffrey Owen, and McCormick, Donna L. (November 1996). “On Developing Trading
Systems.” Technical Analysis of Stocks and Commodities, pp. 46-60.
Katz, Jeffrey Owen, and McCormick, Donna L. (December 1996). “A Rule-Based Approach to
Trading.” Technical An˜alysis of Stocks an,d Commodities, pp. 22-34.
Katz, Jeffrey Owen, and McCormick, Donna L. (January 1997). “Developing Systems with a Rule-
Based Approach.” Technical Analysis of Stocks and Commodiries, pp. 38-52.

<< . .

. 29
( : 30)



. . >>