I concur with David. A 50% change in CROR is more than significant. VV have known of this major flaw for some years. Why would VV not see this as a necessary update?

I concur with the below and wonder why VV has not acted or responded? The backtest result can move by 50% CROR or more!

To know your search is good by using Backtest, it is important that all traded stocks are in the database across the dates being tested. Otherwise the results are skewed owing to survivorship bias. I note that there are comments going back to 2010 so I wonder why there has been no response from VV on why this has not been addressed.

UPDATE: I spoke with VV Support today. They do not have any plans now or in the future to fix this significant deficiency.

I COMPLETELY AGREE. I was a member a few years ago. I uncovered this serious FUNDAMENTAL bug when trying to find a short search that would have been beneficial after the financial crisis. I knew of some tickers that were simply "gone" (Does anyone remember LEHMAN BROS?)

This significant bug basically compromises not only our positions but also several of the proprietary metrics VV uses (e.g., MTI, VVC Composite Index, VV Views, BSR – to a lesser degree, etc).</b>

I find it difficult to believe people are voting “Create Stop Loss Strategy: Buy 10 -- Hold 15” when the source of their data could be misleading.

Without these stocks, are we really left with only the “cream of the crop?” Meaning if you are a “long-only” investor and all stocks that went belly-up during the crisis, your results may be better than you would have actually experienced.

On the flip side, if a company is acquired at a premium, it no longer appears in the searches could reduce some of the results unrealistically.

It really makes the BackTester useless - IMHO - as you really can’t assume that data is realistic.

I am a Oracle database programmer by trade. If VV would like to I could explore trying to restore this information. I wouldn't need any of VV's proprietary information – just the table definitions and a reliable historical dataset.

My fee – contingent on me successfully recreating the data with date effective data – would be lifetime subscription to VV products and seminars. Obviously we would have to hammer out other not financial details but I am willing and able.

It is not clear to me that VV personnel is even viewing these comments as I don’t see any replies to the comments in this thread. Hopefully this is not just a place to vent.

I agree. A large portion of the money I invest is based on VV searches I have developed by using back test. it is extremely important that the back test be based on data that accurately reflects what would have happened during the time of the back test.

Apparently VV deletes stocks from the data base that were delisted which skews the results of the back test. This can result in back test that are significantly different from what would have happened if you had actually traded the stocks during the time of the back test. This greatly reduces my confidence in the data that VV provides. Data used for back test should include all stocks that were available to buy until they were delisted. Fixing this problem should receive a high priority in my opinion.

I agree. Delisted stocks should be removed from the date they were delisted moving forward. not from the entire database thus rendering backtests/search stratagies thru various market conditions skewed and unreliable.

I recently found this while reviewing/comparing previous backtests against a tweaked search strategy. I thought it was something I did wrong, couldn't imagine VV would allow something as critical to their platform as historical data to be skewed.

A search has been built that provides an acceptable, somewhat consistent, pattern over various market conditions spanning the past decade. A hundred or more backtests in the making only now to find the results are skewed rendering a high percentage of the backtests unreliable.

Judging by previous posts it appears this problem has been ignored by VV for some time now, perhaps it's time to reseach another vendor ?

Agree. We need this fix to address surviorship bias to lend more credibility to backtest results. Can we have these, please?

Yes, This is an absolute no brainer! Without this you are completely flying blind with backtests. Why remove stocks in the first place? Leaving them in requires less effort than taking them out. Mind you, I wouldn't want to be the person who has to reinstate them back to when.
Neil

Remove the Survivorship Bias from the back tests database. If those stocks that were removed from the database due to delisting, bankruptcy, bought out, etc. and NO LONGER trade, are removed from the database, it skews the evaluation creating a favorable, but inaccurate, performance of the strategy being tested. This does not represent a true model of actual performance and creates a chain reaction that can result in a disappointing outcome.

This is important for EVERYONE backtesting their ideas! Otherwise the results doesn't mean anything. One stock change can have a dramatic chain reaction to the return figures over time.
For example, I ran a back test 6 months ago and re-ran it a few weeks ago. The difference was a 40% swing from the original backtest to the current backtest in the year 2006 with all the same parameters. The stock that was removed from the (Canadian list) was JDS (JDS Uniphase)
It doesn't show up any more. But it was this stock that got stopped out which triggered the next 5 consecutive successful trades of greater an 30% gains. With this name removed, the picks changed and the outcome was different.

Why can't you have a watch list for delisted stocks? You could add a suffix to the ticker symbol such as XYZ* to show that stock XYZ was delisted and you could have an added column showing the time period XYZ was trading. If the same symbol was reused and delisted a second time, use two asterisks for example. Or code the ticker symbol with the active time [ XYZ970513-040927 could mean XYZ traded between May 13, 1996 and Sept 27, 2004 ]. If needed, you could just have dashes (or other character) as placeholders in the spreadsheet of the watchlist for XYZ for all the days that XYZ did not trade. ----- Victor Hazel vehazel@aol.com

This might turn successful results from certain backtests into disasterous results. I can see why this may not be a priority. But this should be the number one vote getter. Even above an Apple version of VV7.

I agree, but have already spent my votes...

I also find this extremely limiting. Backtester is a great tool, but it only can be as precize as the data it uses. I simply don't understand what's the problem with keeping and showing delisted stocks after they got delisted whatever the reason for that may be.

Take the list of 500 Top US stocks as provided by MoneySense annually and try to load it up in VV. Files provided for 2006-2010 years never loaded fully. Numerous stocks are missing due to delisting and merging.

I hope It'll get resolved in the future.

I have been a VectorVest subscriber for 8 years and have waited patiently for VectorVest to fix this problem

The surviorship bias is extremely important to getting accurate backtest results. My backtests yield significantly different results each time I run them because of this issue.

In fact, I doubt that the confirmed up and confirmed down calls could be accurately reproduced with the current database.

This database is at the core of every product that the company produces and it would be in their best interst to fix it.

I learned the lesson on the first day of the first computer science class I took over 40 years ago: GIGO (Garbage In; Garbage Out)

VectorVest-It's time to fix this once and for all.

A problem I see here is sorting out the bankruptcies from the privatizations from the mergers. When Merril Lynch disappeared it was by acquisition by BAC. Other stocks just went belly-up.

The survivor bias problem is in my opinion a big issue with the VectorVest historical database. My searches in the last year have found stocks that are now bankrupt. If I were to backtest now for the past year, I would get different results than those obtained at the time. I firmly beileve this needs to be fixed.

I agreed, where is ENRON! I back test 10 years and i see very good results in the first 3 years because I did not buy companies that died!.

I was wondering if the VV database had a survivor bias, and this seems to indicate that it does. So, if I chose a strategy that turned up CC and NT, there's a chance that strategy would generate heavy losses for me, yet my backtesting wouldn't reflect it. This is critical for accurate backtesting. Without turning up delisted stocks in historical backtesting, you lose a significant amount of confidence in your backtest.