Dukascopy Tick Data

mike · #1 (**permalink**) 12-20-2009, 09:51 PM

For those who are really serious about backtesting I recently found that Dukascopy have free Data available. This Data is at Tick level not the 1 minute or 5 minute data that is on the MT4 site. Tick Data removes the interpolation that MT4 tries to do so you should get a accurate result. From what I read 99% was achievable as opposed to the 90% that you more commonly see. It is not in a format that is directly acceptable by MT4, however there is a process to convert it. It does seem to be a complicated process to download and convert. Also at Tick level the volume of Data is very large compared with 5 minute data so you will need a fairly fast PC
Google "Dukascopy convert market data" or such like.

Mike

mike · #2 (**permalink**) 03-27-2010, 10:53 AM

This is an addition to the original post above. Below is a comparison between the MT4 Strategy Tester results using first the “MT4's inhouse 5 Min. data” and second using the Dukascopy Tick Data mentioned in the first post above.
The EA I used in both tests was Oewee's “ACG Pro R2” I used this particular EA because firstly it is one of the few EA's that I have and second IMO it is probably one of the best EA's around and so by using it I am not making any comment apart from the above. The identical Set file was used in both Back Tests
From what I understand of the MT4 Strategy Tester, it uses a .FXT file that resides in “tester/history” within the MT4. In this case the .FXT file for the MT4 5 Min data is 677,937KB's and the .FXT file for the Dukascopy Pip Data is 2.0 GB which is the max possible with MT4. Obviously vastly more data points and no interpolation is used. You will notice 90% modeling quality on the MT4 data and 99% modeling quality on the Dukascopy data
I think that the results say it all. If this is all correct and it does appear to be. This is one real good reason why EA's show amazing results on Back Test and promptly blow up accounts in Live Trading.

MT4 Pip Data:

Dukascopy Pip Data:

alan · #3 (**permalink**) 03-28-2010, 09:21 AM

Mike I'm having some problems getting this thing to work. If it's not too much of a hassle, could you perhaps type out a quick "How-To" on downloading and converting the data into a usable MT4 strategy tester format?

Hmm, also I'm wondering is it possible to compress the 2.0 GB Dukascopy tick data file further (as in zip, rar, 7zip, etc)?

mike · #4 (**permalink**) 03-28-2010, 11:44 PM

Alan
Thanks for that Edit.
I am not an expert by any means. I got the instructions from “Birts” EA site. It is a pretty lengthy set of instructions but very complete. The site has the programs/links needed for everything to download then combine the monthly Dukascopy Tick files into one large CSV file. Then the program to convert that CSV into the files that MT4 can load and use and the “FXT Multi Loader.exe” that must be used to start the particular MT4 instance that you will be using to run the back test (if you are running MT4 V225).
If some part of the downloading and converting process did not work for me at the first attempt I went back and re-read the instructions and sure enough I had goofed. Once I followed the instructions meticulously it all worked (on Win XP SP3)
The only variation that I make is that when I back test I start the MT4 using the “Multiloader.exe”. Before I run the script “Dukascopy2FXT” I connect to the MT4 server, wait till it has finished updating then disconnect the MT4 from the server and then run the script to create the .HST and .FXT files. I find that if I don't do things in that sequence I will not get the 99% Modeling Quality on the back test. Apparently all this is all necessary to get around the some shortcomings in the later versions of MT4 ie from V208 onwards. So we can only hope that the new, much vaunted MT5 will be able to read .FXT files (or their MT5 equivalent) larger than 2.0GB and will allow the use of external .FXT files as an option.
Earlier somebody had the tick data up as a Torrent which would have been great but Dukascopy have prohibited redistribution and it was taken down so now its a individual download unfortunately.
Eg:- EUR/USD 30th Feb 2007 to 23rd Nov 2009 = 570 KB download. This later converts to a 4.5 GB .CSV file. It can take a long time!
As a footnote I tried to use MT4 “Optimise” with only 1 parameter at a time but it did not appear to work (Quad Core @ O.C.to 4.0 GHz). Maybe it's just too much data for MT4 to handle.
Mike

alan · #5 (**permalink**) 03-29-2010, 12:05 AM

Dang I did not know about Birt's site, and even that we have a thread here discussing his site:

http://www.forexnirvana.com/f19/birt...iew-site-1248/

But more on the subject at hand...now that I have his instructions I think I will manage to get through it. I can already tell it's going to take a long time, but IMHO it will be worth it.

Yeah with so many data poits it will take a LONG LONG time for the MT4 optimizer to run trough the process, but it should work okay I guess if you just let it run for a long time.

I hope my quad core I7 920 with 12 GB of RAM is enough to take on this huge computing load...I shall find out I guess.

mike · #6 (**permalink**) 03-29-2010, 09:18 AM

Alan
I would be interested to see some backtests on any EA's that you have. I was stunned at the disparity in results between the two sets of Data. A confirmation that the differences are replicated with other EA's would validate the whole process.

Mike

alan · #7 (**permalink**) 03-29-2010, 09:20 AM

Yep I'll post some backtests here..right now I'm at the process of converting the tick data to CSV...man the file is getting huge!!

alan · #8 (**permalink**) 03-29-2010, 09:43 PM

Mike I finally got the Dukascopy tick data working

...here is a backtest comparison. The time period I used was 2007.04.10 21:00 to 2008.03.03 21:05

Using the Dukascopy tick data (99% modeling quality):

Using MetaQuotes history data (90% modeling quality):

Maximum and relative drawdown is the same for both..around 13%..however accuracy is higher on the Dukascopy tick data...98% for the Dukascopy test and 96% for the MT4 data.

The dukascopy data test did end up being more profitable because it took many more trades...

By the way I was using a scalper EA.

mike · #9 (**permalink**) 04-05-2010, 03:09 AM

Hi Alan
Interesting result. Opposite of the EA I tested. Maybe AGC Pro R2 had an element of Curve Fitting for the MT4 data. If you have the AGC Pro R2 and you have time could you run it on the Tick Data just to confirm I got it right? Also have you tried to run any Optimising using the Tick Data. I tried Optimising the basic stock EA's that are on the MT4 and it does run, just very slowly. I dont think it will be a practical process to run it on a complicated EA unless you have a Cray in the basement.

Mike

alan · #10 (**permalink**) 04-05-2010, 04:31 AM

I don't believe I have AGC Pro R2 but I will try to get a hold of it and run a backtest and post the results.

I did and still do run optimizations on their tick data, and although it runs slower compared to MT4 data performance is still acceptable on my system (Core i7 920, 12 GB RAM).

To be honest I'm still not 100% satisfied with the Dukascopy data. I want tick data that goes back all the way to 2000 and up to 2010 03. Do you know of another alternative...even a paid one would be fine as long as the tick data is of high quality and easily convertible to MT4's .FXT format.