Get error when uploading ANcestryDNA

[chrisbonisa]

You are describing a different change that happened last year. I've got 5 total dna samples from ancestry and 3 different formats:

11/24/2014 v1.0 - 701495 lines
03/01/2017 v2.0 - 668961 lines
08/25/2017 v2.0 - 668961 lines
09/29/2017 v2.0 - 668961 lines
10/08/2017 v2.0 - 650429 lines

These are possibly +1 lines if you count the last line-feed. Clearly the last "v2.0" one is in a different format of some kind and its the only one not accepted by FTDNA. Even "changing" the new format one to the other 2.0 formats by removing new snps and filling the missing snps with no-calls doesn't seem to fix it as FTDNA wont accept that new file either with the same error message, although its possible my method of creating this new file is introducing other changes to the file I'm unaware of. I'm willing to provide the last 2 samples to anyone who has the programming skills to attempt what I did. Let me know...

Just to elaborate for anyone else looking at the technical part of this problem, here are the file sizes unzipped:

11/24/2014 v1.0 - 701495 lines - 18,841,144 bytes
03/01/2017 v2.0 - 668961 lines - 18,110,076 bytes
08/25/2017 v2.0 - 668961 lines - 18,110,076 bytes
09/29/2017 v2.0 - 668961 lines - 18,110,076 bytes
10/08/2017 v2.0 - 650429 lines - 17,595,809 bytes

[genealice]

While Dirk's explanation may sound logical when you're only looking at a single file which will not upload, I manage several profiles for my family members and have found that results files from tests run within a single month of each other are of different sizes and some are accepted for upload to ftdna while others aren't. So this can't possibly be due to them using less SNP's for all customers since May 2016.

Hi aprilmcg123,

Oh I agree with you - Dirk's timings are out and it has also occurred to me that Ancestry file sizes are not consistent - I've already put that on this thread or another thread outside this board to which I was posing this question. I thought it was useful that Dirk had confirmed about the SNPs and I'm watching the tekkies here with interest if they can work out a work round for this. I'm not a tekkie - I'm just trying to apply some common sense and get information from others and hopefully something constructive ie a solution to this will eventually be forthcoming.

[CFegley]

While Dirk's explanation may sound logical when you're only looking at a single file which will not upload, I manage several profiles for my family members and have found that results files from tests run within a single month of each other are of different sizes and some are accepted for upload to ftdna while others aren't. So this can't possibly be due to them using less SNP's for all customers since May 2016.

I've previously spoken with Dirk, too, and while he comes across as professional and sincere, the reality is that like others with whom I've spoken, he is simply regurgitating a canned response. If pushed with a logical reason as to why what he is saying is not valid, he will cave, as have many others, including a Francis, a mid-level manager (presumably) who admits that the people in the call center have no knowledge of how the science of DNA works, they are not there to address such questions...their purpose is to explain how you order a test, activate a test, start a tree, update your subscription, etc. It is "not unreasonable" that they might try to figure things out without consulting someone in the area that actually handles the processing of the tests. In fact, they don't even have a way to speak with anyone on the "science" side of the house! Or so they say. Regardless of what FTDNA can or cannot do with the data, and by the way, they are not the only ones who will not accept it, Ancestry owes their customers at least the pretense of respect! I can't tell you how many times their CSRs basically told me I didn't know what I was talking about! If you're going to do something because you don't want your data to be used anywhere else, the have the cahonies to say so and don't belittle my intelligence!

I manage several files whose tests have been processed at different times and their results are consistent with the person (was that you April?) who posted different sizes.

5/2015 - v1.0 - lines: 701,498, size/length: 18,841,299
3/2017 - v2.0 - lines: 668,962, size/length: 18,110,076
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826

Personally, from a business standpoint, I think it would be in their best interest to be consistent with what they test and the results they provide. I also think it would be good for them to accept data from other companies...they already have one of the largest, supposedly, databases (yet my mom's FTDNA matches are nearly double of theirs) as it can only help them...AND their customer base. Refusing to do so shows fear and when you show fear, whether it is to your followers or your adversaries, your credibility will tumble.

[prairielad]

chrisbonisa Sent you a PM

[chrisbonisa]

Like others here, I've been unsuccessful in trying to upload raw data for an AncestryDNA kit which completed its processing on August 22, 2017. When I first downloaded the raw data on the same day it finished, I immediately noticed that the header had a V1.0, something which was mentioned earlier in this thread. However, I didn't try to upload it to FamilyTreeDNA until several days later. When I did so, it was not accepted. I tried the remedies in this thread (changing the header) with no success.

I then decided to try to download a new copy and noticed that the header had changed to match the "older" style which some people had success in submitting. Unfortunately this file was also not accepted.

I noticed that the file size (17,595,826 bytes) did not match the expected 18,110,076 bytes which was mentioned in this thread. In fact, when I first downloaded the raw data and looked at number of lines or SNPs, I noticed it did not match up with the number I expected based on the ISOGG comparison chart.

Yesterday, I decided to download a sample AncestryDNA raw data file posted several weeks ago to the Personal Genome Project in order to compare the lines/SNPs. I found that there are 18532 fewer SNPs in my newer raw data! The older files seem to have 668,962 lines, whereas my newer raw data files only have 650,430.

After doing a basic comparison operation on the two different files, I found that there are 1354 SNPs which are apparently unique to the 17,595,826 byte "broken" file and 19886 SNPs unique to the sample (V2.0) Ancestry file from PGP. Could this be a possible change to Ancestry's chip? If so, it's a shame they're removing more SNPs.

I have to wonder whether these SNPs could have been removed deliberately for medical or other reasons. I have uploaded a list of the unique SNPs to each chip, and a list showing the differences in total SNPs for each chromosome.

Also, today another family member's AncestryDNA kit finished processing and the raw data is the same 650,340 lines.

I've just done a similar comparison using php/mysql database and came up with the exact same results as you with my broken file(2.0b):

v2.0a_count - 668942
v2.0b_count - 650410
diff_count - 18532
v2.0a_missing - 1354
v2.0b_missing - 19886

This suggests its not an error on the part of Ancestry but either a different chip or simply different parsed results for some reason. This also means its most likely FTDNA just not able to handle this new scheme, its probably seeing the "2.0" in the file header and looking for some exact snp schema in the file and since it doesn't match it rejects it. Not counting the new unique snps, this is merely a difference of ~3% which should still be plenty of overlap for FTDNA to handle this, if they would simply try. All other sites are able to accept this type of export without a problem and generate matches from it, I've tried gedmatch, myheritage, wegene, dnaland, etc. FTDNA is the only one erring out...

FTDNA - you are losing money and customers, Ancestry is utilizing this new schema for more and more exports, we need a fix ASAP!

[ltd-jean-pull]

Places like GEDmatch (which has a new section created just for these new chips, called "GEDmatch Genesis") and others like DNA.Land and MyHeritage are using imputation in order to make transferred files work with earlier chip version files. That is not necessarily a good thing; see Roberta Estes' post "Imputation Matching Comparison" (second of what will be three posts on the topic) for one look at it.

I've transferred a couple of kits to MyHeritage from FtDNA. Although each only has 100 or so matches there one or two matches were relatives that had tested there and didn't know they'd get a LOT more matches if they transfer here and to Gedmatch and have since done so.

I also encouraged an adoptee looking for his family to transfer to MyHeritage from Ancestry. I didn't think he had transferred because I couldn't see him, but he had. They share a 64cM segment on Ch2, 62cM on Ch3, 20cM on Ch10, 15cM on Ch16 plus some smaller segments yet MyHeritage doesn't think they're a match!

[Parameswara]

03/01/2017 v2.0 - 668961 lines - 18,110,076 bytes
08/25/2017 v2.0 - 668961 lines - 18,110,076 bytes
09/29/2017 v2.0 - 668961 lines - 18,110,076 bytes
10/08/2017 v2.0 - 650429 lines - 17,595,809 bytes

Maybe AncestryDNA get those samples test in more than one labs. Unlike FTDNA, AncestryDNA don't have their own lab. So, maybe the chips those labs used are not 100% identical each other.

http://labsoftnews.typepad.com/lab_soft_news/2017/07/quest-ancestry-dna-testing.html

[Jim Barrett]

FTDNA - you are losing money and customers, Ancestry is utilizing this new schema for more and more exports, we need a fix ASAP!

The people who are really being hurt, and don't even know it, are those who are buying the down graded product that Ancestry is now selling.

[genealice]

<<<I've previously spoken with Dirk, too, and while he comes across as professional and sincere, the reality is that like others with whom I've spoken, he is simply regurgitating a canned response. If pushed with a logical reason as to why what he is saying is not valid, he will cave, as have many others, including a Francis, a mid-level manager (presumably) who admits that the people in the call center have no knowledge of how the science of DNA works, they are not there to address such questions...their purpose is to explain how you order a test>>>

If this is true, it is one of the reasons why FTDNA should show itself a cut above and clearly indicate in its website layout that there may be times that not all Ancestry files will be accepted so that folks do not spend large amounts of time trying to upload a file which will not upload. I am not biased towards either Ancestry or FTDNA but it is FTDNA that offer the service.

<<< activate a test, start a tree, update your subscription, etc. It is "not unreasonable" that they might try to figure things out without consulting someone in the area that actually handles the processing of the tests. In fact, they don't even have a way to speak with anyone on the "science" side of the house! Or so they say. Regardless of what FTDNA can or cannot do with the data, and by the way, they are not the only ones who will not accept it,>>>

Well, I am on this thread because I would like to know if there is a way FTDNA will accept the Ancestry file I have and that is my main concern. I know some firms do not accept raw data they have not produced. Which firms advertise they accept all files from named outside firms and then do not?

<<<Ancestry owes their customers at least the pretense of respect!
I can't tell you how many times their CSRs basically told me I didn't know what I was talking about! If you're going to do something because you don't want your data to be used anywhere else, the have the cahonies to say so and don't belittle my intelligence!>>>

In the end, very frustrating as it may be for you and me, it is not up to Ancestry to make their files compatible so that they can be uploaded on to other firms’ servers.

I manage several files whose tests have been processed at different times and their results are consistent with the person (was that you April?) who posted different sizes.

5/2015 - v1.0 - lines: 701,498, size/length: 18,841,299
3/2017 - v2.0 - lines: 668,962, size/length: 18,110,076
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826

Personally, from a business standpoint, I think it would be in their best interest to be consistent with what they test and the results they provide.

Interesting - are you saying that the inconsistency is producing sub-standard results?

I also think it would be good for them to accept data from other companies...they already have one of the largest, supposedly, databases (yet my mom's FTDNA matches are nearly double of theirs)

Again, interesting. Are you talking about a comparison between all matches on FTDNA and Ancestry or the quality of the matches?

<<<as it can only help them...AND their customer base. Refusing to do so shows fear and when you show fear, whether it is to your followers or your adversaries, your credibility will tumble.>>>

For myself, as a customer, yes I would appreciate if it was all opened up but if firms choose not to, it’s up to them, whatever their reasons!

[ltd-jean-pull]

Just curious, but if your file still does not work, do you have the older file with an uncompressed size of 18,400 KB or do you have the newer file with an uncompressed size of 17,686 KB? Does anyone have a file with either of these sizes that still will not upload after the V2 fix?

This post dated the 27th August was the first to give us a hint about what might be causing this current issue with uploading Ancestry autosomal data to FamilyTreeDNA being due to a change in file size -i.e. Missing data.

Some might think that FamilyTreeDNA ought to make it a priority to sort out this issue so that they can upload a file that it is a different format from the the files FtDNA is set up to accept, even though FtDNA doesn't get informed by Ancestry that they're going to change the files. Ancestry can't inform their customer service reps or customers of the file change so I doubt they'd pick up the phone and tell a competitor.

Anyway, back to August 27th. What a shame that some computer geek who works for FtDNA who was stuck at home with water lapping around the porch didn't somehow manage to log-in from home and start working on the coding. And how unreasonable of them to close down the server for a few days meaning that e-mails couldn't be received during the flooding.

I can understand people's frustration, but some people need to get some perspective. FamilyTreeDNA's priority is their PAYING customers who tested with them, and in the last few weeks they've been working in very trying circumstances.

Hopefully it will be sorted in due course, but it wouldn't surprise me if Ancestry continues to muck around with file size. I have no idea why they do it.

[chrisbonisa]

This post dated the 27th August was the first to give us a hint about what might be causing this current issue with uploading Ancestry autosomal data to FamilyTreeDNA being due to a change in file size -i.e. Missing data.

Some might think that FamilyTreeDNA ought to make it a priority to sort out this issue so that they can upload a file that it is a different format from the the files FtDNA is set up to accept, even though FtDNA doesn't get informed by Ancestry that they're going to change the files. Ancestry can't inform their customer service reps or customers of the file change so I doubt they'd pick up the phone and tell a competitor.

Anyway, back to August 27th. What a shame that some computer geek who works for FtDNA who was stuck at home with water lapping around the porch didn't somehow manage to log-in from home and start working on the coding. And how unreasonable of them to close down the server for a few days meaning that e-mails couldn't be received during the flooding.

I can understand people's frustration, but some people need to get some perspective. FamilyTreeDNA's priority is their PAYING customers who tested with them, and in the last few weeks they've been working in very trying circumstances.

Hopefully it will be sorted in due course, but it wouldn't surprise me if Ancestry continues to muck around with file size. I have no idea why they do it.

This is completely the wrong approach. First of all there are TONS of us who do the transfer and pay to upgrade it to get the chromosome browser. As a matter of fact, FTDNA isn't of much use to me without that as they have far fewer people than Ancestry or 23andme. FTDNA can only be successful if they allow transfers to build up their user base over time. Also, lots of us, me included, pay for other products and want to have all of our data aggregated in one place. I paid the $160 for y-dna-37 test and will most likely upgrade that(+$100). And the $19 for each transfer upgrade is pure profit for them and likely the same amount of profit they receive from autosomal tests directly through them after lab costs.

Regarding the hurricane, I run an IT company with a datacenter and central office in Tampa, FL.....you know, where hurricane Irma hit. We had emergency contingency plans in place for this sort of thing years ago and because of that, the impact was nill even though power was out for almost all of our local employees and flooding everywhere. We are no where as large as FTDNA and they are more than capable of handling emergencies, and I'm sure they did in fact deal with it in a responsible way. I don't think this problem is in any way related to the hurricane. Not to mention that was like 6 weeks ago.

[aprilmcg123]

I've just done a similar comparison using php/mysql database and came up with the exact same results as you with my broken file(2.0b):

v2.0a_count - 668942
v2.0b_count - 650410
diff_count - 18532
v2.0a_missing - 1354
v2.0b_missing - 19886

This suggests its not an error on the part of Ancestry but either a different chip or simply different parsed results for some reason. This also means its most likely FTDNA just not able to handle this new scheme, its probably seeing the "2.0" in the file header and looking for some exact snp schema in the file and since it doesn't match it rejects it. Not counting the new unique snps, this is merely a difference of ~3% which should still be plenty of overlap for FTDNA to handle this, if they would simply try. All other sites are able to accept this type of export without a problem and generate matches from it, I've tried gedmatch, myheritage, wegene, dnaland, etc. FTDNA is the only one erring out...

FTDNA - you are losing money and customers, Ancestry is utilizing this new schema for more and more exports, we need a fix ASAP!

Unfortunately, this doesn't make sense as being just a new chip issue between Ancestry and FTDNA as I have evidence of several profiles processed using the same chip (V2 according to the file) within days of each other where the size difference is occuring and some of them will upload and others (those with the missing SNPs) will not.

[chrisbonisa]

Unfortunately, this doesn't make sense as being just a new chip issue between Ancestry and FTDNA as I have evidence of several profiles processed using the same chip (V2 according to the file) within days of each other where the size difference is occuring and some of them will upload and others (those with the missing SNPs) will not.

I'm curious about this myself, I wonder if its a new chip not fully rolled out yet? Or perhaps new software? But yes I agree, its inconsistent and confusing.

[WilliamKF]

GEDmatch works fine, but FTDNA rejects

I'm having this issue too with my cousin's results that just posted at Ancestry.com last week. GEDmatch.com accepted the transfer with no issues, same for MyHerritage.com.

[Zack]

AncestryDNA kit received on 10/17/17

Upload attempt (many) on 10/17/17

Can not upload successfully, headers were already the values suggested earlier in the thread.

Error Code: 'The file is an unsupported version or in a corrupt/malformed format.'

Zip file is 5,607 KB
Extracted *.txt is 17,184 KB