Get error when uploading ANcestryDNA

[ltd-jean-pull]

Just curious, but if your file still does not work, do you have the older file with an uncompressed size of 18,400 KB or do you have the newer file with an uncompressed size of 17,686 KB? Does anyone have a file with either of these sizes that still will not upload after the V2 fix?

This post dated the 27th August was the first to give us a hint about what might be causing this current issue with uploading Ancestry autosomal data to FamilyTreeDNA being due to a change in file size -i.e. Missing data.

Some might think that FamilyTreeDNA ought to make it a priority to sort out this issue so that they can upload a file that it is a different format from the the files FtDNA is set up to accept, even though FtDNA doesn't get informed by Ancestry that they're going to change the files. Ancestry can't inform their customer service reps or customers of the file change so I doubt they'd pick up the phone and tell a competitor.

Anyway, back to August 27th. What a shame that some computer geek who works for FtDNA who was stuck at home with water lapping around the porch didn't somehow manage to log-in from home and start working on the coding. And how unreasonable of them to close down the server for a few days meaning that e-mails couldn't be received during the flooding.

I can understand people's frustration, but some people need to get some perspective. FamilyTreeDNA's priority is their PAYING customers who tested with them, and in the last few weeks they've been working in very trying circumstances.

Hopefully it will be sorted in due course, but it wouldn't surprise me if Ancestry continues to muck around with file size. I have no idea why they do it.

[genealice]

<<<I've previously spoken with Dirk, too, and while he comes across as professional and sincere, the reality is that like others with whom I've spoken, he is simply regurgitating a canned response. If pushed with a logical reason as to why what he is saying is not valid, he will cave, as have many others, including a Francis, a mid-level manager (presumably) who admits that the people in the call center have no knowledge of how the science of DNA works, they are not there to address such questions...their purpose is to explain how you order a test>>>

If this is true, it is one of the reasons why FTDNA should show itself a cut above and clearly indicate in its website layout that there may be times that not all Ancestry files will be accepted so that folks do not spend large amounts of time trying to upload a file which will not upload. I am not biased towards either Ancestry or FTDNA but it is FTDNA that offer the service.

<<< activate a test, start a tree, update your subscription, etc. It is "not unreasonable" that they might try to figure things out without consulting someone in the area that actually handles the processing of the tests. In fact, they don't even have a way to speak with anyone on the "science" side of the house! Or so they say. Regardless of what FTDNA can or cannot do with the data, and by the way, they are not the only ones who will not accept it,>>>

Well, I am on this thread because I would like to know if there is a way FTDNA will accept the Ancestry file I have and that is my main concern. I know some firms do not accept raw data they have not produced. Which firms advertise they accept all files from named outside firms and then do not?

<<<Ancestry owes their customers at least the pretense of respect!
I can't tell you how many times their CSRs basically told me I didn't know what I was talking about! If you're going to do something because you don't want your data to be used anywhere else, the have the cahonies to say so and don't belittle my intelligence!>>>

In the end, very frustrating as it may be for you and me, it is not up to Ancestry to make their files compatible so that they can be uploaded on to other firms’ servers.

I manage several files whose tests have been processed at different times and their results are consistent with the person (was that you April?) who posted different sizes.

5/2015 - v1.0 - lines: 701,498, size/length: 18,841,299
3/2017 - v2.0 - lines: 668,962, size/length: 18,110,076
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826

Personally, from a business standpoint, I think it would be in their best interest to be consistent with what they test and the results they provide.

Interesting - are you saying that the inconsistency is producing sub-standard results?

I also think it would be good for them to accept data from other companies...they already have one of the largest, supposedly, databases (yet my mom's FTDNA matches are nearly double of theirs)

Again, interesting. Are you talking about a comparison between all matches on FTDNA and Ancestry or the quality of the matches?

<<<as it can only help them...AND their customer base. Refusing to do so shows fear and when you show fear, whether it is to your followers or your adversaries, your credibility will tumble.>>>

For myself, as a customer, yes I would appreciate if it was all opened up but if firms choose not to, it’s up to them, whatever their reasons!

[Jim Barrett]

FTDNA - you are losing money and customers, Ancestry is utilizing this new schema for more and more exports, we need a fix ASAP!

The people who are really being hurt, and don't even know it, are those who are buying the down graded product that Ancestry is now selling.

[Parameswara]

03/01/2017 v2.0 - 668961 lines - 18,110,076 bytes
08/25/2017 v2.0 - 668961 lines - 18,110,076 bytes
09/29/2017 v2.0 - 668961 lines - 18,110,076 bytes
10/08/2017 v2.0 - 650429 lines - 17,595,809 bytes

Maybe AncestryDNA get those samples test in more than one labs. Unlike FTDNA, AncestryDNA don't have their own lab. So, maybe the chips those labs used are not 100% identical each other.

http://labsoftnews.typepad.com/lab_s...a-testing.html

[ltd-jean-pull]

Places like GEDmatch (which has a new section created just for these new chips, called "GEDmatch Genesis") and others like DNA.Land and MyHeritage are using imputation in order to make transferred files work with earlier chip version files. That is not necessarily a good thing; see Roberta Estes' post "Imputation Matching Comparison" (second of what will be three posts on the topic) for one look at it.

I've transferred a couple of kits to MyHeritage from FtDNA. Although each only has 100 or so matches there one or two matches were relatives that had tested there and didn't know they'd get a LOT more matches if they transfer here and to Gedmatch and have since done so.

I also encouraged an adoptee looking for his family to transfer to MyHeritage from Ancestry. I didn't think he had transferred because I couldn't see him, but he had. They share a 64cM segment on Ch2, 62cM on Ch3, 20cM on Ch10, 15cM on Ch16 plus some smaller segments yet MyHeritage doesn't think they're a match!

[chrisbonisa]

Like others here, I've been unsuccessful in trying to upload raw data for an AncestryDNA kit which completed its processing on August 22, 2017. When I first downloaded the raw data on the same day it finished, I immediately noticed that the header had a V1.0, something which was mentioned earlier in this thread. However, I didn't try to upload it to FamilyTreeDNA until several days later. When I did so, it was not accepted. I tried the remedies in this thread (changing the header) with no success.

I then decided to try to download a new copy and noticed that the header had changed to match the "older" style which some people had success in submitting. Unfortunately this file was also not accepted.

I noticed that the file size (17,595,826 bytes) did not match the expected 18,110,076 bytes which was mentioned in this thread. In fact, when I first downloaded the raw data and looked at number of lines or SNPs, I noticed it did not match up with the number I expected based on the ISOGG comparison chart.

Yesterday, I decided to download a sample AncestryDNA raw data file posted several weeks ago to the Personal Genome Project in order to compare the lines/SNPs. I found that there are 18532 fewer SNPs in my newer raw data! The older files seem to have 668,962 lines, whereas my newer raw data files only have 650,430.

After doing a basic comparison operation on the two different files, I found that there are 1354 SNPs which are apparently unique to the 17,595,826 byte "broken" file and 19886 SNPs unique to the sample (V2.0) Ancestry file from PGP. Could this be a possible change to Ancestry's chip? If so, it's a shame they're removing more SNPs.

I have to wonder whether these SNPs could have been removed deliberately for medical or other reasons. I have uploaded a list of the unique SNPs to each chip, and a list showing the differences in total SNPs for each chromosome.

Also, today another family member's AncestryDNA kit finished processing and the raw data is the same 650,340 lines.

I've just done a similar comparison using php/mysql database and came up with the exact same results as you with my broken file(2.0b):

v2.0a_count - 668942
v2.0b_count - 650410
diff_count - 18532
v2.0a_missing - 1354
v2.0b_missing - 19886

This suggests its not an error on the part of Ancestry but either a different chip or simply different parsed results for some reason. This also means its most likely FTDNA just not able to handle this new scheme, its probably seeing the "2.0" in the file header and looking for some exact snp schema in the file and since it doesn't match it rejects it. Not counting the new unique snps, this is merely a difference of ~3% which should still be plenty of overlap for FTDNA to handle this, if they would simply try. All other sites are able to accept this type of export without a problem and generate matches from it, I've tried gedmatch, myheritage, wegene, dnaland, etc. FTDNA is the only one erring out...

FTDNA - you are losing money and customers, Ancestry is utilizing this new schema for more and more exports, we need a fix ASAP!

[prairielad]

chrisbonisa Sent you a PM

[CFegley]

While Dirk's explanation may sound logical when you're only looking at a single file which will not upload, I manage several profiles for my family members and have found that results files from tests run within a single month of each other are of different sizes and some are accepted for upload to ftdna while others aren't. So this can't possibly be due to them using less SNP's for all customers since May 2016.

I've previously spoken with Dirk, too, and while he comes across as professional and sincere, the reality is that like others with whom I've spoken, he is simply regurgitating a canned response. If pushed with a logical reason as to why what he is saying is not valid, he will cave, as have many others, including a Francis, a mid-level manager (presumably) who admits that the people in the call center have no knowledge of how the science of DNA works, they are not there to address such questions...their purpose is to explain how you order a test, activate a test, start a tree, update your subscription, etc. It is "not unreasonable" that they might try to figure things out without consulting someone in the area that actually handles the processing of the tests. In fact, they don't even have a way to speak with anyone on the "science" side of the house! Or so they say. Regardless of what FTDNA can or cannot do with the data, and by the way, they are not the only ones who will not accept it, Ancestry owes their customers at least the pretense of respect! I can't tell you how many times their CSRs basically told me I didn't know what I was talking about! If you're going to do something because you don't want your data to be used anywhere else, the have the cahonies to say so and don't belittle my intelligence!

I manage several files whose tests have been processed at different times and their results are consistent with the person (was that you April?) who posted different sizes.

5/2015 - v1.0 - lines: 701,498, size/length: 18,841,299
3/2017 - v2.0 - lines: 668,962, size/length: 18,110,076
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826
8/2017 - v2.0 - lines: 650,430, size/length: 17,595,826

Personally, from a business standpoint, I think it would be in their best interest to be consistent with what they test and the results they provide. I also think it would be good for them to accept data from other companies...they already have one of the largest, supposedly, databases (yet my mom's FTDNA matches are nearly double of theirs) as it can only help them...AND their customer base. Refusing to do so shows fear and when you show fear, whether it is to your followers or your adversaries, your credibility will tumble.

[genealice]

While Dirk's explanation may sound logical when you're only looking at a single file which will not upload, I manage several profiles for my family members and have found that results files from tests run within a single month of each other are of different sizes and some are accepted for upload to ftdna while others aren't. So this can't possibly be due to them using less SNP's for all customers since May 2016.

Hi aprilmcg123,

Oh I agree with you - Dirk's timings are out and it has also occurred to me that Ancestry file sizes are not consistent - I've already put that on this thread or another thread outside this board to which I was posing this question. I thought it was useful that Dirk had confirmed about the SNPs and I'm watching the tekkies here with interest if they can work out a work round for this. I'm not a tekkie - I'm just trying to apply some common sense and get information from others and hopefully something constructive ie a solution to this will eventually be forthcoming.

[chrisbonisa]

You are describing a different change that happened last year. I've got 5 total dna samples from ancestry and 3 different formats:

11/24/2014 v1.0 - 701495 lines
03/01/2017 v2.0 - 668961 lines
08/25/2017 v2.0 - 668961 lines
09/29/2017 v2.0 - 668961 lines
10/08/2017 v2.0 - 650429 lines

These are possibly +1 lines if you count the last line-feed. Clearly the last "v2.0" one is in a different format of some kind and its the only one not accepted by FTDNA. Even "changing" the new format one to the other 2.0 formats by removing new snps and filling the missing snps with no-calls doesn't seem to fix it as FTDNA wont accept that new file either with the same error message, although its possible my method of creating this new file is introducing other changes to the file I'm unaware of. I'm willing to provide the last 2 samples to anyone who has the programming skills to attempt what I did. Let me know...

Just to elaborate for anyone else looking at the technical part of this problem, here are the file sizes unzipped:

11/24/2014 v1.0 - 701495 lines - 18,841,144 bytes
03/01/2017 v2.0 - 668961 lines - 18,110,076 bytes
08/25/2017 v2.0 - 668961 lines - 18,110,076 bytes
09/29/2017 v2.0 - 668961 lines - 18,110,076 bytes
10/08/2017 v2.0 - 650429 lines - 17,595,809 bytes

[aprilmcg123]

Ah, you’re so psychologically astute about me - yes, I’m a nice person who skips through life thinking happy thoughts about everybody .

Whatever the timing involved, it does seem likely that it’s about the change in file size which Dirk from Ancestry did address:

“ ... Yes, as far as I have been told the kits tested since the update described in the blog will be smaller as snps were refined so there will be less text or lines of data."

As I’ve said, it would be good practice for FTDNA, as it offers and advertises the service, to alert people about the smaller Ancestry files and that they will not upload. For me, it’s been a waste of time and effort, although I have got to meet nice people like you - and Dirk! lol

While Dirk's explanation may sound logical when you're only looking at a single file which will not upload, I manage several profiles for my family members and have found that results files from tests run within a single month of each other are of different sizes and some are accepted for upload to ftdna while others aren't. So this can't possibly be due to them using less SNP's for all customers since May 2016.

[chrisbonisa]

FTDNA is really missing the boat here......I've actually paid for all my autosomal transfers to upgrade($19), and wanting to do the same thing to my non-working file but cannot. I've also paid for a y-dna 37 test ($160). This non working file unfortunately is one of the most important I need to be able to import as it's my mom and my dads dna is not available. Clearly we need a fix here, and they are losing lots of money not fixing it!

[chrisbonisa]

I have just had a live chat with Dirk at Ancestry. He says the change to the raw data files was announced on 5/12/2016. https://blogs.ancestry.com/ancestry/...estrydna-chip/

The files are smaller, Dirk says, because:

"It is due to certain snps being removed as they don't pertain to what we are testing for... Not necessarily a fault, but that there have been changes made and we can't determine what they [FTDNA] do on their end ... Yes, as far as I have been told the kits tested since the update described in the blog will be smaller as snps were refined so there will be less text or lines of data."

So it looks as if the ball is in FTDNA's court. As my file uploaded ok on Gedmatch and My Heritage, it does seem to be about FTDNA.

You are describing a different change that happened last year. I've got 5 total dna samples from ancestry and 3 different formats:

11/24/2014 v1.0 - 701495 lines
03/01/2017 v2.0 - 668961 lines
08/25/2017 v2.0 - 668961 lines
09/29/2017 v2.0 - 668961 lines
10/08/2017 v2.0 - 650429 lines

These are possibly +1 lines if you count the last line-feed. Clearly the last "v2.0" one is in a different format of some kind and its the only one not accepted by FTDNA. Even "changing" the new format one to the other 2.0 formats by removing new snps and filling the missing snps with no-calls doesn't seem to fix it as FTDNA wont accept that new file either with the same error message, although its possible my method of creating this new file is introducing other changes to the file I'm unaware of. I'm willing to provide the last 2 samples to anyone who has the programming skills to attempt what I did. Let me know...

[genealice]

Ah, you’re so psychologically astute about me - yes, I’m a nice person who skips through life thinking happy thoughts about everybody .

Whatever the timing involved, it does seem likely that it’s about the change in file size which Dirk from Ancestry did address:

“ ... Yes, as far as I have been told the kits tested since the update described in the blog will be smaller as snps were refined so there will be less text or lines of data."

As I’ve said, it would be good practice for FTDNA, as it offers and advertises the service, to alert people about the smaller Ancestry files and that they will not upload. For me, it’s been a waste of time and effort, although I have got to meet nice people like you - and Dirk! lol

[ltd-jean-pull]

I have just had a live chat with Dirk at Ancestry. He says the change to the raw data files was announced on 5/12/2016. https://blogs.ancestry.com/ancestry/...estrydna-chip/

The files are smaller, Dirk says, because:

"It is due to certain snps being removed as they don't pertain to what we are testing for... Not necessarily a fault, but that there have been changes made and we can't determine what they [FTDNA] do on their end ... Yes, as far as I have been told the kits tested since the update described in the blog will be smaller as snps were refined so there will be less text or lines of data."

So it looks as if the ball is in FTDNA's court. As my file uploaded ok on Gedmatch and My Heritage, it does seem to be about FTDNA.

That's nice that you think Dirk understands the issue. This is NOT related to the changes that were made in May 2016 when Ancestry started to use a new chip. FtDNA had to make some changes to accept uploads from the new chip so uploads were not possible for anyone tested with the new chip for several months. Then it was possible again, until fairly recently. Files that are being downloaded now are often missing several SNPs and will not upload to FtDNA.

Dirk is probably keen to help but doesn't know what he is talking about.