Big-Y matches seem broken. Possibly don't include snp pack kits

[FuriousGeorge]

I have no matches, even at my 2x Great Grandparent clade of E-CTS5856, which I believe is the biggest sub-branch of V13 (could be wrong there, but it's big).

I suppose it's possible that no one on ftdna took the Big-Y from my branch, but I know from the M35 group that at the very least people have taken SNP packs that put them in lower clades then E-CTS5856, and it looks as if some people took The Big-Y itself, also.

I reported this as a bug to ftdna, and have not heard back.

I'm trying to be charitable about this, but it's difficult.
On top of this, .BAM files can't be released to members at this time, and there is no estimation as to when that will change. This has been the case at least a few weeks, probably much longer.

It really seems like FTDNA is selling a broken product, and one that isn't cheap to boot. Judging by their response to my message (radio silence), they look more interested in taking the money than providing a working product.

Assuming this is the case for everyone, I don't think anyone should be buying the test until this is sorted out, nor should they be selling it.

Meanwhile, I'm getting messages that I have a Y12 'match', that does not meet their threshold for a match at 25 markers. Of course they know this person is not related to me, unless you count the common ancestor of everyone M35, ~4700 ya.

Rather than doing that, they should cross reference the SNPs data in their own database, and give me results who are actually potentially related to me, even if it's at GD 20.

I suspect part of the reason they don't do it is because they sell SNP tests, and that would often get a user a more specific snp, without them having to pay for it. They advertise access to other user's data, users want to share this data, and they prevent that by hiding it. Probably legal, definitely not ethical.

Alternatively, they could show nothing, and not misrepresent the data users paid to get, on purpose.

[spruithean]

The new matching algorithm for Big Y requires a difference of 30 SNPs no more. As opposed to the original 4 SNP difference. Thus, less matches are visible.

In the meantime you can download your VCF if you are wishing to upload to Yfull.

Their lack of response is because it is not a bug, just part of the new algorithm, however disappointing.

[The_Contemplator]

I have no matches, even at my 2x Great Grandparent clade of E-CTS5856, which I believe is the biggest sub-branch of V13 (could be wrong there, but it's big).

What do you mean by your 2x great grandparent clade? Does this mean you know of a paternal 3rd cousin who tested positive for CTS5856?

I reported this as a bug to ftdna, and have not heard back.
...
It really seems like FTDNA is selling a broken product, and one that isn't cheap to boot.
...
Assuming this is the case for everyone, I don't think anyone should be buying the test until this is sorted out, nor should they be selling it.

I looked for E-CTS5856 over at YFull and the estimated age is over 4000 years old. Well for the branch it is still part of which is E-CTS1273 as it hasn't been split away from it with submitted samples yet. So the reason you aren't seeing matches is because none of the other E-CTS5856 people are closely related to you within the past few thousands of years. The new Big Y algorithm will only show matches with 30 variant differences or less. So if you have about 15 or more unnamed variants and so does your distant paternal cousins, they won't show up on your list.

Meanwhile, I'm getting messages that I have a Y12 'match', that does not meet their threshold for a match at 25 markers.

Sounds like you don't have any close STR matches, which means you probably won't have close Big Y matches.

Rather than doing that, they should cross reference the SNPs data in their own database, and give me results who are actually potentially related to me, even if it's at GD 20.

I suspect part of the reason they don't do it is because they sell SNP tests, and that would often get a user a more specific snp, without them having to pay for it. They advertise access to other user's data, users want to share this data, and they prevent that by hiding it.

The Big Y will only give you matches of others who have taken the Big Y. SNP tested individuals that have not taken a Big Y are not shown in the Big Y matching list. As for getting a user "a more specific SNP without them having to pay for it". They did pay for the SNP test, but those tests don't always get you to a "more specific SNP". The Big Y is really the one that can get you there.

If a SNP test can get you to the same place that a Big Y can, then it means some distant yet close enough cousin has already been tested and those SNPs have been added to SNP packs. A SNP pack will only get you so far. SNP packs will only contains SNPs that have been discovered in tests like the Big Y. SNP packs can't discover new SNPs.

Let us say there are 3 paternal cousins that descend from a common ancestor in the past several centuries. One takes a SNP pack, one takes a Big Y, and the third takes a Big Y years later. Maybe the first got to E-CTS5856 with their SNP pack but that is where their testing stopped and nothing will change without further testing. The second person may have also reached E-CTS5856 with his Big Y. No matches and it may seem like an expensive way to reach the same point as the first person. The 3rd person finally tests and shows up as a match to that other Big Y tester. FTDNA may decide at that point to compare those two kits and create a new branch with their common unnamed variants. Without even taking more tests, their haplogroup designation is updated to some new branch E-BY#####. The SNP pack tester won't see this change as they did not take the Big Y. Only the two Big Y testers and future matches will see it.

So you may have to recruit distant and possible paternal cousins to take the tests to show up as matches to you. If t hey don't show up as close enough STR matches, no sense bothering with them taking a Big Y.

[dtvmcdonald]

I agree that it is a very big mistake not to compare and generate matches between BigY and SNP Pack results.

I'm biased because I'm admin for a large project and can do the matching myself for all in the project, which includes essentially 100% of my own Y-chromsome matches. Its very very
valuable. True, I designed the SNP Pack and at the time it was designed it was as complete as possible, including one SNP at that time unique to me.

As more BigYs in this area arrive, the pack is less complete, but it will direct people
to the correct cheap extension tests at yseq.net.

FTDNA really should do matching with SNP Packs + BigY.

[FuriousGeorge]

Thanks for the reply.

What do you mean by your 2x great grandparent clade? Does this mean you know of a paternal 3rd cousin who tested positive for CTS5856?

No, just meant in the sense of the y-tree.

I looked for E-CTS5856 over at YFull and the estimated age is over 4000 years old. Well for the branch it is still part of which is E-CTS1273 as it hasn't been split away from it with submitted samples yet. So the reason you aren't seeing matches is because none of the other E-CTS5856 people are closely related to you within the past few thousands of years.

I'm not looking for close relatives. What I had in mind is the case of a person who's done SNP testing, but not as much as another. He may match, or he may not, but based on the SNP test it would be more accurate than getting a Y12 match that doesn't match at 37.

Sounds like you don't have any close STR matches, which means you probably won't have close Big Y matches.

Yes, that's correct. Although, I'd be more interested in people related to me 'historically' anyway.

The Big Y will only give you matches of others who have taken the Big Y. SNP tested individuals that have not taken a Big Y are not shown in the Big Y matching list. As for getting a user "a more specific SNP without them having to pay for it". They did pay for the SNP test, but those tests don't always get you to a "more specific SNP". The Big Y is really the one that can get you there.

If they paid for an SNP test, and someone else paid for TBY, and they match that person with a low GD, then they can know their likely terminal clade.

They can do it now by manually looking through the csv data, and with considerable exta effort, at least they can for the M35 project. I don't want to sound dramatic, and I prefer to be charitable when possible, but there seems to be a clear incentive not to give people who have taken SNP tests this result, even though the person who took TBY almost universally wants that person to get it.

If the GD between two members is beyond the threshold for a close match, they won't be able to contact each other to confirm this, even if they are members of the same project. Perhaps not coincidentally, the 'contact member' feature of ysearch is broken, and ftdna no longer supports that site.

I have someone with GD8 who might have taken an SNP test. It's important to note that I don't know for sure if he tested, or if was assigned that clade by one of the project admins, but based on our genetic distance (he's not closer to anyone else, but not very close to me either), and because I know that most BY5022s took SNP tests, I'm inclined to believe he did.

If he were one or two markers closer, he'd be in my STR results, and then I'd be able to contact him. One of the things I would point out is that he is likely in my terminal clade. I'm not sure if, because the terminal clade was discovered with me, he could already order a relatively inexpensive test to check.

If you manually look through the project data, you see he's from the same part of the world (not that I'd want ftdna to consider that), and it's clear that he is likely my closest hit on here, much closer than the Y12 and Y25 matches I have now, which are probably only related to me through the common V13 or even M35 ancestor, in pre-history.

In my case, as I mentioned, I can't be sure. However, if SNP tests are not included, then what I've just described is definitely the case for at least some people.

[FuriousGeorge]

Thanks for the reply.

The new matching algorithm for Big Y requires a difference of 30 SNPs no more. As opposed to the original 4 SNP difference. Thus, less matches are visible.

In the meantime you can download your VCF if you are wishing to upload to Yfull.

I believe there are some caveats with using the vcf over the bam, and the only way around them is to pay again to upload a bam. I will look into it, as I could be wrong on.

[spruithean]

Last I heard YFull had said they would allow VCF users to upgrade to BAM free of charge when BAM became available again.

[FuriousGeorge]

Last I heard YFull had said they would allow VCF users to upgrade to BAM free of charge when BAM became available again.

You're right. Thanks for letting me know.

[The_Contemplator]

No, just meant in the sense of the y-tree.

I see, well whatever haplogroup you are right now would also be the same for multiple generations before you given that it isn't remotely recent. Singling out a specific ancestor just confused me so I wanted to be sure.

I'm not looking for close relatives. ... I'd be more interested in people related to me 'historically' anyway.

Sorry I wasn't clear. By "close" I meant different things relative to the type of test I was talking about. For Big Y, it wouldn't just include distant cousins with a common ancestor in genealogical times (like within 500 years or more for some). It could include matches with a common ancestor more than a thousand years ago. It varies lineage to lineage based on the number of SNP mutations that may have occurred.

If they paid for an SNP test, and someone else paid for TBY, and they match that person with a low GD, then they can know their likely terminal clade.

They can do it now by manually looking through the csv data, and with considerable exta effort... there seems to be a clear incentive not to give people who have taken SNP tests this result...

Like you said it is considerable extra manual work. There is no incentive for FTDNA to do that for free. Which is why they love project admins. Project admins will take on such tasks and are not FTDNA employees. It is also worth noting that there can be errors. If the gaps in the data are big enough, people can guess the wrong terminal clade. Some lineages also suffer from dealing with convergence. Which means that other distantly related lineages will appear closer due to having similar STR haplotypes. With one surname I follow, I can guess the terminal clade for Y37 matches easily. For another surname I follow, I have to bump that up to Y111 to be reasonably sure. Still risky in that lineage even then.

If the GD between two members is beyond the threshold for a close match, they won't be able to contact each other to confirm this, even if they are members of the same project. Perhaps not coincidentally, the 'contact member' feature of ysearch is broken, and ftdna no longer supports that site.

If the near-match is in the same project as you, contact your project admin. Have them relay a message that you wish to contact that user. Then it will be up to that near-match to reply back. As for YSearch, I don't recall why FTDNA has let it die, but it has nothing to do with stopping anyone from contacting others.

I have someone with GD8 who might have taken an SNP test. It's important to note that I don't know for sure if he tested, or if was assigned that clade by one of the project admins, but based on our genetic distance (he's not closer to anyone else, but not very close to me either), and because I know that most BY5022s took SNP tests, I'm inclined to believe he did.

Project admins are not able to change the haplogroup assignment listed by the kit's results. Only testing done or FTDNA updates can assign those. If the person with SNP or Big Y testing is in a project, you can check out the project's SNP results page. Assuming the admin did not disable it of course.

Generally when someone has taken a Big Y, you will see a longer list of SNP results for that person in the SNP result page of the project. If the list is a single one, then that person only took a single SNP test or it was part of the test FTDNA does when they can't predict a haplogroup designation. If the list is a short one, it will be something like the old Deep Clade testing or a SNP pack. If the person is an STR match, they will have "BigY" listed with the other tests they have taken. This is a relatively recent change.

I'm not sure if, because the terminal clade was discovered with me, he could already order a relatively inexpensive test to check.

Many of the SNPs can be ordered through the haplotree page by the person. However, not all of them are available. Any recently discovered ones won't be purchasable. A project admin would need to request for it to be an option. FTDNA would then look into it and see if it is viable. If only a few people have it, they may not go through with it as it won't be worth it for them. However, you could look up YSEQ and that may be a better option for you.

[dtvmcdonald]

Thanks for the reply.

If the GD between two members is beyond the threshold for a close match, they won't be able to contact each other to confirm this, even if they are members of the same project.

I don't understand that. Anybody in my large project can contact anybody else if the ask me for their email.

[dna]

W. (me) has a 4th cousin (C.).

Big Y results for C. are missing the final SNP that W., W. grandfather brother, and other men (unrelated to W. or C.) have.

Once that SNP (of BY variety) becomes available for purchase, we will of course order it. However, the discussion impressed on me that a positive result would not be incorporated into C. Big Y results.

That makes no sense whatsoever, since FTDNA position is very clear:

Due to the exploratory nature of Big Y, we cannot guarantee results for specific, known SNPs because results will vary from sample to sample. Single specific SNPs are better ordered as such.

The above comes from https://www.familytreedna.com/learn/...g/big-y/big-y/

? ? ?

Mr. W.

P.S.
Paperwork and FF results agree on W. and C. being the 4th cousins.

[Jim Barrett]

I don't understand that. Anybody in my large project can contact anybody else if the ask me for their email.

I hope you don't share the email address without the approval of the kit holder.

[The_Contemplator]

However, the discussion impressed on me that a positive result would not be incorporated into C. Big Y results.

Perhaps whoever was discussing meant, that the Big Y results won't be changed because the single SNP test is a separate thing. Like if a person takes a Big Y and a SNP Pack, the SNP pack results aren't added into the Big Y results/pages. They are separate. They do affect haplogroup if the SNP pack had downstream SNPs that the Big Y did not pick up of course. They would also show up in the haplotree section which is part of the Y-DNA buttons.

Have you tried looking at the VCF of cousin C? Look for the position of the SNP they were not shown results for. It could be that the Big Y did pick it up but in low quantities that they were excluded from Big Y results page.

[dna]

[----] Have you tried looking at the VCF of cousin C? Look for the position of the SNP they were not shown results for. It could be that the Big Y did pick it up but in low quantities that they were excluded from Big Y results page.

I have asked FTDNA for the VCF files.

Mr. W.

[wkauffman]

W. (me) has a 4th cousin (C.).

Big Y results for C. are missing the final SNP that W., W. grandfather brother, and other men (unrelated to W. or C.) have.

Once that SNP (of BY variety) becomes available for purchase, we will of course order it. However, the discussion impressed on me that a positive result would not be incorporated into C. Big Y results.

That makes no sense whatsoever, since FTDNA position is very clear:

Due to the exploratory nature of Big Y, we cannot guarantee results for specific, known SNPs because results will vary from sample to sample. Single specific SNPs are better ordered as such.

The above comes from https://www.familytreedna.com/learn/...g/big-y/big-y/

? ? ?

Mr. W.

P.S.
Paperwork and FF results agree on W. and C. being the 4th cousins.

One can NEVER expect that two independent samples will provide EXACTLY the same coverage and corresponding calls. If you are looking at a lower quality call and it isn't present as expected in a given sample it comes back to differences in quality and coverage.

FTDNA should NEVER edit in a missing call for a sample based upon other results. The undermines the basic integrity for all of the information which is dependent upon a consistent mechanism for analyzing the data. If individuals were allowed to get calls added back in to results on a case by case basis think of the effect that would have on potential downstream analysis efforts such as age estimation. You skew the statistics with manual edits.