Challenge Accepted! After being insulted like anything, I set upon myself to prove the world. I finally did an AB test with a friend of mine who took care of changing the tubes etc while I listened.
SETUPSpeakersAktimate Micro
Swan M50W
PreampProject Ember
DACAsus STX
All connected through Belkin Rockstar splitter
math.stackexchange.com/questions/151810/probability-of-3-heads-in-10-coin-flipswww.google.co.in/search?newwindow=1&site=&source=hp&q=binomial+calculator+online+free&oq=binomail+calc&gs_l=hp.3.2.0i10l10.552.3399.0.5279.17.16.0.0.0.0.238.2064.5j9j2.16.0....0...1c.1.27.hp..7.10.1009.Xa-YE_Y8EGceasycalculation.com/statistics/binomial-distribution.phpAB TESTING METHOD 1We selected 2 tubes at a time. Wrote the name of each 5 times on a paper. So a total of 10 names. My friend cut it into pieces and selected the one which came to his hand. I was in the other room waiting for the changes or lack of changes
After he randomly selected the tube, I had to guess which one was it based on a small section of a particular song or two I had already listened it with, before we started.
Initially, I was using a song from Bhaag Milkha Bhaag but after the first round changed to a song from Chennai Express because the first song was more difficult to clearly articulate the differences. Both are bollywood flicks (Hindi films).
Round 1: Lorenz 12AU7 v/s Sylvania Conn 1959 12AU7I got 6/10.
The differences were there but very subtle. Although I could tell a difference I still wasn't sure which was which every time.
Probability of error: Approx 20%
Round 2: Shu Guang Hifi 12AU7 v/s RCA ClearTop 12AU7I got 8/10.
The differences were there. Subtle but noticeable, definitely not night and day!
Probability of error: Approx 4%
Round 3: Siemens 6922 v/s Shu Guang Hifi 12AU7I got 10/10
Probability of error: Less than 0.1%
Round 4: Lorenz 12AU7 v/s Sylvania Conn 1959 12AU7I got 7/10.
Being disappointed with my first try, I tried again. Was a bit easier with the second song. But the differences were still subtle.
Probability of error: Approx 12%
AB TESTING METHOD 2
I asked my friend to randomly choose any of the tubes first, and then put the second tube. From one set at a time (from the 3 sets above). This means, I knew he was choosing between RCA and Shu Guang, but didn't know which was used first and which was used second. And so on. Used the same song from Chennai Express.
I was allowed to hear each tube as many times as I wanted, without him telling me another except No.1 or No.2. And I had to tell which was No.1 and which was No.2.
Almost always, just on hearing the first tube the first time, I was very confident as to which it was and immediately spelled it. I rarely waited to hear the second tube, leave alone for more than 1 round. And still got very few of them wrong
Round 1: Lorenz 12AU7 v/s Sylvania Conn 1959 12AU7I got 3/5.
I got the first two wrong because I got confused in how they sounded. However, I got the final 3 right in a row!
Again, differences are there, but subtle and not very imminent!
Probability of error: Approx 31%
Round 2: RCA ClearTop 12AU7 v/s Shu Guang Hifi 12AU7I got 5/5
What more should I say?
Probability of error: Approx 3%
We didn't try Siemens 6922 v/s Shu Guang Hifi 12AU7 because I got 10/10 in the previous test. So this wasn't needed.
Please NOTEMy friend was shocked as to how I could notice the subtle differences in the tube. He was in awe and refused to take the test because he felt the differences were rather small
ConclusionNow, this is a complex formula I used to calculate the overall error probability, you can't just add or multiply stuff together for this. But being from a mathematical background, I have tried my best to be as accurate as possible. For doubts, please ask Frans or Jeremy as they know more maths than me as they use it in their daily life!
Overall probability of error: Approx 9.2%
The differences in tubes can be summed up as:5% Very subtle differences
5% Somewhat noticeable differences
5% Difference in dynamics
25% Difference in bass
30% Difference in gain
30% Perception Bias
I admit that there is some degree of perception bias that does creep in, regardless of whether we feel it or not. No matter how much difference we feel, some of it is perception bias. Not all of it, for there are differences in tubes. But the differences are usually subtle or at most moderate.
Between two tubes which are very different in sound signature or sound quality, the difference could be immediate and huge. But that is not the case with most tubes, especially not in the same price range. Although the differences exist between each tube, and although you can't deny that they can indeed be appreciated with keen ears, the differences are not as big as they seem initially, especially once you come down to AB testing.
I still believe that there are differences in tubes, and each has an unique flavor, nevertheless, AB testing proves that although the differences are there, they are not huge. They are on the subtle and somewhat noticeable side. Once you get a tube you like, only some tweaks are possible with a change of tube, and that perfect tube may or may not be out there, and even if it is, it definitely won't make a night and day difference. But the difference will certainly be worth it with a good tube, and the experience will be worth it, and very significant, just not night and day.
I hope I have proved that I wasn't boasting. A part of it was indeed perception bias, but the changes I hear are indeed very real and do exist with my setup (for somebody with very keen ears), but not as huge as I thought they were. They exist, are worth buying tubes, but not worth wasting a fortune
Thank You
Aakshey