Uber voting ratios - revisited

[Cathy]

The results of the first round of stage 3 voting have brought the issues raised in my April thread to the forefront. The suspect voters, who gained voting rights after around three months of testing and experience with the suspects, and who voted in the poll before it was closed (nine people did not vote in time), decided to remove Garchomp from the next round of stage 3 by a margin of 44 to 41, which is three people out of 85, or 3.5% of the voting body.

If Garchomp had received three more votes for "not uber" then he would be allowed in the next round of stage 3, but he would not be immediately allowed back into standard yet. This raises the question of whether the vote was convincing enough to remove Garchomp from the next round of testing—or whether more testing of Garchomp should be permitted in the next round. It is tempting to consider only the present question before us—whether to remove Garchomp from the remainder of stage 3—but eventually stage 3 will come to an end, so we have to consider the long term effects of our policy on what ratio of the voters voting "uber" is good enough to consider the pokemon uber.

If we accept that 50%+1 (a "simple majority") is not good enough to consider a pokemon uber then we need to recognise all of the effects this has on the competitive community, namely allowing into tournaments and the ladder a pokemon that a simple majority of people consider uber. Allowing this "simple majority uber" pokemon into the standard metagame could potentially have negative effects on the game, or on the community at large. On the other hand, we need to weigh these negatives against the advantages of allowing a pokemon to receive more testing, namely making sure we are creating the best rules for all players of the game—including future players who are not yet Smogon members.

The purpose of this thread is to establish under what conditions we are willing to declare a suspect "uber, somewhat permanently" where "somewhat permanently" means as far as stage 3 results is concerned. By voting for a condition, you mean as a corollary that if the condition is not satisfied then the pokemon is not considered to be uber yet. There are several different models we can adopt for this. My list here is definitely not exhaustive, but it lists quite a few options, and you should select the direction in which you want to proceed. We want to see which one has the most support. Here is an explanation of each option:

1. Simple majority to consider a pokemon uber

This is the status quo. Garchomp is not tested further in stage 3. As a corollary of supporting this condition, if and only if 50% or less of the voters vote the suspect uber, it is not considered to be uber yet.

2. Some ratio larger than 50%+1 to consider a pokemon uber

The advantage of a larger ratio is to prevent a pokemon from being banned based on a close vote. If you vote for one of these, you agree the advantages of more testing outweigh any disadvantages, including possible long term disadvantages to tournament and ladder play. In particular, all of these would allow Garchomp to be tested further in stage 3.

Which ratio above 50%+1 we adopt is ultimately arbitrary. There is no mystical significance to 3/5 or to 2/3 or to 4/5. However, there are all greater than 50%+1 and all represent more inertia to declare a pokemon uber. If you support a ratio above 50%+1, vote for whichever you prefer, or some combination thereof: there are very few justifications to be made on this point.

As a corollary of supporting this condition, if fewer people than the chosen ratio vote the suspect uber, it is not considered to be uber yet.

If this option wins, then we will later vote on exactly which ratio to adopt since it is, after all, arbitrary.

3. Two simple majorities in a row in the same stage to consider a pokemon uber

This condition means that a pokemon has to be voted uber by a simple majority twice in a row within the same stage of suspect testing to be considered uber. For example, under this condition, Garchomp is allowed into the next round of stage 3, but if he is voted uber by a simple majority again, he is declared uber.

If we choose to proceed in this direction, then we can also set an arbitrary ratio > 50%+1 that, if achieved for the "uber" side, allows a suspect to be declared uber without a second round of voting.

As a corollary of supporting this condition, a suspect is not considered to be uber until it has voted uber by simple majority twice in a row in the same stage of suspect testing or voted uber by a supermajority > 50%+1 to be determined later (e.g. 2/3+1).


Note: This poll is a straw poll in the sense that its results are not binding on anybody. The goal is to establish what the community favours.

 

[jumpluff]

I still can't vote on the poll for some absurd reason (considering I can post), so I am convinced by option 2, a ratio > 50%+1.

 

[Philip7086]

I agree with the notion that on-the-fence Pokemon, such as Garchomp, may have been erroneously banned and deserve to be re-tested for certainty's sake. However, I do believe that ultimately, the tiers should be determined by what the majority of users (>50%) deem fit. I do not see the need for an arbitrary majority (2/3, etc.) to be the threshold for banning Pokemon, if said Pokemon is consistently being voted Uber by a simple majority. In the end, you run the risk of allowing a Pokemon whom most people think is Uber to roam free in the OU environment if you make the Ubers requirement anything more than 51%.

However, this brings into question how to determine if a Pokemon needs to be re-tested. For instance, if option 3 is selected as the optimal criteria for banning a Pokemon to Ubers, then Garchomp, Latios, and Shaymin-S must all be re-tested. I think a distinction needs to be made between an on-the-fence Uber and an unquestionable Uber to weed out the obvious Ubers from being thrown into the next stage of testing. For this distinction, I see no problem with an arbitrary majority being the deciding standard. There can be no question about a Pokemon's tier status if 2/3s of the voters thought it was Uber.

To sum it all up, I think two simple majorities in a row is a reasonable standard for banning a suspect to Ubers. However, if the first vote ends in an arbitrary (to be determined) majority, such as 2/3, then a second simple majority should not be needed.

 

[Sapientia]

You cannot avoid that a vote is decided by few votes, if it needs 2/3 of the vote, because you still can have 64 percent or 70 percent. Also a problem is, if you need 66 percent to be considered as an uber that if it has 60 percent it will be tested again, even though the majority of the voters don't want it to be tested, what is non-sense in my opinion.
Also if you need 2/3 you will not have many results and the suspect test will take almost forever, because iirc the first Garchomp vote was the only vote that had an arbitrary majority and I don't think that votes in the future will hit this 2/3 of the votes

 

[Aldaron]

I would like a 4th option: "Some ratio larger than 50% + 1 OR voted uber twice by simple majority in stage 3."

This is necessary for 2 reasons: if we go by only the ratio larger than 50% + 1 "garchomp might never get banned." This is obviously a practical over idealogical concern; if Garchomp is frequently erring on the side of slightly uber in stage 3, I would rather take the practical, time efficient approach. The second reason is that if we only go by voted by a simple majority twice, we would have to retest skymin / latios, and I think it is pretty clear that both don't need retesting (since both were voted uber by over 66%).

Anyway, since this suspect process has been all about compromises, I see no reason why for the sake of further testing (not declaring ou / uber, just further testing), we can't have option 4: "Some ratio larger than 50% + 1 OR voted uber twice by simple majority in stage 3."

 

[Blue Kirby]

I agree wholeheartedly with Aldaron's proposal.

 

[reachzero]

As it so happens, we are wrestling with this issue in UU as well (regarding the possibility of retesting Raikou and Froslass, both of which are banned on very narrow margins), so this actually has some relevance beyond Stage 3, as well. Aldaron's proposal sounds pretty handy to me, in that it allows you to get the "obviously Uber" stuff banned right away, while putting a reasonable limit to how long we can argue stuff that will always be contentious and split opinion (that's you, Garchomp and Shaymin-S).

 

[Articanus]

I would have gone with Aldaron's proposal but since that is not available I selected #3.

 

[Hipmonlee]

When I suggested a second simple majority I intended exactly what Aldaron proposed.

I expect that was what was meant by #3 but I am unsure so I wont vote until you clarify that point.

Have a nice day.

 

[Flashstorm1]

As with most everyone else here, I am also in support of Phillip and Aldaron's proposal. I would see no large disadvantages in allowing Garchomp to be retested, as with Latios and Skymin out of the picture, Garchomp might be able to earn a supermajority of votes in round 2 were it allowed to be retested. In addition, Garchomp never proved itself to be overly broken as Latios and Skymin (shown by the votes), so I feel it is certainly a worthy candidate for further testing.

Edit: Will refrain from voting for now until Aldaron's proposed "option #4" is either accepted or rejected.

Edit #2: Voted for option #3 in response to Colin's post below mine, as it seems like a fair compromise for both sides

 

[Cathy]

What I intended was that the option you select represents a general direction to proceed. So if we decide to proceed in the direction of #3, we can afterward set an arbitrary ratio that allows us to skip a second vote. Normally changing the meaning of poll options after starting a poll would be highly questionable, but all three people who have voted for #3 so far (Philip, BlueKirby, and Articanus) agreed with this model, and I just going to clarify in the opening post that this was intended to be a feature of #3.

 

[Elevator Music]

For the same reasoning as Flashstorm etc, I'm voting option 3 as it seems like the best idea.

 

[Jumpman16]

Also if you need 2/3 you will not have many results and the suspect test will take almost forever, because iirc the first Garchomp vote was the only vote that had an arbitrary majority and I don't think that votes in the future will hit this 2/3 of the votes

Manaphy was voted OU by a 58% majority, which serves to strengthen your point. If we were to put any of the non-simple majority proposals in place, we would have to honor this both ways, meaning that, for whatever reason, if Manaphy were to be voted uber by simple majority or Garchomp voted OU by simple majority, we would need a Stage 3-3 to test them further.

"Then what?" Neither would have been given the same Stage 3 tag twice in a row, which is the only way they would be decided on with any finality under the given proposals. What if they're voted on the same way they were in Stage 3-1? Is there a Stage 3-4 in that case? Or, if we would be content to say "best two out of three", how much did we need to perform Stages 1 and 2 in the first place, given that at this point we would very much be trying to sort out the giant explosion of a mess to which playing with all the Suspect at first last summer was originally likened?

Either way, those who champion "some ratio larger than 50%+1" absolutely need to decide on what that ratio is. Tangerine has stated he would be ok with 55%. Others cite 67%. This is very important to decide on.

I am willing, at least for the sake of argument, to accept Aldaron's proposal, because I agree with it more than the second and third options. That said, there is a sense of hindsight being applied here that I'm not very thrilled with. Changing the voting rules of Stage 3 in the middle of the Stage is likely less ideal then honoring a simple majority once, given everything that's gone into making sure our voting pool was the most qualified. This element of "supermajority?" either would not be present if I ignored every proposal, or should have been present from the very beginning. I prefer either to having the element change considerably in the middle of Stage 3. It will stand to make people lean more towards the final conclusive "tag" (Uber/OU) in subsequent stages, since if the tags always alternate, we will "never finish".

 

[lati0s]

I voted for two simple majority's in a row, though I agree aldronon's suggestion is probably best. I think all close votes should be retested due to the nature of the stage 3 voting.

 

[Tangerine]

The biggest reason I dislike the double simple majority option is that people are so fixated in what they believe that no amount of retesting is going to change these user's minds. Meaning that it's very likely it's going to be the same users voting the same options again, so we're just leaving it out to circumstances to make our decisions.

I don't have enough faith in the members of this community (and who can blame me really) to see double simple majority as a plausible option. I think the only way people will be convinced if they simply are forced to play with it for a long time to the point where 1) people find ways to deal with it, 2) people overwhelmingly decide it is broken and they want it gone - either of the options should lead to a stronger result than a 50/50.

 

[MythTrainerInfinity]

I really don't like using simple majorities either, but the third choice seems to be the best option on the poll. Also, agreeing that Aldaron's proposal is better than the poll's options.

I'm agreeing with Tangerine that 55% is good (although I'd be cool with anything up to 60%, but not lower than 55%). 2/3s does seem like a bit much and with how some of the votes have been going it would take us even longer to finish a cycle...

I mentioned in the other thread I'd love to see the voters express their opinions publicly in a discussion open to only them. I think that not only battling skill is important to these tests, but the ability to discuss and make the best vote possible. The last thing I'd like to see in these tests is a voter's misconception going unchecked.

We better not find someone telling us that the princess is in another castle at Stage 3-4...

 

[Cathy]

If we agree to adopt some ratio above a simple majority then we can compromise on what ratio later. I don't see a compelling reason to decide on it before first deciding to adopt some ratio above a simple majority. As I have emphasised many times, there is no special significance of any particular ratio above a simple majority: which one we will adopt is just the one that turns out to the most amenable to the community.

The first thing I want to establish is whether there is a will to change the status quo at all. (So far only 21.88% of the votes have been for the status quo, so you could say it is looking that way.)

 

[Earthworm]

The biggest reason I dislike the double simple majority option is that people are so fixated in what they believe that no amount of retesting is going to change these user's minds. Meaning that it's very likely it's going to be the same users voting the same options again, so we're just leaving it out to circumstances to make our decisions.

I don't have enough faith in the members of this community (and who can blame me really) to see double simple majority as a plausible option. I think the only way people will be convinced if they simply are forced to play with it for a long time to the point where 1) people find ways to deal with it, 2) people overwhelmingly decide it is broken and they want it gone - either of the options should lead to a stronger result than a 50/50.

I think a better solution to this is to not allow the closed-minded users (assuming they exist) to vote in the first place and that it is a problem with the voters rather than the percentage of votes that we should require. Assuming the voters are qualified and voting responsibly according to whatever criteria is required of them, I think that a simple majority is how the community should be tiering anything if we are going to use a vote at all, because it doesn't make sense to me that a minority of a group of qualified users should prevent a tiering change.

If I'm to believe that the players that qualified have shown themself to be sufficiently experienced with the suspects and that they understand (and care) that they are responsible for shaping the game for the community in general and therefore are voting by their beliefs, then I don't see why a supermajority should be required.

 

[Tangerine]

I think a better solution to this is to not allow the closed-minded users (assuming they exist) to vote in the first place and that it is a problem with the voters rather than the percentage of votes that we should require.

How would you filter that out? That kind of solution is unenforceable.

Assuming the voters are qualified and voting responsibly according to whatever criteria is required of them, I think that a simple majority is how the community should be tiering anything if we are going to use a vote at all, because it doesn't make sense to me that a minority of a group of qualified users should prevent a tiering change.

If you have ever read stark, you would know that it's no "minority" - it's an obvious majority.

If I'm to believe that the players that qualified have shown themself to be sufficiently experienced with the suspects and that they understand (and care) that they are responsible for shaping the game for the community in general and therefore are voting by their beliefs, then I don't see why a supermajority should be required.

Because 99.9% of the people who profess that they understand the game enough don't really understand it and just voting things based on whatever they feel like rather than actual reasons based on solid theory.

 

[Earthworm]

How would you filter that out? That kind of solution is unenforceable.

Well, having to write paragraphs should deter the people that don't care about voting responsibly. Then having their reasoning judged by someone that maintains a neutral-as-possible stance would filter out stupid reasoning, and presumably also help with the 'not understanding the game' thing.

If you have ever read stark, you would know that it's no "minority" - it's an obvious majority.

I meant the voting minority, so the 4X% that voted Garchomp OU for example is a minority and in my opinion should not be preventing a tier change or not a tier change.

Because 99.9% of the people who profess that they understand the game enough don't really understand it and just voting things based on whatever they feel like rather than actual reasons based on solid theory.

If this deep understanding is really what is required and we really have 0.1% people that do have this, then I think we should make this the requirement for voting rather than allow closed-minded users to make next to worthless votes and tier based on the simple majority of the 0.1%. However, I don't think this percentage is accurate and even though my opinion is that nobody really understands the game at this really deep level you're referring to, I think that we still have users that understand the game well enough to determine what is uber and what is not to a reasonable degree of accuracy. We have plenty of statistical data, players with experience, and game theory plus non-formal debates across a wide range of metagames, one of which is or will hopefully be the ideal OU metagame. All of the voters at least have access to these, so I'd argue that if we have determined which of the users have enough experience to vote and the users are voting responsibly and taking everything into account, then a simple majority is enough to determine the suspects' fates.

 

[Tangerine]

Well, having to write paragraphs should deter the people that don't care about voting responsibly. Then having their reasoning judged by someone that maintains a neutral-as-possible stance would filter out stupid reasoning, and presumably also help with the 'not understanding the game' thing.

So again.

How will this filter out closed minded users?

I meant the voting minority, so the 4X% that voted Garchomp OU for example is a minority and in my opinion should not be preventing a tier change or not a tier change.

Ah, I've misread. In that case it's completely irrelevant to my point or the topic at hand so I don't know why you responded to it in that manner - you're missing my objection completely.

I think that we still have users that understand the game well enough to determine what is uber and what is not to a reasonable degree of accuracy.

No they don't. Or else this entire process would have never happened to begin with.

We have plenty of statistical data, players with experience, and game theory plus non-formal debates across a wide range of metagames, one of which is or will hopefully be the ideal OU metagame.

Have you ever read these debates? People don't know how to argue at all. Secondly, data means jack shit if they don't have a theory to interpret it - which they 100% don't - they just look at it and say "lol look at those pretty numbers, here's what i think about it" yet half of the people can't even process causes and effects. Statistics mean absolutely nothing.

All of the voters at least have access to these, so I'd argue that if we have determined which of the users have enough experience to vote and the users are voting responsibly and taking everything into account, then a simple majority is enough to determine the suspects' fates.

Yeah everyone has access to all the statistical data we have out there.

Are we all experts?

no.


EDIT: Public apology to earthworm... I really shouldn't have said that and I apologize. Sorry :(

 

[lilyhollow]

All of the voters at least have access to these, so I'd argue that if we have determined which of the users have enough experience to vote and the users are voting responsibly and taking everything into account, then a simple majority is enough to determine the suspects' fates.

I disagree. When you increase the credibility of the entire voter pool, that means the minority's opinion just got that much more valid, just like the majority did. The nature of this conflict will not have been changed at all; you'll just be getting rid of bad voters and that's it.

In that same vein, I feel that Hipmonlee/Aldaron's proposal in no way suggests a more "convincing" conclusion to any Suspect vote. That isn't to say that it's necessarily a bad proposal (I obviously prefer it to our current system because it automatically makes it more difficult for a Pokemon to be voted Uber), just that it isn't any more convincing for a Pokemon to get voted Uber by 2% twice, three times, or any number of times greater than one; again, no matter how you look at it, a sizable portion of our voters are going to be sitting around wondering what the hell is wrong with the rest of us, and need to be taken seriously if our sole objective is to guarantee (or close to it) a correct decision.

Obviously Aldaron already pretty much stated that this is an issue of practicality, but I just want to clarify by saying that I think this is only an issue of practicality. If Garchomp gets voted Uber by 2% twice, "fuck it we're tired of voting for Garchomp anyway, let's just throw him in Ubers already." That's the "kind" of compromise this is, not one where we're sacrificing "some" credibility for "some" practicality; it's purely practical, and if that wasn't your mentality when you made your vote, I'd seriously be questioning it right now. I'm saying all this basically because I'm reading through this thread thinking that I'm running through some happy flowery meadow of wonderful compromise, and I'm worried that some people might not be looking at what that compromise actually "means." If that isn't the case, you can pretty much ignore this post because, again, I still feel that option 3 has plenty of merit all its own.

 

[Philip7086]

Tangerine, your argument about close-minded people works the other way around too, you know. As in, there are people who will think a Pokemon deserves to be in OU no matter what the evidence is against it. If that's the case, then why should anything more than a simple majority determine a Pokemon's tiering? You say that because if something is indeed broken, then after a while, even the stubborn people will break and think it is Uber, well I disagree. If they are close-minded by definition, then they will continue their beliefs that a Pokemon is OU for as long as they play. And even if what you assume were to be true, how long are we all supposed to wait? Garchomp has been "tested" for over a year, and it's still on the fence with its tiering. If we opt for anything more than a simple majority, we risk having a minority of close-minded OU voters decide the metagame for a long time. So long, in fact, that I don't think we will ever reach a balanced metagame before the next generation comes out. You act as if we all have an infinite amount of time to run all of these tests, but that's simply not the case. I, for one, would like to see a stable environment in which I can confidently build a team that I won't have to tweak at the next whim of ignorant voters.

I definitely think the issue is with the quality of voters selected, not the quantity voting in either direction. Of course there's no way to get a group of omnicient voters who understand everything perfectly; or else all votes would be 100% for either tier. Instead, we should focus on weeding out those who don't really care or don't have a solid understanding of the metagame. I personally know of at least 5 people who voted in Stage 3 that didn't give a fuck and voted Garchomp to OU for fun, because they wanted to spice up the metagame. These types of people can easily be taken out of the equation by forcing voters to submit a paragraph explaining their votes, like before. As for weeding out people who don't have a solid understanding of the metagame, I think a more strict selection criteria should be implemented. I think it is evident that the criteria for Stage 3 voters was too relaxed, when considering how many people laughed at the fact that they qualified. I do not mean to undermine the efforts of Jump and everyone else involved with the selection process. In fact, I am very grateful that they are putting the effort required to get this done in the first place. However, I'm sure they would not want all their efforts to be in vain because less-than-desireable voters were allowed to skew the results.

If the voter selection process is changed to weed out the poor voters, then I think Aldaron's proposal (which is exactly what I said two posts before him, but put in proposal form and worded better) is the best method of determining tiers. I understand that it takes a lot of effort to do the kind of filtering required to select the best possible voters, but I think expanding the staff of people qualified to read voters' paragraphs could make things run faster and take some of the burden off of Jump and Aeolus's shoulders. This would allow for the best possible voting environment.

 

[Jumpman16]

Have you ever read these debates? People don't know how to argue at all.

If this is actually true, and...

99.9% of the people who profess that they understand the game enough don't really understand it and just voting things based on whatever they feel like rather than actual reasons based on solid theory.

...this is also true, then what exactly do you think we're going to get out of the Suspect Test process or any tiering process, anyway?

I'd argue that if we have determined which of the users have enough experience to vote and the users are voting responsibly and taking everything into account, then a simple majority is enough to determine the suspects' fates.

And I believe that the way we have filled our voting pool has allowed us to determine, quite literally, "which of the users have enough experience to vote". No matter what anyone has to say about the merits of SEXP, I am confident that the voters all knew what they were talking about.

Also Aeolus and I aren't reading 120-150 bold votes every iteration of Stage 3. That's actually the main reason the pool is four times as large as previous pools, and with more of an emphasis on actual experience with the Suspects to make the pool arguably more pure than past pools in spite of the larger size, there's much less of a need for bold voting.

If we agree to adopt some ratio above a simple majority then we can compromise on what ratio later. I don't see a compelling reason to decide on it before first deciding to adopt some ratio above a simple majority.

"Later" doesn't mean a thing to me when we are already figuratively compromising the integrity of the Stage 3 process by potentially changing what "majority" we go by in the middle of the Stage, as I've pointed out. You may clarify that your "later" would come before the votes are actually made, but that is much more important than you have indicated thus far. How would a compromise on a ratio after knowing Garchomp or Manaphy was voted uber or ou by 58% or 61% in Stage 3-2 even remotely be unbiased, especially when we know which one will mean at least a month more of testing?

 

[Aeolus]

What sort of majority do we need to determine what sort majority we want to make decisions?

I don't think that requiring 55%, 60% or 66% to declare a Pokemon uber any more meritorious than 50% + 1. I understand and support the idea that the default position of any Pokemon is "unbanned"... but I don't think that requiring a super-majority (or other ratio) necessarily supports that precept.

If a simple majority of what I consider to be a highly qualified group of people vote a Pokemon uber, then that is good enough.