Pokemon Testing: Standard Ladder or Suspect Ladder?

[Aeolus]

Ok, soon we will be making a decision about how to handle the next round of suspect testing. With participation at a record low for the most recent test of Latias (only 24 people qualified for voting consideration), the idea that testing could be carried out on the Standard ladder surfaced. Essentially, the decisions hinges on one tradeoff: participation vs. stability of the standard ladder.

If we conduct the test on the standard ladder, participation will be through the roof and many more users will qualify to vote on the fate of the suspect. However, there is the chance that many users will be turned off by the appearance of a formerly uber pokemon on the ladder and will leave Smogon's server for an alternative server.

If we conduct the test on the suspect ladder, participation will be low and the decision to (semi-permanently) alter the standard game after the test will rest with very few. The upside to this idea is that our standard ladder remains stable and the status quo of our dominance atop the shoddy registry remains in tact.

We want your input.

 

[Great Sage]

As a compromise, perhaps "reverse" the current positions of the standard and suspect ladders? The suspect ladder will become the default ladder like standard is now, but there will still be an alternative ladder available to play "real" OU.

 

[Imran]

I am all for keeping the suspect ladder, I outlined my reasoning here.

EDIT: That post is a little wordy, in short I'm worried about participation on the standard ladder, as you outlined in the OP

 

[kd24]

i think we should just use the standard ladder for testing. 1.) i really doubt we would lose many people, if any at all. people are attracted to smogons name because many battlers on here are the best and there is still a uu and uber ladder to use. secondly, the people "testing" the suspect will become huge. It will become much easier to determine the metagame and to achieve the proper voting rating. the more people voting, the better chance that the desicion is correct.

 

[Caelum]

the more people voting, the better chance that the desicion is correct.

What in the world ever gave you that idea.

Before I make a comment, would you be resetting the standard ladder ratings each month? If so, that's not fair to people that do not want to participate.

 

[kd24]

'-' i meant that having a pool of 60-80 voters or so should be much more effective than a small pool of 20, because you could easily have 10 of those 20 people making the wrong desicion for a clearly broken pokemon ;-;

 

[Bologo]

Using the standard ladder for testing sounds like an okay idea, and like KD said, I really doubt we'd lose that many people. We already commonly go over our user limit on the server, and besides, testing is usually an exciting thing.

However, if we do this, we have to reset ratings this time. In the Skymin test, there were a ton of people who didn't even participate in the test (ie. they didn't play for several weeks), and they still qualified due to their high rating the month before. This is probably the biggest drawback to testing on the standard ladder, because I'm not sure how many people want their rating to be resetted without wanting to test.

We could try that older suggestion of registering a new shoddy nick in some thread so that people would have a fresh rating, and they would be people that want to test. That could be our alternate to resetting ratings if people are into that.

Also, remembering the Wobb test on the Official server, we need to give people a lot of warning that the tests will occur on the standard ladder. However, I'm sure that's a given.

 

[reachzero]

I am in favor of Great Sage's suggested compromise. If the Suspect turns out to be nearly unbeatable (that is to say, Uber on merit), it would quickly become very disruptive and upset the metagame (arguably the way Abomasnow/Walrein is in UU). As a result, the metagame will become much less enjoyable, and their would be no metagame to fall back on for those who want to opt out. However, if Suspect is the "default" ladder, no one will be able to plead ignorance, and I see participation rising significantly.

 

[Jumpman16]

As I have shared privately with Aeo, Doug and Mekk, I feel that "participation" during the actual past Suspect Test may not have been as low as we are stating. When we keep in mind that we are definitely going to go through with the idea of "Suspect EXP" that I originally raised to Aeo and Doug and that Doug has the tools to easily calculate now, we are reminded that we stand to gain many qualified "participants" that just missed out on out the 1655/65 requirements. I am prepared to make people like Bolts and ipl actually vote regardless of whatever numbers they can put up if they don't use the Suspect throughout the whole test, and besides that, the numbers we have clearly indicate that these two were the only ones in the pool of 24 qualifiers who didn't use the Suspect. We stand to gain many, many more, and my point is that the number of people we gain is entirely up to us.

Doug is going to send me, Aeo and Mekk some numbers on who would have passed on the "unpublished requirement" of 1600/70. I will reiterate that those numbers were totally pulled out of my ass. However, the point of the actual numbers Doug is going to get us can serve as the benchmark for the number of voters we feel will be enough for each Suspect Test round. We have not yet decided how much Latias Experience is "enough", but we also have those numbers (Suspect Used and Suspect Sent) and can decide what's high enough.

I do feel that totally doing away with the Suspect Ladder is, given our goal to increase "participation", not only unnecessary, but potentially very harmful. We are going to be testing some very wacky things in the coming months after we're done with Lati@s and Manaphy, including Evasion, OHKOs and possibly Species Clause. We don't want to shy away from testing these things, but it will definitely cease to be "standard pokemon" as we even remotely know it, and if the only options every battler has are to "play with Evasion or play unrated (or UU, ubers etc) or play somewhere else", it's going to be really shitty. And I don't think making a secondary ladder that will be just like the old Standard for people to play rated battles on is necessarily a good solution, given that we may very well have solved our problem already by deciding on the correct Suspect EXP requirements.

 

[Kevin Garrett]

I support the notion that fewer voters isn't necessarily a bad thing. Since Jump, Aeolus, and Doug can now determine how many voters they want, I don't think the varying activity of the suspect ladder will be a major problem. Since the best way to do it on standard would be to clear everyone's rating, it wouldn't appeal to countless people. And the Smogon Tour also represents that most recent tiers on each of the standard ladders, so it would interfere with that as well.

 

[Aeroblacktyl]

random question: when will we actaully get to play 'real ou' without ever having to worry about team/metagame changes etc

 

[X-Act]

I don't care where the testing is done, really. But I'll warn you that if the testing is done in the Standard ladder, the 1655/65 requirements should be thrown out of the window.

Those requirements assume that everyone starts from a 1500/350 rating/deviation. If someone on the Standard ladder is already at a rating of, say, 1950/30, he would need to play just one or two games, lose both of them, and qualify to vote. This is what happened with the Shaymin-S vote, and that is the main reason why it was utterly flawed.

So if you choose to do it on the Standard ladder, we'll need other criteria to get the people with voting rights. I'll gladly try to help with these criteria if the Standard ladder is chosen, though.

 

[DougJustDoug]

If we test on the Standard ladder, how do we answer the user that says:

"I came here to play the normal pokemon metagame. And now I'm forced to face all these uber pokemon. I don't want to be part of this test. How do I get out of this test?"​

I don't think we should force people to play a metagame that we fully acknowledge may be broken. The word "suspect" means "We suspect it may not be broken". But the reverse is also true, we also "suspect it may be broken". If we know that the suspect metagame may be broken, I don't think we should be putting that metagame front-and-center at Smogon.

It would be like the FDA testing out new food additives by putting them in the foods in every grocery store in the world, and then seeing if the general populace starts puking their guts out. It's one thing if willing test subjects sign up to test experimental food and drugs. Those people know what they are getting into, and they do it willingly (probably for pay). But to subject the general public to something that MAY be harmful -- that's just not right.

I know we are not talking about people's lives here. This IS just pokemon, so maybe the analogy is a bit strained -- but I think the general point still holds true. We should not make everyone participate in an experiment that may be "harmful".

I'm not too worried about losing people to other servers. I think the general Smogon populace will probably stay here, even if we subject them to all sorts of metagame abuse. But, it just doesn't seem "fair" to me.

We need to encourage participation, not force it. I think there are plenty of people interested in experimenting with the metagame. We need to figure out a way to allow us to conduct the experiment, and collect meaningful feedback on the results -- without making everyone in the community become a test subject, willing or not.

 

[Aeolus]

I've altered my position some since I last spoke with Jump... but I have to say that I now agree that testing must be done on the suspect ladder rather than the standard one. I do like the idea of making the suspect ladder the default just to boost awareness of the test... but we'll talk about that later. We've also come up with a small incentive for people who choose to actively participate... but you'll learn more about that later. Thanks for all the input guys.