What a month!
Jaxcalibur certainly deserves credit for ushering in competitive Pokémon’s “AI era” with a bang — shortly after its launch, AI developers came out of the woodwork, causing community tumult, and inspiring me to get back into the swing of things with pkmn.
One thing that certainly has changed from 6 months ago — frontier LLMs are really, really good, and no longer are struggling to grok the Pokémon problem space. Where previously they required tons of handholding and I viewed them as more trouble than they were worth when writing quality code, the game has definitely changed and they’re now an effective tool for testing out designs / uncovering bugs / proposing optimizations / etc. As such, the pkmn organization’s policy on LLM use is in need of updating:
- LLMs will still never be used to write any text output intended for human consumption and checked into a pkmn repository
- flagging typos and grammatical errors is OK, but actually writing or rewriting sections is not
- text that can incidentally be read by humans but which is not intended for humans and is labeled as AI-generated is acceptable, e.g., the work-in-progress
IMPLEMENTATION.mdof the pkmn.ai arena
- LLMs will be allowed to write pkmn code, but every line requires manual review (“vibecoding” by Karpathy’s original definition is only allowed for throwaway programs with verifiable output)
The big usages of LLMs for pkmn this month have been:
- cleaning up pkmn/engine to remove workarounds for outdated Zig bugs and reworking the build so that it would stop poisoning Zig’s build cache
- the white whale of engine work: finishing the long-languishing
hazebranch which adds support totransitionsfor Haze, something which has been blocking development on the engine for ages - fixing the Smogon teams scraper workflow
- helping to finally update pkmn/ps after numerous large upstream changes (e.g., tags and translations) necessitated an overhaul
- helping to work out kinks in the design for the forthcoming pkmn.ai arena
I did manage to do some good ol’ fashioned coding this month as well in the form of decompose & recompose helpers for the engine, but admittedly even those were inspired by an idea open-cover’s LLM had.
Future work includes getting the engine back in sync with upstream, the major protocol overhaul (which will impact @pkmn/protocol and the engine), finishing items on the engine’s roadmap, potentially doing targeted Generation II work, and actually implementing the pkmn.ai arena.
On that note, while agents have been assisting with the coding, much of my month has been spent focused on handling the influx of Pokémon AI projects. Furthering the field of Pokémon AI has always been a major driver for pkmn’s work in general, and it’s nice to be positioned to help benefit this burgeoning community. There has been a great surge in membership in the pkmn Discord, and while some newcomers are likely struggling to adapt to the IRC LARP that is “The Cabal” what with its single channel and lack of embeds1, long-time members are likely reeling from the burst of activity (or, activity that isn’t just angry development monologues…). The Discord will evolve over time just like its membership has, but it feels to be in a fairly good place right now where ideas are being shared and there’s a good deal of signal in the noise.
I’ve been collaborating with Smogon and Pokémon Showdown to help hammer out community AI policy — there will likely be a joint summary post on this at some point in the near future. I ended up publishing the current responsible release guidelines a little early to be able to have something to point to in this still very early period of AI acceptance in competitive Pokémon. I have worked to help develop policies, alternatives, and countermeasures that I am optimistic will be able to allow for Pokémon AI to flourish while still protecting the competitive integrity of the game for humans.
I’m most excited to soon be launching the pkmn.ai arena (Olympus Mons) — rough timeline given that the hardware has just arrived, the team data is approved but still forthcoming, and I’ll be travelling for most of November (Japan!):
- publish rules and allow for registration starting sometime before the end of next week
- hopefully begin accepting ladder submissions around November 15
- first hosted tournament details (whatever the upcoming VGC OTS Bo3 ruleset is) announced mid December, to occur sometime in the last 1/3rd of the lifetime of the format2
The design for this hosted sandbox was quite challenging as there are a lot of important tradeoffs3 to consider, but it probably helps that I’ve been planning on building this for several years at this point and have had a lot of time to think. While I’m OK waiting to make sure everything is perfect for the long-term arena, I don’t want developers to need to wait to have a good environment for testing their work. As such, I’ve partnered with pelle who has done a great job meeting the community’s needs with vgc.bot and we’re releasing a sister site, pkmn.bot, for Smogon / Pokémon Showdown formats! This site will serve as one way of qualifying for entrance into the pkmn.ai arena and is expected to just be generally useful for getting immediate feedback on your project’s performance.
— pre
-
Every so often someone new joins who has never used IRC (and who has never learned about the benefits of “lurking”) and proposes that things could be improved if only there were a little more structure. The lack of channels is intentional. It is a feature, not a bug. The whole point is that everyone is forced to read about projects or topics that they think might not completely be relevant to their narrow interests (cross-pollination).↩︎
-
The hosted VGC ladder is going to be seasonal and the current plan is to retire the old ladder and stand up the new ladder around ~1/3 of the way into each new format (to allow developers time to retool and retrain their models) and then host a tournament near the end of the season. New live ladders on vgc.bot are likely to be set up soon after the new format gets implemented on Pokémon Showdown.↩︎
-
A hosted arena with blind-pair matches from a fixed team pool for a limited number of frozen formats is a great solution for an evaluation server but doesn’t allow for observability or expressing and testing agents in every way one might hope. Periodic hosted tournaments help remove hardware resouce contraints and broaden format options, and the live pkmn.bot ladder also allows for quick transparent feedback during testing and expressing strength through team selection. All these options are still not completely sufficient at exploring the many different ways to compare Pokémon AI agents, but probably cover a pretty relevant subset.↩︎