feat: may progress update

blogposts/6-2-25.md

@@ -0,0 +1,79 @@
+---
+title: "May Progress Update"
+author: "Jon"
+author_image: "https://www.github.com/jonbarrow.png"
+date: "June 2, 2025"
+caption: "Updates covering the first half of the year"
+cover_image: "/assets/images/blogposts/6-2-25.jpg"
+---
+
+Hey all, Jon here! We know it's been quite a while since our last blog post. We planned to get these out more often, but alas more important issues and updates came first. We apologize for the lack of blog posts, and promise to get them out as often as we can. With that, halfway into the year seems like as good a time as any to summarize what we've been up to!
+
+(Note: This blog post will not cover *all* changes made since the last post, as there are far too many to list. We will be highlighting the more relevant/larger ones here)
+
+# Technical Updates
+
+Many of you voted to have more technical updates in blog posts from now on, and today marks the first time we do!
+
+## Splatoon Database Performance
+Around May 26th, 2025 we noticed random instability in Splatoon. The game would work for a while, allowing players to create new, and join existing, matches, before seemingly randomly locking up. This locking up would cause subsequent requests to never get a response from the server, resulting in failed connections. Developers [shutterbug](https://github.com/shutterbug2000) and [quarky](https://github.com/ashquarky) quickly began investigating. We were unsure where the issue was at, but we had a pretty good idea of where to look. The locks could be happening in 2 places:
+
+1. The application layer. A few examples being:
+	1. A locked [mutex](https://en.wikipedia.org/wiki/Lock_(computer_science)) which never releases. Our servers are written in [Go](https://go.dev/), and make use of [goroutines](https://go.dev/doc/effective_go#goroutines) for concurrency. Primitive Go types such as [slices](https://go.dev/doc/effective_go#slices) and [maps](https://go.dev/doc/effective_go#maps) can only be accessed by one goroutine at a time. To ensure this, a mutex is used to lock access when a goroutine needs the data, and is released when it's done with it. If a goroutine were to lock up and never release the mutex, then future goroutines will never be able to access it, and also be locked up waiting for the mutex to release.
+	2. A poorly structured SQL query could lock up the database connection at the application layer, causing future requests to no longer be able to access the database.
+2. The database layer. A few examples being:
+	1. Database corruption could lead to queries failing or getting stuck, locking up the connection.
+	2. Poor indexing causing full table scans on massive tables, locking up the connection.
+	3. Bad queries causing the table to infinitely loop and lookup in itself, locking up the connection.
+
+Shutter began by checking the code for locked mutexes and the server logs for any abnormalities. Our game server libraries are split into 3 separate modules, one of which we colloquially refer to as ["common"](https://github.com/PretendoNetwork/nex-protocols-common-go). This module holds all the "common" default implementations of features used by many games. Quarky began by investigating the database itself. Together, they were able to narrow down the problem areas to 2 functions. Lockups would happen in the [`MatchmakeExtension::AutoMatchmake_Postpone`](https://github.com/PretendoNetwork/nex-protocols-common-go/blob/50586821d228cdc2bae728797febdae51e67fe5b/matchmake-extension/auto_matchmake_postpone.go) and [`MatchMaking::UnregisterGathering`](https://github.com/PretendoNetwork/nex-protocols-common-go/blob/50586821d228cdc2bae728797febdae51e67fe5b/match-making/unregister_gathering.go) functions. These functions are responsible for creating/joining multiplayer sessions, and for deleting them when the game is over.
+
+There did not appear to be any areas where a mutex would not release, so we switched our attention back to the database itself. We had noticed some wonky behavior from the database earlier as well, later fixed by [Will](https://github.com/binaryoverload), and I had a hunch that this is where the issue would lie as well. Upon investigation, we found several instances of poor database performance, such as a lack of indexes on some "hot spots" resulting in slower queries. Most notably, we found 2 queries that had become locked, running over 7 and 9 hours respectively. These queries are what was causing the database to lock up, and requests to not get responses. But we still didn't quite know WHY they were locking. It was not initially clear if the issue was due to a looping query, poor indexes, or database corruption. The database in question only held ephemeral data about temporary matches, however, so we opted to just rebuild the database and add in the missing indexes at the same time. Doing so would cover both possibilities of database corruption and missing queries causing the locks, and so far things have once again become stable.
+
+During the investigation I also noticed a lack of a proper timeout mechanism in the Go functions that made the database queries. Regardless of the reason for *why* the queries locked up, having a timeout would prevent things from looping *forever*.
+
+This investigation has highlighted some key areas of improvement in regards to our database performance, and we have already begun investigating using even better indexes (currently being tested in Minecraft) and will begin to implement proper query timeouts. So while it is unfortunate that Splatoon was down, in the end this will result in even better performance moving forward.
+
+## `DataStore` Protocol Rework
+As mentioned above, many games share a "common" implementation of certain features. This allows us to define this functionality once and simply tell a server to use it. In the past, our philosophy was to be as unopinionated as possible in this regard, only implementing basic boilerplate for the functions and providing developers a series of configurations and hooks to use. The actual *logic* of the functions was still up to the developers to set. This was done so that developers outside of our team could more easily integrate our code into their systems, by just hooking up whatever stack they already use.
+
+We have recognized the limitations of this approach however, and have decided to become more opinionated. This first took place in the matchmaking code, made by [Dani](https://github.com/DaniElectra), making it so that all the logic for matchmaking now lives in our "common" module. This now allows our code to be more "plug-and-play", simply telling servers to use matchmaking and letting our logic do the rest.
+
+Another protocol which used this unopinionated philosophy was our [`DataStore`](https://nintendo-wiki.pretendo.network/docs/nex/protocols/datastore) implementation. `DataStore` is a protocol developed by Nintendo allowing for "object" (file) storage in games, essentially acting as a frontend for [S3](https://en.wikipedia.org/wiki/Amazon_S3). This protocol is what powers games like Super Mario Maker (courses are "objects"), Animal Crossing: New Leaf (dreams are "objects"), Mario vs. Donkey Kong: Tipping Stars (levels are "objects"), and more.
+
+We have now begun the process of moving our `DataStore` implementation into a more opinionated implementation, allowing for the same sort of "plug-and-play" nature as seen in the matchmaking changes. Progress can be seen [here](https://github.com/PretendoNetwork/nex-protocols-common-go/pull/53). Once finished, this will be our first *fully complete* protocol implementation, allowing for MANY games to get their "object" features working without any extra work. This is also why Mario vs. Donkey Kong: Tipping Stars is still offline, it is so old that it needs to be rewritten from the ground up and is mostly just `DataStore`, making it the perfect candidate to test these new changes against.
+
+Eventually we hope to do this with *all* protocols in the future.
+
+# New Games/Updates
+
+Several new games have joined us since our last blog post! Both existing games coming out of beta, and entirely new games!
+
+Also a big thank you to everyone outside of our team as well for helping debug and suggest changes!
+
+## Splatoon
+As a belated celebration for Splatoon's 10th anniversary (we deeply apologize that the stability issues mentioned above prevented us from doing anything on time), for the next round of rotations (starting June 7th) our rotations will feature previously banned stages! Allowing banned stages has been a highly requested feature for a long time, so we felt this was a good time to try it out!
+
+## Animal Crossing: New Leaf
+Thanks to the work by [shoginyan](https://github.com/shoginyan) and [shutterbug](https://github.com/shutterbug2000), Animal Crossing: New Leaf has officially left beta testing and is available to everyone! Note that Dream Suite is not yet implemented (see [`DataStore` Protocol Rework](#datastore-protocol-rework)), but multiplayer is supported! We've also identified an issue regarding best-friend messages, of which a fix is being actively tested.
+
+## Mario Kart 7 Communities
+Mario Kart 7 now has partial support for communities! Leaderboards are not currently implemented, as Mario Kart 7 uses a version of the [`Ranking`](https://nintendo-wiki.pretendo.network/docs/nex/protocols/ranking/legacy) that is very different to the one used by modern games.
+
+## Yo-kai Watch 2 and Yo-kai Watch Blasters
+[shoginyan](https://github.com/shoginyan) has also kick-started Yo-kai Watch 2 and Yo-kai Watch Blasters both into development! Yo-kai Watch 2 has left beta and is available to everyone, while Yo-kai Watch Blasters (and variants) is currently in beta testing.
+
+## Monster Hunter 4 Ultimate
+Monster Hunter 4 Ultimate has officially left beta testing! We have also removed the region lock, allowing for experimental crossplay between MH4U and MH4G players! You may encounter technical difficulties when playing across regions due to this experimentation, so please report any issues you encounter to [the game's issue tracker](https://github.com/PretendoNetwork/monster-hunter-4-ultimate/issues/new/choose).
+
+## Swapdoodle
+Swapdoodle was one of the first titles Dani wanted to work on, prior to him joining our core dev team. After joining, focus was shifted elsewhere and work stopped on Swapdoodle fairly early. However thanks to the work by outside contributors [Silver-Volt4](https://github.com/Silver-Volt4) and [CenTdemeern1](https://github.com/CenTdemeern1), work on Swapdoodle has once again picked up! They took it upon themselves to implement the server entirely, with promising results! For more information, see:
+
+- https://forum.pretendo.network/t/we-implemented-swapdoodle-and-would-like-to-contribute-it/16204
+- https://github.com/PretendoNetwork/swapdoodle/pull/1
+
+## Dr. Luigi and Dr. Mario: Miracle Cure
+Both Dr. Luigi and Dr. Mario: Miracle Cure have officially left beta testing! Both titles are now available to the public!
+
+## The Legend of Zelda: Tri Force Heroes
+After several months of downtime, The Legend of Zelda: Tri Force Heroes is officially back online! This was one of our legacy games requiring upgrades to our latest libraries, and has finally had its issues ironed out. Due to some recent changes to our general matchmaking code, the games multiplayer region lock was unintentionally returned. This is being looked into and should be fixed in due time.