The gba payload was broken for the BUILD_LANG variant of the gba payload. So if the PTGB rom was english, the english payload would be broken.
The reason for this was because the text_helper ended up trying to convert the same entries twice, because the <BUILD_LANG> RSEFRLG table
would be converted during generate_text_tables() first, and then it would be converted again during generate_rsefrlg_tables_for_all_langs()
But convert_item is a destructive operation, because it replaces the ['bytes'] dictionary field.
Fix: add a "converted" boolean to guard against converting more than once.
Do some more work for multilang gba payloads. Revise the patch structure and make it use the right RSEFRLG table.
But it's broken. For some reason the english text doesn't show up when running the payload. Don't ask me why yet.
first_time entries would be longer, therefore the reference pointers inside the payload would shift.
To avoid that, we now reserve space for the longest entry, knowing that the texts will get stripped after
the payloads have been generated.
Doing so ensures that the text offsets remain constant, no matter if we're dealing with the first_time or othertimes payload.
This also means we need to take that reservation into account on the PTGB rom side of things of course.
To do so, I created a pair array indicating the first time entries and there non-first_time counterparts.
We'll determine which entry is the longest and ensure we use that length for the offset of the next entry.
This commit makes mystery_gift_injector re-insert the text entries into the gba payload at runtime.
You may or may not remember that I made gba-payload-generator strip the text entries again after generating the payloads.
I did this to improve compression: it removes the variability of the text lengths. It also ensures that texts are not really part of the BPS patch files
that are generated to transform the base payload into a specific language and game variant.
Therefore, we don't run at risk of storing certain text entries multiple times (or partially multiple times) due to them being stored at FileContainer boundaries.
It should make patch files smaller and more predictable.
Stripping the text entries also made it easy to reinsert: almost every rom uses the same offset for the first entry.
We now only need to store each language variant of the RSEFRLG (variants aren't being generated yet) once. That should be a big boon.
By first generating the payloads and THEN stripping the texts, we also ensure that the pointers/references within the payload remain intact and will become valid
again once the texts get reinserted.
Anyway, this specific commit does the re-insertion of those text entries into the payload (script + section30) using the RSEFRLG table.
Because the text stripping takes out these variable length texts, the text insertion offsets always end up the same, no matter the language:
Section30: 0x97E
Script:
- Ruby/Sapphire: 0x268
- Fire Red/Leaf Green: 0x278
- Emerald: 0x298
So, one of my concerns is that the BPS patches are too big and become even worse once the first_time flag and actual other payload translations
are being taken into account (which is not the case yet). Especially because we have to store our patch files into FileDataContainer chunks
and as a chunk boundary is crossed, compression context is lost. That means we might end up storing the same texts multiple times.
What we always wanted to do is to inject the texts at runtime into the payload, but have the payload itself generated at build time.
But what stopped us before is that not injecting the text shifts pointers (variable references, text offsets, ...) around.
But I had an idea: what if we DO inject the texts at first, thereby establishing the right reference addresses inside the generated payload
and THEN strip the texts again?
This way, our BPS patches would only have to account for the pointer differences between game/language variants.
And we would only have to store one compressed RSEFRLG text table per language. This will allow us to inject the texts at runtime to fixed
payload offsets and take care of both the compression problem and the "first_time" text difference all in one go.
As referenced in the title, this is still a work-in-progress commit. While the text stripping seems to be complete, there are still tasks to be done:
- Generate all language variants of the RSEFRLG text table
- Determine the text injection offset per game type. Many will be the same, but I mostly expect differences
between Ruby/Sapphire vs Emerald vs FireRed/LeafGreen.
- Inject the texts accordingly at runtime in the mystery_gift_injector.cpp
Problem: several variables were used in mystery_gift_builder before they were initialized.
Fix: initialize them.
I also made sure to memset the entire buffer before generating a payload to avoid garbage data to seep in.
This commit integrates the pregenerated payload into mystery_gift_injector.
It currently hardcodes the English Ruby v1.2 payload for testing purposes though.
The next test I'll have to do is to see if the transfer process works with this specific rom.
If it works, I can start integrating Pi's work in regards to delta patching using BPS.
I changed the approach a bit.
I realized that injection of stuff like text and the looker sprite at runtime, would result in every var
reference requiring an address shift.
That would be much more difficult to track and patch at runtime.
So I decided to do things a little differently:
- Copied the grit-*generated* lookerRSE.c, lookerFRLG.c and associated headers.
- text injection will also be done at compile time. To do so, I'll take the path to the uncompressed
RSEFRLG table (which is generated by text_helper) as a cmdline arg and use it to inject those strings
at compile time.
So the only runtime injection required would be the pokémon themselves. And the size of those is static.
So that works out!
This is the start of reworking mystery_gift_builder to run outside of PTGB and generate (most) of the payload
at compile time.
The idea is to leave most of the code intact, but inject some things at runtime.
It looks currently like that would be:
- Looker sprite (avoid LZ77UnCompWram AND having to bin2o the sprite file)
- text (avoid LZ77UnCompWram AND having to bin2o the text into this tool project)
- Pokémon box data
To that end, the goal is to generate a metadata struct with offsets of where to insert those things
at runtime.
But this commit is incomplete. It already tackles a bunch of stuff, but it can't be compiled yet.
TODO:
- Replace injection of the aforementioned things at compile time with something to record the offsets into
struct.
- Make it compileable
- output the payload to a (set of?) file(s)
- replace mystery_gift_builder IN PTGB with the pregenerated payload
- Inject the aforementioned things into the pregenerated payload at runtime in PTGB.
- Validate the pregenerated payload by comparing with mg_array and section30 dumps
from the previous implementation.
