pokeplatinum/docs/datafiles/overview.md

Overview of Processing Tools for Data Files

The build pipeline uses a suite of custom compiler tools in tools/dataproc to bridge plain-text data files into game-ready binary data. These tools leverage a common interface for querying any given data markup format defined by lib/dataproc.c. Additionally, this common interface provides validation routines for common types and rich error reporting.

The driver implementations of these tools (e.g., src/speciesproc.c) also make use of shared infrastructure from src/common.c to:

1. Parse C enums and preprocessor definitions from headers into lookup tables,
2. Generate output header files using .template files from the data subdirectory,
3. Create output NARC files using libnitroarc, and
4. Write dependency files at runtime for correct builds.

Tools

Tool	Input	Output
dexproc	Regional Pokédex JSON	Mappings of species to their regional and national Pokédex order.
frontierproc	res/trainers/frontier/{data,pokemon}/*.json	Battle Frontier Pokémon pools and trainer definitions.
itemproc	res/items/data/*.json + include/constants/items.h	Item metadata, hidden-item script definitions, item-to-icon mappings, TM-to-move mappings.
moveproc	res/moves/<name>/data.json + include/constants/moves.h	Move metadata, in-game battle text, and animation scripts.
npctradeproc	res/npc_trades/*.json + include/constants/npc_trades.h	Pokémon offered by NPCs for an in-game trade.
resdatproc	Sprite template JSON + NARC ID maps	Serialized sprite templates to be loaded from the filesystem.
speciesproc	res/pokemon/<species>/data.json	Pokémon base stats, evolutions, learnsets, and sprite positions.
trainerproc	res/trainers/data/*.json	Trainer metadata, including their parties, in-battle messages, and overworld messages.

Common Patterns and Conventions

Each tool follows a similar structure. The following static variables are first defined to provide a base environment:

1. Constants to be loaded into lookup-tables, defined in an array enums[]. These lookup tables enable referencing in-game constants from JSON files from their identifiers.
2. Any archives to be created which are sized according to the input, defined in an array archives[]. For example, moveproc defines move_data.narc here, which is known to contain one entry for each in-game move.
3. Any C headers to be generated, defined in an array headers[]. Headers are instantiated with a header and footer according to a corresponding .template file in tools/dataproc/data. For example, itemproc produces item_tm_move_map.h, which is constructed using tools/dataproc/data/item_tm_move_map.h.template.
4. Any text banks to be generated, defined in an array textbanks[]. For example, npctradeproc produces npc_trade_names.json, which contains the nickname and trainer name applied to each in-game trade.

Then, the processing code executes in the main loop. The following snippet illustrates the processing code from moveproc, with additional comments annotating the control flow:

int main(int argc, char **argv) {
    int        errc = EXIT_SUCCESS;
    datafile_t df   = { 0 };

    parse_args(&argc, &argv);

    // Instantiate the processing environment. This will load the input filename
    // passed as a program argument and look for "enum Move" inside its content.
    // This enum provides the base number of files for the output archives.
    //
    // The init_orderfiles function passed to .hook_before will be executed after
    // the base library is instantiated, but before any output files are
    // initialized.
    enum_seq_t moves = common_initenum(
        filename, "Move",
        .sourcefile  = __FILE__,
        .depfile     = depfile_path,
        .outdir      = output_dir,
        .format      = DATAPROC_F_JSON,
        .enums       = enums,
        .archives    = archives,
        .textbanks   = textbanks,
        .extra_files = 3,
        .hook_before = init_orderfiles,
    );

    // Process individual files, using the names from the input enum as an
    // authority for subdirectories in the asset tree.
    for (size_t i = 0; i < moves.size; i++) {
        const char *stem = moves.members[i].name + lengthof("MOVE_");

        char *basename = strlower(stem);
        char *filepath = pathjoin(base_dir, basename, "data.json");
        declare_dep(filepath);

        if (dp_load(&df, filepath) == 0) {
            proc_move(&df);
            proc_text(&df, i, basename);
            prep_scripts(basename);
        }

        // Report any diagnostics found during processing.
        if (dp_report(&df) == DIAG_ERROR) errc = EXIT_FAILURE;

        free(basename);
        free(filepath);
        dp_free(&df);
    }

    // Handle any additional processing tasks before sealing the output files
    // and exiting the program.
    pack_extra_moves();
    prep_extra_scripts();
    return common_done(errc, close_orderfiles);
}