Summary:
Test Plan:
Read any configuration values through bemanitools's
configuration API abstraction. This still had to be
hooked up when a hook dll is being loaded. The
current implementation relies on an environment
variable providing the path to the original inject
configuration file, then it extracts the relevant hook
configuration section for the currently loaded hook
library.
This adheres to the limitations of not having any
other entry points than DllMain for a hook library
(as of now).
Module to handle environment setup for hooks that use DllMain and
calls the bemanitools 6 API for hooks. This way, all hooks that use inject
(no AVS available) can be refactored and use the same API as any hooks that use launcher (with AVS available).
Remark: Module namespacing is not proper. This should move to a separate
part of the project that’s purely holding SDK related code that can be used
by other developers on other projects without requiring the whole
bemanitools project checkout. This problem is considered out of scope for now.
- Split into modules similar to launcher to improve overall structure
- Use property (node) API for configuration
Remark: "MVP" without IAT hooking working, command line arguments and overriding. Will be added (again) later
Add wrapper functions that fatal on any error.
Saves a lot of error checking code on occasions
where we want to fail anyway because some
configuration value could not be retrieved
successfully.
To provide a unified backend for all games, no
matter if they have AVS 2 available or now, use the
property (node) abstraction for any means of
configuration. This is somewhat annoying and a pain
because the property API isn't amazing. However,
the past has shown that having different means of
providing and handling (structured) configuration
data isn't great either.
Thus, have a non AVS implementation that allows
AVS independent applications/old games to use
the same core bemanitools API.
Summary:
Test Plan:
Summary:
Test Plan:
There is no need for different implementations
for this depending on the property (node) backend.
Thus, have these as ext(ension) functions that
already use parts of the actual API to implement
logging property (node) structures.
I checked out the few xml implementations that
are available as pure C and this one seemed like
the best choice so far. It's fairly small regarding
foot print, similar API to the property (node) API,
and I could get it to work fairly easily with a few
sandbox examples thanks to decent documentation.
This serves as the base for a bemanitools property
(node) implementation for games that do not come
with AVS 2 and the property API.
The previous implementation just went along with how AVS
property models and deals with attributes: attributes are just
another type of node that follow the same rules as standard
xml nodes/elements.
This design already showed several flaws in application as
these nodes always had to be referred to with an appended
‘@‘ on the attribute keys. This was mitigated with an “ext”
function that hides the whole details of “find the attribute node
before reading the attribute” steps.
By adding another property_node implementation with mxml,
the AVS style abstract layer showed incompatibilities with
the “an attribute is a node” approach. mxml doesn’t treat them as a type of node and just addresses them directly using their keys. This is lot simpler and aligns with how handling
attribute is done throughout the code thus far.
Refactor the property_node interface and the current AVS
implementation to also adapt this. Hide the detail that AVS
treats attributes as nodes and also the whole “append @“
to the keys notation.
Solves similar issues with the old games not using
the AVS logging system (because there is none).
Threads calling the log functions are still logging
using OutputDebugStr, but the dispatching in the
debugger module in inject hands those messages
off to the async log sink which avoids blocking
the calling thread (for long).
We want to keep this feature around as there are
several games actually having non-removed
OutputDebugStr calls which can provide some
useful debugging information.
Furthermore, neutral hooks only using DllMain
and not the bemanitools API can still channel
log messages this way to bemanitools’s logging
system (though less efficiently than calling it
directly).
Further changes to existing hooks will be applied
to migrate their logger usage to using the logger
directly through the bemanitools API.
A long awaited solution to address several
long-standing problems:
* Remove the need for log-server band-aid for iidx
versions using ezusb and the modern AVS logging
System, i.e. iidx 19 to 24. Any non AVS thread calling
the AVS log functions will crash due the mutex
implementation in AVS being not compatible with
non AVS threads (see the journal entry in the dev docs
for the whole story).
* Discontinue using the AVS log functions instead of
bemanitools’s own ones. There was never a technical
reason to do that, actually. It was neat to share one common
interface, but that was about it.
* Fixes performance issues on several games such as stuttering
brief slowdowns and de-syncs
Why not make this an integral part of the logger? There are
situations where async logging is not desired, e.g. ensure
predictiable logging output during debugging/development
tasks, and performance doesn’t matter at this point.
With the decision to implement this as a sink, the async
component can be easily composed arbitrarily with the other
existing log sinks.
Make this configurable. Normally the buffer size
doesn’t need to be 64k, that’s only required for
very verbose debug/development logging output.
Furthermore, the parameter needs to shared and
aligned with the async logger which has it’s own
buffer size. This avoids tight coupling between
the async sink and the logger.
This is required to ensure colored log messages
are executed as an atomic operation. Otherwise,
the color code information written to the terminal
could interleave with concurrent log messages
which results in “randomly” colored messages or
parts of them.
Remove usages of AVS threads and logging functions from the bemanitools backend. Use the bemanitools 6 API instead
which propagates interfaces to call thread and log APIs.
Prior design decisions for using these were:
- have a single log stream through the AVS logging system for “neatness”, log output redirection options in launcher, and theoretically logging to an xrpc endpoint
- IIDX 19 to IIDX 24 crash the AVS logging engine in libavs due to IO hook code running in a non AVS thread (threads in ezusb library) causing a stackoverflow when trying to acquire a mutex (see dev journal 2018-02-10-logging-breakdown-avs.md for details)
With a flexible log sink architecture in the core of bemanitools, and a separate writer function that is hooked up as a log writer to AVS, log streams are still unified on a sink level. This takes care of having all log messaged sinked to the same targets no matter how many logging engines are using them.
This avoids going through the AVS logging engine with the IO related hooking code in iidx, which just uses the bemanitools logging engine instead.
Furthermore, this removes any needs to having to switch thread and logging implementations once AVS is booted which significantly simplifies the runtime orchestration during bootstrapping.
The log-server, which was specifically implemented for iidx to solve the threading issue, can also be removed now. Unfortunately, this also caused performance issues such as stuttering due to it’s rather simplistic implementation.
**THIS IS A HIGHLY WORK/DEVELOPMENT IN PROGRESS VERSION**
**THINGS ARE BROKEN AND EVERYTHING IS SUBJECT TO CHANGE**
First cut after massive refactoring with most critical and fundamental changes implemented.
We need to start somewhere, and this might be as good as anything else, so we can get started
with testing, bug fixing and iterating for the next releases.
The following list is non-exhaustive, does not guarantee anything does work, yet, and is supposed
to give a high level idea of what all of this is about. Updated documentation will reflect all of
this at some later point in time in more detail.
* A common "core" now abstracts logging, thread, property and configuration infrastructure and
provides a common interface. This is used by bemanitools internally as well as all tools, hooks
and APIs provided and don't depend on the game, version of the game or AVS version available
anymore
* New bemanitools (public) API
* Versioned API allowing for handling incremental API changes as well as breaking changes by
providing a new/different version when necessary
* Unified interfaces for bemanitools core API, i.e. logging, threads, configuration
* SDK with examples (TBD)
* Dogfooding approach: Bemanitools uses its own (public) API to implement and provide fundamental
features like configurable keyboard implementations for IO or hooks for different games and
versions
* All bemanitools hooks and IO libraries have been or are about to be re-worked to use the new APIs
* New hook API allows for more fine grained runtime control when stages of the hook are to be
executed, i.e. pre AVS, before main game, iat hooking instead of relying purely on DllMain
(which is still a compatible option though)
* launcher as a replacement for bootstrap: Bring it significantly closer to the original bootstrap
by supporting completely vanilla data and bootstrap.xml configurations to run the games. Note
that bemanitools does not include any code or means to run DRM'd data, only decrypted
* inject is also being reworked to use as much of the same "infrastructure" as launcher to provide
a more seamless bootstrapping process for games that keeps pre-eapki data as vanilla as possible
Summary:
Test Plan:
Summary:
Test Plan:
Summary:
Test Plan:
Summary:
Test Plan:
A general debugging tool. 3rd party applications such as
"procmon" (same name) provide these capabilites and even
more. But, they are more difficult to run with bemanitools
and don't provide a unified look at the output in combination
with the log output by bemanitools.
Provide an initial set of system call hooks that have already
supported debugging efforts. More can be added when needed
later.
Apply a simple heuristic to provide the user with more
specific information regarding which vcredist package
might not have been found on their system.
This is a common problem as different games require
different versions and different versions of windows
and installations might already come with some versions
already pre-installed.
Also improve the readme regarding that and provide links
to all versions that are required by one game or another
today.
Because we are using mingw, we can't just use window's
dbghelp library as the symbols created are in dwarf format.
Fortunately, the dwarfstack library already provides all the
facilities to easily print very descriptive stacktraces,
including function names, file names and line numbers,
when dwarf symbols are available.
This moves the incomplete exception handling portion from
signal to a separate module as well to improve scoping.
Kudos to Shiz for providing the groundwork for this.
Fundamentally re-think how launcher operates and
bootstrapping the games is managed and configured.
This brings it significantly closer to how the original
bootstrap is doing the job: launcher now utilizes the
data (structures) provided by the bootstrap.xml configuration
file. This creates compatibility with vanilla data dumps
and original stock images. Note that bemanitools does not
include any code or means to run DRM'd data, only decrypted.
But, this allows users to keep decrypted dumps as stock as
possible which means:
* No copying around of property files anymore
* Keep the modules/ folder with the binaries
* Have bemanitools binaries separate in the data
* No need to edit/customize the original configuration files
A list of key features of the "new" launcher:
* Boostrap games by following the configuration provided by
stock game's bootstrap.xml files
* Custom launcher.xml configuration file that adds further
launcher configurable features, composability of
bootstrap.xml configuration(s) as well as configuration
overriding/stacking of selected types of configurations,
e.g. eamuse config, avs-config. The latter eliminates
the need for modifying stock config files in the prop/
folder
* Unified logging system: launcher and AVS logging uses
the same logger, all output can now be in a single file
* Original features such as various hook types still
available
Due to the significant architectural changes, this also
breaks with any backwards compatibility to existing
launcher setups. Thus, users need to migrate by re-applying
the new configuration format and migrating their config
parameters accordingly.
Further migration instructions and updated documentation
will be provided upon release.
Co-authored-by: Shiz <hi@shiz.me>
Keep this a separate commit because this also removes
inject's own logging engine and replaces it with the
streamlined core API. The core API provides all the
features of inject's own logging engine which also
performed horribly. The entire logging operation
was locked which included expensive operations
that formatted the log messages and required
memory allocations and copying around data.
The core API's implementation at least only
synchronizes the actual IO operations
(though this can be improved further with an
actual async logging sink, TBD)
