230 KiB
August 2023 client migration — local work, not yet playable
Target: C:\RecRoom\2023-08-29T06-53-33Z.
The version stored in this client's globalgamemanagers is 20230825,
and its Unity version is 2020.3.33f1.
Implemented
- Server default game version is 20230825 (earlier supported versions remain).
- The metadata patcher recognizes this release's shuffled 256-byte header using a SHA-256 fingerprint, excluding only the two fields it intentionally updates. Unknown headers still fail closed.
- Literal table offset/size fields are at 0x2C/0xBC; literal data offset/size fields are at 0xC8/0xC0. All 52,189 literal records fit the original blob.
- Rewrite verification checks unchanged literal contents and all original bytes outside the literal records and the two edited header fields.
- The patch marker records 20230825 rather than the old hardcoded March version.
- Local development exposes HTTPS on localhost:3030 and HTTP on 127.0.0.1:3031, and advertises HTTPS service URLs.
Original client metadata and boot configuration are preserved beside the files
as .orig. No March BepInEx files or generated interop assemblies were copied
into the August client. No Git push or remote server deployment was performed.
Verification and remaining blocker
The patcher's real-file verification passes, including a second dry run with zero changes. Server startup succeeds and the version endpoint returns ValidVersion=0 / VersionStatus=0. The 38 targeted VersionCheck, Nameserver and SplitTestCompatibility tests pass.
Initial startup failures (before the August plugin):
- HTTP: the client's request constructor throws
Exception: Invalid URI scheme. - Direct desktop launch with HTTPS: embedded BestHTTP/BouncyCastle rejects the
local certificate with
user_canceled(90)/ nameserver errorsummer. - EAC bootstrapper launch: native crash near
Connecting to RecNet.
The new 2023Patch/RadicadeAugustClient project builds against August interop
and replaces the March plugin locally. It installs embedded TLS, nameserver,
image signature key, hub authentication, and August EAC challenge hooks without
March-only UI patches. The old DLL is preserved as
BepInEx/plugins/RadicadeEacBypass.dll.march-backup; the old configuration is
preserved as com.radicade.eacbypass.cfg.before-august.
Live verification: August Client 2.0.0 loads without a plugin-load exception,
TLS accepts the configured localhost endpoint, the game fetches the nameserver,
/api/gameconfigs/v1/all and /api/config/v1/amplitude successfully, and logs
the Steam profile. It then reports Launch validation failed. Is Rec Room
installed correctly? Login and room joining remain unverified. No broad
launch-validation or error-suppression patch was installed.
Launch-validation root cause and bypass
Native analysis with the tools/AugustInterop disassembler traced the error:
NDIIPBKGGIG::CBNCPLACNIE(15)is the only call site producing the error screen; value 15 isJKBPKDAGEHN.LAUNCH_VALIDATION_FAILED.- Its sole caller chain is
NFPDGJNDFOF::GBLLEJBCFGI -> NFPDGJNDFOF::GBAMNAKGEGD. GBAMNAKGEGDis the boot validator invoked fromBootSequence::DFCEILLEFOKviaADJOODNBCKG::BEFDPLAELIH. It resolves an obfuscated static-data file reference, opens the referenced file, parses its PE headers, and computes a SHA-256 over selected sections in 0x400-byte chunks on a background thread, then hard-rejects (CBNCPLACNIE(15)) when the hash does not match the expected value baked intoGameAssembly.dll's static data.- The check is not satisfiable in a patched installation (BepInEx loader
metadata redirect and localhost server wiring), so the plugin now installs
AugustLaunchValidatorPatch, a prefix onADJOODNBCKG.NFPDGJNDFOF::GBAMNAKGEGDthat resolves the boot completion promise (PFDOLDLEIBA.OLKGFACPIPC, with a direct handler-list fallback) and skips the native body. It must be registered viaharmony.PatchAllin the pluginLoadmethod — an earlier attempt defined the patch class but never registered it, which is why the error screen still appeared once.
Rebuild and reinstall RadicadeAugustClient.dll after this change; the log
should show Bypassing native launch validator before boot proceeds.
Build validation: August plugin builds with zero warnings/errors. The March plugin cannot be compiled unchanged against August's renamed types.
The final client target is https://localhost:3030. Start the server with
dotnet run --project Server/Server.csproj. For diagnostics, direct desktop
launch uses RecRoom.exe +forcemode:screen; it now passes TLS and encounters
the launch-validation blocker described above.
BepInEx interop generation fixed
The installed BepInEx 6.0.0-be.697 initially failed Cpp2IL's magic-number check.
The original .orig metadata has the same shuffled header: this is the release's
format, not evidence of download corruption or a URL-patch error.
tools/AugustInterop/normalize.cjs creates a separate tooling copy with a
standard v27 header. It also repairs the 305 synthetic <Module> records, whose
name indices and generic-container indices otherwise break assembly generation.
Only verified module records are repaired. This copy must NEVER replace the
game's global-metadata.dat; the native runtime expects the custom format.
The installed loader reads global-metadata.interop.dat for Cpp2IL. Its original
is backed up at BepInEx/core/BepInEx.Unity.IL2CPP.dll.before-august-interop.
The loader modification changes only RunCpp2Il's metadata filename literal.
Verified with the actual August GameAssembly.dll: 42,537 types and 283,224
methods parsed, native registrations resolved, interop generation completed,
335 files written including the hash and caches, and the live loader logged
Chainloader startup complete. Interop generation was first verified with no
plugins; the August connection plugin was then built and installed separately.
See tools/AugustInterop/README.md for reproduction and rollback.
August login "code: block" root cause and fix
Symptom: title screen showed Cached/Manual Login failed with: Unable to login (code: block) right after a POST /matchmaking/player/login log line whose
[Net] line prints when the request is queued, not when the response lands.
Server-side truth (server-console.log + PlayerDataDebug): the login POST
actually returned 200 both times. The real 405 came from
POST /api/PlayerReporting/v1/moderationBlockDetails, which the server only
mapped as GET; Kestrel answered the POST with 405 Method Not Allowed. The
August login flow runs its moderation-block check inside the login promise, and
a failed check maps to JKBPKDAGEHN.LOGIN_CODE_BLOCK - hence "code: block",
which reads like a moderation block but was a route mismatch.
Fix: added [HttpPost("v1/moderationBlockDetails"), Authorize] GetBlockDetailsPost()
in Server/API/Moderation/PlayerReportingAPI.cs, delegating to the existing GET
action so both verbs serve the identical body. Verified live: POST now returns
401 without a token (auth layer reached, no more 405) and the GET shape is
unchanged for older clients. Server rebuilt and restarted on 3030/3031.
Remaining non-fatal August 404s (client continues without them):
GET /matchmaking/player/qos, GET /matchmaking/player/connection-info,
GET /chat/thread/party, GET /cdn/room/. RudderStack analytics cannot
resolve its host and self-disables after 15 failures.
Verified in-game after the fix: POST /matchmaking/player/login -> 200,
POST /matchmaking/player/exclusivelogin -> 200,
POST /api/PlayerReporting/v1/moderationBlockDetails -> 200 (86-byte empty
block-details body), hub WebSocket connected, and the avatar spawned in the
Dorm. The August 2023 build boots fully into the Radicade server. The RSA
neutralizer correctly passes through verification calls signed with non-p1
keys (observed on a cdn/sigs/... fetch).
Remaining August 404s fixed
The four 404s the August client hit after login were implemented, using the recflare reference server's response shapes where documented:
GET /matchmaking/player/qos- unauthenticated BARE ARRAY of QoS probe regions (id + host:port address); the client pings these and reports back throughPUT /matchmaking/player/photonregionpings(already a no-op ack).GET /matchmaking/player/connection-info- authed{ success, value, error }envelope carrying a freshly minted HS256photonAuthToken(sub/rn.platid/ rn.plat/rn.deviceclass/rn.env claims, 1h TTL, signed with the server JWT key), the configured Photon app ids,photonRegion, and thephotonRoomIdof the caller's current room instance (presence first,roomInstanceIdquery param as fallback). Theexperimentsblock mirrors the reference verbatim;shouldUseGameServerNetworkingstays false.GET /chat/thread/party- empty object (recflare's documented stub: parses as "no party" without failing the client's deserializer).GET /platformnotifications/crm/me/config- empty object (same stub convention; the 404 here raised one console error per login).
Photon app ids and region are configurable (Radicade:Photon:RealtimeAppId,
VoiceAppId, ChatAppId, Region in appsettings) and default to empty ids /
us region, exactly as recflare behaves with no Photon configured: the client
is told empty ids and connects to nothing. A deployment that runs a Photon
server sets the three ids and the client connects normally.
Corrections after the first live run with those routes (all observed in BepInEx LogOutput.log):
chat/thread/party: the August client deserializes this as a LIST - the empty-object stub made it throwexpected:'[', actual:'{'. Answers a bare empty array now.platformnotifications/crm/me/config: the empty-object stub made the manager throwDeserialization returned null. The client parses a counted envelope here (same asconfig/categories), so it answers{ Results: [], TotalResults: 0 }now.connection-infowith empty Photon app ids is WORSE than a 404: the client's GetRegions call getsEmpty application id(ReturnCode -2) and Photon drops the session with InvalidAuthentication. The ids must be the ones the client actually uses - they live in the game'sBepInEx/config/com.radicade.eacbypass.cfgunder [Photon] and are now set in appsettings (22dda1bb-...realtime,82766593-...voice,e8ea33f3-...chat), matching the AppIdOverride the March plugin was already forcing.
2026-09-27: the same empty-app-id trap, hit again on a second host
Moving the client to the public server reproduced the connection-info failure above,
from the other direction. The loader reported GOING TO ^DormRoom 0% PREPARING.. and
sat there; the server looked healthy throughout (presence, matchmaking and the room
routes all 200), and the client's own log showed the real story:
[Network.Photon.LoadBalancingClient] Queued Network action failed due to exception: NullReferenceException
[RoomLoad.RoomManager.PresenceUpdates] Error joining room instance RoomInstance - ID:21516 DORM_ROOM:25/32
JOKAIBMCFDA::Failed to connect to region "us"
IKDHNNILPMJ::Unexpected Photon Disconnect due to cause: InvalidAuthentication
with Empty application id in Player.log exactly once and InvalidAuthentication eight
times, retrying roughly every 7 s. The cause was config, not code: the ids were set in
appsettings.Development.json and not in appsettings.Production.json, so a host
running the Production environment kept the DBConfig defaults ("") and answered
connection-info with an empty photonRealtimeAppId. Photon's GetRegions then fails
with ReturnCode -2 and drops the session, which is the documented behaviour quoted
above - connection-info with empty ids is worse than a 404.
Two changes so the next environment does not have to be diagnosed from a loading screen:
Server/appsettings.Production.jsoncarries theRadicade:Photonblock (same app as the dev config and the plugin'sAppIdOverride).Server/Program.csprints eitherPhoton: realtime app id configured, region 'us' (voice set, chat set).or a full WARNING naming the four keys to set, at startup. The warning path is verified by booting withRadicade__Photon__RealtimeAppId=set to empty.
Note the failure is silent by design from the server's point of view: nothing 500s and no route errors. Anything that shows up as "the room load hangs at 0%" should have the client's Photon state checked first, because the REST surface will look perfect.
Run-3 hard crash root cause: RSA passthrough access violation (fixed)
The fully-green-server run (20:25) still crashed the client on the loading
screen with the infamous DILGBLCELMP: Exception of type 'DILGBLCELMP' was thrown. at DLHKINDAIHK+MICFNEHOAMC.MoveNext() right after
GET /platformnotifications/crm/me/config (200) and
[ImageSig] RSA verify called #1 with non-p1 key - passing through.
That obfuscated exception was a red herring: the real fatal is in BepInEx
ErrorLog.log:
Fatal error. System.AccessViolationException
at Il2CppInterop.Runtime.IL2CPP.il2cpp_runtime_invoke(...)
at DynamicClass.DMD<RSACryptoServiceProvider::VerifyHash>(...)
Chain of events:
- After
exclusivelogin, the client downloads the file-integrity manifestGET /cdn/sigs/6fae9a40-edf5-44c5-a263-36d7b5b4a91f(served since an earlier blocker) and starts verifying the listed game files' signatures. - It verifies with the game's OWN embedded RSA key (not p1.rec.net), which
reached
RsaVerifyNeutralizerPatches.VerifyPrefix's deliberate "non-p1 -> pass through to original" path. - Invoking the il2cpp original through the Harmony DMD re-implementation
access-violates (interop cannot safely re-enter that native method). Unity
surfaces the AV as the unmappable
DILGBLCELMPil2cpp exception inside the sigs-check async chain (DLHKINDAIHK+MICFNEHOAMCawaited the RecNet HTTP task, then died during verify) - hence the crash after the crm 200 line even though crm/party responses were parsed fine.
Fix (client plugin, RadicadeAugustClient.Plugin.cs): the RSA verify prefix now
forces true and skips the original for EVERY intercepted verify, p1 key or
not - the passthrough path is never taken because it is un-invokable. Rebuilt
(dotnet build 2023Patch/RadicadeAugustClient/... -c Release, 0 errors) and
deployed to BepInEx/plugins/RadicadeAugustClient.dll (2023-09-23 21:07).
Lesson: on this build, any Harmony-prefixed il2cpp method whose prefix returns true (executes the original) risks the DMD AV. Neutralizer-style patches must always skip the original when the original's failure is the thing being patched.
Run-4 follow-up: file check fails gracefully -> manifest parser bypassed (fixed)
After the RSA-forcing fix, the client survived the sigs manifest (RSA verify
forced true, DILGBLCELMP now a caught/logged error) but the checker
(KKHDHILFOGO) threw while PARSING the placeholder manifest:
IndexOutOfRangeException in KKHDHILFOGO.PFNKMGPPFGP -> "File check failed
KKHDHILFOGO+LPIKGKOLBJH: Error validating file Failed to process verification
file due to internal exception" -> POST /api/PlayerReporting/v1/hile ->
PreparingToExit -> clean shutdown. Not a crash, but still an exit.
The real manifest format encodes per-file hashes inside the RSA-signed blobs, so no manifest we can serve without Rec Room's private key parses.
Fix (client plugin): new FileCheckManifestParserPatch prefixes
KKHDHILFOGO.PFNKMGPPFGP and returns an empty file->hash dictionary - with no
files listed the per-file verify loop completes as success. Rebuilt and
deployed (2023-09-23 ~21:20). Look for [FileCheck] manifest parser bypassed
in the next BepInEx log.
Run-5: manifest parser patch is NOT viable -> sigs route reverted to 404 (final)
The FileCheckManifestParserPatch attempt failed differently: HarmonyX's DMD
cannot compile the detour for KKHDHILFOGO.PFNKMGPPFGP's out-byref generic
signature (Dictionary<string, Il2CppStructArray<byte>>& out) and throws
InvalidProgramException: Common Language Runtime detected an invalid program
at (il2cpp -> managed) PFNKMGPPFGP detour time - the checker then fails with
NullReferenceException and the same hile report + PreparingToExit follows.
Empirical summary across runs (all 2023-09-23):
| cdn/sigs response | client outcome |
|---|---|
| 404 (run 1) | reaches the Dorm (known-good) |
| 200 placeholder (run 4) | IndexOutOfRange in parser -> hile -> exit |
| 200 + parser patch (run 5) | InvalidProgramException (DMD) -> hile -> exit |
Conclusion: the check TOLERATES a 404 and fails hard on any 200 we can produce.
The parser patch was removed from the plugin and SigsCDN.cs now deliberately
answers 404 (with the full analysis in its header comment). Plugin rebuilt and
redeployed; server rebuilt, restarted (PID 4768), verified: sigs 404,
gameconfigs/qos 200. The RSA verify neutralizer stays for p1-keyed image
signatures; the DMD passthrough AV lesson from run 3 still applies to any new
patch that would execute an il2cpp original.
Run-6 correction: 404 is NOT tolerated; final sigs/RSA combination
Run 6 disproved the "client tolerates a 404" theory from run 5's writeup:
OBHEMJEJNMK: HTTP Error 404 -> "File check failed ... Dowload failed" ->
hile report -> PreparingToExit. Reconciling ALL six runs gives the real
rules of the file check (KKHDHILFOGO):
- Missing manifest (404) -> fatal (run 6)
- Manifest present, non-p1 verify PASSES through -> DMD access violation to il2cpp original (run 3, flaky)
- Manifest present, non-p1 verify forced TRUE -> parser decodes the fake 16-byte blobs as a hash table -> IndexOutOfRange (run 4)
- Manifest present, non-p1 verify forced FALSE -> parser skips blob decoding, check completes -> known-good (run 1, Dorm reached)
- A client-side parser prefix is impossible (DMD InvalidProgramException, run 5) - the out-byref generic signature is un-detourable.
FINAL combination (all live 2023-09-23 ~21:35): SigsCDN.cs serves the
placeholder manifest again (with byte-range support) AND
RsaVerifyNeutralizerPatches now answers __result = isP1 - TRUE only for the
embedded p1.rec.net key (image signatures, boot-critical), FALSE for every
other key (the file check, which must fail cleanly to skip blob decoding).
Never passthrough (AV), never unconditional TRUE (IndexOutOfRange).
Plugin rebuilt/deployed, server rebuilt/restarted (PID 42708), verified:
sigs 200 placeholder, gameconfigs/qos 200. Watch for
[ImageSig] RSA verify #N (game key) forced false in BepInEx.
Run 11 outcome (22:36) + Run 12: exit-state veto (deployed 22:55)
Run 11: suppression armed, file-check bypass fired, Dorm load started at 11.13s - then the process DIED mid-scene-load at 16.26s with no exception, no hile POST, no exit sequence. The IJHOPKJFKBI prefix crashed the process inside the DMD stub BEFORE the prefix body ran: the scanner thread is a native, unattached thread (CallerCount(1), no managed callers), and il2cpp detours on such threads access-violate - the run-3 VerifyHash AV class. winhttp.dll cannot be dodged at the source: it is the EAC-bypass door DLL (the only non-stock DLL in the game root) and loads BepInEx itself.
Run 12 therefore ALLOWS the hile report to happen (HileAPI logs it at info)
and vetoes the MANAGED exit state machine instead:
CCKNJCHHBLL.KALEFEDAHCC (AppExitState setter, enum Running/PreparingToExit/
ReadyForExit) is prefixed; transitions OUT of Running are blocked for 120s
after load, then pass through. The observed shutdown always begins with the
managed "AppExitState changed: Running => PreparingToExit" line, so this
cuts the shutdown off at its head from attached managed-thread territory
where detours are proven safe. Alt+F4 still works (native path).
Run 12 markers: [ExitVeto] blocked AppExitState -> PreparingToExit at ~16s
in BepInEx, hile POST visible server-side but session SURVIVES, room load
completes, player spawns in the Dorm. If the game still exits: the scan
thread exits natively without the managed transition - next lever is
thread-freezing or attacking the scan itself.
Run 10 outcome (22:31): hile suppression DID NOT FIRE - registration bug, fixed
Run 10 replayed run 9's ending exactly: file-check bypass fired, Dorm room
load ran 6.1s, hile POST at ~18.3s, PreparingToExit 160ms later, room load
cancelled. No [Hile] suppressed line, no armed startup line. Cause:
HileReportSuppressionPatch was added as a class but never registered -
this plugin patches per-class (harmony.PatchAll(typeof(...))) and the new
class was missing from the Awake chain. Fixed + startup log line
"[Hile] report suppression armed" added; deployed 22:34:43.
Server-side wins confirmed in run 10: gameserverregionpings PUT no longer
404s ("Failed to upload cluster pings" gone); curatedlists 404 gone but
the client now logs non-fatal "Deserialization returned null" + "No such
list" for GET /lists/curatedlists/bulk?id=17859340 - the [] shape is
tolerated but wrong; may want a Results/TotalResults envelope or a list
object later (NOT the current blocker).
Run 11 markers: [Hile] report suppression armed at startup, then
[Hile] suppressed report 0 (Obscured) message='winhttp.dll' at ~18s,
NO POST .../v1/hile, no PreparingToExit, player spawns in the Dorm.
Run 9 outcome (22:16): file check DEAD, new blocker = native module scan (winhttp.dll)
The both-entries bypass WORKED: [FileCheck] entry bypassed fired after
exclusivelogin, no GET /cdn/sigs, no game-key RSA verify, no
"Signatures don't match!". Boot went deeper than ever: matchmake/dorm ->
room 25/32 instance 20830 -> Photon connect -> LoadScene(dormroom2) ->
Initial Room Load '@TestPlayer's Dorm' completed its 5.3s flow - then the
session died: hile POST at 20.48s, PreparingToExit at 20.64s, room load
cancelled.
The hile body decoded server-side: Type=Obcured Message=[0:winhttp.dll]
- the native module scanner flags our EAC-bypass proxy DLL (game root, 26KB, deployed 15:18) as an obscured module. The hile enum (MEHDFJHKLBK. JBLJIENBBHK) decodes [0:...] = (Obscured, "winhttp.dll"). Same pattern in the short 22:15:40 session.
Also fixed (server, 22:29): GET /lists/curatedlists/bulk -> [] (was 404, caused one "Deserialization returned null") and PUT /matchmaking/player/gameserverregionpings -> {} (was 404, "Failed to upload cluster pings").
Run 10: hile funnel suppression (deployed 22:29)
Client: HileReportSuppressionPatch - prefix on
KDJDKJNPELI.IJHOPKJFKBI(MEHDFJHKLBK.JBLJIENBBHK, string) (CallerCount(1)
funnel for EVERY hile report; enum Obscured=0/Time=1/Inject=2/.../
AppData_Boot_InvalidSignature=310/...) returns false and logs the report;
finalizer on ModerationManager.BJGDALDIOGI is insurance for paths
skipping the static funnel. No hile report is legitimate in a Radicade
session. Server: HileAPI log downgraded warning -> info.
Success markers: [Hile] suppressed report 0 (Obscured) message='winhttp.dll'
in BepInEx, no POST .../v1/hile in server log, no PreparingToExit, room
load 25/32 completes and the player spawns in the Dorm. If a DIFFERENT
hile type appears later, the suppression log will name it.
Run 9: entry patch v2 - both entries (deployed 22:08)
Run 8 (21:54) failed exactly like run 7: no [FileCheck] entry bypassed line,
GET /cdn/sigs/... still ran, RSA verify #1 (game key) forced false, then
"Signatures don't match!". The public wrapper GGIFLDPHCIP is never invoked.
The decomp explains it: GGIFLDPHCIP is marked CallerCount(0) (dead public
wrapper) while the private twin GOCCLPJDJIC (same signature) is
CallerCount(3); the no-arg NPIILCCIPNF is also CallerCount(0), so it
needs no patch. FileCheckEntryPatch now uses TargetMethods() to prefix
BOTH entries and returns the already-resolved success promise
(PFDOLDLEIBA.PLJINLOCHGM(KKHDHILFOGO.BJGODLEIDGE.NOMPJGOIAEB(...))).
No download, no parsing, no RSA verify; the consumer sees a resolved
MCFFIFIPNKE<BJGODLEIDGE> exactly like a passing check produces. Success
markers for run 9: [FileCheck] entry bypassed - resolved success promise returned in BepInEx, and NO GET /cdn/sigs / no RSA verify / no
POST .../v1/hile; boot should continue past the file check into the Dorm.
Run-7/8: forced-FALSE also fails; final bypass = resolved success promise (deployed)
Run 7 (__result = isP1, non-p1 -> false, manifest 200) failed with an
explicit validation error: File check failed KKHDHILFOGO+LPIKGKOLBJH: Error validating file verification.sig: Signatures don't match! at
KKHDHILFOGO.AFHNPNGLCML called from the parser. The complete matrix now
(six live client outcomes, one theory):
- 404 manifest -> "Dowload failed" (run 6)
- verify false -> "Signatures don't match!" (run 7)
- verify true -> blob decode IndexOutOfRange (run 4)
- verify passthrough -> flaky DMD access violation (run 3)
- parser prefix -> impossible, DMD InvalidProgramException (run 5)
- run-1 "Dorm reached" -> actually a race: the flaky AV path returned native false before the AV could trigger; the run-1 theory of "false skips blob decoding" is dead.
Conclusion: NO server response + verify-result combination passes this check.
DILGBLCELMP is the checker's own exception class (KGABAOLHEHL-derived,
string ctor), logged caught in every run - it is a symptom, never the cause.
Run-8 hypothesis before the decomp check: patch the public entry
KKHDHILFOGO.GGIFLDPHCIP(string, byte[], byte[], long, string) (DMD-safe
parameter profile) and return an already-resolved success promise. Run 8
disproved it (see Run 9 above); the fix now covers both entries.
The RSA neutralizer keeps forcing TRUE only for the p1 key (image signatures); with the entry bypass active the file check never reaches RSA verify at all.
Run 12: AppExitState veto - process survives but zombie (deployed 22:55)
Client: ExitStateVetoPatch - prefix on private setter
CCKNJCHHBLL.KALEFEDAHCC (nested enum CCFIKLBABNB), blocks any transition
out of Running for 120s after load, then allows. Safe to detour from managed
threads (fired dozens of times, no crash). HileReportSuppressionPatch fully
removed (run-11 lesson stands: NEVER detour KDJDKJNPELI.IJHOPKJFKBI).
Outcome: the veto WORKS as a patch ([ExitVeto] blocked ... PreparingToExit
right after the hile POST at 17.6s) but is INSUFFICIENT. The shutdown path
does not run through AppExitState: immediately at hile time the client also
cancelled the room load ("Room load to 20833 cancelled" AFTER the 6s load
flow completed), tore down EAC ("[EAC] Tearing down and releasing runtime"),
POSTed logout + reportjoinresult, and threw OBHEMJEJNMK: Aborted. Then an
infinite ~1.35s loop: NRE + "An error occurred" + EAC teardown +
veto-blocked PreparingToExit AND ReadyForExit, from 17.6s to 65.6s+.
VoiceClient timed out after 30s; Backtrace "Cannot resolve destination
host". Result: alive-but-unusable zombie. The setter's CallerCount(3)
matches the three state drives visible in the loop.
Conclusion: the exit has at least two arms - the AppExitState machine AND a parallel shutdown initiator (EAC teardown / room-load cancel / Photon disconnect). Vetoing the state machine alone wedges the process.
Run 13: server-side hile-response experiment (pending)
Untested hypothesis: the native exit fires in response to the hile POST
RESPONSE (exit began ~160ms after the POST completed in runs 9/10/12).
HileAPI.PostHile is now env-selectable via RADICADE_HILE_BODY:
ok-true (default, unchanged shape) | ok-false | nocontent (204) |
servererror (500). Test pins ok-* to OkObjectResult+bool; 204/500 are
diagnostic-only.
Procedure: kill Server.exe (PID 20224 at 23:00), dotnet build, start with
RADICADE_HILE_BODY=ok-false, relaunch client, compare:
- Exit GONE -> the native scanner waits on the server verdict; tune the response (then try nocontent/servererror to map the contract).
- Exit STILL at ~17.6s -> trigger is client-local; next step is decomp of the managed exit initiators (CCKNJCHHBLL.BEFDPLAELIH/GIEEBIOOFML/ GBGMBMGDGGD, all run on attached threads) and prefixing them, or native handling of the scanner thread (run-11 lesson: do NOT detour the funnel).
Run 13: hile-response experiment - exit is CLIENT-LOCAL (23:11)
Server returned Ok(false) for the winhttp.dll hile report (verified in
server-console.log: "Hile experiment: returning Ok(false)"). Result: the
client STILL exited - same signature as run 12 (veto blocks, EAC teardown
loop, OBHEMJEJNMK Aborted, ~94s death). The server response does NOT drive
the exit.
Two new facts:
- The IMMEDIATE post-report exit was skipped. Run 12: exit began 160ms after the hile POST and killed the room load. Run 13: room load COMPLETED (FinishRoomLoad 21.5s), spawn completed ~23s, and the player had ~30s of real gameplay before the first PreparingToExit attempt at 52.5s. First-time NRE + "InvalidOperationException: Cannot error an already completed promise" at 32s looked suspicious but did not kill.
- The 52.5s exit is the same driver as the 17.6s exit (veto + EAC + Aborted loop). Interpretation: the native scanner re-scans on its own schedule; with Ok(false) the first post-report exit did not fire, but a later scan still flagged the module and restarted the shutdown.
Conclusion: stop DETECTION, not the shutdown. Next lever: make winhttp.dll invisible to the module scan itself.
Run 14: PEB module hiding - hide winhttp.dll from the scanner (deployed 23:27)
Client: PebModuleHider.HideModule("winhttp.dll") at plugin Load (before
any Harmony patching). Walks PEB->Ldr->InLoadOrderModuleList, matches
BaseDllName (x64 offset 0x58 from the InLoadOrder node; 0x60 was wrong -
FullDllName at 0x48 is a 16-byte UNICODE_STRING), unlinks the entry from
all three public lists (InLoadOrder / InMemoryOrder /
InInitializationOrder). Pure user-mode pointer writes on the main thread -
no detours, no scanner-thread contact, no DMD. Module ENUMERATION
(Toolhelp, EnumProcessModules, PEB-walks) no longer sees the DLL; module
RESOLUTION (LoadLibrary/GetModuleHandle) is untouched (loader hash tables),
so the BepInEx door keeps working. AllowUnsafeBlocks enabled.
Server: hile restored to canonical Ok(true) (ok-true, default env) so run 14 tests exactly one variable. Server PID 34420.
Success markers: [ModuleHide] winhttp.dll unlinked from PEB module lists
in BepInEx at startup, NO POST /api/PlayerReporting/v1/hile in the whole
session (the scanner finds nothing to report), no PreparingToExit, room
load completes, spawn, stable gameplay past 2min. If hile STILL appears,
the scanner is not PEB-based (try hiding via handle-table wipe / ImagePath
alternate next). Keep ExitStateVetoPatch armed as belt-and-suspenders.
Run 14: PEB unlink - scanner does NOT use module lists (deployed 23:27, failed)
PebModuleHider.HideModule("winhttp.dll") fired successfully
("[ModuleHide] winhttp.dll unlinked from PEB module lists"). BaseDllName
offset correction: 0x58 on x64, not 0x60 (FullDllName at 0x48 is 16 bytes).
Outcome: the hile POST STILL arrived at 16.3s with
Message=[0:winhttp.dll], immediate exit followed 150ms later (veto loop as
before). Server log confirms exactly one fresh report at 23:32:58.
CONCLUSION: the scanner does not enumerate via PEB module lists. It walks committed image regions (NtQueryVirtualMemory-style) and classifies each region by parsing PE headers - unlinking the lists hides the NAME but the region still self-identifies via its MZ/PE signatures.
Note the response correlation across runs: ok-true -> immediate exit (12, 14); ok-false -> deferred exit (13). The exit is native local state, the POST is only telemetry.
Run 15: PE signature wipe - declassify the mapped image (deployed 23:42, FAILED)
PeHeaderWiper.WipeSignatures("winhttp.dll") at plugin Load (after the PEB
unlink, before Harmony). Zeroes the two structural signature tokens:
"MZ" at base+0 and "PE\0\0" at base+e_lfanew (6 bytes total, VirtualProtect
RW for the write then restore). A region-walking scanner that classifies by
parsing headers can no longer name this region; GetProcAddress is unaffected
(it follows e_lfanew into the intact optional/export headers without
validating the signature bytes) and winhttp IATs bound at process start.
Success markers: both [ModuleHide] lines OK, NO POST /v1/hile in the whole session, no PreparingToExit, load completes, spawn, stable past 2min. Failure modes to distinguish in the log:
- hile still appears -> scanner hashes raw region bytes against the module's on-disk image (need bytes-level rework of the mapped image or targeted section wiping) -> next run.
- no hile but exit at ~16-18s -> an independent native watchdog (e.g. door DLL fingerprint check) -> revisit native path.
Run 15 outcome (14:28 Sep 24, FAILED)
Both hides active in the boot log ("[ModuleHide] winhttp.dll unlinked" AND
"PE signatures wiped"). The hile POST fired anyway at 19.3s session time
(client [Net] log: Post .../api/PlayerReporting/v1/hile), immediate exit
150ms later: veto blocked PreparingToExit, EAC teardown, room load
cancelled, OBHEMJEJNMK: Aborted, NRE/"An error occurred" loop ~1.3s -
identical to runs 12/14.
Server-side note: server-console.log stopped receiving entries after 23:33 Sep 23 even though the server process survived (its console redirection died with the previous agent session), so run-15 POSTs are evidenced from the client log only. Server was alive and answering gameconfigs 200 during the whole session.
CONCLUSION after runs 14+15: the scanner neither enumerates PEB module
lists nor classifies regions by PE signature tokens. It still flags
winhttp.dll as Type=Obcured (note the misspelling - the flag is
"obscured", i.e. it is reporting our tampering itself, not a name it
got from the headers). Remaining hypotheses: (a) hashes raw mapped
bytes against the on-disk image, (b) fingerprints the door DLL through
the loader record / LoadLibrary bookkeeping, (c) an independent native
watchdog.
DECISIVE observation for run 16: across ok-true runs the exit fires ~150ms after the POST, while run 13's ok-false (response that never satisfied the handler) deferred the exit until the next scan. The exit is triggered by the report's RESPONSE arriving, not by the scan itself. So: drop the report request at the HTTP layer before it is ever sent.
Run 16: HileGate - drop the hile POST at the HTTP layer (deployed 14:50 Sep 24, FAILED - prefix never skipped the original)
HileGate (in NameserverConnectionPatches.cs): a prefix check inside the
existing BestHTTP HTTPManager.SendRequest(HTTPRequest) patch - the same
managed, main-flow funnel the [Net] logger uses, proven detour-safe
(unlike the scanner-thread funnel KDJDKJNPELI.IJHOPKJFKBI, run 11 fatal).
When the request URI path ends with /api/PlayerReporting/v1/hile, the
prefix returns before any processing: the request is never queued, never
sent, and no completion callback can fire, so the response-gated exit
path never runs. The scanner cannot observe the drop - it only sees that
its report never completes. Opt-in via HileGate.Enabled (armed in the
August plugin Load; the March plugin source stays inert). The
ExitStateVetoPatch stays armed as a second layer.
Success markers: [HileGate] dropped anti-cheat report POST around 16-20s,
NO hile POST reaching the server, NO PreparingToExit, load completes,
spawn, stable past 2 min.
Failure modes to distinguish in the log:
- No [HileGate] line but exit at ~16-20s -> the scanner sends through a different stack (native winhttp directly, not BestHTTP) -> shim the door DLL's winhttp exports or block at the socket layer -> next run.
- [HileGate] line fires, no POST, still exit -> trigger independent of report completion (native watchdog) -> revisit native exit initiators (CCKNJCHHBLL.BEFDPLAELIH / GIEEBIOOFML, tokens 100667097/098).
- Periodic re-scan attempts every ~30s should each be dropped silently; a rising [HileGate] count with stable gameplay is a WIN state.
Run 16 outcome (14:53 Sep 24, FAILED - implementation bug, model intact)
The gate FIRED ([HileGate] dropped anti-cheat report POST at 16.9s) but
the POST went out anyway: server-console-run16.log line 12835 shows
POST /api/PlayerReporting/v1/hile -> 200 at 14:53:55.431, exit machinery
started ~150ms later, veto loop as usual. ROOT CAUSE: the run-16 prefix was
VOID - in Harmony an early return from a void prefix does not skip the
original; only a bool prefix returning false does. The drop was a no-op, so
this run is evidence-wise just another ok-true repetition (12/14/15).
FIX (deployed 14:58): the prefix is now bool Prefix(HTTPRequest request, ref HTTPRequest __result); on a hile report it logs, sets
__result = request (the overload's fluent return value - decomp confirms
SendRequest_Public_Static_HTTPRequest_HTTPRequest_0) and returns false so
the original never queues the request. Success markers unchanged; next run
must show the [HileGate] line AND no POST /v1/hile on the server, then no
PreparingToExit at all.
Run 16 RETRY (boot 14:59:23 Sep 24) - SUCCESS
All success markers met for the first time in 16 runs:
[HileGate] dropped anti-cheat report POSTfired at 16.9s session time (LogOutput.log line 174) - exactly where the scanner always reported.- ZERO
POST /api/PlayerReporting/v1/hilereached the server for this boot (server-console-run16.log: the only hile entry is the buggy 14:53 run's). The drop now actually skips the original. - ZERO
[EAC] Tearing down, ZEROOBHEMJEJNMK: Aborted, ZERO NullReferenceException/"An error occurred" - the post-report zombie loop never started. - Room load COMPLETED: joinresult 200 at 14:59:43 (instance 20845), RunFor.Authority handlers finished, phases advanced through DESERIALIZE_CONNECTABLES -> POST_DESERIALIZE_OBJECTS (attached log tail at 20s), avatar/keepsake/invention fetches all flowed.
- Session end (14:59:52 logout, 16 veto blocks 0-10s, process left ~15:00:03) was a USER-INITIATED quit ~9s after spawn, NOT an anti-cheat exit: no EAC teardown, no hile around the logout, and the native quit path bypasses the veto by design (documented run-12 behavior). The 120s veto window had not expired; the native path took the process down cleanly.
VERDICT: the anti-cheat kill chain is broken at its response gate. The report is dropped client-side, the response never arrives, the exit handler never fires, and the game survives into the Dorm.
PENDING VERIFICATION (next soak run): the scanner's ~30s periodic re-scan. Prediction: every subsequent report is dropped identically (rising [HileGate] count, stable gameplay). If a later scan instead switches to a different report channel (native winhttp direct), the first symptom is a server-side hile POST - watch for it in a 5+ minute soak.
Note: keep the ExitStateVetoPatch armed. With reports dropped it should never fire again except for genuine user quits inside the 120s window; if it starts blocking repeatedly WITHOUT a [HileGate] drop preceding it, a second exit trigger exists and the native initiators (CCKNJCHHBLL.BEFDPLAELIH / GIEEBIOOFML) become the next patch target.
Run 17: second exit trigger + veto removed by a parallel edit (Sep 24
afternoon)
Three more boots with the fixed gate (14:59, 15:07, 15:13) reshaped the evidence:
- 14:59 boot: gate dropped the report at 16.9s, room load COMPLETED (joinresult 14:59:43), then 16 veto blocks (PreparingToExit x8, ReadyForExit x8) at 14:59:52-15:00:03 - the exit re-request came ~9s AFTER load finished, and the process left once the veto window ended. No hile, no EAC teardown, no crash dump, no Windows fault.
- 15:07 boot: gate dropped reports TWICE (17s, room-change), survived 7 minutes, then died through the managed shutdown cascade at 15:08:42 - exactly when the run-12 veto (120s from 15:07:40) expired. Same clean shape: the exit kept being re-requested and the veto kept absorbing it until the window closed.
- 15:13 boot: a PARALLEL EDIT (Plugin.cs modified + rebuilt at 15:11, not by this thread) REMOVED ExitStateVetoPatch entirely. That boot had [HileGate] armed but no [ExitVeto] armed, died ~40s in with nothing absorbing the exit. This confirms the veto was doing real work.
MODEL REVISION: the exit is NOT purely response-gated. The scanner keeps RE-REQUESTING exit on a cadence (runs show 10-40s intervals), plausibly a report-completion TIMEOUT on the dropped report (run 13's ok-false exit at ~30s fits: no satisfying response within the deadline = failure = exit). Dropping the request cannot win alone; something must absorb the re-requests forever. The veto did that until its window expired; a parallel edit removed it.
Run 17 change: ExitInitiatorTracerPatch prefixes the managed exit initiators CCKNJCHHBLL.BEFDPLAELIH (100667097), GIEEBIOOFML (100667098) and FNOEHNDLHHG (100667101), logs the FULL managed stack of whoever calls them, and blocks the call. These are attached-thread, detour-safe targets (run-14 prep). Deployed 15:42. Success markers: [ExitInit] stack traces identifying the second trigger's caller, and sessions that no longer die at 10-40s. If sessions still die with all three blocked, the trigger is native-only (TerminateProcess-class) and the blockades prove it by absence.
Run 18: "crash on boot" was a disabled Doorstop proxy, not the tracer
Symptom: the user reported a crash on boot immediately after the 15:42 run-17 deploy.
Evidence chain:
- BepInEx/LogOutput.log still had mtime 15:14, i.e. no plugin log was written for any boot after the 15:13 session. No boot has ever printed [ExitInit] armed.
- The Unity log (AppData/LocalLow/Against Gravity/Rec Room/Player.log and Player-prev.log, mtime 15:59:47 / 15:59:33) contains ZERO BepInEx/Doorstop/Radicade lines: two VANILLA boots of ~14 KB each. Player.log shows "Could not resolve host 'ns.rec.net'" at 3.6s, then AppExitState Running => PreparingToExit at 3.9s, then "[EAC] Tearing down and releasing runtime", process gone by ~7s. EAC was live because the Doorstop/EAC-bypass door never loaded, and the game was still pointed at the real nameservers.
- Cause: winhttp.dll (26,112 b, Doorstop 4.3.0 proxy, md5 b2a4d0cb...) had been renamed to winhttp.dll.baseline-disabled at 15:17:15 by a parallel action, which also created baseline-backup-20260924-151633/ containing a copy of winhttp.dll, Player-before-baseline.log (the 15:13 session log), the pristine 173 b RecRoom_Data/boot.config and the pristine 38,654,794 b il2cpp_data/Metadata/global-metadata.dat.
- No new WER 1001 event and no new CrashDumps entry: the game exits through its orderly quit path, it does not fault. That is why the crash left no crash report and why only the Unity log has evidence.
- Fix: the snapshot's winhttp.dll was copied back to the game root (md5 identical). The rest of the door chain was verified intact: doorstop_config.ini enabled=true, .doorstop_version 4.3.0, BepInEx/core/BepInEx.Unity.IL2CPP.dll, dotnet/coreclr.dll, 335 interop assemblies, 64 unity-libs.
Consequences:
- Run 17 has NEVER booted. It was deployed at 15:42, 25 minutes after the door was disabled, so ExitInitiatorTracerPatch has zero runtime data and its block/trace behaviour is still unproven.
- The "baseline" experiment cannot work as designed. With the door removed there is no patch stack at all, so the game dies at ns.rec.net DNS plus EAC long before any comparison is possible. A vanilla control run needs the door present and the patch groups switched off from inside the plugin, never a removed proxy DLL.
- Parallel edits remain the biggest source of false signal: ExitStateVeto Patch was deleted from the working tree at 15:11 and the door was disabled at 15:17. Both were uncommitted, so neither the veto nor any boot data exists in the run-17 build.
The next boot is run 17's first real run. Expected markers: [ModuleHide] x2, [HileGate] armed, [ExitInit] armed, a hile drop ~17s in, and [ExitInit] stack traces naming the second exit trigger's caller.
Run 17's first real boot (16:10): the exit block broke the room load
The door restore worked and the boot reached the Dorm, but the player saw a
BLACK SCREEN: the debug overlay showed Room: 'DormRoom: Home', RoomId 2, SubroomId 2883, Players 1 (2 TestPlayer, Loading), FPS 910 (nothing being
drawn). Run 17's own markers were all present: [ExitInit] armed, ONE blocked
initiator call, and four [HileGate] drops.
The one block is the problem. It fired ~0.05s after Chainloader startup complete, before any networking, and its caller trace is:
at System.Environment.get_StackTrace()
at ExitInitiatorTracerPatch.TraceAndBlock()
at DMD<CCKNJCHHBLL::BEFDPLAELIH>()
at (il2cpp -> managed) BEFDPLAELIH(Il2CppMethodInfo* )
Read that trace correctly: in an IL2CPP build ALL game code is native, so "no managed caller above the trampoline" does NOT mean an outside native component. It only means the caller is another il2cpp method - ordinary game code. A boot-time call to an exit initiator is therefore most likely part of normal startup, and blocking it is not free.
What followed, compared against the last good session (15:13, pre-run-17, preserved as baseline-backup-20260924-151633/Player-before-baseline.log):
- 15:13: load of 25/32 runs 14.08s -> COMPLETE at 19.77s (FinishRoomLoad). FetchDeserializationDataTask.Run('25.32') finishes in 1ms: DownloadAsset Bundle, then DownloadAndParse of MKMOLMPLENB, JGEHMKKALNE and OBHEFDFCAIC all return in 2-7ms, then the phases march through DATA_DOWNLOADED, INIT_OBJECTMODEL ... COMPLETE.
- 16:10 (run 17): the SAME room, the SAME sequence - and
START: DownloadAndParse('OBHEFDFCAIC)')never returns. The load sits in DOWNLOADING_DATA until the client is closed 64s later. Everything after the load never happens: norooms/25/experience/player, nokeepsakes/rooms/25, noroomkeys?roomId=25, and noGET /rooms/photon_access_token, so the dorm's Photon join is never even attempted and the save can never arrive. That is the black screen. - Server side is exonerated:
GET /rooms/rooms/25?include=5421...returned byte-identical payloads (len=2174) in every session including 16:10, and the server log shows no [RoomUtils] self-heal firing. The only client-side delta between the 15:13 build and the 16:10 build is run 17's tracer (the veto was already gone at 15:13). Four consecutive pre-run-17 boots (14:53, 14:59, 15:07, 15:13) all completed the dorm load.
Run 18 change: ExitInitiatorTracerPatch still traces every call, but it now
ALLOWS the call (returns true) until HileGate.DroppedCount > 0, i.e. until
the scanner has actually filed a report, and only blocks from then on. The
exit that matters is the one the dropped report triggers; a call that arrives
before any report exists is startup traffic. HileGate grew the DroppedCount
counter (NameserverConnectionPatches.cs, shared into the August assembly), the
log line is now "[ExitInit] armed - ... traced; blocking switches on after the
first dropped hile report", and an allowed call logs
"[ExitInit] allowed managed exit initiator (no report filed yet)".
Deployed 16:16 (34304 b). Success markers for the next boot: an [ExitInit] allowed ... line during startup instead of a block, the dorm load reaching
COMPLETE / FinishRoomLoad, a visible Dorm, then the first hile drop and - from
that point - [ExitInit] blocked ... traces.
Run 18 gating result (16:17 boot): the hang is gone, the load still fails
The gate did what it was built to do:
[ExitInit] allowed managed exit initiator (no report filed yet)at startup (the boot-time BEFDPLAELIH call now passes through),- four
[HileGate] dropped anti-cheat report POSTlines, and - one
[ExitInit] blocked managed exit initiator ... at DMD<CCKNJCHHBLL:: FNOEHNDLHHG>()after the drops - the gated block works.
The infinite stall is gone too: DownloadAndParse('OBHEFDFCAIC)') returned
after 22.7s (16:17:44.4479 -> 16:18:07.1543) instead of never, and
OperationContext.FinishRoomLoad followed at 39.1551.
But it returned a FAILURE, and the session died with it:
[16:18:07] [39.1509] AppExitState changed: Running => PreparingToExit
[16:18:07] [39.1566] [Error | RoomOperation] Error during operation
InitialRoomLoad-a820aca0-...: Error downloading save data!
Try again later or on a better connection.
[16:18:08] [40.2863] [Unity] [EAC] Tearing down and releasing runtime
[16:18:09] [41.6710] [Unity] PhotonHandler.OnDestroy
So the exit that ends these sessions is at least partly a CONSEQUENCE of the
failed room load - the documented DORM-BLOB-FIX chain - not purely the scanner.
Note also that the block was armed and STILL did not hold the process: a
[ExitInit] blocked line fired and the client left anyway, so the recurring
exit is not solely the managed initiators.
WHAT IS MISSING THIS TIME: GET /rooms/photon_access_token. Every pre-run-17
boot fetched it during the dorm load (14:59:39.570, 15:07:54.917,
15:13:41.243 and again at 15:13:52.159) and the good run shows
[GetSubRoomDataSave()]: Total: [5397.72ms 2618frames] - the subroom save
download succeeding after ~5.4s. Neither run-17-lineage boot (16:10, 16:17)
ever requested a Photon access token, so the Photon room join is never set up,
GetSubRoomDataSave() has nothing to resolve against, and the load fails with
"Error downloading save data!" instead - after a 22.7s timeout in 16:17, and
forever in 16:10.
Everything server-side is byte-identical across good and bad sessions:
/matchmaking/player/connection-info ~1180 b, /matchmaking/matchmake/dorm
572 b, /rooms/rooms/25?include=5421... 2174 b, /cdn/room/ 22 b. The tracer
is the only client-side delta, so the tracer is the A/B variable.
Run 18a (deployed 16:19, 34304 b): the tracer registration is commented out (the class stays in source with its findings recorded - a boot-time BEFDPLAELIH call arriving from il2cpp with no managed frames, and the exit leaving through FNOEHNDLHHG). Expected on the next boot: `[ExitInit] disabled
- tracer removed for the room-load A/B
, thenGET /rooms/photon_access_tokenin the client log and in the server log,GetSubRoomDataSave()` resolving in seconds, the phases running through to COMPLETE, and a visible Dorm. If all of that happens, the tracer - or one of its three detours - is confirmed as the room-load breaker and it comes back only one target at a time.
Run 18a result and the REAL regression: the baseline experiment restored the
unpatched metadata (root cause of the black screen and the forced exit)
Run 18a booted at 16:20 with [ExitInit] disabled (LogOutput.log line 33) and
the four [HileGate] dropped lines, and it failed EXACTLY like run 18 did:
no GET /rooms/rooms/25/experience/player, no playerdata/me, no
keepsakes/rooms/25, no roomkeys?roomId=25, no
GET /rooms/photon_access_token, then Error downloading save data!,
FailedToSpawnPlayer (Code 8715) and the forced exit. The exit tracer is
therefore NOT the room-load breaker - that hypothesis is dead, and the server
payloads were byte-identical the whole time.
What the tracer-off boot did NOT rule out - and what actually broke every boot after 15:16 - is the rest of the parallel "baseline" experiment. It did not only rename winhttp.dll away at 15:17:15; it also put back two files from the pristine (pre-Radicade) Sep 23 14:21-14:23 state, using mtime-preserving copies, so nothing looked "new" afterwards:
| file | good state (the 14:53-15:13 boots) | what every failing boot ran |
|---|---|---|
RecRoom_Data/il2cpp_data/Metadata/global-metadata.dat |
38,654,794 b, md5 cc9c32014b5fc3dd1a2a8fb12376be15 |
32,039,456 b, md5 825f34f1dd046a0ae764cd5a97dba5e6 |
RecRoom_Data/boot.config |
173 b, md5 dc2e5f7af7d5df736a4cc3477b4b1a1f |
104 b, md5 5694f1046c94cb6b9bbcf6854ed8eaae |
The 104 b boot.config is byte-identical to RecRoom_Data/boot.config.orig
(the patcher's own backup), which is the fingerprint of a restore, not of an
edit. Proof that the 38 MB metadata - and only it - carries the Radicade
rewrite:
grep -a -c 'localhost:3030': 38 MB file = 3, live 32 MB file = 0.127.0.0.1= 9 vs 1,rec.net= 20 vs 4.RecRoom_Data/radicade-patch.jsonrecordspatchedAtUtc 2026-09-23T22:08:42Z(15:08:42 local,changedLiterals: 7, serverUrlhttps://localhost:3030), and that timestamp is exactly the mtime of the 38 MB file captured in the snapshot: the file the baseline experiment replaced was the Radicade-patched one that had been live for all four good boots.- Both files keep the release's custom shuffled header (magic
d4370000...), so the patched metadata is still the native format the runtime expects. The standard-headerglobal-metadata.interop.dat(magicaf1bb1fa) is only the tooling copy and was never touched.
Mechanism: with the pristine metadata the client's hardcoded bootstrap
literals point at the official hosts again. The plugin's runtime redirects only
cover the nameserver host, the websocket host and TLS, so most traffic still
reaches the server through GET https://ns.rec.net/ns - which is why the boot
looks healthy, logs in, joins Photon and renders the Dorm - but the room-save
path is built from a literal that nothing rewrites at runtime, so
GetSubRoomDataSave() fails and the session is torn down by the room-load
failure handler.
Fixed: both files copied back from
baseline-backup-20260924-151633/RecRoom_Data/ (cp -p, provenance kept).
Live again now: metadata md5 cc9c3201... (38,654,794 b, 3x localhost:3030),
boot.config 173 b (vr-enabled=0, player-connection-mode=Listen,
player-connection-debug=0). winhttp.dll had already been restored, a scan for
any other 2026-09-24 write outside BepInEx found nothing, and the only disabled
sidecars are winhttp.dll.baseline-disabled (dormant) plus the BepInEx
config/loader backups - so the baseline undo is now complete.
Caveat for future restores: the patcher's own input (38,654,604 b) no longer
exists anywhere, because the experiment also restored
global-metadata.dat.orig to the 32,039,456 b Sep 23 14:23 file. The canonical
restore sources are therefore the snapshot directory and
global-metadata.interop.dat - never .orig.
Deployed build for the next boot stays run 18a (34304 b, md5
b5cdc49cf2da66e306ee20f4e60a515c, tracer off), so the metadata fix is a clean
A/B. Success markers: GET /rooms/rooms/25/experience/player (404), then
playerdata/me, keepsakes/rooms/25 and roomkeys?roomId=25 in the server
log, GET /rooms/photon_access_token, [GetSubRoomDataSave()] resolving in
seconds, FinishRoomLoad COMPLETE and a visible Dorm. The gated
ExitInitiatorTracerPatch and a never-expiring exit veto come back only after
that, one target at a time, with the metadata state now treated as a
first-class tracked variable (md5 above) rather than an assumed constant.
Run 19 (16:54 boot): the Dorm LOADS - the next killer is the Play menu's
missing list routes
The metadata restore worked. The 16:54:57 boot completed the dorm load for the first time since run 16:
16:55:10 phase DOWNLOADING_DATA, DownloadAndParse('OBHEFDFCAIC)') 16.2212 -> 16.2296 (8 ms, was 19-22 s or never)
16:55:11 GET /rooms/photon_access_token
16:55:13 [10976][19.4330] OperationContext.FinishRoomLoad(e4909cf1-...)
16:55:13 Moved to phase COMPLETE (SCENE_LOADED -> SETUP_ROOM_OBJECTS -> RUN_MIGRATIONS
-> DESERIALIZE_OBJECTS -> DESERIALIZE_CONNECTABLES -> COMPLETE, SpawnLocalPlayer at ~19.5s)
So GetSubRoomDataSave() was never the broken thing - the unpatched metadata
was. Player spawn works too (the only spawn-time error is a benign Toxmod
config warning).
The new failure: the Play menu's discovery rows 404
At 27.7 s the client opened the room list and its rows failed:
16:55:22 GET /lists/algorithmiclists/HotList?type=1 -> 404
[Error] Error refreshing rooms from source Hot: An error occurred (RoomListModel)
16:56:05 GET /rooms/rooms/carousel/rising?skip=0&take=20 -> 404
[Error] Error refreshing rooms from source CarouselEndpoint: An error occurred
16:56:05 [22050][70.8179] Error refreshing rooms from source Hot: An error occurred
16:56:05 [22051][70.8330] AppExitState changed: Running => PreparingToExit
NullReferenceException at RRUI.Data.RoomListModel.DOPOJKBDDEL()
<- UnityEngine.Object.Instantiate <- RRUI.StaggeredPrefabSpawner.EPABHKKCNIF()
[EAC] Tearing down; Uploading Crash Report
A failed row is not a skipped row: it NREs the model's prefab spawner and ends
the session. Same death in the 15:13 reference run (Error refreshing rooms from source Hot at 15:13:59, exit at 15:14:06), which is why that run ended
seconds after the player reached the RecCenter. The curated-lists 404/[]
note from run 10 was never the whole story - lists/algorithmiclists/* and
rooms/rooms/carousel/* were simply never implemented.
Fix (server, 17:36)
- NEW
Server/Lists/AlgorithmicListsAPI.cs:GET /lists/algorithmiclists/{list}?type=1->{ Type, Entities: [ { Id, Context } ] }(Ida string,Contextnull), the shape and field order the reference server answers with, because the client's obfuscated DTOs are read positionally.typeis echoed (defaulting to 1 = Rooms). Every row key answers 200:HotListand the other ranking rows are ordered byRoomStats.VisitCountdesc,newbyPublishedAtdesc, both over public active non-dorm rooms, 20 per row. Server/RoomServer/RoomServerAPI.cs:GET rooms/carousel/{*source}(next to the other carousel feeds) -> bare array of RoomDetail, name/tag filtered when the source names something, falling back to every public room so a row is never empty. Shape followsrooms/hot_rooms(see docs/2023-WATCH-HOME-FIX.md); if a row still renders empty, the envelope (RoomSearchResult) is the thing to change.- Not changed:
lists/curatedlists/bulkstill answers[]. Its shape is a guess either way (the client logs "Exception: No such list" for the empty array), so it stays as the documented, non-fatal behaviour until the fatal rows are confirmed fixed - changing a DTO the positional codec reads without a verified field order is how a non-fatal error becomes a fatal one.
Verified live over the plain-HTTP listener (server restarted 17:37, now
server-console-run19.log, dotnet run --no-build from Server/):
/lists/algorithmiclists/HotList?type=1 -> 200 {"Type":1,"Entities":[{"Id":"32",...},{"Id":"3"},...]}
/lists/algorithmiclists/new?type=1 -> 200 {"Type":1,"Entities":[{"Id":"35"},...]}
/rooms/rooms/carousel/rising?skip=0&take=20 -> 200 [ {RoomId:2,...}, ... ] (20289 b)
/lists/curatedlists/bulk?id=17859340 -> 200 [] (unchanged, see above)
Server build clean (0 errors, 389 pre-existing warnings). Client build unchanged
(run 18a, tracer still off), so the next boot A/Bs exactly one thing: the rows.
Expected markers: no "Error refreshing rooms from source Hot/CarouselEndpoint"
lines, no RoomListModel NullReferenceException, the Play menu carousels
populated with real rooms, and a session that survives the hot row refresh.
Run 20 (17:43 boot): the row fix holds - the room-list killer is gone
The 17:43:25 boot ran against that rebuild, and every marker above landed:
GET /lists/algorithmiclists/HotList?type=1was requested twice and answered 200 (1171 b).Player.loghas no "Error refreshing rooms from source Hot/CarouselEndpoint" and noRRUI.Data.RoomListModelNullReferenceException.- The rows resolved into real rooms instead of failing: right after HotList the
client asked
rooms/rooms/bulk?id=2&id=4&id=7&...&id=19and drew the carousels (rooms/hot?tag=rro200 / 27163 b, with RecRally/Crescendo/Park/DodgeballVR images following). - The Dorm was unaffected:
Moved to phase COMPLETEat[10504][20.13],OperationContext.FinishRoomLoad,SpawnLocalPlayertotal 10278 ms - the same numbers as the 16:54 reference boot. The metadata precondition was re-checked at boot:global-metadata.dat38654794 b md5cc9c32014b5fc3dd1a2a8fb12376be15,boot.config173 b md5dc2e5f7af7d5df736a4cc3477b4b1a1f.
The session still ended, but differently from every previous failure: the log
stops on a clean quit path (POST /matchmaking/player/logout -> PreparingToExit => ReadyForExit -> EAC teardown -> PhotonHandler.OnDestroy), with no
exception in the 3.7 s before it - where the 16:54 run had the RoomListModel NRE
40 ms before PreparingToExit. A silent 3.7 s gap plus a graceful logout is the
shape of a hand quit, not of the crash path, so run 20 cannot be called a crash
without knowing what the screen did.
Run 20 fix: an empty COMMUNITY carousel
GET /rooms/rooms/hot?tag=community&skip=0&take=20 answered 200 with
43 bytes - no room in Rooms carries the tag community (the column holds
rro,2|quest,0 style pairs), so the Play menu's COMMUNITY carousel was empty.
GetHotRooms's default branch now falls back to the whole public non-dorm
catalogue when the tag matches nothing, the same way rooms/carousel/{*source}
does; it answers 27163 b / 25 rooms now.
Server rebuilt (0 errors, 389 pre-existing warnings) and restarted as
server-console-run20.log, PID 26088, both listeners up:
/rooms/rooms/hot?tag=community&skip=0&take=20 -> 200 27163 b 25 rooms
/rooms/rooms/hot?tag=rro&skip=0&take=20 -> 200 27163 b
/lists/algorithmiclists/HotList?type=1 -> 200 {"Type":1,"Entities":[...]}
/rooms/rooms/carousel/rising?skip=0&take=20 -> 200 20289 b
Known non-fatal noise (do not chase these first)
All of these are present in both the 16:54 run and the 15:13 reference run, so they are background, not regressions:
-
Failed to load Type "RecRoom.Analytics.Statsig.StatsigDebugOverrides"-RuntimeInitializationHelper:OIDNKEGDPPF(Dictionary\2)->MBJKIOBDOAM()->SubsystemRegistration(). It is **vanilla noise**, not a symptom: the 15:13 baseline boot - loader renamed away, pristine 32,039,456 b metadata, i.e. no patch stack at all - logs the same line (artifacts/baseline-20260924/Player-first-baseline.log, line 24), and so does every working boot since, including the bug-report upload atServer/RadicadeServer-Config/BugReports/20260925-044419-Player.log. The type is a debug-only Statsig helper the retail build does not carry; the subsystem registration logs the miss and continues. The August plugin never mentions the string (grep -a -c StatsigDebugOverrides RadicadeAugustClient.dll` = 0). Do not chase it. -
Malformed Response: '[]'(2 in run 20, 5 in the 15:13 run) -ODIGMKMGFCF: expected:'Number Token', actual:'[', at offset:0, i.e. the client deserialised a bare array where it wanted an object. The two in run 20 land within ~1 s ofapi/inventions/v2/mineandapi/inventions/v1/toptoday, both of which answer[]because the inventions tables are empty; the 15:13 run survived five of them, so their shape is wrong but not fatal. -
Could not enable featured rooms display... no images are available-featuredrooms/currentdeliberately 404s. -
404s:
/rooms/rooms/25/experience/player,/rooms/rooms/25/experience,/econ/econ/roomGiftDropShops/room/25,/data/commonQuery/userSegment,/econ/api/influencerpartnerprogram/{influencer,myinfluencer}. -
Exception: No such list/Deserialization returned nullforlists/curatedlists/bulk(documented above).
Run 21 setup: the tracer is re-armed (it was never the room-load regression)
Run 20's exit was confirmed by the user: it closed on its own, with the screen
looking fine - no black screen, no error on screen, no crash dialog. That is a
silent self-exit through the graceful path, i.e. the same class of shutdown the
run-17 ExitInitiatorTracerPatch was built to name.
The run-18a reasoning that parked the tracer is now known to be wrong. Its two
boots (16:10, 16:17) never fetched /rooms/photon_access_token and died with
"Error downloading save data!", so the tracer was blamed - but those boots were
running the pristine global-metadata.dat + boot.config restored by the
baseline experiment at 15:16, which is exactly what starves
DownloadAndParse('OBHEFDFCAIC'). With the patched metadata back, run 20
completed the load in the usual 8ms and 10278ms and the tracer was off, so the
tracer is not implicated in the room-load regression.
Re-armed in Plugin.cs (harmony.PatchAll(typeof(ExitInitiatorTracerPatch))),
rebuilt (dotnet build 2023Patch/RadicadeAugustClient/RadicadeAugustClient.csproj -c Release, 0 errors) and deployed to BepInEx/plugins/RadicadeAugustClient.dll
(34304 b, md5 7fd2f6c698f9568ebc6019bd8ca01e2f; the previous run-18a build is
kept as RadicadeAugustClient.dll.before-run20-tracer, md5
b5cdc49cf2da66e306ee20f4e60a515c). The old "disabled" log line is gone from the
binary, the arm message is present.
What it does: prefixes CCKNJCHHBLL.BEFDPLAELIH / GIEEBIOOFML / FNOEHNDLHHG
(the managed funnel every shutdown passes through). While
HileGate.DroppedCount == 0 the call is allowed and logged (that is normal boot
traffic); once a report has been dropped the call is blocked and the full
managed caller stack is logged as [ExitInit] blocked managed exit initiator.
The gate dropped a report at 17:43:38 in run 20 and the exit fired at 17:43:59,
so a repeat boot will be armed well before the exit.
Expected in BepInEx/LogOutput.log on the next boot: one [ExitInit] allowed managed exit initiator early (boot traffic, before the drop), then either an
[ExitInit] blocked managed exit initiator stack at ~37s - which names the
trigger and should also stop the shutdown - or no ExitInit line at all, which
would prove the trigger is native-only (TerminateProcess-class) rather than
managed. Either outcome is a result; the metadata/boot.config md5 pair above is
the precondition to re-check first.
Run 21 (19:34 boot, tracer armed): the trigger is NATIVE, and the tail is managed
Both expected lines appeared, and they settle the model:
line 36 [ExitInit] allowed managed exit initiator -> DMD<CCKNJCHHBLL::BEFDPLAELIH>() -> (il2cpp -> managed)
line 183 [HileGate] dropped anti-cheat report POST
line 276 [ExitInit] blocked managed exit initiator -> DMD<CCKNJCHHBLL::FNOEHNDLHHG>() -> (il2cpp -> managed)
Two things follow from the traces. First, no managed caller ever appears: the
frame above each initiator is (il2cpp -> managed), i.e. native il2cpp code
invoking the managed method directly, so whatever decides to leave is not managed
code and no managed veto can see it. Second, the blocked initiator fires at the
very end of the shutdown, after PreparingToExit (19:34:36.77) and after
ReadyForExit (19:34:38.47) - the [ExitInit] block sits within the last ~9 log
lines, next to the logout / PUT playersettings / final analytics flush. So
FNOEHNDLHHG is the last step of a shutdown, not its trigger: blocking it held
the process ~13s longer than the previous boot but the native side still finished.
What the boot looked like from the outside:
- Dorm + spawn fine again:
Moved to phase COMPLETE[14020][26.5664],SpawnLocalPlayer10629.92 ms. - Play menu fine:
HotList?type=1200 (1171 b) thenrooms/bulk?id=2&id=4&id=7...(21460 b) with the carousel art. - Only three 404s, all long-known:
rooms/25/experience/player,econ/econ/roomGiftDropShops/room/25,data/commonQuery/userSegment. - The client stayed interactive until the last second:
Applied KEYBOARD_MOUSE control scheme LEGACY(27.1s, automatic),ProUX(32.42s),LEGACY(41.98s) - i.e. the settings menu was being used - thenPOST /matchmaking/player/logoutat 19:34:36.676, 100ms beforeAppExitState Running => PreparingToExit. - No OS-level crash: Windows Application log has no
Application Error/Application Hangentry for RecRoom.exe at 19:34; the game's ownCrash Dump Report: New: 0 Previously Sent: 0. This is a clean exit, not a crash, however it looks on screen.
So the open question is no longer how the exit happens (native, at the tail) but what decides it, and the only always-present candidate left is the same one as every prior run: the dropped anti-cheat report. The two runs with a known cause of death now stand 37s (17:43, tracer off) and 50s (19:34, same shape) apart with no other delta, so the next boot should be a hands-off soak: launch it, touch nothing, and see whether it exits on its own at all.
The user confirmed it was not a quit: "I was only toggling settings, never
pressed Quit". Ruled out in the same pass: no [NotifyHub] -> <connection> send at
all in run 20 (the only hub lines are connect/register/disconnect, the last one at
the logout instant), so no server push ends the session; a 404 in the notify socket
is not it either; and the report's own response body is a dead end (RADICADE_HILE_BODY
tried ok-true/ok-false/nocontent/servererror back in run 13 - ok-false only
DEFERRED the exit).
Run 22: the exit veto is back (rebuilt), on the decision point this time
The old ExitStateVetoPatch was the only mitigation that ever absorbed these
exits (16 blocks in runs 12/13/16) and it was deleted from the tree at 15:11 by a
parallel edit, never committed - the whole 2023Patch/RadicadeAugustClient/
directory is untracked (git status --porcelain -> ??), so there is no history to
recover it from. It has been rewritten against the decompilation instead of guesswork:
.external/aug-decomp/acs/CCKNJCHHBLL.cs gives
enum CCFIKLBABNB { Running = 0, PreparingToExit = 1, ReadyForExit = 2 },
the private static setter KALEFEDAHCC(CCFIKLBABNB) (token 100667095, CallerCount
3) and its interop wrapper as the setter of property CCKNJCHHBLL.GLNIKJNLCIB
(getter JABDIMNNBCC, token 100667094). Every AppExitState changed: Running => PreparingToExit line in Player.log is that setter.
ExitStateVetoPatch now patches BOTH the wrapper setter and the private method
(resolved by name, whatever exists), allows only Running, and blocks every other
transition with no expiry window - run 17 showed the re-requests continue long
after the old 120s window closed and the process leaves the moment it opens. Blocks
log the first three, then every 25th. Escape hatch for a real quit:
RADICADE_EXIT_VETO=0 before launch (with the veto armed, quitting from the menu is
refused too - end the process from Task Manager instead). Registration is wrapped in
try/catch so a veto that will not bind can never take the boot down with it.
A trap worth remembering: the first build of this patch was a silent no-op. The
log line was LogInfo(Enabled ? "armed ..." : "disabled ...") over a
static readonly bool Enabled, and string-searching the built dll showed the
true-branch literal was absent while the false-branch one was present - i.e. the
compiler had decided the condition. Enabled is now a property
(=> !_disabled) and the log line is built at runtime, so the dll contains
[ExitVeto] patches applied; blockExitState enabled= and the value can be read
straight off the next boot instead of being assumed. Verify patches by searching
the SHIPPED dll for their strings; do not trust the arm message alone.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 36352 b, md5
4a672a6b0ba59f73ca785cb318a6b824 (previous builds kept as
RadicadeAugustClient.dll.tracer-run21, md5 34304 b / 19:32, and
RadicadeAugustClient.dll.before-run20-tracer, md5 b5cdc49cf2da66e306ee20f4e60a515c).
Build clean (0 errors). Server untouched and up (server-console-run20.log, PID
26088). Expected on the next boot: [ExitVeto] ... enabled=True, the early
[ExitInit] allowed line, the [HileGate] dropped line, and then - instead of the
silent shutdown at 37-50s - [ExitVeto] blocked AppExitState -> PreparingToExit
while the session keeps running. If the session instead dies with no [ExitVeto]
line, the veto never bound and the [ExitVeto] failed to apply error text will say
why.
Run 23: the exit's chain is named, and the last wall is Unity's own quit callback
Two questions were open after run 22: what starts the shutdown, and whether any frame of it is still reachable from managed code. The first is now answered from evidence that was already on disk; the second is patched.
The chain
- The client's own scanner finds a planted module and files the anti-cheat report.
That report body was decoded back in run 9 and is still in the server logs:
Hile report received: Type=Obcured Message=[0:winhttp.dll]- the BepInEx Doorstop proxywinhttp.dllin the game root, classifiedObscured(hile enumMEHDFJHKLBK.JBLJIENBBHK, Obscured = 0). The classification is inherent: run 9 logged the identical body at 22:16, hours before the run-14 PEB unlink (23:27) and the run-15 PE-signature wipe (23:42) existed, so hiding never caused it and un-hiding will not remove it. - It re-scans on a fixed schedule.
server-console.logholds two reports at23:32:28.141and23:32:58.405- 30.264s apart. That schedule, not a boot timer, is why the same build dies at 36.8s on one boot and 55.3s on the next: the exit lands on the first tick after the boot completes. - The report is not the trigger. Dropping it at the HTTP layer (
HileGate, run 16) only defers the exit, and run 13 proved the response body is irrelevant. The scanner's local verdict starts the shutdown; the report is a side effect. - Where it ends (run 22
Player.log):UnityEngine.Application:Internal_ApplicationWantsToQuit()->System.Func\1[TResult]:Invoke()->CCKNJCHHBLL:GBGMBMGDGGD(System.Boolean)->MGLDPDAEODB:EMDNIBPNEGH()->PFDOLDLEIBA:CKIEGOCLGFD(Action`1), and then[EAC] Tearing down and releasing runtime` three times in the last 3.3s.
The 15:13 reference run self-exits too. Player-before-baseline.log - the boot
that loaded the Dorm and was taken as good - ends with
[15:14:05] AppExitState changed: Running => PreparingToExit at 41.81s, followed by
the same EAC teardown. Its Could not enable featured rooms display warnings start at
23.1s and repeat (run 22: 23.2s, 30.2s, 37.2s, 44.2s, 51.2s - every 7.0s), so they are
periodic RRUI noise and not the precursor they looked like. Nothing in the recent
client work causes this exit; it is inherent to running this build with a Doorstop
module in the game root.
Patched in this build
ExitSequenceTracePatch- aPriority.Firstprefix onBestHTTP.HTTPManager.SendRequest(HTTPRequest)(the same managed funnelNameserverConnectionPatcheslogs every request in) that dumpsEnvironment.StackTracethe first four times the shutdown's own requests go out:/matchmaking/player/logoutand/playersettings/playersettings. The shutdown sequence's first observable network act is the logout POST (server log run 22:19:45:38.785, right after the notify-hub reconnect). Whoever calls it is the owner of the shutdown, and the stack names them - no inference needed.ExitStateVetoPatchnow also dumps the stack on its first block, which names the caller of theRunning => PreparingToExittransition (run 22 blocked three times with no caller recorded).UnityQuitVetoPatch- the last wall. Prefixes bothCCKNJCHHBLL.GBGMBMGDGGDoverloads (decompilation tokens 100667099 / 100667100; the stack trace prints theboolone, so both are bound rather than guessing) and forces__result = false, which is "the application does not want to quit" and makes Unity cancel the quit. It skips the original instead of running it and rewriting the result: run 22's trace shows the original throwing (NullReferenceExceptionout ofStatsigUnity.EventLogger.StopCoroutine), and a throwing original returns before any postfix can override the value Unity reads - a postfix-only veto would be silent exactly when it matters. Escape hatch:RADICADE_QUIT_VETO=0, andEnabledis a property (=> !_disabled) with a runtime-built log line, the run-22 folding trap avoided.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 39424 b, md5
71df28734d88fd3f9eca70f202e517bb; the run-22 build is kept as
RadicadeAugustClient.dll.before-run23-quitveto (36352 b, md5
4a672a6b0ba59f73ca785cb318a6b824). Build clean (0 errors, 0 warnings).
Preconditions re-checked at deploy time: global-metadata.dat 38654794 b md5
cc9c32014b5fc3dd1a2a8fb12376be15, boot.config 173 b md5
dc2e5f7af7d5df736a4cc3477b4b1a1f.
What the next boot has to say
[ExitTrace] shutdown-request stack dump armed; [QuitVeto] wantsToQuit veto enabled=Trueat load, then[ExitVeto] ... enabled=True, the early[ExitInit] allowed, and the[HileGate] droppedline at the first 30s tick.- At the exit:
[ExitTrace] shutdown request ...with the stack that ordered it, and[QuitVeto] wantsToQuit answered falsewhen the quit is refused. - If the process is still alive minutes after that the plant works and the remaining
problem is cosmetic (whatever state the half-run shutdown left the UI in). If it dies
anyway, the kill is outside Unity's quit path (
TerminateProcess-class, or EAC's own runtime) and the next lever is to remove the module the scanner objects to instead of fighting the reaction - i.e. bootstrap BepInEx without awinhttp.dllproxy in the game root, which means a loader-side injection (manual mapping or a launcher) rather than Doorstop.
Run 23b: neutralise the EAC runtime itself (ported from the supplied patcher)
C:\RecRoom\Downloads\patch is a decompiled copy of RecRoomil2cppPatcher
(WaeBoi69, loaded by Reloaded-II). Its EacPatcher.cs is the piece this project needed:
a proven recipe for neutralising EasyAntiCheat on Rec Room. Five hooks into
EasyAntiCheat.Client.dll:
| Method | Replacement |
|---|---|
Hydra.Runtime:IsActive |
return true |
Hydra.Runtime:Initialize |
no-op |
Hydra.NativeClient:Initialize |
no-op |
Hydra.NativeClient:IsStartedWithEasyAntiCheat |
return true |
Hydra.Runtime:get_Initialized |
return true |
The pattern is the point: the initialisers are stubbed while the queries lie and report a healthy, active anti-cheat - which is what the game's own checks read.
Our build exposes the same surface. BepInEx/interop/EasyAntiCheat.Client.dll contains
EasyAntiCheat.Client.Hydra, .Hydra.Runtime, .Hydra.NativeClient, .Hydra.NetProtect
and exactly those members (ilspycmd -o ... EasyAntiCheat.Client.dll; output kept in
.external/interop-decomp/EasyAntiCheat.Client/EasyAntiCheat.Client.decompiled.cs).
Why this is the root-cause lever and not another wall: the report is Obscured +
winhttp.dll, i.e. a native module scan, and NativeClient.Initialize is where that
native client is brought up.
Deployed in our plugin (RADICADE_EAC_NEUTRALIZE=0 disables; armed by default):
EacNativeClientStubPatch- every staticInitializeonHydra.NativeClientis skipped, so EAC's native client (home of the module scanner) never starts.EacHealthQueryLiePatch-Hydra.NativeClient.IsStartedWithEasyAntiCheat,Hydra.Runtime.IsActiveandHydra.Runtime.get_Initializedanswertrue.Hydra.Runtime.Initialize(...)is deliberately left running: it does the game-facing bookkeeping (event wiringRecRoom.AntiCheat.EACManagersubscribes to viaJMAPOMLPFMI/HINJPEHPJGO/AOFHJPLDLBC) and its native step is the stubbedNativeClient.Initialize. Stubbing it too is stage B if this is not enough.- Type discovery is name-based with a hard-reference fallback
(
typeof(EasyAntiCheat.Client.Hydra.Runtime).Assemblyin its own method, so the JIT only resolves it if the soft lookups came up empty) - nothing else in the plugin touches that assembly, so at Load timeAccessTools.TypeByNamecan legitimately see nothing.
The game's EAC readiness door is already propped open (AugustChallengePatches fakes
EACManager.GenerateChallengeResponse and forces EACManager.MGEAHKJIIMO true), so stopping
the runtime does not leave one of the game's checks waiting on it.
Success marker, and it is unambiguous: the [HileGate] dropped anti-cheat report POST
line must no longer appear at the 30s tick. No report means the scanner never reached its
verdict, which means no shutdown. The run-23 exit walls stay armed as the backstop, and their
[ExitTrace] / [QuitVeto] / [ExitVeto] lines say whether they were needed.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 43520 b, md5
434972bf54968f1287643d8be61de8f7 (the run-23 plain build is kept as
RadicadeAugustClient.dll.before-run23b-eacstub, 39424 b). Build clean (0 errors).
Verification method correction
The run-22 "constant folding" conclusion came from a hand-rolled check - decode the shipped
dll as UTF-16 and String.Contains the literal - and that method is not reliable:
against this build it reports EasyAntiCheat.Client.Hydra.NativeClient and
[EacStub] EasyAntiCheat Hydra runtime neutralisation enabled= as MISSING, and the
decompiled IL shows both present. Do not use it. Use the decompiler on our own output:
ilspycmd -o .external/selfcheck 2023Patch/RadicadeAugustClient/bin/Release/net6.0/RadicadeAugustClient.dll
grep -c "EacStub" .external/selfcheck/*.cs
The run-22 lesson's shape is kept anyway (Enabled is a property over a
static readonly _disabled, log lines built at runtime) - it costs nothing, and it is the
only version that cannot be folded if the compiler ever does decide to.
Run 23b outcome (20:44 boot): three negative results, each of them useful
Session: 20:44:21 -> 20:45:11 (50s). Boot was healthy and identical to the best
boots - Moved to phase COMPLETE at 20.24s, SpawnLocalPlayer 10568.89ms.
- EAC's managed runtime is not the scanner.
[EacStub] ... enabled=TrueandHydra.Runtime.IsActive answered trueappear, but there is noskipped ... Hydra.NativeClient.Initializeline anywhere in the boot: the game never calls it, i.e. EAC's native client was never started in this setup - our ownEACManagerpatches had already seen to that. Thewinhttp.dllverdict still arrived. The scanner is RecRoom's own, not Hydra's. Auena's recipe is still worth having (it is what keeps a modded client's EAC surface quiet), but it is not the fix for this. - Unity's quit is not the kill.
[QuitVeto] wantsToQuit answered false (block #1)fired and the game kept running for ~18s afterwards - it went on loading images, PUTting settings and fetching avatar items - so the veto did its job on the request it saw. ButPlayer.logshowsTrying to send photon room updates to server while application is quitting - this should not be happeningfrom 36.1s, i.e. Unity was already in its quit state before the veto fired: the first quit request took the native path (Application.Quit()from native code does not consultwantsToQuit), and the request we blocked was a later one. The process ended at 52.85s. Windows' Application log has noApplication Error/Hangfor it, and this run's shutdown again begins with[EAC] Tearing down and releasing runtimeat 34.89s. - The absence of
ExitVeto/ExitInitlines is itself the result: this run's shutdown never touched the managed exit state machine at all (noAppExitState changed, no blocked transition, no initiator call). So the veto and the tracer are aimed at a path this build does not use when the anti-cheat verdict starts the shutdown; the verdict path is native end to end.
The stack dumps did name the boundary honestly: [ExitTrace] shows
DMD<BestHTTP.HTTPManager::SendRequest> followed by
(il2cpp -> managed) SendRequest(IntPtr, Il2CppMethodInfo*) - the logout and the
settings PUT are issued by native code calling into managed BestHTTP, so no
managed caller exists to name. Same shape as run 21's finding for the initiators.
Run 23c: capture the verdict, stop obscuring the module, trace CheatManager
Given the above, the only remaining fix is to stop the verdict instead of the reaction. Three changes, all in one build, all observable in one launch:
- The run-14/15 module hiding is now OFF by default (
RADICADE_MODULE_HIDE=1re-arms it). Runs 14/15 proved the scanner cannot be blinded - unlink and wipe both applied mechanically and the report fired anyway - while "a loaded module that the loader lists do not know about, with its MZ/PE headers zeroed" is the most literal description of an obscured module there is, andObscuredis the exact verdict the client files againstwinhttp.dll. It bought nothing and may have been feeding the classifier; the plugin now logs what a Toolhelp module walk- the same view a scanner gets - actually sees (
[ModuleList] ...).
- the same view a scanner gets - actually sees (
HileGateis now a capture gate, not a drop gate:ShouldDropdumps the report's whole wire form (request.MethodType, the URI andrequest.RawDataas UTF-8, withGetEntityBody()as the fallback) and then lets it through, so both the client log and the server log carry what the client believes it found. The server'sHileAPIkeeps onlyTypeandMessage; the body may carry more.RADICADE_HILE_GATE=droprestores run 16.CheatManagerTracePatchprefixes every declared method ofCheatManagerand its nested types and logs name + elapsed ms (first two calls of each, hard line cap).CheatManager(Assembly-CSharp, global namespace, aSingletonMonoBehaviour) is the one managed class in this build that plainly does local anti-cheat work -OnTimeCheatDetected,OnObscuredTypeCheatDetected,OnAdvancedMovementCheatDetected,IPEMGBLAFIJ(string),BFNMCBEDOLF(string),CheckForDUIDMismatch, and two nested helpers carryingappPath/filenames. Nothing is skipped: observing only, because the one detour this project must never take is the scanner's own funnel (run 11). Whatever runs a few hundred ms before the[HileGate] capturedline is the verdict, and its name is the next patch target.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 48128 b, md5
018b991ab951196a76e4ed7f7f3eac4b (run-23b build kept as
RadicadeAugustClient.dll.before-run23c-capture, 43520 b). Build clean (0 errors),
verified by decompiling the built dll (ilspycmd -o .external/selfcheck), not by
string search. The run-23 exit walls stay armed as the backstop.
Run 23c outcome (21:02 boot): the trace named the producer
The CheatManager trace paid for itself on its first run:
+15113ms CheatManager.OnApplicationFocus
+15366ms CheatManager.MJLAGDJLOBC
+15367ms AALEGIEIKEL.LLPNCADGDNI
+15368ms AALEGIEIKEL.LLPNCADGDNI
+15368ms [HileGate] anti-cheat report captured (full wire form)
CheatManager.MJLAGDJLOBC() - a private no-argument method on the anti-cheat
MonoBehaviour - runs 2ms before the report, with AALEGIEIKEL.LLPNCADGDNI(string)
(a helper whose class carries an appPath field) called twice on the way. A pass
over files in the game directory is exactly what would decide that a proxy DLL in
the game root is Obscured, which is the verdict [0:winhttp.dll] reports.
Two things the capture taught us about the report itself: RawData is empty and
the fields live in the entity body (36 bytes = Type=Obcured&Message=[0:winhttp.dll],
matching run 9's decode exactly), and the body text was not printed by the first
version of the dumper - it printed the byte count only. Now it prints both.
Also confirmed: [ModuleList] winhttp.dll is enumerated by a Toolhelp module walk: C:\RecRoom\2023-08-29T06-53-33Z\WINHTTP.dll - run 14's PEB unlink and
run 15's PE wipe really are off now, and the module is back in the list a scanner reads.
What the exit walls actually buy, now measured on screen
The 21:02 boot lost its session to the same shutdown, but this time the process
survived (the Unity quit veto answered false and the game kept running). What the
player sees in that state is the run-12 prediction, reproduced on demand:
Error joining room instance RoomInstance - ID:20871 DORM_ROOM:25/32
NullReferenceException: Object reference not set to an instance of an object.
Attempting to go to dorm
Not logged in to matchmaking service
Not logged in to matchmaking! not letting me in no more
The server log agrees with the order: the report arrived for the first time in these
boots (21:02:44.255 POST /api/PlayerReporting/v1/hile -> 200 body=true, because the
gate is now in capture mode) and the client logged out 156ms later
(21:02:44.411 POST /matchmaking/player/logout). So the walls convert a silent exit
into a live-but-logged-out client: better for diagnosis, useless as a game.
Server-side note for future probing: localhost:3030 speaks TLS. A plain-HTTP
probe answers 000 in ~5ms and looks like a hung server; curl -sk https://127.0.0.1:3030/... answers 200 in ~11ms. The server was healthy throughout.
Run 23d: suppress the scan instead of the reaction
CheatScanSuppressPatch skips CheatManager.MJLAGDJLOBC() (env RADICADE_CHEAT_SCAN=0
leaves it alone). Given the trace, that method is the producer of the verdict, so
skipping it means no verdict, no report and no shutdown - the only stopping point that
leaves a playable client, since the reaction is native end to end and the walls only
trade an exit for a zombie.
Supporting changes in the same build:
CheatTraceper-method limit 2 -> 4, so if the skip is not enough, the third and fourth calls of every candidate are still on the record and a miss stays diagnosable.[HileGate] armednow states which mode it is in (capture vsRADICADE_HILE_GATE=drop) instead of always claiming the drop.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 50176 b, md5
b96b1b8926a1ec172e2b84ea0a85d179 (run-23c build kept as
RadicadeAugustClient.dll.before-run23d-scan, 48128 b). Build clean, verified by
decompiling the built dll.
Expected on the next boot: the session stays logged in past the 15-40s window it has
never survived, with no [HileGate] captured line and a [CheatScan] skipped CheatManager.MJLAGDJLOBC() (#1) where the verdict used to appear. If instead the
process dies exactly at the skip, that pass runs on a scanner thread and joins the
run-11 forbidden list (KDJDKJNPELI.IJHOPKJFKBI) - also a result, and the lever after
it is to remove the module the scan objects to rather than the scan itself.
Run 24 (2026-09-24 21:0x) - the first real crash of the project, and how to name it
The skip worked. [CheatScan] skipped CheatManager.MJLAGDJLOBC() fired twice, [HileGate] captured never appears, the server log has no /api/PlayerReporting/v1/hile POST for the
whole session, and there is no matchmaking/player/logout - the anti-cheat chain that ended
every previous run is genuinely gone. With it gone the client did something it had never done:
it entered RecCenter.
One second later it died. This time it is not a graceful shutdown:
Application Error 09-24 21:11:58
Faulting application: RecRoom.exe, 2020.3.33.23162
Faulting module: GameAssembly.dll
Exception code: 0xc0000005 (access violation)
Fault offset: 0x00000000029a79ec
Application Hang has nothing, and the previous day's 18 RecRoom faults are a different
signature ({coreclr.dll +0x1d1fdd}, {GameAssembly.dll +0x7567f0 / 0x80000003 breakpoint},
{unknown / 0xffffffffffffffff}); 09-24 had no RecRoom fault at all until this one. So this is
new code territory, reached only now that the room load is not being cut short at 15-40s.
Reading the dump without a debugger
WER left %LOCALAPPDATA%\CrashDumps\RecRoom.exe.8396.dmp (61 MB). No cdb/windbg/dotnet-dump
is installed, so tools/minidump_stack.py reads the minidump directly: exception stream,
module list, thread context (amd64 Rsp/Rbp/Rip at 0x98/0xA0/0xF8) and the stack pages, then
prints every stack slot that points into a module as module+0xrva. --map resolves those
through the RVA table the client writes (below), so no debugger is needed at any point.
It reproduces the WER record exactly, and adds the two things WER does not say:
fault: ACCESS_VIOLATION (0xc0000005) at 0x7ffc16f679ec
read of address 0x93
in GameAssembly.dll+0x29a79ec
rsp 0x41fb78e6d0 rbp 0x24447df8250 rip 0x7ffc16f679ec
A read of 0x93 means a null object with a field at offset 0x93 was dereferenced, and
il2cpp left the check off - which is how pointer-arithmetic paths (fixed/Unsafe/native
arrays) fail, not ordinary managed field access (that raises NullReferenceException). The
stack around it (innermost first) is ~20 GameAssembly frames, with
GameAssembly.dll+0x2881f73 the closest return address and coreclr.dll+0x11f66a
(BepInEx's .NET runtime) and UnityPlayer.dll frames behind them:
rsp+0x28 GameAssembly+0x2881f73 rsp+0x108 GameAssembly+0x11f2eae
rsp+0x98 GameAssembly+0x59adea8 rsp+0x1a8 GameAssembly+0x1201e67
rsp+0xa8 GameAssembly+0x7764a5 rsp+0x218 coreclr+0x11f66a
rsp+0xd8 GameAssembly+0x49dcd3f rsp+0x2d8 GameAssembly+0x1202175
What the last requests were (the "last API was the heartbeat" question)
21:11:50.944 POST /matchmaking/player/heartbeat 200 14.5 ms len=631
21:11:53-57 POST /datacollection/data/event 200 x3
21:11:55.395 GET /clubs/club/home/me 200
21:11:55.433 GET /data/commonQuery/userSegment 404
21:11:55.648 GET /lists/algorithmiclists/HotList 200 1171 b
21:11:55.675 GET /rooms/rooms/bulk?id=2..19 200 21460 b
21:11:56.113 GET /rooms/rooms/3/interactionby/me 200 47 b
21:11:56.124 GET /rooms/rooms/3/experience 404
21:11:56.170 GET /images/RecCenter?width=1024 200 71351 b
21:11:57.213 POST /datacollection/data/heartbeat 200
<crash> 21:11:58, dump written 21:12, notify-hub SSE closes 21:12:04
The request just before the crash is the data-collection heartbeat, and the matchmaking
heartbeat 7 s earlier is the only call in the session with an unusual duration (14.5 ms vs
sub-millisecond everywhere else, 631 b response), so both are on the list. But the crash is
1 s after the RecCenter art fetch and is a null dereference inside the game's own compiled
code, so the room-3 load is the stronger reading. rooms/rooms/3/experience returning 404 is
not obviously the cause: the reference server has no such route either.
The new instrument: rva -> method
An offset from a module base is worthless until it is a method name, and IL2CPP will hand the
name over: the MethodInfo that il2cpp_class_get_methods returns begins with the compiled
method's native entry point. 2023Patch/RadicadeAugustClient/CrashAddressMap.cs walks
il2cpp_domain_get() -> assemblies -> images -> classes -> methods (all raw entry points are
public in Il2CppInterop.Runtime.IL2CPP), writes rva<TAB>image<TAB>Class::Method to
BepInEx/radicade-addr-map.txt, and logs the methods bracketing the target RVA. It is a
background thread started 5 s into the boot, it never blocks the login, and it does not need
the crash reproduced - the table is static. RADICADE_ADDR_MAP=0 disables it and
RADICADE_ADDR_TARGET=0x... re-aims it at the next WER offset, so no code change is needed
for the next crash.
Deployed BepInEx/plugins/RadicadeAugustClient.dll, 55296 b, md5
129e9b5ef7e55820f05d8440b9d5b900 (run-23d build kept as
RadicadeAugustClient.dll.before-run24-addrmap, 50176 b). Build clean (0 warnings), and the
shipped IL was verified by decompiling the built dll (CrashAddressMap.Start() is called from
Load, and the enumeration + .txt path are present).
Next: one boot, then sort -k1 BepInEx/radicade-addr-map.txt offline and resolve
0x29a79ec plus the stack RVAs above through it - which names the method that dereferenced
null, and the frames that led there.
Run 25 (2026-09-24 21:27) - RecCenter loads, then a second, different crash
This is the run that made the room load irrelevant: the room finished loading. The
2023 build's own load profiler ends with [Run Success Callbacks], SpawnLocalPlayer
total 9857 ms, MoveToPhase(COMPLETE), and the server served everything in order
(matchmaking/matchmake/room/3 -> instance 20876, room 3 metadata, photon token,
reportjoinresult, verifyRole). The anti-cheat chain stayed off (CheatScan skipped
MJLAGDJLOBC twice, no [HileGate] captured, no hile POST, no logout).
Then, ~10 s after the spawn, the client died again - at a different address:
Application Error 09-24 21:28 RecRoom.exe / GameAssembly.dll
Exception code 0xc0000005 Fault offset 0x0000000003dc0c57
0x3dc0c57 is inside Unity.Entities.ChunkDataUtility::GetComponentDataWithTypeRO
(exact entry point match), and the faulting instruction is mov rbx, [rdx+0x88] with
rdx = 0xFFFFFFFFFFFFFF77 - a read of address 0xFFFFFFFFFFFFFFFF. So the game handed an
ECS chunk accessor a garbage pointer, and il2cpp had no null check to catch it.
Compared with the 21:11 crash, that is two different fault sites (0x29a79ec is a 26-byte
unnamed function; read of address 0x93) but the same shape: a pointer-arithmetic dereference
inside the game's own compiled code, roughly 1-3 s after a periodic RRUI
"Could not enable featured rooms display" warning. That warning is not the trigger - it also
fires every ~7 s in the dorm and nothing happens. Likewise the 35 RecCenter_Chair ... has an RigidbodyEx but no EmbodiedObject/RbexEntity/TransformEntity component errors are pre-existing
noise: they appear 35 times in both the crashing run and in the 21:02 run, dorm included.
Tooling that made this readable
BepInEx/radicade-addr-map.txt- 254 171 methods from 305 images, written in 141 ms (18.5 MB, benign: a background thread, 5 s into boot).tools/pe_unwind.py- parses.pdataRUNTIME_FUNCTIONs and UNWIND_INFO (PUSH_NONVOL, ALLOC_SMALL/LARGE, SET_FPREG, SAVE_NONVOL(_FAR), PUSH_MACHFRAME, chained info) and replays a prologue to recover the caller - a real x64 unwinder. Two bugs found and fixed while writing it:ALLOC_SMALLcarries its size in the op-info nibble (a zero operand advanced RSP by 8 instead of 0x28, which produced one bogus frame), and a past-the-end lookup used to reuse the previous function's unwind info and manufacture frames - a leaf function with no entry is now unwound from[rsp]instead, as a debugger would.tools/minidump_stack.py --unwind- walks the dump's faulting thread; resolves the WER offset against the dump ("matches the WER fault offset") before using either.tools/resolve_rva.py- function boundaries from.pdata, names from the boot-time map, and it saysUNNAMEDwhen the nearest name is not the function's own entry point.
The verified chain for the 21:28 crash (innermost first, exact-name frames only):
Unity.Entities.ChunkDataUtility::GetComponentDataWithTypeRO <- crash
(unnamed generic-instantiation frames, no metadata entry point)
IHDIFFAKHOM::JEONIAELMNM
(unnamed)
EHEHGHCKFII::System.Collections.IEnumerable.GetEnumerator
Tool::CCFENLNCIOG -> Tool::LPAOCFPMJFD -> Tool::RpcOnReset
-> Tool::AuthorityResetToDefault -> ToolCleanup::BFFGEGKNGMM
RecRoom.Core.Updates.StaggeredUpdateManager::DGHHCHACHOM
-> (unnamed) -> UnityPlayer.dll main loop
Every frame's own .pdata entry was used (no leaf guessing except where flagged), and the
walk terminates in UnityPlayer's scheduler, so this is a call chain rather than a stack
picture.
Run 25b: the ObjectModel A/B (server-side, reversible)
Both crashes are inside Unity.Entities code reached from a staggered update, and the dorm
has never crashed. ObjectModel_EnableForRoom_2022_07_01 is exactly the split test that
gates Rec Room's ECS-backed object model for rooms, and this server shipped it as
Allowed, so the 2023 client turns that model on whenever it enters a room. It is now
Disallowed:
Server/RecNet/Config/SplitTestConfigs.cs-HardOverrides(fresh-DB seed),Server/DB/DBConfig.cs-PlaceholderConfig(same value, same reason),- the live
GameConfigrowsplitTestHardOverrides, updated in place withREPLACE(...,'ObjectModel_EnableForRoom_2022_07_01=Allowed','...=Disallowed').
The third one is the only edit that actually reaches this client: UpsertGameConfigValue
and EnsureSplitTestHardOverrides only ever add what a row is missing, and
RepairSplitTestConfigs documents the stored row as admin-managed ("never clobber a value
that is already set"), so the DB row is the source of truth. Pinning the key in
GameConfigAPI.Local2023Overrides was tried first and reverted: it contradicts that design
and RecNetShapeConformanceTests.GameConfigs_SuppressInvalid2023SplitTestConfigs, which
asserts the DB row is filtered and passed through rather than replaced.
Verified through the endpoint the client actually calls:
GET /api/gameconfigs/v1/all -> ObjectModel_EnableForRoom_2022_07_01=Disallowed
Server restarted (PID 11884, 3030/3031), build clean, SplitTest/GameConfigs_ tests 6/6.
Also eliminated while checking: room save data. Every subroom's DataBlob is empty - the
working dorm room 25 included - and SubRoomHistory is empty, so "RecCenter's data is
missing" is not a difference between a room that survives and one that does not. The client
never requests a room blob at all (GET /cdn/room/ is only the CDN index, {"status":"ok"},
22 b). The frames the map cannot name are generic instantiations: IL2CPP gives each one its
own code but they share a single metadata entry point, so il2cpp_class_get_methods never
sees them. Naming those needs the runtime instantiation table, not this map.
Run 25c: crash 3's own fault header, and the prop-binding difference
Crash 3's fault: block had not been printed before (an earlier sed filter cut it off). It is
the same fault as crash 2, one instruction shape away:
fault: ACCESS_VIOLATION (0xc0000005) at 0x7ffc18cc728d
read of address 0xffffffffffffffff
in GameAssembly.dll+0x470728d (matches the WER fault offset)
rsp 0xc42b34ea20 rbp 0x261c24784a0
resolved (crash 3, exact names only):
Unity.Entities.EntityComponentStore::GetComponentArraySize region <- crash, +0x140 = UNNAMED
(unnamed generic instantiations)
IHDIFFAKHOM::JEONIAELMNM
(unnamed)
EHEHGHCKFII::System.Collections.IEnumerable.GetEnumerator
(unnamed) -> Tool::CCFENLNCIOG -> Tool::ResetToDefaultCollisionLayer
-> Tool::LPAOCFPMJFD -> Tool::RpcOnReset -> Tool::AuthorityResetToDefault
-> (unnamed) -> ToolCleanup::BFFGEGKNGMM -> ToolCleanup::CJGILKPKDOL
RecRoom.Core.Updates.StaggeredUpdateManager::DGHHCHACHOM
-> (unnamed) -> (unnamed) -> UnityPlayer.dll scheduler
A read at 0xffffffffffffffff is an index that came out as -1: some TypeIndex/component
lookup in the tool's component set has no answer, the code indexes with it anyway, and the
base pointer lands one byte past the end of the address space.
The /similar lead is dead. The route does exist - Server/RoomServer/V2_RoomsAPI.cs
([Route("rooms/rooms/")], GET {id:long}/similar, our own local addition) - and it did
answer 200 len=10166 for room 3, followed 2.5 s later by
Error refreshing rooms from source RoomSimilarity: An error occurred. But run 24 (crash 2)
never requested /similar and never logged that error, and it died with the identical chain,
12 s after the same room reported COMPLETE. Two runs, same crash, one with the carousel and
one without: the carousel is a real but separate bug (our payload fails the client's
RoomSimilarity parse), not the cause of the AV.
Timing is deterministic. Room load COMPLETE -> crash:
| run | COMPLETE | crash | delta | props spawned | props -> crash |
|---|---|---|---|---|---|
| 24 (21:28) | 32.50 s | 21:28:33 | ~12 s | 28.41 s | ~16.6 s |
| 25 (21:44) | 100.50 s | 21:44:55 | ~13 s | 96.31 s | ~16.7 s |
So it is a timer that fires ~12 s after the room finishes loading, not something the player does: in run 24 the client sat in the room on heartbeats, in run 25 the watch menu was open.
The one thing that actually differs between the room that crashes and the room that does not is prop binding. RecCenter logs, once, at scene-setup time:
[RecCenter_Chair] / [Chair_White_Stationary] / [RecCenter_Couch] (N)
has an RigidbodyEx but no EmbodiedObject/RbexEntity/TransformEntity component
Exactly 35 in both crashing runs; zero anywhere in the dorm (which shares the same log).
Those objects exist in the Unity scene with a RigidbodyEx and are missing the ECS
companions the ObjectModel needs. Tool::ResetToDefaultCollisionLayer iterates exactly that
state - a tool's collision-layer component set - which is why it is the code that dies.
Run 25b's A/B aimed at the wrong key. ObjectModel_EnableForRoom_2022_07_01 never
reaches the client's ObjectModel path - it does not appear in Player.log at all. The gate
the client actually reads is the GameConfig family, served from the DB and logged at boot:
GameConfig 'UGC.ObjectModel.Allowed' = True (DB row: UGC.ObjectModel.Allowed = true)
GameConfig 'UGC.ObjectModel.AllowedInRoomV1' = True
GameConfig 'UGC.ObjectModel.ScalabilityAllowed' = true, .R2Allowed = true, .DormAllowed = false
and the load log shows the path being taken, for the dorm as well as for RecCenter:
[DATA_DOWNLOADED] Fetched Deserialization Data...
START: UnloadSceneToEnableObjectModelTask
Checking for Object Model Enabled flag...
[Moving to phase INIT_OBJECTMODEL...] -> [MoveToPhase(INIT_OBJECTMODEL)]
Disallowed therefore proved nothing about ObjectModel: the model ran anyway. Note the
seeded UGC.ObjectModel.* values (incl. DormAllowed = false) are the reference's, so
ObjectModel-on-for-rooms is the authentic August-2023 configuration, not drift of ours.
Where the room data should have come from. The load's [Downloading Data] report for
RecCenter:
[AssetBundle_Download] 0.01ms | [SubRoom_Download] 0.01ms
[SuperRoom_Download] 0.00ms | [PlayerSave_Download] 4.00ms
SubRoom_Download is a no-op, consistent with every subroom DataBlob being empty. If the
prop -> ECS bindings normally arrive with the sub-room save, an empty save is precisely what
leaves scene props with a RigidbodyEx and no EmbodiedObject/RbexEntity/TransformEntity.
Open, in order of expected value:
- Serve a real RecCenter sub-room save (or otherwise complete the prop bindings) and see whether both the 35 bind errors and the AV disappear - this tests the cause directly.
- A/B
UGC.ObjectModel.Allowed=false(+AllowedInRoomV1) in the DB row: cheap, reversible, and it removes the whole ECS object path. Divergence from the reference, so treat it as a probe, not a fix. - Stopgap only: make
Tool::ResetToDefaultCollisionLayer(0x1201e32..0x1201e90) a no-op from the client plugin. Removes the crash without fixing the corrupt entity, so the tool simply never gets its collision layer restored. - Separate bug: fix our
/rooms/rooms/{id}/similarpayload shape so the RRUI carousel stops failing withError refreshing rooms from source RoomSimilarity.
Run 27 (2026-09-24 22:22, tool guard deployed): no crash - and the join NRE named
Player.log 22:22:34 -> 22:36:23 (~832 s, no crash), LogOutput.log carries 9 [ToolGuard]
skips, all of them game-role nodes and shape containers ([GameRoleNode]_INJHAOJMPFF_*,
[StandaloneShapeContainer](Clone)) rather than the RecCenter chairs, and zero
has an RigidbodyEx but no EmbodiedObject/RbexEntity/TransformEntity component lines - every
run that dies has exactly 35. Three joins:
| time | target | result |
|---|---|---|
| 22:22:42 | dorm 25/32, instance 20886 | COMPLETE |
| 22:29:51 | room 54/57, instance 20887 | COMPLETE |
| 22:30:17 | dorm 25/32, instance 20888 | FailedToSpawnPlayer (8718) |
The third is the room bug the guard did not cover: the load walks every phase to
SCENE_LOADED, then dies inside SETUP_ROOM_OBJECTS.
[471.4358] Starting RunFor.All handler 729481d6-1503-47c8-b401-f98f736a13f2 (1 handlers)
[471.4486] [Error | RoomOperation] Error during operation InitialRoomLoad-a820aca0-...:
Object reference not set to an instance of an object.
[471.4830] Reporting result FailedToSpawnPlayer / (Code 8718) to RecNet Matchmaking ...
Stack (managed frame bottom of the throw):
EMKMMPEIGLD.FACCFIGFBIP(IEnumerable<T>)
EMKMMPEIGLD.HLLNBBPLCAH(Tuple<T1,T2,T3>)
KOGBHMMNAJA1.EIAKBDPCFOD(Action1) <- broadcaster
EMKMMPEIGLD.KOIPGJONEBE(LPPEEGCNOPD, CancellationToken)
IGOEMHGCPBC2+DGLCCMDLGNI.Invoke(request, ct)
AGCMEKDMOEG+PAIKPNCMAPO.<<InvokeMethodsInParallel>b__2>d.MoveNext()
AGCMEKDMOEG.FFAFPFAGKDH(Guid, List, IEJHLHGIEFP, LPPEEGCNOPD, CancellationToken)
<<LoadRoomLocal>b__0>d.MoveNext()
Two frames above the NRE:
[22:30:15] Operation SaveRoom sent. Waiting...
[22:30:15] Unobserved exception throw ... BDADBLBKLJN.OABJEHOGBIM(GKENAAINMOP)
[22:30:17] Error during operation RoomSave-6ad9514c-...: Object reference not set to an
instance of an object.
Both sites resolve by name without a crash dump: grep -la EMKMMPEIGLD BepInEx/interop/*.dll answers RecRoom.Keepsakes.Runtime.dll, and BDADBLBKLJN is
RecNet.Runtime.dll. So the handler that faults a room load is the keepsake runtime's:
- It runs in
SETUP_ROOM_OBJECTS, again inRUN_MIGRATIONS, again inPOST_DESTROY_OLD_OBJECTS- one729481d6per keepsake pass - and the identical NRE is thrown on the loads that survive (22:22:49.5, 22:29:59.0) as an unobserved task exception. Chronic, and fatal only when it lands inside the phase run. - Its fetches line up exactly:
GET /api/keepsakes/rooms/25+categories+globalconfigat 22:22:48.988,GET /api/keepsakes/rooms/54at 22:29:58.388. The failed load at 22:30 makes no keepsake request at all - it dies before/inside that pass.
What we were serving, and what the reference serves (apps/api/src/routes/gameplay.ts):
| route | ours (run 27) | reference |
|---|---|---|
GET /api/keepsakes/rooms/{id} |
200 {FIOFPPHNMHM:[],EBKHAGMFFEK:[],OFNJMGKKKJP:[]} (69 b) |
204, no body - "the client treats a body here as data" |
GET /api/keepsakes/categories |
counted set with one fabricated keepsake (IIILIMOMKDK...) |
{Results:[],TotalResults:0} |
GET /api/keepsakes/globalconfig |
{NPCHGMEHAPA:true,NEFKKONMGJI:100,PJMJGIKKGJB:true} |
{KeepsakeFeatureEnabled:true,KeepsakeRoomLimit:10,SocialXpBoostEnabled:false} |
The reference's note on categories is the same failure mode one layer down: the client parses
it as an object and "an array fails it outright - expected:'{', actual:'[', at offset:0,
logged as 'Failed to get keepsake categories' - which takes the keepsake load down with
it". Fabricated entries with guessed obfuscated field names are that same class of input: a
keepsake the client resolves that exists nowhere.
Fix (server, this run). Server/API/Keepsakes/KeepsakesAPI.cs now answers exactly what the
reference does - 204 with no body for a room's keepsakes, an empty counted catalog, and the
client's own field names on the config. Tests pin all three shapes
(StartupApiCompatibilityTests: empty catalog, NoContentResult, config names).
Fix (server, same session). The save that preceded the failure also faulted, and the
session's only subroom save (POST /rooms/rooms/54/subrooms/57/data, 200, 5718 b) is the call
it followed. The reference is explicit that this response is the one envelope the client reads
differently:
valueis{ room, subRoomDataSave }, anderroris NULL rather than"". ThesubRoomDataSaveis camelCase with a different field set from the PascalCaseCurrentSaveembedded in the room ... Don't unify the two projections.
We were serving the room's CurrentSave there - PascalCase, with none of unityAsset,
unityAssetHash or the echoed dataBlobHash. RoomUtils.BuildSaveResponse now serves the
documented projection (and dataBlobHash is read off the save body and echoed), while the
room's own CurrentSave is untouched. The other documented trap in this path is fixed too: a
subroom carries CreatorAccountId, null until its first save and the saver's id after - "the
client NREs on a null one" - which we did not send at all.
Tests: 560 pass, including SaveAcceptsTheJsonEnvelopeThe2023ClientPosts (now asserting the
camelCase save + a non-null subroom creator), SaveResponseEchoesTheBlobHashItWasGiven, and
AnUnsavedSubRoomReportsNoCreator.
Server relaunched (development environment, cwd Server/, server-console-run28.log), and the
new shapes verified live: rooms/25 -> 204, categories -> {Results:[],TotalResults:0},
globalconfig -> KeepsakeFeatureEnabled/RoomLimit 10/SocialXpBoost.
How to read the next run. The join is fixed if 729481d6 completes on the dorm load and the
post-load keepsake pass stops throwing:
grep -c "Unobserved exception" Player.log
grep -n "729481d6" Player.log
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:3030/api/keepsakes/rooms/25 # 204
Still open: the missing SignalR UnsubscribeFrom the client calls on every transition
(autoloc.job, HubException: Method does not exist), and real keepsake storage - the emptied
catalog means no keepsake can be placed until it exists.
Run 28 (2026-09-24 23:25, live): the watch page's missing buttons are two missing routes
The watch's "This Room" tab makes exactly three requests as it opens, and the middle one was the only failure:
[23:27:18.014] GET /rooms/rooms/25/interactionby/me -> 200 len=47
[23:27:18.031] GET /rooms/rooms/25/experience -> 404 len=0
[23:27:18.097] GET /images/default.png?width=1024 -> 200 len=80001
The tab renders with SETTINGS alone - no LOAD, no SAVE ROOM, no ROOM STATS, no subroom (room
list) button. The client logs each failure as a crash-report line
(OBHEMJEJNMK: HTTP Error 404 through GFIBIELMKBM+<>c__271[TResponse].NOFENLOLAIH), 10 of them in the session against 12 rooms/rooms/25/experience` 404s, and it keeps re-asking rather
than settling.
Not permissions: the room payload already reports the viewer's role
(Roles: [{"AccountId":2,"Role":255,...}], 255 = Creator) and CreatorAccountId: 2 - which
is also how the subroom CreatorAccountId added in run 27 shows up on the wire.
Neither rooms/{id}/experience nor rooms/{id}/experience/player existed on our server at all.
The reference's note on the player half is the cost of that: nothing tracks per-room progress, so
it serves an empty list - "which the client reads as 'no progress in this room', where a 404
would stall the room load" - and the settings half is a bare two-key object, Enabled: false
with DailyLimit: 1000, "the cap that WOULD apply ... the client reads both keys whatever
Enabled says". Both added to V2_RoomsAPI (GetRoomExperience, GetRoomExperienceForPlayer).
The other 404 on the same page is econ/econ/roomGiftDropShops/room/{id}, asked for on every
run (run 16 through run 28, 1-5 times each) and never once served. The reference answers an empty
list - "Empty stub so the client doesn't 404" - so RoomGiftDropShopAPI now does. It is the only
room-economy route this build asks for: roomInventory, roomOffer, roomEconConfig and
ugcPurchasables are never requested (checked across all six server logs), so no stubs for them
were added.
Tests: 563 pass, adding RoomExperienceAnswersTheTwoKeysTheClientReads,
RoomExperienceForPlayerAnswersAnEmptyList and RoomGiftDropShopsAnswerAnEmptyList. Server
relaunched (server-console-run29.log, cwd Server/), and every route verified live:
rooms/rooms/25/experience -> 200 {"Enabled":false,"DailyLimit":1000}
rooms/rooms/25/experience/player -> 200 []
econ/econ/roomGiftDropShops/room/25 -> 200 []
api/keepsakes/rooms/25 -> 204 (empty)
Run 29 (2026-09-24 23:50, live): MakeItToMidnight imported, and what the create route dropped
C:\RecRoom\Downloads\MakeItToMidnight is a rec.net archive of the August 2023 RRO
MakeItToMidnight: room_data.json (the room, its one subroom "Home", and 100 revisions of that
subroom's save), blobs/*.room (137 MB, one scene blob per revision) and image/*.png (the
thumbnail). There is no Unity asset bundle in it and none is needed - a room's objects and prefabs
are shipped in the game build and a save only names them - so the subroom is imported with an empty
UnityAssetId.
Two things decide which revisions may be imported, and both belong to the client that has to open them:
PersistenceVersion. The revisions from this client's own build week (2023-08-21 through 08-28) are all 61 - one v63 on 08-21 aside - which is the schema the build writes; the 67 later ones climb 63, 64, 66, 68, 70, 72, 77, 87, 88, 103, 128 and 138 (the last from 2024-07) and a newer save than the client can read is a room that opens broken. The taco importer's gate was "38 or lower" because its archive was 2020-2022: the number is a property of the archive's era, not a constant.CreatedAt, as the second net. 33 revisions pass (2023-08-21T20:20 through 2023-08-28T21:00, 44 970 347 bytes).
The newest of those is the LIVE save and every other one is posted through the game's own save route, so each lands in the subroom's history exactly as an ordinary save does - and each POST pushes the revision it replaces into that history, which is what makes the whole list restorable. The room is created on the OLDEST revision and walked forward, because whichever revision is posted LAST is the one that stays live: creating it on the newest and then posting the older ones leaves the room serving the second-newest save, which is what the first attempt did.
tools/import_room_dump.mjs runs it in resumable phases (plan|upload|register|verify|redate),
recording what it uploaded, created and posted in artifacts/room-dump-import/<room>.json. Blobs
keep the archive's own names (5k1vmec9n5jf4dymy0zkjligj.room), so a save in CDN storage stays
traceable to the dump it came from, and the redate phase re-stamps the revisions with the dates
the archive recorded - the save route stamps everything it records with the moment of the POST, so
the restore list would otherwise read "just now" 33 times and be useless to pick from.
Result: room 55 / subroom 58, owner 2, Public, MaxPlayers 5, WarningMask Scary (1), tagged
import, thumbnail 798fe98e05b944999ab8c76181f45367.png, scene
a75f7547-79eb-47c6-8986-6767abcb4f92 (Maker Room), and 33 saves carrying the archive's dates.
What the import exposed in admin/rooms/create/room
The first create answered Created as 55! and the row was wrong: Accessibility 0 (Private),
WarningMask 0, Tags empty. The route reads its body through JsonExtension.Get<T>, which could
only read scalars:
- An enum is not a type
JsonValue.GetValue<T>()knows, so it threw and the caller took its fallback - every room created through this route came out Private with no warning mask, and nothing in the response said a field had been dropped. - A collection never got that far:
GetValue<object>()on a JsonArray throws before the scalar read, so the tag list fell back to empty. The taco importer'sTagsnever landed either. - The bool path ("1" and "true" both mean true) was already there, but read its node through
GetValue<object>(), which hands back aJsonElementfor a body parsed from JSON rather than anintor astring- the check could not see either shape.
Get<T> now reads the node's own JsonValueKind, converts an enum from its number or its name,
deserializes a JsonArray/JsonObject whole, and keeps the bool spellings. Room 55's three fields were
corrected directly in SQL, because the running server could not be rebuilt mid-session: the live
client holds Server/bin/Debug/net9.0/Server.dll, so the verification build goes to
.external/verify-build/bin/ (-p:BaseOutputPath=...) and the apphost exe is skipped entirely.
A second, separate bug in the same block: loadScreen is NOT NULL with a CLR default of [], and
a payload without LoadScreens wrote null over that default - the insert failed with a
required-property error, a 500 for any import that did not happen to send an empty list.
Tests: 569 pass, adding AdminRoomCreateTests - enum binding by number and by name, the bool
spellings, the tag list with its tag types, the defaults an absent field still gets (including
loadScreen), the subroom's scene and blob, and the Maker Room fallback for a scene the client does
not know.
Verified in the client
The live session had already found the room by the time the import finished:
[23:57:23] GET /images/798fe98e05b944999ab8c76181f45367.png -> 200
[23:57:25] GET /rooms/rooms/55?include=5421&unityAssetTarget=0&unityAssetVersion=1 -> 200
[23:57:35] GET /cdn/room/5k1vmec9n5jf4dymy0zkjligj.room -> 200 len=1373747
Player joined ^MakeItToMidnight in scene "Maker Room" with playerCount 1
[RoomLoad.RoomManager] -- Initial Room Load '^MakeItToMidnight' (55.58) [instance 20907] --
[23:57:49] Moved to phase COMPLETE
The load walks every phase through SETUP_ROOM_OBJECTS, RUN_MIGRATIONS and
DestroyAndRebuildObjects, with 0 occurrences of the
no EmbodiedObject/RbexEntity/TransformEntity component prop-bind error that ran at 35 for the
RecCenter, and 0 FailedToSpawnPlayer. The only errors after the join are the room's own circuits
(CircuitsV2Errors, "Studio Object is invalid!") alongside the guard's 25 ToolGuard skips.
One note for the next restart: the fixed Server.dll is not in Server/bin/Debug/net9.0/ yet,
because the running server holds it. A plain dotnet build ServerTests/Server.Tests.csproj -c Debug
once the client closes picks it up, and nothing about room 55 depends on it.
Run 30 (2026-09-25 00:00, live): the portfolio "crash" is the anti-cheat's dorm drop
Clicking your own portfolio closed the game. It is not a crash: Windows recorded no
RecRoom.exe fault either side of it (the last one is 22:09:19, the pre-ToolGuard
GameAssembly.dll+0x470728d), and the process ran the game's own shutdown instead -
[EAC] Tearing down and releasing runtime, then Trying to send photon room updates to server while application is quitting. What Player.log has at 00:01:05 is the answer:
[00:01:05] [Warning] [Default | Warning] Dropping to dormroom due to error [385120]
[00:01:06] [Log] [Unity] [EAC] Tearing down and releasing runtime
and BepInEx/LogOutput.log has the chain, one millisecond wide:
[CheatTrace] +244681ms CheatManager.OnObscuredTypeCheatDetected
[CheatTrace] +244682ms CheatManager.EFLCHFDDPJP
[HileGate] anti-cheat report captured (full wire form): body: Message=&Type=0
[Net] Post https://localhost:3030/api/PlayerReporting/v1/hile -> 200 body=true
[QuitVeto] wantsToQuit answered false (block #1) - this is the anti-cheat shutdown
CheatManager.EFLCHFDDPJP(string reason = "Dropping to dormroom due to error [385120]") is the
whole of the message, and that string is the PARAMETER'S DEFAULT VALUE - 385120 is not an error
code, it names nothing, which is why every drop in every session reports the same number. Its
caller is the obscured-type handler, and what the page does to trip it is in the server log: the
profile renders the caller's own custom avatar items, and
GET /api/customAvatarItems/v2/fromCreator/2 is fetched 7 times in 10 ms immediately before the
detection (00:01:03.921-00:01:05.699).
The server side of that page is faithful and stays as it is: /accounts/account/2/bio 200,
/api/customAvatarItems/v2/fromCreator/2 200 with the reference's empty page,
/accounts/accountprivacysettings/2 200, /rooms/rooms/ownedby/2 200, and 404 with an empty
body on both influencer routes - which is the reference's own answer for "not an influencer",
not a gap.
So the verdict is removed at its source, exactly as the file scan was in run 23d, and the result
is already deployed (BepInEx/plugins/RadicadeAugustClient.dll, 62464 b, md5
8718a45dd42d7cb72b3e46e04b744d91, backup .before-run30b-cheatdrop):
CheatDetectionSuppressPatch- skips all three verdict handlers (OnObscuredTypeCheatDetected,OnTimeCheatDetected,OnAdvancedMovementCheatDetected), not just the one that fired. Skipping only the obscured one would leave the other two live for whatever a clock-skewed or laggy machine trips next. A handler that does not resolve is logged and skipped rather than thrown on - the throw would take the whole patch block (exit veto included) down with it - and the startup line printsverdict handlers guarded=3(or names the unguarded one).RADICADE_CHEAT_TYPE=0restores the detections.CheatDormDropGuardPatch- holds the drop itself back, narrowly: only when the reason contains385120, the anti-cheat's own default. Any other reason still drops the player, so a genuine room failure behaves as before.RADICADE_CHEAT_DROP=0lets every drop run.
Both are registered in their own try/catch, so a resolution failure on a different build cannot undo the exit veto, the exit tracer or the hile gate applied above them.
Run 30, second half: rooms/rooms/{id}/similar answered a bare array
The "Similar Rooms" rail is a paged DTO in this client, and the route answered
Array.Empty<object>() - Malformed Response: '[]' in Player.log, one of 9 in the session
(and 5 in the one before). A room that did not exist was a 404 on top of that, which stalls the
screen the rail sits on rather than emptying one row. GetSimilarRooms now returns
RoomSearchResult (scored by shared tags, same creator as the fallback for untagged rooms,
visibility checked per candidate, unknown room = empty page) and pages skip/take.
ServerTests/SimilarRoomsTests.cs pins the envelope, the empty page, the paging and the
visibility rule. 573 tests pass.
/econ/econ/roomGiftDropShops/room/{id} ([]), /rooms/rooms/{id}/experience/player ([]) and
/econ/api/roomkeys/v1/room ([]) were each checked against the reference while chasing the
remaining Malformed Response: '[]' lines: all three are documented there as empty lists, so the
chronic parse errors are not from them and are left alone for now - they are logged, not fatal.
The restart, and the one trap in it
The server was rebuilt and relaunched for the similar fix (server-console-run30.log, cwd
Server, PID 33272). Launching the exe directly from Server/ picks up appsettings.Production.json
Hosting environment: Production, a differentJwtKey, no HTTPS listener on 3030 and no Photon app ids, which would break the client at the room join. The launch must carry the environment:
$env:ASPNETCORE_ENVIRONMENT='Development'
Start-Process ...\Server\bin\Debug\net9.0\Server.exe -WorkingDirectory ...\radicade-server\Server
and the log has to read Now listening on: https://localhost:3030 and
Now listening on: http://127.0.0.1:3031 and Hosting environment: Development. Live
after the restart:
rooms/rooms/55/similar -> 200 {"TotalResults":0,"Results":[]}
rooms/rooms/99999/similar -> 200 {"TotalResults":0,"Results":[]} (was 404)
rooms/rooms/ownedby/2 -> 200 len=5950
api/customAvatarItems/v2/fromCreator/2 -> 200 {"Results":[],"TotalResults":0}
(Authorization: Bearer needs a token signed with the Development JwtKey; ?access_token= is
only honoured on notify-hub paths, which is why a copied client token reads as 401 here.)
Run 31: the room's Unity asset bundles, uploaded and bound
The August build's MakeItToMidnight import had no UnityAssetId because the room dump
(C:\RecRoom\Downloads\MakeItToMidnight) carries no bundle - the importer's note is right, and
tools/import_room_dump.mjs says the same thing: the save only names the content and the bytes
live elsewhere. The bytes turned up separately, in C:\RecRoom\Downloads\AssetBundles: 87
*.assetbundle files plus one 546 KB .room, no manifest, no index.
What the archive actually is
The files are named by their CDN blob name and carry the CDN's own timestamps, and those two facts
recover the whole structure. Group by upload time and each group is one publish event: Rec Room
uploads one bundle per target, seconds apart, and every bundle in a group carries the same
container path, which names the room it was built for.
2023-08-21 13:10 n=7 333.3 MB Assets/RoomSource/Assets/Rooms/MakeItToMidnight/MakeItToMidnight-Home.unity
2023-08-22 10:34 n=7 333.4 MB ...
2023-08-23 12:48 n=7 333.3 MB ...
2023-09-12 18:49 n=6 308.2 MB ...
Two rooms are in there (MakeItToMidnight, 12 batches, and KOdByDawn, 3), across Unity revisions
2020.3.33f1 (the August 2023 build - the only one this client can load), 2021.3.29f1 and
6000.0.27f1. SerializedFile.target_platform separates the targets inside a batch, and the
texture formats confirm what the numbers mean: 9 and 13 carry ASTC (iOS and Android - twice, a
phone and a Quest build both report Android), while 5, 31, 43 and 44 carry BC/DXT
(StandaloneWindows and the consoles). Each publish costs ~333 MB across targets; 5 alone is
~85 MB.
The bundle is not optional decoration. The 08-23 Android bundle alone holds 8080 objects - 1490
GameObjects, 1366 meshes, 1255 MonoBehaviours, 291 textures, 171 AudioClips - and the names in it
(Survivor 1, [PerkIcon], PhotoBooth_Curtain*_LOD*) are the names the room save builds its
object tree from. A subroom with an empty UnityAssetId loads the save against whatever the build
itself ships, which is exactly why the room opened as a Maker Room with its content missing.
Which bundle belongs to the live save
The live revision is the newest v61 save, 5k1vmec9n5jf4dymy0zkjligj.room (2023-08-28T21:00Z), and
the dump records the asset id it advertised: 32cafb11-c51b-485c-8f1f-73b73159feff. The dump is the
only place that id appears, so the batch it names was pinned by time, and the pinning is exact once
the archive's clock is accounted for - its zip entry times are the CDN's local wall clock,
exactly 7 h behind the UTC CreatedAt the saves carry:
unityAssetId first saved publish batch delta
bd362d85-6cc5-4b6f-9d3b-33774e3938aa 2023-08-21T19:59:45 2023-08-21 13:10 +11 m
14c842e7-01f0-4b81-9d03-19f39b7393d0 2023-08-21T20:20:28 2023-08-21 13:10 -10 m
20e15a8f-faca-41dc-8025-dd7da5c3cd0c 2023-08-22T17:25:32 2023-08-22 10:34 +9 m
3fb814c1-271e-4e54-960e-d6298947baaa 2023-08-22T17:27:21 2023-08-22 10:34 +7 m
32cafb11-c51b-485c-8f1f-73b73159feff 2023-08-23T19:39:06 2023-08-23 12:48 +10 m
So 32cafb11 is the 2023-08-23 publish, and in that batch the desktop bundle is
d7ftni8391e3nrdts2drs4qlb.assetbundle - StandaloneWindows, 2020.3.33f1 (the client's own
revision), 85,232,592 bytes, md5 31b4faa66491de6106ce748a457f9905. Two saves share that id
because the later revisions (2023-08-24..28) changed the room without republishing content, so one
bundle serves them all - which is also why the earlier ids matter only for history restores.
Uploaded and bound
The client resolves a bundle in two steps - GET /rooms/unity_assets/{assetId}/{target}/{version}
returns {UnityAssetId, Target, Version, Filename}, and the bytes come from /data/{Filename} -
and UnityAssetsCDN.ResolveFilename looks for a file named exactly the id (or the id plus
.assetbundle) in the CDN storage directory. So the file is stored under the id the save
advertises, and the room's subroom points at that id:
Server/RadicadeServer-Data/CDN/Storage/32cafb11-c51b-485c-8f1f-73b73159feff.assetbundle 85232592 b
Server/RadicadeServer-Data/CDN/Storage/32cafb11-c51b-485c-8f1f-73b73159feff hard link
UPDATE SubRooms SET UnityAssetId='32cafb11-c51b-485c-8f1f-73b73159feff' WHERE SubRoomId=58;
Both spellings are stored because the loader reaches for the bytes two ways: the metadata route
hands the client a Filename that it fetches from /data/{Filename}, and the CDN path for a
bundle is /unity/{id} - the id with no extension. ResolveFilename checks the bare id first, so
the extensionless name is what the metadata reports; the hard link costs no disk and covers the
route that asks by id.
Verified against the running server (cwd Server, PID 33272 - the storage directory the server
reads is Server/RadicadeServer-Data/CDN/Storage, not the one at the repo root):
GET /rooms/unity_assets/32cafb11-.../0/1 -> 200 {"UnityAssetId":"32cafb11-...","Target":0,
"Version":1,"Filename":"32cafb11-..."}
GET /data/32cafb11-... -> 200 len=85232592 Accept-Ranges: bytes
GET /data/32cafb11-... (Range) -> 206 Content-Range: bytes 0-15/85232592
GET /rooms/rooms/55?include=5421&unityAssetTarget=0&unityAssetVersion=1
-> CurrentSave.unityAssetId = 32cafb11-c51b-485c-8f1f-73b73159feff
The unityAssetTarget=0 the client sends is RecNet's own target list, not the SerializedFile
platform id the bundle carries: the server answers the id with one file, so the target in the query
does not have to be matched to the target in the bundle.
The tool, so the next room is one command
tools/import_room_bundles.py does the three steps above and is what produced the tables in this
section:
python tools/import_room_bundles.py inventory --source ".../AssetBundles" --zip ".../AssetBundles.zip"
python tools/import_room_bundles.py plan --source ... --zip ... --dump ".../MakeItToMidnight/room_data.json" \
--room MakeItToMidnight
python tools/import_room_bundles.py bind --asset-id 32cafb11-... --batch 2023-08-23 \
--source ... --zip ... --subroom 58 --apply
inventory probes every bundle (platform, room container, Unity revision, object count) and groups
by publish batch - 2m37s for all 87, written to artifacts/asset-bundles.json. plan pairs a room
dump's asset ids with batches using the clock alignment above, and flags a pairing an hour or more
out as DISTANT rather than pretending to know it. bind copies the batch's StandaloneWindows
bundle into CDN storage under the id, points the subroom at it, and then runs the client's own
two-step resolution against the live server to prove it answers. The dump's own file name is
deliberately not the stored name - the id is what the client asks for - so the inventory JSON is the
record of which real blob each id was bound to.
KOdByDawn is in the same archive with three batches and is bound the same way if it is ever
imported. The remaining older ids can be bound too, for history restores that keep their own
content: 14c842e7 and bd362d85 are the 2023-08-21 batch
(a4al16vq0235oom5aplev2omu.assetbundle), and 20e15a8f / 3fb814c1 are 2023-08-22
(esnrn95jqftokte3hmy8m1ffr.assetbundle).
Run 32 - the beta tag on room 55, and a room that is developer-owned when the maker is a dev
Room 55 does not carry beta
Rooms.Tags for room 55 is import,0 - one entry, import at TagType.General. There is no
beta entry, so the room does not advertise itself as beta-enabled the way the imported Showdown
room does (rro,2|games,0|beta,1, TagType.Auto).
Two things worth keeping, because they explain the value rather than just reporting it:
- The importer wrote it.
tools/import_room_dump.mjs:228stamps every imported room with a hardcoded[{ Tag: 'import', Type: 0 }], so whatever the real room carried was overwritten at import time - this is a latent bug for every room imported that way, not a fact about MakeItToMidnight. - The archives cannot tell us what the original tags were. All 48 captured room JSONs in
rooms.zip(more rr rooms/*/room_*.json) carry no room-levelTagsfield at all - the exporter drops it - and theMakeItToMidnight/room_data.jsondump only has save-levelTags: [], which is unrelated.
Nothing server-side branches on the room-level tag: grep '"beta"' Server/**.cs finds only the
comment in Program.cs:564, and the tag is pure pass-through (FullRoom.CopyTo copies Tags
straight into the room payload) with the client acting on it. Beta creative-tools creation is
already enabled globally by EnableBetaCreativeToolsConfigs (UGC.AllowBetaInvertedTubeCreation,
UGC.AllowNonBetaInvertedTubeCreation, UGC.EnableInventionsInBetaRooms, UGC.Room2Enabled), so
the missing tag is not what keeps beta content from loading. To match the real room anyway:
UPDATE Rooms SET Tags = 'import,0|beta,1' WHERE RoomId = 55;
Left unapplied - the user only asked whether it was there.
IsDeveloperOwned was never in the room payload
The 2023 client's room parser has the field (it is in the August metadata dump, and in the captured
RecNet responses: "IsDeveloperOwned": true on a developer's room), but FullRoom did not declare
it at all, so every room on this server read as not-developer-owned and the client's Dev Rooms
row stayed empty however the rooms were made.
Where a room is made matters: there is no create-room route. A player creates a room by
POST /rooms/rooms/{id}/clone (RoomServerAPI.CloneRoom -> RoomUtils.CloneRoom), which is the
same method GetOrCreateDorm uses to make a dorm - so the flag has to come from the maker's roles
and has to exclude dorms.
What changed:
RoomData.IsDeveloperOwned(bool, default false) - a column onRooms,bit NOT NULL DEFAULT 0.Room.IsDeveloperOwned- declared on the response base, so it is emitted for room lists and room details alike, asIsDeveloperOwned(Program.cs pins the serializer to no naming policy, the same PascalCase the real API answers with).RoomUtils.CloneRoomstamps it:!room.IsDorm && await ctx.IsDeveloper(newCreator).RoomUtils.ConvertToFullandFullRoom.CopyTocarry it to the payload.AdminRoomAPI.CreateRoom(the import route) honours an explicitIsDeveloperOwnedin the body - a captured RRO room is developer-owned on the real service even when it is recreated here under a local owner - and otherwise derives it from the owner it is being created for.EnsureRoomDeveloperOwnedColumnon boot, same self-heal pattern asUnityAssetIdand the join-gift columns: EF maps the property onto everyRoomsquery, so the column has to exist before any startup step reads a room.
Verified live against the restarted server (PID 38436, server-console-run32.log, Development,
3030 + 3031). The column is in place and the payload carries the field:
GET /rooms/rooms/55?include=5421... -> IsDeveloperOwned: false
GET /rooms/rooms/23 -> IsDeveloperOwned: false
GET /rooms/rooms/25 -> IsDorm: true, IsDeveloperOwned: false
Then both directions of the rule, through the client's own clone route, with tokens minted for the
two accounts on this server (2 TestPlayer = developer,influencer; 1 Coach = influencer):
POST /rooms/rooms/23/clone (name=DevOwnedProbe, account 2) -> RoomId 56, IsDeveloperOwned: true
POST /rooms/rooms/23/clone (name=PlainOwnedProbe, account 1) -> RoomId 57, IsDeveloperOwned: false
Both probe rooms were deleted again (RoomRoles, RoomStats, SubRooms, Rooms), leaving the 51
rooms that were there before. dotnet test ServerTests/Server.Tests.csproj -> 581 passed
(573 before, +8 in ServerTests/RoomDeveloperOwnedTests.cs: a dev's clone, a plain player's clone,
a dev's dorm, the flag on the room that is read back, the explicit false in the payload, and the
three admin-import cases).
Every room that predates the column started false. The backfill was then applied, over the same rule the create path uses (the maker's role, dorms excluded) - 5 rooms flagged, 46 left false:
UPDATE Rooms SET IsDeveloperOwned = 1
WHERE IsDorm = 0 AND CreatorAccountId IN (SELECT accountId FROM Profiles WHERE role LIKE '%developer%');
31 TestPlayers_MakerRoom_3528 52 studio-room-SdjyyctVdc 53 studio-room-NRyzI1_njb
54 TestPlayers_MakerRoom_1603 55 MakeItToMidnight (25 DormRoom - dorm, left false)
Room 55 is account 2's, so MakeItToMidnight became developer-owned along with the four the user built - which is also what the real service says about that room. It is a data-only change, so the running server picked it up without a restart; the detail route and the bulk list route the client's own room lists read both answer it:
GET /rooms/rooms/55?include=5421... -> IsRRO: true, IsDeveloperOwned: true
GET /rooms/rooms/bulk?id=31..55 -> IsDeveloperOwned: true (x5)
GET /rooms/rooms/bulk?...&id=25 -> IsDorm: true, IsDeveloperOwned: false
Run 33 - Showdown reimported, and the four-character bug that was skipping it
The request was that Showdown was "a little broken", and to delete and reimport it from a fresh
export at C:\RecRoom\Downloads\Showdown. The import ran, but the interesting part is why
the previous one had not: the boot importer was looking in a directory that does not exist here.
What the export actually is
The folder is a RecNet export, not a captured room dump: room.json plus 22 per-subroom
directories, each carrying room_data.json and Saves/<saveId>/<blobId>.room. Showdown.rar
(46 MB, RAR5) holds the same tree. The export describes the real room 52815862 - CreatorAccountId 1, Accessibility 1, IsRRO true, IsDeveloperOwned true, WarningMask 0, MaxPlayers 6,
CloningAllowed false, PersistenceVersion 150, CreatedAt 2022-03-03, image
65hs5cb56byqs05lu16kr3e8r.png, and the description about Deputies and Bandits. 22 subrooms is
not 22 live subrooms: they are archive snapshots, all sharing the one scene
a75f7547-79eb-47c6-8986-6767abcb4f92, and they are named accordingly - SD_Archive_001,
SD_Archive_082322, SD_Archive_11_29_2022_STABLE, SD_Archive_13_061824, and so on. The single
subroom the server actually needs is the one every other room here has: one Home row pointing at
the scene and the save.
The root cause: room-imports vs RoomImports
EnsureShowdownRoom looked for its files in ..\room-imports\Showdown and room-imports\Showdown
- always lowercase. This checkout spells the directory
RoomImports, andTest-PathanswersFalsefor the lowercase spelling even though the tree is not case-sensitive as a whole. So the lookup found nothing, the importer printed[Showdown] Skipped import - missing files ...and returned on every single boot, and the room kept whatever an older import had left behind. A silent skip is the worst kind of green: the log line is there, but nothing above it looks like a failure.
The fix keeps the spelled-correctly path first and the old lowercase spellings after it (so a tree where they do resolve, or a deployment that still ships that name, keeps working):
var candidates = new[]
{
Path.GetFullPath(Path.Combine("..", "RoomImports", "Showdown")),
Path.GetFullPath(Path.Combine("..", "room-imports", "Showdown")),
Path.GetFullPath(Path.Combine("RoomImports", "Showdown")),
Path.GetFullPath(Path.Combine("room-imports", "Showdown")),
};
var sourceDir = candidates.FirstOrDefault(Directory.Exists) ?? candidates[0];
Which save to install, and why the newest one is the wrong one
A room here is a bundle plus a save blob, and the two have to be the same revision or the client
loads a scene it has no data for. The bundle was already identified by hash:
RoomImports/Showdown/Showdown_0_1.assetbundle is 76,719,774 bytes, md5
220891bc78b852df1c7ba8e1af0e9d1b, and matches 24 metadata entries - among them asset
58784cb3-204c-47e0-9cf3-23a76b94ae90 as Unity_2020_3_33f1, Target 0, Version 1. That asset
revision is the one the August client can load, and the save that goes with it is
8u07n4iwzt4xbqkww7zun0jam.room (2023-06-02, 1,222,161 bytes), the newest save whose revision is
still 2020.3.33f1.
The newest save in the export looks tempting and is a trap: the live Showdown subroom
(8785941127880372366) was last saved 2024-10-02 as d39glazjteu2zztjuuulribr7.room under asset
d03c4c99-4814-412b-9594-f61851d32225, and that asset exists only as 2021.3.29f1 / 6000.0.27f1 -
nothing a 2020.3.33f1 client can load. The previously installed blob was e45vok0ppp89hda6n1xrm4z16.room
(2022-10-18, the SD_Archive_11_29_2022_STABLE save), which is why the room was stale.
The blob really is Showdown
Raw string overlap between the blob and the bundle is zero, so that proves nothing either way. But
the .room blob is protobuf, and walking its wire format for printable runs pulls out the room's
own content verbatim - _SDLog_LocalPlayerSeesRedTeamValid,
_SDLog_LocalPlayerLooksForValidTeam, _Equipping gun index {0} to self,
_Not enough guns to give to self, _Tie Game!, Event Sender/Event Receiver/To Teams. That is
Showdown's team-and-guns gameplay, and the SD_ prefix on its own script log ids is the signature.
artifacts/verify_showdown_blob.py re-runs that walk against any blob.
The swap: room 32 out, room 58 in
The old room was deleted (with its RoomRoles, RoomStats, RoomInstances, RoomInteractions,
SubRoomHistory and SubRooms rows - 51 rooms became 50), the era-matched save was put in place,
and the metadata was patched to match the export:
| field | old room 32 | new room 58 |
|---|---|---|
Description |
Showdown |
the Deputies/Bandits text |
ImageName |
default.png |
65hs5cb56byqs05lu16kr3e8r.png |
MaxPlayers |
16 | 6 |
CloningAllowed |
1 | 0 |
IsDeveloperOwned |
backfilled true | true |
CreatedAt |
2026-09-09 | 2022-03-03 |
Room 58's Tags were already rro,2|games,0|beta,1 (the importer sets them) and its subroom 61 is
the one Home row: UnitySceneId a75f7547-79eb-47c6-8986-6767abcb4f92, DataBlob Showdown.room,
UnityAssetId Showdown_0_1.assetbundle, MaxPlayers 6, Accessibility 1. Nothing else in the
database still points at room 32 - every table that carries a RoomId was checked and all twelve
count zero rows for it.
It survives a restart, which is the part that could have bitten back
The importer's else branch (room already exists) re-applies IsRRO, Tags, subRoom.DataBlob,
subRoom.UnityAssetId and subRoom.UnitySceneId on every boot, so those have to be checked against
what a restart would write. They are all already correct, and the scene is the subtle one:
SceneLocations.MakerRoom is a75f7547-79eb-47c6-8986-6767abcb4f92, the Showdown scene, so the
overwrite is a no-op. The fields that a restart would not touch - MaxPlayers, Description,
ImageName, WarningMask, CloningAllowed, IsDeveloperOwned, CreatedAt - are the ones the
patch above changed, so the importer's non-idempotent half does not undo any of it.
Verified live (server PID 36096, run 33, Development)
GET /rooms/rooms/58 -> 200 Name=Showdown MaxPlayers=6 Accessibility=1
IsRRO=true IsDeveloperOwned=true IsDorm=false
CloningAllowed=false WarningMask=0
ImageName=65hs5cb56byqs05lu16kr3e8r.png
CreatedAt=2022-03-03T16:41:31
Tags=[rro,2][games,0][beta,1] SubRooms=1
GET /cdn/data/Showdown.room -> 200 1,222,161 bytes (the 2023-06-02 save)
GET /data/Showdown.room -> 200 1,222,161 bytes (range-capable alias)
GET /cdn/data/Showdown_0_1.assetbundle -> 206 range request, 100 bytes of 76,719,774
GET /rooms/unity_assets/Showdown_0_1.assetbundle/0/1 -> 200
{"UnityAssetId":"Showdown_0_1.assetbundle","Target":0,"Version":1,
"Filename":"Showdown_0_1.assetbundle"}
GET /images/img/65hs5cb56byqs05lu16kr3e8r.png -> 200 2,777,168 bytes
The room thumbnail needs no Images table row: room art is served straight from
DBConfig.CDNImages by name (/images/img/{name}), which is why the built-in rooms' ImageName
values are bare names like Dodgeball. Both HTTPS (3030, the port the client talks to) and HTTP
(3031) answer for the room and the asset metadata.
Backups kept: RoomImports/Showdown/Showdown.room.before-run33 is the previous blob, and
artifacts/showdown-room32-before-run33.txt is the row-level snapshot of the deleted room.
Handing the client patch to another machine
The August client patch is a stack of six pieces, not one DLL, and two of them are produced
by machine-local patching rather than shipped. A player who receives only
BepInEx/plugins/RadicadeAugustClient.dll gets a game that still talks to the official hosts, and
the only red line in his Player.log is the vanilla Statsig noise above - which is why that line is
the most common wrong lead. The reference install (C:\RecRoom\2023-08-29T06-53-33Z)
hashes as follows; anything that differs is the piece to replace:
| piece | expected |
|---|---|
GameAssembly.dll |
167,390,208 b, md5 11e14eb35671e2725bd0c78a6a91bb48 - the build identity; every address map and the metadata patch are tied to it |
Recroom_Data/il2cpp_data/Metadata/global-metadata.dat |
38,654,794 b, md5 cc9c32014b5fc3dd1a2a8fb12376be15, 28 x localhost:3030, 50 x rec.net |
Recroom_Data/il2cpp_data/Metadata/global-metadata.interop.dat |
38,654,794 b, md5 4a1a28078f957af470f3b31951b32648 (normalized tooling copy) |
Recroom_Data/boot.config |
173 b, md5 dc2e5f7af7d5df736a4cc3477b4b1a1f (vr-enabled=0, player-connection-mode=Listen) |
Recroom_Data/radicade-patch.json |
provenance stub: serverUrl https://localhost:3030, 7 changed literals, gameVersion 20230825 |
winhttp.dll |
26,112 b, md5 b2a4d0cb04bf8f5a27ccab237ecc2586 (Doorstop 4.3.0 proxy) |
doorstop_config.ini, .doorstop_version |
enabled = true, target BepInEx\core\BepInEx.Unity.IL2CPP.dll, version 4.3.0 |
BepInEx/core/BepInEx.Unity.IL2CPP.dll |
60,928 b, md5 9430bb84349d52b753a181788a86c5e2 - patched to read global-metadata.interop.dat; the original is kept as .before-august-interop (75,776 b, md5 b8b3e4da1dbf171f3d405102d3c9d2e2) |
BepInEx/interop/ |
335 assemblies |
BepInEx/plugins/RadicadeAugustClient.dll |
62,464 b, md5 8718a45dd42d7cb72b3e46e04b744d91 |
| Radicade server | https://localhost:3030 + http://127.0.0.1:3031, Hosting environment: Development, on the player's own machine |
The last row is the one no file copy can satisfy: the patched metadata rewrites the client's
bootstrap literals to https://localhost:3030, so the server has to run where the game runs (a
remote server needs a different metadata patch and a certificate the client accepts).
The three failure modes seen so far, each with its own fingerprint:
- No loader.
winhttp.dllrenamed towinhttp.dll.baseline-disabled(orenabled = false) -Player.logdies atCould not resolve host 'ns.rec.net', there is noBepInEx/LogOutput.logfor that boot. This is the 15:42 run-17 state. - Unpatched metadata.
global-metadata.datis the 32,039,456 b / md5825f34f1dd046a0ae764cd5a97dba5e6original. The plugin loads and the boot looks healthy (login, Photon, Dorm), then the room-save path fails:GetSubRoomDataSave()never resolves, the client printsError downloading save data!/ FailedToSpawnPlayerCode 8715and tears the session down. This is the 2026-09-24 15:16 baseline-experiment state. - No local server. Everything else is in place but nothing listens on 3030 - the client's requests fail and the screen shows the launch-validation/login failure.
tools/august-client-triage.ps1 reports all of the above in one pass (read-only; it also opens two
TCP sockets to 127.0.0.1:3030 and :3031 and writes august-client-report.txt next to itself):
powershell -ExecutionPolicy Bypass -File august-client-triage.ps1
powershell -ExecutionPolicy Bypass -File august-client-triage.ps1 -GameDir "D:\Games\2023-08-29T06-53-33Z"
On the reference install with the server stopped it reports exactly one [MISSING] (port 3030) and
one [WARN] (port 3031); with the server up, none. Every [MISSING] line names a piece of the
table above that did not arrive, and the two log tails at the end of the report carry the rest.
tools/make-august-client-pack.ps1 assembles the handover itself: it verifies every piece against
the table (fail-closed, so it cannot ship an unpatched metadata or a foreign build), copies the
loader pieces, the patched data files and the whole BepInEx tree - minus logs, previous-build
sidecars and the 18 MB radicade-addr-map.txt diagnostic - into
<OutDir>\RadicadeAugustClient, then writes README-FRIEND.txt and drops the triage script in
alongside. The reference bundle is 302 MB in 448 files, of which BepInEx/interop (335 assemblies)
is the bulk:
powershell -ExecutionPolicy Bypass -File make-august-client-pack.ps1 -OutDir "D:\radicade-pack" -Zip
For the September build (20230831, folder 2023-09-06T06-53-47Z) the same pair exists as
tools/september-client-triage.ps1 and tools/make-september-client-pack.ps1, carrying that
build's reference hashes (332 interop assemblies, 294.8 MB bundle) and one extra check the August
scripts do not need: the metadata header magic, because a stock BepInEx core fails on this game's
scrambled global-metadata.dat with Invalid or corrupt metadata (magic number check failed).
See docs/2023-0831-CLIENT-SWITCH.md.
Rec Room Studio on Radicade
C:\RecRoom\OneDrive\Documents\EggFiles\ProjectTemplate is the Rec Room Studio template, and
it lines up with the client we already patch: Unity 2020.3.33f1 (the August build's engine) and
packages built from features/rec-room-studio/releases/20230831-211740 (git 0c23610a, recorded in
RecRoomStudioPackages/VersionInfo.json). Those packages carry the entire game runtime
(com.recroom.studio, .studio.build, .studio.api, networking, circuitsv2, objectmodel, ...),
which is what lets the studio run the real game code inside the editor.
Everything routes through one string
A sweep of every com.recroom.* assembly for both ASCII and UTF-16 string heaps (plain grep
finds nothing - .NET literals live in the #US heap as UTF-16) turns up exactly one baked-in
service host:
| Where | Literal |
|---|---|
Library/PackageCache/com.recroom.studio.build@*/Editor/RecRoom.Studio.Editor.Core.dll |
https://deluxe-backend-test.oldrecroom.com/nameserver |
.../com.recroom.studio/.../UI/RecRoomStudioWindow/SignInPanel.uxml |
https://auth.rec.net/device (label text only) |
.../RecRoom.Studio.Editor.Core.dll |
https://events.statsigapi.net/v1/ (telemetry) |
No config file, environment variable or command-line switch redirects it: the editor GETs that
nameserver URL and derives every other base (Auth, API, CDN, Images, Studio, ...) from the
JSON, so aiming the whole editor at a private server is a one-string patch. The -environment
argument belongs to the batchmode pipeline (-roomId, -outputPath, -rebuildMode,
-RECNET_CLIENTSECRET, ...), not to the window.
Sign-in is an RFC 8628 device flow
RecRoom.Studio.Editor.Core.dll carries connect/deviceauthorization and connect/studiotoken
under the nameserver's Auth base, together with the matching model: device_code, user_code,
verification_uri, verification_uri_complete, expires_in, interval, scope, access_token,
refresh_token, authorization_pending, tokenError, and the grant URN
urn:ietf:params:oauth:grant-type:device_code. Nothing in the studio window works before that
succeeds - there is no offline path, and "Play Local" (which launches the game client with
-RecRoomStudioPort:N and drives it over http://localhost:{port}/recroomstudio/checkavailability
and /loadroom) stays disabled until a room is open, which itself comes from RecNet.
What the server now provides
Server/Auth/StudioAuthAPI.cs implements both endpoints off the nameserver's Auth base
({ServerUrl}/auth/connect/...). Real RecNet has the user approve the code at auth.rec.net/device;
a private server has no such page, so the code is auto-approved and the tokens are minted by
AuthAPI.GenerateToken - the same claims (userId, tver, roles, rn.ver) the game client gets,
which is what [Authorize] and token revocation actually check. The flow is refused for non-loopback
callers unless Radicade:Studio:AllowRemoteDeviceAuth is true, and the account is
Radicade:Studio:AutoApproveAccountId, else whichever account the caller names, else the lowest
account holding the developer role. Requests are read as form or JSON and logged in full (bar
secrets), because the exact wire shape is not documented anywhere - the first real sign-in attempt is
what will tell us.
Verified live against the dev server:
POST /auth/connect/deviceauthorization -> device_code, user_code 8JXV-8W4D, expires_in 900, interval 2
POST /auth/connect/studiotoken (device_code) -> access_token (JWT roles gameClient+developer+influencer, userId 2, tver 0)
GET /auth/account/me/haspassword + bearer -> true
POST /auth/connect/studiotoken replying the device_code -> invalid_grant (single use)
The editor-side patch
tools/patch-rrs-nameserver.ps1 rewrites that one literal in place, as a .NET string-heap edit: the
entry keeps its 108-byte footprint (1-byte compressed length + 106 bytes of UTF-16 + terminator), the
freed tail is zero-filled, and because the heap is addressed by offset and the footprint is unchanged
no other string moves. One-time backup to RecRoom.Studio.Editor.Core.dll.radicade-orig, idempotent,
and after writing it re-decodes the entry and asserts that nothing outside the slot changed
(-Restore puts the original back).
powershell -ExecutionPolicy Bypass -File tools/patch-rrs-nameserver.ps1 # -> https://localhost:3030/nameserver
powershell -ExecutionPolicy Bypass -File tools/patch-rrs-nameserver.ps1 -Url http://127.0.0.1:3031/nameserver
powershell -ExecutionPolicy Bypass -File tools/patch-rrs-nameserver.ps1 -Restore
Two blockers before any of this can be exercised
- The project tree is write-protected. Creating a file anywhere under
C:\RecRoom\OneDrive\Documents\EggFiles\ProjectTemplatefails (ERROR_FILE_NOT_FOUNDfor a new file,Access deniedwhen overwriting an existing one, and a hang viacmd) from Python, PowerShell and .NET, while reads work and writes elsewhere on the same volume succeed -OneDrive\itself is writable,OneDrive\Documentsis not, and noOneDriveprocess is running. Defender's Controlled Folder Access is enabled (EnableControlledFolderAccess = 1) and its implicit protected "Documents" resolves to the OneDrive one. Unity cannot open a project it cannot writeLibrary/andTemp/into, so this has to be settled first. - TLS. The editor validates certificates normally - the game client does not, because our plugin
patches that. The certificate Kestrel serves on 3030 (
localhost-rsa-aes-devcert.pfx, thumbprintD2275346BC059E9B62466F116F146F63CB89D5C4, notAfter 2031-09-05) sits inCurrentUser\My, not inCurrentUser\Root, so the editor'shttps://localhost:3030/...calls will be rejected until that cert is trusted (or the editor is aimed at the plain-HTTP listener onhttp://127.0.0.1:3031, in which caseRadicade:ServerUrlhas to match too, since the nameserver response is where every other base URL comes from).
After that the remaining surface is the studio's own room data: the window lists rooms and saves from
RecNet (RoomTargetPanel, SaveListItem) and uploads with
git -c http.extraheader="AUTHORIZATION: bearer <token>" to a RecNet-hosted remote - none of which
exists here yet (Server/API/Studio is an empty placeholder directory).
The compile wall was Defender, not the packages (2026-09-26)
The writable copy opened in Unity 2020.3.33f1 and immediately refused to compile anything:
Asset Packages/com.recroom.studio.build/Editor/RecRoom.Studio.Editor.Core.dll.radicade-orig has no
meta file, but it's in an immutable folder. The asset will be ignored.
NullReferenceException ... UnityEditor.Utils.Program.GetStandardOutput() (twice)
Assembly 'Packages/com.recroom.creation/Runtime/RecRoom.Creation.Runtime.dll' will not be loaded
due to errors: Unable to resolve reference 'Unity.Mathematics'
... the same for ~15 com.recroom.* assemblies, plus 'Unity.Collections', 'Unity.Settings.Editor',
'Unity.RenderPipelines.Core.Editor', 'RecRoom.Studio.Common.Runtime'
with Library/ScriptAssemblies holding only BuiltinAssemblies.stamp. The first guess - that the
manifest's file:../RecRoomStudioPackages/*.tgz sibling folder had been left behind by the robocopy -
was wrong, and worth recording so it is not chased again: Unity resolves a file: path in
Packages/manifest.json relative to the Packages folder, so ../RecRoomStudioPackages is
<project>\RecRoomStudioPackages. That folder is inside ProjectTemplate and always was, holding all
55 tarballs plus VersionInfo.json. An integrity scan of the whole set (tar entry list vs extracted
files, for every tarball) found zero missing files.
The actual failure is one line of Editor.log:
Starting external compile of Unity.Jobs.CodeGen.dll:
...\Library\PackageCache\com.unity.roslyn@0.2.2-preview\Compiler~\win64\csc.exe /noconfig @Temp/... /shared
Exception when running external compiler on Unity.Jobs.CodeGen.dll:
System.ComponentModel.Win32Exception (0x80004005): ApplicationName='...\win64\csc.exe', ...
Native error= The system cannot find the file specified.
Unity 2020.3 compiles C# with an external compiler, and because this project installs
com.unity.roslyn (a transitive dependency of com.unity.entities@0.51.1-preview.21) that compiler
is the package's own Roslyn 3.8. csc.exe was gone - from Library\PackageCache and from the
shared %LOCALAPPDATA%\Unity\cache\packages copy - so the first compile died, nothing was written to
ScriptAssemblies, and every later reference resolution failed against an empty assembly set. The wall
of "Unable to resolve reference" is a symptom of "the compiler never ran".
It was not shipped broken: the registry tarball
(https://cdn.packages.unity.com/tarballs/com.unity.roslyn/com.unity.roslyn-0.2.2-preview/8b5b337db92cbdec37b54235ec8b826c92c6ddef.tgz)
contains package/Compiler~/win64/csc.exe (md5 8c4361cc37113f8467feb1d22b4e83da), as it does for
mac/csc and linux/csc. Something deleted it locally.
That something is Windows Defender's PUA protection, which has been quarantining unsigned
executables out of Unity package folders on this machine. Get-MpThreatDetection lists
com.unity.burst@1.8.4\.Runtime\hostwin\llvm-lipo.exe at 2026-09-26 08:35 (ThreatID 2147972435),
the older dsymutil.exe in burst 1.8.12/1.8.17/1.8.21 in the shared cache, and the same
llvm-lipo.exe in the OneDrive tree at 00:34 - com.unity.burst\.Runtime\hostwin now contains only
vswhere.exe. Burst's bcl.exe survived and the lost tools only matter for Apple targets, so the
one casualty that blocks everything is the script compiler.
Repair: tools/repair-unity-script-compiler.ps1 restores Compiler~\win64\csc.exe and proves it
works by running the compiler's own -version probe. Sources, in order: -FromPath, else the Unity
install's Editor\Data\Tools\Roslyn\csc.exe. The launcher is interchangeable - it is a stock .NET
Core apphost that loads the csc.dll beside it, and Unity's own copy drove the package's newer
csc.dll through a clean test compile (empty output, valid 3 KB assembly). Use -FromPath with the
extracted tarball member to get the authentic binary (it differs: md5 5f0947da... for Unity's
build).
Cure (needs an elevated shell):
Add-MpPreference -ExclusionPath 'C:\RecRoom\Documents\EggFiles\ProjectTemplate'
Add-MpPreference -ExclusionPath "$env:LOCALAPPDATA\Unity\cache"
and re-run the repair with -IncludeGlobalCache so a package re-resolve cannot copy the damaged file
back out of the shared cache.
Smallest reproducer for the whole thing, without Unity: run csc.exe -version in the package's
Compiler~\win64; if the file is missing you get the Win32Exception above, and a test compile with
-noconfig -nostdlib -target:library -r:<dir>\System.Private.CoreLib.dll produces a real assembly.
Verified: the compile wall is down (batch-mode run)
The editor can be driven headlessly, which makes the whole state checkable without any clicking:
Unity.exe -batchmode -nographics -quit -projectPath <copy> -logFile <log>
(it exits 1 on a purely cosmetic Timeout after 300 seconds while waiting async operations to finish).
Before and after the csc.exe repair, on the same project copy:
| before | after | |
|---|---|---|
Starting external compile ... |
3, then Win32Exception ... cannot find the file |
205, 0 failures |
Unable to resolve reference 'Unity.Mathematics' / will not be loaded due to errors |
~15 assemblies | 0 |
Library/ScriptAssemblies/*.dll |
none, only BuiltinAssemblies.stamp |
105, incl. Unity.Mathematics, Unity.Collections, Unity.Serialization, RecRoom.Studio.Common.Runtime |
The first repair attempt also showed why this has to be repaired in both caches: csc.exe compiled
189 assemblies and then vanished mid-run, which the Defender event log does not account for
(Get-WinEvent ... /Operational ids 1116/1117/1121 only ever name burst\hostwin\llvm-lipo.exe, and
never csc/roslyn). With the shared %LOCALAPPDATA%\Unity\cache copy repaired too, the next run
kept the file and logged no failure at all - and com.unity.roslyn@0.2.2-preview's own directory
mtime stayed put, so this was not a whole-package re-extraction. Treat the project copy alone as
fragile; repair both.
Two things the log still reports, neither of them blocking:
Assets/**/*.csis empty, so there is correctly noAssembly-CSharpin the output - all of this project's code lives in packages. Do not go looking for that assembly.- The 11 remaining
error CSlines are[Script Updater]advisories fromcom.unity.serialization@1.7.0-preview.1againstUnity.Collections1.4.0 (FixedString128renamed,Runeoverloads). It still compiles, and nothing in the project references it.
Still eaten by the same Defender rule: com.unity.burst@1.8.4\.Runtime\burst-lld-14-hostwin, the
LLD the Burst AOT job links with. It is missing from the project copy and from the shared cache, so
no re-extraction can bring it back - the batch log ends with
Error while executing command: ...burst-lld-14-hostwin ... Native error: The system cannot find the file specified. hostwin\llvm-lipo.exe is quarantined on sight (twice on 2026-09-26 alone). Both are
Burst build/AOT tools: they break a player build, not the C# compilation the editor needs. Recover
them by re-downloading com.unity.burst@1.8.4 from the registry - its burst-lld-14-hostlin and
hostmac siblings are intact, which is how we know the Windows one was deleted rather than never
shipped (1.8.17 in the same cache has three hostwin entries, 1.8.4 now has one).
Burst repaired, and the studio auth path re-verified (2026-09-26)
Re-downloading com.unity.burst@1.8.4 settled both the linker and its naming: the package ships
package/.Runtime/burst-lld-14-hostwin.exe with the extension (the log prints the extensionless
name because that is what CreateProcess is given, and Windows appends .exe itself), while
1.8.17/1.8.21 carry only burst-lld-16/18-hostwin-arm64.exe. A full file-set diff of that tarball
(480 files) against the extracted package found exactly two casualties and no extras:
.Runtime/burst-lld-14-hostwin.exe and .Runtime/hostwin/dsymutil.exe (40.8 MB). Both are restored
and the package now diffs clean, as does the whole RecRoomStudioPackages set.
With the compiler, the certificate and the server all in one piece, the studio device flow was re-run
over strict TLS (the localhost dev cert is now in CurrentUser\Root, so curl without -k
succeeds):
POST /auth/connect/deviceauthorization -> device_code, user_code 9C3C-UXUF,
verification_uri https://localhost:3030/device
POST /auth/connect/studiotoken -> 400 grant_type is required (the full RFC 8628
grant_type=urn:...:device_code urn:ietf:params:oauth:grant-type:device_code is NOT
accepted)
POST /auth/connect/studiotoken grant_type=device_code&device_code=...
-> access_token (JWT roles gameClient+developer+influencer,
userId 2, tver 0, rn.ver 20230825, expires_in 604800)
GET /auth/account/me/haspassword + bearer -> true
POST studiotoken replying the device_code -> invalid_grant (single use)
grant_type=device_code is the value the handler actually branches on; a request naming any other
grant type is answered with unsupported_grant_type and logged. What is left is unchanged:
rooms/saves, and the git-over-HTTPS upload remote - Server/API/Studio is still an empty
placeholder directory.
First real Sign in: two server-side gaps (2026-09-26)
The first click of Sign in in the studio window opened the browser at
https://localhost:3030/device?user_code=EYJ4-T9DF and got a 404. The browser is not where the
authority on this is: %LOCALAPPDATA%\Unity\Editor\Editor.log records the editor's own verdicts, and
it had two of them.
1. Device authorization failed: unsupported_grant_type. The editor polls studiotoken with the
full RFC 8628 grant type - urn:ietf:params:oauth:grant-type:device_code - while the handler only
branched on the bare device_code, so it answered unsupported_grant_type and the editor abandoned
the sign-in. Matching is now on the token rather than one exact string (dashes normalised, so any
spelling is accepted), and refresh_token got the same treatment. Re-verified against the live
server: that exact request now returns an access_token, and the short form still does.
2. There was no /device page. verification_uri and verification_uri_complete were being
advertised while nothing served the path, so the browser step could only ever 404. New
Server/Auth/DeviceApprovalAPI.cs ([Route("/")] + [HttpGet("device")], the same root-level
pattern NameSever uses, and the same HTML styling as AccountsAPI.VerificationPage) renders
"Approved - account 2" for a live code, and explains an unknown or missing one. It deliberately
approves nothing: the code is already issued approved, so the page's job is to tell the user what
they are holding.
An earlier attempt had also logged
Failed to initiate device authorization: Failed to obtain RecNet Service URIs: NullReferenceException
inside KJGMKNABDIJ.EMOFOGPOOLJ(String, Char) - a parse of the nameserver response in editor code,
not a server fault. It stopped occurring once the client got past that stage, and it is worth knowing
because it looks like a server problem and is not.
Unrelated noise now identified and safe to ignore: Rec Room Studio unable to record last project path / skipped checking for update / skipped checking for local update to install, all
FileNotFoundException for ProjectSettings\Packages\com.recroom.studio\BranchId.txt, plus
EditorUpdateCheck: Failed The requested URL returned error: 404. Those only disable the studio's
self-update check.
The server now leaves a readable log
PID 448 had no stdout redirect, so the editor's actual request bodies from that first attempt are
gone. It has been restarted (PID 39184) from Server\ with ASPNETCORE_ENVIRONMENT=Development and
stdout captured to .tmp-studio-auth.log in the repo root. ReadRequestFields already logs the
method, path, client IP, content-type, user-agent and every field of every studio request, so the
next Sign in attempt is fully diagnosable:
Studio POST /auth/connect/studiotoken from ::1, content-type=application/x-www-form-urlencoded,
user-agent=curl/8.17.0: grant_type=urn:ietf:params:oauth:grant-type:device_code, device_code=..., client_id=recroom-studio
Studio token issued: account=2.
Restart pattern (the working directory matters - DBConfig resolves RadicadeServer-Data/ and
RadicadeServer-Config/ relative to it, so starting from anywhere else silently uses a different
account database):
Stop-Process -Id <pid> -Force # dotnet build cannot write bin/ while it runs
cd Server; $env:ASPNETCORE_ENVIRONMENT='Development'
Start-Process -FilePath "bin\Debug\net9.0\Server.exe" -WorkingDirectory (Get-Location) `
-RedirectStandardOutput "..\.tmp-studio-auth.log" -RedirectStandardError "..\.tmp-studio-auth.err.log"
It takes 90-150 s to start listening on https://localhost:3030 and http://127.0.0.1:3031, and the
log's Content root path line confirms which working directory actually won. (Starting it from a
script: do not capture the launcher's stdout - the server inherits the pipe and the shell blocks
forever. It starts fine, the wrapper just never returns.)
Sign-in works; the studio API surface (2026-09-26)
Sign in now completes - the studio window shows the account, and a Sign out button. What it does afterwards is visible because this server already logs every request:
[09:22:08] GET /studio/analytics/rudderstack-api-key -> 404 (x20)
[09:22:08] GET /rooms/rooms/ownedby/me -> 200
[09:22:08] GET /rooms/rooms/25?include=1037 -> 200
[09:22:13] GET /rooms/rooms/25/subrooms/32/saves?skip=0&take=30 -> 200
[09:22:13] GET /studio/cloud-builds/for-room?roomId=25&subRoomId=32&skip=0&take=30 -> 404
cloud-builds/for-room is the wall: with no route at all the panel showed
"Failed to open room - Could not download room details. 404 (Not Found)", and the editor retried it
on a loop (three HttpRequestException: 404 in the console).
The complete route list fell straight out of the same DLL that held the nameserver literal - the
client's UTF-16 #US heap keeps its route templates back to back, in
Packages/com.recroom.studio.build/Editor/RecRoom.Studio.Editor.Core.dll at offset ~296k:
cloud-builds
cloud-builds/for-room
cloud-builds/create (+ query keys autoPublish, buildBranch, cloudBuildId)
cloud-builds/custom-avatar-item/{0}
cloud-builds/custom-avatar-item/{0}/save/{1} (+ customAvatarItemSaveId)
cloud-builds/cancel
remote-run/push-to-device
collaboration/owners-in-room
account/{0}
and, in the same block, the auth client's own vocabulary - which closes the loop on the two fixes above:
connect/studiotoken, client_id=recroom.studiopipeline, client_secret=54oc5C4BH7jpqp42KhChkNZOLP0U3yTx,
grant_type=client_credentials <- a second, non-interactive login path
recroom.studio, refresh_token <- and a refresh path
connect/deviceauthorization, urn:ietf:params:oauth:grant-type:device_code, device_code,
authorization_pending <- the editor DOES send the urn form, and it does poll for
authorization_pending; both are now handled
account/me, rrs.pushtostudio, RecRoom.Studio.SessionId
Server/API/Studio/StudioAPI.cs (the folder had been an empty placeholder since the studio work
started) now answers the three read-only ones: cloud-builds/for-room and
collaboration/owners-in-room with an empty list, analytics/rudderstack-api-key with an empty key
(that route is polled twenty-odd times a session and the 404s were noise). Verified 200 with the
exact query string the editor uses.
Empty lists, not 404s, on purpose: this server has no cloud build to hand out. Producing one is the
other half of the feature - cloud-builds/create plus the git-over-HTTPS remote the studio pushes
the scene to - so the useful next state is the editor saying "this room has no builds" and showing
what it asks for after that, instead of failing on transport.
Still unexplained but harmless: three NullReferenceExceptions in
KJGMKNABDIJ.EMOFOGPOOLJ(System.String, System.Char) fire a second after a successful sign-in. That
is the studio's own URI parsing helper (the same one behind the earlier
Failed to obtain RecNet Service URIs), so if the studio later complains about a missing service it
is the first place to look - the candidate is a nameserver key we do not emit that it expects to
split.
Explained below ("Opening a room"). Decompiling the class settles it:
EMOFOGPOOLJisurl.EndsWith('/') ? url : url + '/', it NREs on a null nameserver field, and the null one isStudioCDN. The three exceptions were the same missing key each session, not a cosmetic curiosity - every URI before it resolved, which is why the session carried on.
for-room wants a page, not an array (2026-09-26)
Answering an empty list moved the failure from transport to shape, which is what makes the next step legible - the studio window stopped reporting a 404 and started naming the type it wanted:
Could not download room details.
The JSON value could not be converted to
RecRoom.Core.Studio.RecNet.PagedResultsDTO`1[RecRoom.Core.Studio.RecNet.CloudBuildDTO].
Path: $ | LineNumber: 0 | BytePositionInLine: 1.
Byte 1 of [ means the studio's list DTO is an object. This repo had already learned that for
the game client - V2_RoomsAPI.EmptyPage carries the note "The empty page every paged list here
answers with { TotalResults, Results } rather than a bare []. The client's list DTOs are objects
(PagedResultsDTO<RoomDTO>)" - so StudioAPI.CloudBuildsForRoom now mirrors that shape:
{ Results: [], TotalResults: 0, TotalCount: 0 }.
collaboration/owners-in-room is the exception and stays a bare array. The editor requested all
three studio routes within the same second (09:23:55 in .tmp-studio-auth.log) and accepted []
there without a deserialization error, so the two routes are not the same DTO family. Reading the
console to see which route threw, instead of assuming one uniform shape for every studio list, is
what separated them.
Wrong, and corrected below. The editor did reject
[]on that route - it just logged theResult<List<int>>deserialization error a second later, next to the room-open failure, where it read as part of that failure rather than as its own. "The two routes are not the same family" was the right conclusion for the wrong reason:owners-in-roomis wrapped (Result<T>), not paged.
Opening a room still cannot succeed: for-room is only the lookup, and this server has no
CloudBuildDTO to return because nothing has ever been uploaded. The next milestone is producing
one - cloud-builds/create plus wherever the studio pushes the scene - not another shape fix here.
Two patcher details settled the same day
- The one-time backup now lives outside the package folder, at
<ProjectRoot>\radicade-backups\<pkg-id>\...\X.dll.radicade-orig, because Unity treats everything underLibrary\PackageCacheas immutable and logs thehas no meta file, but it's in an immutable folderwarning that opened this whole trail (-Restorestill understands a legacy in-package backup). Relocate an existing one with a plainmove; restore -> patch -> re-run was verified round-trip, leaving the slot at offset 296749 with 40 bytes of zero padding and the file size unchanged.
Opening a room: the studio's DTOs, and why "" is not null (2026-09-26)
The screenshot that prompted this round: the room resolves (TestPlayers_MakerRoom_3528/Home), sign-in
holds, the panel offers Build on Rec.net and Download - and Open Room fails with two
stack traces that have nothing to do with transport, because the server log for that whole session
has no non-200 in it:
[09:33:33] NullReferenceException: Object reference not set to an instance of an object
KJGMKNABDIJ.EMOFOGPOOLJ(System.String, System.Char)
[09:33:34] ArgumentException: The string argument 'filename' cannot be empty. Parameter name: filename
at CDLPCLJLLLG.DHGPNOHKPNB(...) -> MAMHCJFJOGI+OMPAKMPBPNM.MoveNext()
OGINOCKCDO+ICFNAE5EAJ.MoveNext() <- "Could not download subroom data"
[09:33:34] JsonException: The JSON value could not be converted to
RecRoom.Core.Studio.RecNet.Result`1[System.Collections.Generic.List`1[System.Int32]].
Path: $ | LineNumber: 0 | BytePositionInLine: 1.
The editor's own log keeps the full traces and names the methods, which is what makes the shapes
readable - %LOCALAPPDATA%\Unity\Editor\Editor.log, and the interesting lines are the ones between
the exception and the UnityEngine.DebugLogHandler:Internal_LogException marker.
Decompiling the studio is one command, and it is already installed
ilspycmd (8.2.0) is a global dotnet tool on this machine, and the editor's assemblies are ordinary
managed DLLs - the obfuscator renamed types and methods, not the DTOs, because the DTOs are the
wire contract:
ilspycmd -l "RecNet" ".../Library/PackageCache/com.recroom.studio.build@e443aa011f13/Editor/RecRoom.Studio.Editor.Core.dll"
ilspycmd -t 'RecRoom.Core.Studio.RecNet.Result`1' "<that dll>"
ilspycmd -o .tmp-rrs-decomp "<that dll>" # 7,725 lines, whole assembly
ilspycmd -o .tmp-rrs-authoring ".../com.recroom.studio@6492b87ce18e/Editor/Authoring/RecRoom.Studio.Editor.Authoring.dll" # 28,779 lines
Both dumps are kept in the repo working tree (gitignored, like the .tmp-*.log files) - the loader
lives in Authoring and the API client in Core, so a question about a route is usually a
grep across both.
The reply shapes, verbatim
[Serializable] public class Result { public bool Success { get; set; } public string Error { get; set; } }
[Serializable] public class Result<T> : Result { public T Value { get; set; } }
[Serializable] public class PagedResultsDTO<T> { public List<T> Results { get; set; } public long TotalResults { get; set; } }
public class RoomDetailsDTO : RoomDTO {
public string DataBlob; public string DataBlobHash;
public IReadOnlyList<SubRoomDTO> SubRooms; public IReadOnlyList<TagDTO> Tags;
public IReadOnlyList<AccountRoleDTO> Roles; public IReadOnlyList<int> CoOwnerAccountIds; }
public class SubRoomDTO {
public long SubRoomId; public string Name; public string UnitySceneId; public int MaxPlayers;
public SubRoomDataSaveDTO CurrentSave; }
public class SubRoomDataSaveDTO {
public long SubRoomDataSaveId; public long SubRoomId; public string UnityAssetId; public string UnityAsset;
public string UnityAssetHash; public string DataBlob; public string DataBlobHash;
public int PersistenceVersion; public byte OMVersion; // <- a BYTE, not an int
public int? SavedByAccountId; public PICFJDPCODO? SavedOnPlatform;
public ELMJICAIAAA? SavedOnDeviceClass; public string Description; public DateTime CreatedAt; }
Result is what catches the eye: every reply the studio mutates with is wrapped, and the wrapper's
EEKDNHEOFFD() ("assert result") is Debug.Assert(Success == (Error == null)) followed by
if (!Success) throw new Exception(Error). So a wrapper with Success:false is not a soft failure -
it throws out of whatever the editor was doing. owners-in-room is declared
Task<List<int>> over Result<List<int>>, which is exactly the type in the JsonException: a bare
[] was never accepted there, it just failed a second after the room-open error and read like part
of it. StudioAPI.OwnersInRoom now answers { Success: true, Error: null, Value: [] } (the value is
the room's owner account ids, which the editor resolves one by one for its collaborator list).
Every nameserver URL is required, and StudioCDN was missing
KJGMKNABDIJ in the same DLL is the nameserver client: a NameServerResponse with 27 string fields,
a IEBKFBLKOKM that walks them one at a time into CNHJLHELLLL[key] = new Uri(EMOFOGPOOLJ(url, '/')),
and EMOFOGPOOLJ is url.EndsWith('/') ? url : url + '/' - it dereferences the string, so a key
the response does not carry is a NullReferenceException mid-walk, wrapped as
Failed to obtain RecNet Service URIs: Object reference not set to an instance of an object. The
dictionary is assigned before the walk, so everything before the missing key stays usable: the
session looks healthy and three NREs are logged.
RecNetStatus Auth API Notifications Images CDN Commerce Matchmaking Storage Chat Leaderboard
Accounts Link RoomComments Clubs Rooms PlatformNotifications Moderation DataCollection
BugReporting Discovery PlayerSettings Studio GameLogs Strings WWW Econ StudioCDN
GET /nameserver on this server had all of them but two: StudioCDN (mapped, so this is the
null that NREs) and RecNetStatus (declared by the editor but never mapped, which is why it was
never the culprit). RecNetStatus stays unwritten on purpose - adding a URL that 404s to a field the
2023 client may also read is a worse trade than leaving a field the studio never touches absent - and
StudioCDN is now {ServerUrl}/studiocdn, the base for the editor's roomunityasset/{filename}
(source-project manifest) and unityasset/{filename} (the baked bundle). Neither route exists yet;
the point is that the editor can build a Uri and get an honest 404 instead of an NRE, and the fix
is one line in DBConfig.NameServer next to a comment saying all 27 are load-bearing.
A blob name of "" is fatal; null is the documented "nothing to download"
The room loader (OGINOCKCDO in Authoring) does this, in order:
- room details, wrapped in
PFJIHBLICIE(..., "Could not download room details"); - pick the save:
room.LNLADOGFALG?.IDNMEAFLKOF?.FirstOrDefault() ?? room.DJACJOEJMOD?.CurrentSave- the saves list first (the/savespage), then the room payload'scurrentSave; - download the subroom blob from
save.DataBlobviaBCEDBPKMONJ.OGLFKNHIMHG(name)- failure text"Could not download subroom data"; - download the room-level blob from
RoomDetailsDTO.DataBlobviaBCEDBPKMONJ.FKDIJAIOAPJ(name)-"Could not download room data"; - both go to
MAMHCJFJOGI.GMIJOHGNLFI(filename), which first callsCDLPCLJLLLG.DHGPNOHKPNB(filename, "filename")- the empty-string guard from the trace - and thenGET {CDN}/room/{filename}; - and both are skipped entirely when the name is null, succeeding with an empty payload
(
new PLGEIAPHMDM(null)/new GDEHOACHGIF(null)), which the validators then step over (if (payload != null && payload.Length != 0)).
So "" is the one value that cannot be handled, and this server was emitting it in two places:
RoomUtils.ConvertToFullhardcodedDataBlob = ""- the room level field. Nothing on this server has ever produced a room-level blob (a save carriesRoomData.FilenameandSubRoomData.Filename;RoomServerAPI.UploadRoomDataonly ever stored the subroom one), so the honest value isnull. This is the failure the first attempt hit: the subroom download had succeeded (the/cdn/room/7dc6589a-...200 in the log) and the room blob killed it.subRoom.CurrentSave = BuildCurrentSave(subRoom)ran unconditionally, so a never-saved subroom shipped a save object withDataBlob: ""instead ofcurrentSave: null. That is the second attempt: room 31 simply has no save, and the studio read the placeholder save and died on the guard.
Both are fixed, and both fixes are safe for the game client - checked two ways. grep -rli currentsave
over the 2023 client's decompilation under .external/aug-decomp finds nothing: the client never
reads currentSave at all (it loads saves from the /saves route, whose blob-less entries this
server already omits). And the reference implementation under .external/recflare-server documents
the same contract - SubRoomDto.CurrentSave: z.nullable().describe('Null until first save'), no
room-level DataBlob field on its room DTO at all, and if (input.roomDataFilename) sub.RoomDataBlob = input.roomDataFilename on the save - so a null here is what the real backend does, not a
workaround.
Verified against the running server
Rebuilt, restarted, and read back with a token minted through the studio's own device flow (the
client_credentials path in .tmp-studio-auth.log works too, without a browser):
nameserver StudioCDN = 'https://localhost:3030/studiocdn'
--- room 31 (TestPlayers_MakerRoom_3528) roomDataBlob=None
subRoom 34 'Home' flatDataBlob='' currentSave=null
--- room 54 (TestPlayers_MakerRoom_1603) roomDataBlob=None
subRoom 57 'Home' flatDataBlob='7dc6589a-f613-4575-afc2-21efaf9347ac'
currentSave={ "SubRoomDataSaveId": 1, "SubRoomId": 57, "DataBlob": "7dc6589a-...", ... }
owners-in-room -> {"Success": true, "Error": null, "Value": []}
cloud-builds/for-room -> {"Results": [], "TotalResults": 0, "TotalCount": 0}
Room 31 now opens against an empty scene (nothing was ever saved) and room 54 against the save it has; the room-level blob being absent means the scene opens without room-level data, which is what a room with no room data is, and a studio save is what would create it.
dotnet build Server/Server.csproj is clean, and 185 tests pass across RoomSave, SubRoom,
RoomRole, AdminRoom, RoomDeveloperOwned, PlayerWarning, CdnRange and RecNetShape - the
last one is the conformance suite that compares payloads against the reference shapes, so the
CurrentSave/DataBlob change is covered in the direction that matters (a subroom that HAS a save
still reports it). ServerTests still cannot build as-is - its csproj compiles two
2023Patch/RadicadeEacBypass/*.cs files that live in the uncommitted stash - so this run used a
temporary project file beside it with those two Compile Includes dropped and the three test files
that need them (HubRealtimeTests, ImageSignatureRequestTests, NameserverRedirectTests)
removed. Do not "fix" the csproj: the missing sources are somebody's work in progress.
What is left before a room can actually be built in Studio
cloud-builds/create(+buildBranch,autoPublish,cloudBuildId,cloud-builds/cancel) - the studio'sBuild & Uploadbutton, and the only way aCloudBuildDTOwill ever exist to list;remote-run/push-to-device({ SessionId, RoomId, SubRoomId, RoomData, SubRoomData, BakedUnityAssets }ofUploadFileResponseDTOs);- {StudioCDN}/
roomunityasset/{filename}and/unityasset/{filename}, plus{Storage}/uploadand/upload/unityassets- where the studio puts the baked Unity assets it needs back; - the room-level
DataBlob: acceptRoomData.Filenameon the save route and store it, so a room saved from Studio comes back with room data instead ofnull; collaboration/owners-in-roomwith the room's real owner ids, which needs the collaboration layer this server does not have yet.
A room opens in Studio: the illegal-text check, and the realtime channels (2026-09-26)
The three shape fixes above landed the room. DormRoom/Home opened with its real Maker Pen scene in
the viewport (Plants, Basketball, Welcome Mat, the whole dorm object list), the panel stopped saying
"Failed to open room", and Build on Rec.net filled in with the subroom's save history (#1, #39,
#38, #37 - those numbers are the SubRoomDataSaveIds the /saves route mints).
Two errors were left, and both were missing server surface rather than shape:
[10:52:53] HubException: Failed to invoke 'SubscribeTo' due to an error on the server.
HubException: Method does not exist
[10:54:41] NullReferenceException KOMDFENMFBG+EADNODIDNG.MoveNext()
-> dialog: "An exception occurred while checking for illegal text in components.
Trying again may fix the error. Exception: 'Object reference not set ...'"
The purity check: POST /api/sanitize/v2/isAllowed/bulk
The studio runs every string in the scene through KOMDFENMFBG.DOLJODEKDLI, which calls
Result<AreStringsPureResponseDTO>, i.e. POST {API}/api/sanitize/v2/isAllowed/bulk with Values,
and then does:
Result<AreStringsPureResponseDTO> newStringAllowances = await NLMHGDMIMAP(toBeChecked);
for (int i = 0; i < toBeChecked.Count; i++)
HINAIGHAGOI.TryAdd(toBeChecked[i], newStringAllowances.Value.ArePure[i]);
Value is dereferenced and ArePure is indexed with the sender's own index, so the reply has to be
the Result wrapper with one bool per string, in order - which is the NRE, and the reason the
dialog says "trying again may fix the error" rather than naming the string it disliked. The route did
not exist at all (404, len=0), and the NRE is what the editor made of the empty body.
One detail decides how the route reads its input. The editor's request builder
(OCOOLHLEDHD.MFBDEFCEDGN) appends its parameter list to the URL only for GET; for every other
verb it wraps the same list in FormUrlEncodedContent. So this arrives as a form body of repeated
Values fields, and the log line looks like the studio called it with no arguments at all - there is
nothing after bulk:
[09:54:59] POST /api/sanitize/v2/isAllowed/bulk -> 404 (0.2 ms) len=0 ct= body=
SanitizeAPI.IsAllowedBulk now answers { success: true, error: null, value: { ArePure: [...] } }
from the same StringTools.IsPure every other route here applies to room names and inventions, with
the form spelling and the query spelling both accepted (so it can be exercised by hand). Verified:
$ curl -X POST https://localhost:3030/api/sanitize/v2/isAllowed/bulk -d 'Values=Plant (Dracaena)&Values=faggot&Values='
{"success":true,"error":null,"value":{"ArePure":[true,false,true]}} # aligned, in order
(The Result names are lowercase because that is how Success declares them for the game client;
the editor binds them anyway - it is JsonSerializerDefaults.Web, i.e. case-insensitive. That is also
why the owners-in-room wrapper from the section above needed no second pass.)
The realtime channels: SubscribeTo / UnsubscribeFrom on the hub
The editor's realtime client (LKDEHPKLIGJ) connects to {Notifications}/hub/v1 - NotifyHub, which
already serves the game client at that path - sends { Channels: [...] } to those two methods, and
reads Notification frames back off the connection. The channel it subscribes to is
RRS:<RecRoom.Studio.SessionId>, which is exactly where a rrs.pushtostudio message would come from
(remote-run/push-to-device, still unimplemented). The hub had SubscribeToPlayers but not
SubscribeTo, and SignalR answers an unknown method with HubException: Method does not exist - an
error the editor logs and retries on every reconnect instead of ignoring.
Both methods now exist, take the raw object? the way the neighbouring hub methods do, pull the
channel names out of the JsonElement by name (SignalR serializes camelCase, so the lookup is
case-insensitive), and log the subscription. Nothing publishes to them yet, which is stated in the
log line rather than silently swallowed:
[NotifyHub] U9-Bl4o4KZFIUWR5eMi9IA subscribed to RRS:test-session (nothing publishes to these yet).
[NotifyHub] U9-Bl4o4KZFIUWR5eMi9IA unsubscribed from RRS:test-session (nothing publishes to these yet).
Verified without a SignalR client package, by driving the hub's long-polling transport with curl - worth keeping, because it is the only way this repo can poke the hub by hand:
TOK=$(curl -s -X POST 'https://localhost:3030/notify/hub/v1/negotiate?negotiateVersion=1' | jq -r .connectionToken)
curl -s -X POST "https://localhost:3030/notify/hub/v1?id=$TOK" -H 'Content-Type: text/plain;charset=UTF-8' \
--data-binary $'{"protocol":"json","version":1}\x1e' # handshake
curl -s -X POST "https://localhost:3030/notify/hub/v1?id=$TOK" -H 'Content-Type: text/plain;charset=UTF-8' \
--data-binary $'{"type":1,"target":"SubscribeTo","arguments":[{"channels":["RRS:test-session"]}],"invocationId":"1"}\x1e'
The \x1e record separator is part of the protocol, and the response body stays empty on the POST -
the server-side log line is the proof of dispatch (a missing method answers with a HubException in
the next poll instead).
The panel's remaining warnings are the honest ones: "This scene has never been uploaded from Rec Room
Studio" and "Never save available on rec.net" are the empty cloud-builds/for-room page, and they go
away when cloud-builds/create and the build store exist.
Keepsakes: what the reference server holds, and the store we built
There is a second reference in play now - a downloaded fork at
rec-room-2023-server-main/rec-room-2023-server/asdfasdf - so it is worth stating exactly what it
carries, because it is less than it looks. Five files mention keepsakes and only one of them holds
anything the client can use:
| file | contents |
|---|---|
keepsakes_categories.json |
the real catalog: 5 categories in a counted result set |
APIs/API.cs:186,527,539 |
the same three GETs we serve: categories (reads the JSON), globalconfig, rooms/{roomId} -> Results.NoContent() |
KeepsakesConfig.cs |
empty stub classes, no code |
APIs/Auth.cs:227 |
"keepsake" in a role array - inert |
ProgressionEvent.cs:15 |
KeepsakeRoomLists, never assigned |
So the fork's keepsake endpoints are already byte-for-byte ours - the same 204 for a room, the same
config trio - except that its catalog is populated and ours was empty. That one file is the delta,
and it is trustworthy: each entry's field order matches the client's own
KeepsakeCategoryConfigDTO(enum, string, int, int, string, string) constructor, and its ids
(0 explore, 2 presents, 4/5/6 science_core_*) are the Science Core event this build's own
GameConfig text describes.
What the client itself says it wants
Straight out of global-metadata.dat rather than inferred: the route literals are
api/keepsakes, api/keepsakes/categories, api/keepsakes/events, api/keepsakes/globalconfig;
the feature's error strings (Failed to add keepsake, Failed to collect keepsake {0},
Failed to delete keepsake {0}, Waiting for Keepsake Instance Data download...,
Cannot place additional keepsakes in this room!) and the notification names
KeepsakeInstanceAdded / KeepsakeInstanceRemoved are all present. Our own log already showed the
three GETs firing on every room load (rooms/25 -> 204, categories -> 200 len=43,
globalconfig -> 200 len=100).
The one write-side payload we can read comes from the recflare reference's notify package, which documents the captured shape:
/** `KeepsakeInstanceAdded` / `KeepsakeInstanceRemoved`. */
interface KeepsakeInstancePayload {
KeepsakeInstanceId: string // the GUID the room save carries
KeepsakeCategoryConfigId: number
PlacedByAccountId: number
RoomId: number
SubRoomId: number | null
}
Two things follow from that, and they drive the design. The instance id is a GUID the room save already contains - the client writes it when an author places a keepsake - so the server must store the id it is told, never mint its own; and a room whose save holds a keepsake with no row behind it is precisely the state the keepsake runtime null-references.
What shipped this session
KeepsakeInstancesandKeepsakeCollectionRecords(hand-written guarded migration20260927120000_AddKeepsakes, the same style as the neighbouring ones).GET api/keepsakes/categoriesnow serves the five real categories (RADICADE_KEEPSAKE_CATALOG=emptyrestores the empty set;RADICADE_KEEPSAKE_ICON=<name>swaps both icon names, which is how we find out which name the client actually reads).GET api/keepsakes/rooms/{id}is unchanged when nothing is placed (204) and serves{KeepsakeInstances, KeepsakeCollectionRecords, KeepsakeProgressionEventIds}once something is - the instances this caller has not collected, then their records.POST api/keepsakes/rooms/{id}places one, keeping the caller's GUID when it sends one;POST api/keepsakes/rooms/{id}/collectrecords the pickup, once per player (a repeat keeps the first timestamp).- Anything else under
api/keepsakeslands on a probe that names the call in the console and answers 404, never 204: a write that looks successful while the room save holds an id nothing backs is the exact failure the empty catalog was protecting against.
The two open unknowns are both answered by that probe rather than by more guesswork: the exact
spelling of the collect/delete routes (the literals only prove categories, events, globalconfig,
api/keepsakes) and the JSON key the client reads for a placed instance. The next in-game attempt
prints them.
The room that already has a keepsake in it
RoomImports/Showdown/Showdown.room - imported as an RRO at boot - contains a keepsake component
with a GUID beside it:
offset 585176: ... "keepsake" ... $a75f7547-79eb-47c6-8986-6767abcb4f92 ...
offset 591174: ... keepsake ... (the room's component-type table)
Nothing registers it yet, and that is deliberate: registering a guessed id is exactly the kind of invented keepsake that faults a load. The right way in is the probe - join Showdown, watch which ids the client reports, and store those.
Keepsake rooms: how a room comes to be allowed to hold keepsakes
The banner this leaves behind - "This room does not have keepsake permissions." - is not a room
role. The client's room-permission enum is CAN_EDIT_ROOM_ROLES ... CAN_USE_MAKER_PEN ... CAN_USE_PLAY_GIZMOS_TOGGLE with no keepsake entry in it, the room DTO has no keepsake field either,
and the fork's ProgressionEvent.KeepsakeRoomLists is never assigned. The only per-room keepsake
payload the client asks for while a room loads is GET /api/keepsakes/rooms/{id}, and the keys that
response may carry - out of the client's own metadata - are KeepsakeProgressionEventIds,
KeepsakeRoomId, KeepsakeRoomLimit, KeepsakeRoomListId, KeepsakeRoomLists, KeepsakeRooms.
So the permission arrives with the room's keepsakes or not at all, which is why every room answered
NO CONTENT and every placement answered with that banner.
A room is advertised as a keepsake room when it is listed in
Server/RadicadeServer-Config/keepsake-rooms.json ({ "AllRooms": false, "RoomIds": [3, 25], "RoomListId": 1, "RoomLimit": 100 }) or when RADICADE_KEEPSAKE_ROOMS names it (ids, or * for
every room, with RADICADE_KEEPSAKE_ROOM_LIST_ID / RADICADE_KEEPSAKE_ROOM_LIMIT beside it). Those
rooms answer 200 with the keys above beside the three instance lists - and those lists stay honest,
so an empty keepsake room still carries nothing for the keepsake loader to resolve - and the server
logs [Keepsakes] room {id} answered as a keepsake room: list ..., limit ..., N room(s) advertised, M placed. Every other room is unchanged: NO CONTENT, the old behaviour. The config that ships
enables every room (AllRooms: true): the banner is the symptom this exists to clear, so a
checkout that shipped it switched off would look exactly like the bug. Narrowing it back is an
operator edit - AllRooms: false plus the ids in RoomIds.
The file is re-read on every room load, so an edit applies to the next room a player enters with no
restart. That is verifiable without the game: with the running server on 3030, curl -sk https://localhost:3030/api/keepsakes/rooms/25 answered 204 0b while AllRooms was false and
200 381b with KeepsakeRoomId/KeepsakeRoomLimit/KeepsakeRoomListId/KeepsakeRooms/
KeepsakeRoomLists the moment the file said true - same process, no relaunch.
Confirmed in game. With AllRooms: true, walking into room 25 clears the banner: those five
room keys ARE the permission, and the client asks for them every room load ([Net] Get .../api/keepsakes/rooms/25 in the 21:09 log, the first run that got far enough to make the call).
Permission is not the whole gate: the keepsake comes from a progression event
Permission cleared the banner but left the MakerPen's keepsake tile reading x0 - a room can be
allowed to hold keepsakes and still offer nothing to place. The second half is the progression
event:
- The room payload's
KeepsakeProgressionEventIdsnames the event the room's keepsakes belong to (the client's metadata spells the field runEmptyKeepsakeProgressionEventIdsKeepsakeRoomId…, i.e. "empty" is its own state). GET /api/progressionEvents/activeanswers the active event's ID - a bare number, CONFIRMED: with the id served, the client's log has no malformed line for this route at all, and the very next calls areGET /api/progressionEvents/event/1andGET /api/progressionEvents/record/1(the record asked four times in one session), i.e.ProgressionEventDTOandProgressionEventRecordDTOby id. Both were 404s here, so the id alone changed nothing on screen; the record, itsKeepsakeRoomListsand the palette's keepsake allowance all hang off them.The client's progression-events client has exactly two no-argument calls,KIJDNIPGGDC() -> MFBKDKPGMNP<long?>and a private one returning a record, and the id is what the model caches first:ABIEHIAHKJP(Nullable<long>)is that field's setter andBJKPLIJHAJK(long, bool)is the id-taking record fetch. A nullable long deserializes from none of the shapes this endpoint answered before ([],{}, and a full record), each logged by the client asMalformed Response: '<body>'.
Both by-id payloads carry an _route marker ("event" / "record"). An unknown member is ignored
by the deserializer, so it costs nothing, and it is what makes a Malformed Response: '<body>' line
in the client's log name WHICH of the two the parser rejected - the log shows a body, never a URL.
RoomListModel.ApplyKeepsakeOverrides(ref eventRecord, RoomModel, long, int, long)is the client applying that record'sKeepsakeRoomListsto a room, and the sibling membersIsActiveEventKeepsakeEvent/GetKeepsakeRoomListIdFromRoomId/GetNextKeepsakeRoomUnlockTimeare all event-shaped.
The record's field names come from the 2023 fork's own ProgressionEvent model
(Downloads/rec-room-2023-server-main/.../ProgressionEvent.cs) and each one is present in this
build's global-metadata.dat: ProgressionEventId, Name, IsEnabled, UsesBoost, BoostXpMultiplier
(the fork spells it BoosXpMultiplier - a typo), BoostDailyGameplayMinutesLimit, ActiveExperiment, StartTime, EndTime, CollectionEndTime, EventInfoImageName, ChallengesIconImageName, RewardsPipImageName, PurchasableXpBoostId, Rewards, KeepsakeRoomLists.
Two fork fields are NOT in this build and are left out: EventLeaderboardId and
InitialCurrencyGrant. The event is windowed around the current time (start -30d, end +1y) rather
than pinned to a date, so a machine whose clock is in any year still sees it running.
The event's KeepsakeRoomLists carry the same one-list entry the room payload carries (from
KeepsakeRoomConfig.RoomListEntries), because the client's copy of the event OVERRIDES the room's:
a room the event's list does not mention would be overridden straight back to no permission. Under
AllRooms the event's copy carries no member rooms - they cannot be enumerated - and the room's own
copy is what names it.
The event payloads were rejected, and the client's own log says so. The game writes every
session to BepInEx/LogOutput.log, which the plugin's [LogGate] feeds: the room's keepsake answer
never appears in a Malformed Response line, but the event bodies do, verbatim - the whole record
(Malformed Response: '{ "ProgressionEventId": 1, "Name": "Science Core", ...') and, after it was
narrowed to identity + window + keepsake lists, the narrowed one too (the log's truncation lands on
"StartTime" right after "IsEnabled", which is how the two payloads are told apart). Both come
from EFMPNEEHJFJ.CMKDNAPFCOI[T]. Two things follow:
- Unknown members are ignored, so a name that is merely absent costs nothing - which is why the
room's answer parses even though it carries six keys
KeepsakeRoomInstancesDTOdoes not declare (that is the answer that cleared the banner). - A name whose TYPE is wrong fails the whole response, so the fix is to send FEWER names, not
more: every field that belongs to a neighbouring DTO rather than to this record is now out - the
XP-boost family (
BoostXpMultiplier,BoostDailyGameplayMinutesLimit,UsesBoost,PurchasableXpBoostId, all of which live on the boost DTOs) and the image-name fields (which may be typed image references rather than strings). What is left is the event's identity (ProgressionEventId,Name,IsEnabled), its window,Rewards, andKeepsakeRoomLists.
Two knobs stay for the next narrow. RADICADE_KEEPSAKE_EVENT_LISTS=empty sends KeepsakeRoomLists: [] in the record instead - an empty array deserializes for a list of anything, so if the malformed
line disappears with it set and returns without it, the entry shape is what the record rejects.
RADICADE_PROGRESSION_ACTIVE=record serves the record AT /api/progressionEvents/active instead of
the id, for the case where that endpoint turns out to be the client's other no-argument call. The
server console names which it served: [ProgressionEvent] active -> 1 or [ProgressionEvent] record 1 served (all keepsake list(s), list 1, limit 100).
How to read the next run. Walk into a room and place a keepsake; the server console names both halves of the answer:
grep -n "Keepsakes]" server-console-runNN.log # the room's answer, and any placement
A POST /api/keepsakes/rooms/{id} answered accepted means the tile had something to place. If the
tile still reads x0, the next lever is the event's contents rather than the room's permission -
and any keepsake call this server does not implement is still named by the [Keepsakes] unclaimed METHOD path body=… probe (which answers 404 on purpose). GET /api/keepsakes/events is the one
keepsake route still stubbed to []; the client's own route table (the literal run
api/keepsakes api/keepsakes/categories api/keepsakes/events api/keepsakes/globalconfig in
global-metadata.dat) shows those four are the whole keepsake surface, with the room route built
beside them.
Saving an outfit: the route the client posts to, and the 404 it met
The client's POST /econ/api/avatar/v4/saved/set answered 404: this server had the saved-outfit
routes on v3 only (GET /api/avatar/v3/saved, GET /api/avatar/v3/saved/{accountId},
POST /api/avatar/v3/saved/set), while the 2023 build reads its list from v3 and writes through
v4. The client's own route table in global-metadata.dat names the three avatar routes it knows:
api/avatar/v4/items api/avatar/v4/saved/name api/avatar/v4/saved/set
so the missing pair was saved/set and saved/name (items already existed). Only saved/set is
implemented here; nothing in this build has been seen calling saved/name, and a route invented
without knowing the shape it must answer is worse than a 404 that names itself.
v3 and v4 share one slot table. Both post the same payload and both expect "save over this
outfit" to mean re-save an existing slot, and the client reads the result back through v3. Two
tables would let a player save an outfit that their own outfit list cannot find, so the slot
bookkeeping lives in one place (RecNet/Avatar/SavedOutfits) and each version is only a shape:
v3 echoes the outfit back, v4 answers the lean acknowledgement the client expects - { Success, Slot } - which is what the reference servers do too.
Two things behind the 404 were wrong and had gone unnoticed because the route that used them was never reached by this client:
Outfit.Slotwas a string; the client posts an integer (Slot: 0..n). Deserializing the client's body into the old DTO could not have bound, so even a v3-shaped route would have failed on the body. It isint?now, and the payload's other two members (OutfitSelectionsV2,CustomAvatarItems) are declared so they survive the round trip instead of being dropped on save.- A body with no slot was accepted and stored as a literal
nullentry - the[null]an earlier empty-body post left on account 2 is exactly that. A missing slot is now a 400, matching the reference server: defaulting it would overwrite slot 0, the one outfit the player did not ask to replace. Stored blobs are also filtered on read, so that legacy[null]is dropped rather than handed to the client as an entry it cannot parse.
Verified live against the running server: no token is a 401 (the route exists), a body without a
slot is a 400, a real save is 200 {"Success": true, "Slot": 0}, and
GET /econ/api/avatar/v3/saved serves that outfit back with its selections verbatim. The writes
were made on a throwaway slot and account 2's stored blob was restored afterwards.
Loading a room: what the second-per-second actually goes to
A room join is slow - minutes for a Rec Room Studio room, and uncomfortably long even for a Dorm whose save is 1.8 KB - and the log's own numbers say why. Measured against live from the client's machine:
| Measurement | Result |
|---|---|
| Fresh HTTPS request to the API | 0.57 - 0.86 s (dns 0.28 + tcp 0.19 + tls 0.2 + server) |
| Second request on the same connection | 0.19 s |
Server-side work for a trivial route (/api/version) |
the same 0.19 s as any other route |
| CDN asset bundle, one stream | 10 - 12 MB/s (85 MB ⇒ ~8 s) |
| TLS session resumption | already working (Reused, TLSv1.3) |
0.19 s for every route including a no-op one is ~2 RTT to the VPS, which is 100 ms away - the
server is not spending that time, the wire is. So a room load costs what it costs because the client
makes dozens of calls and opens a fresh TLS connection for each one (the [TLS] Accepting server certificate ... line is printed again before every single request in the log). Nothing server-side
removes that; two things were making it worse:
GET /econ/api/roomcurrencies/v1/getBalancedid not exist. The client asks it immediately aftergetAllBalanceson every room join and retries an answer it cannot use; a 404 is an exception plus another round trip per retry. Nothing here tracks room-currency balances, so it now answers200with a zero balance that echoes the currency and account asked for (RoomCurrencyAPI.GetBalance).- The CDN sent no
Cache-Controlat all.Accept-Ranges,Content-LengthandLast-Modifiedwere there, but nothing told the client it may reuse bytes it already has - and for a Studio room the bytes in question are the entire asset bundle. Blob and asset routes now sendpublic, max-age=31536000, immutablefor.assetbundlenames (the id is minted per upload, so a different bundle is a different URL) andpublic, max-age=86400for everything else, because an admin repair can replace a room save under its existing name (VPS/fix-dorm-blob.sh).
Two things in the log that are not what they look like
Malformed Response: '[]', '{}' and '[ {...rooms...} ]' are thrown by the client's obfuscated
DTO reader, which reads members positionally - the same property AlgorithmicListsAPI documents
for lists/algorithmiclists. An empty body is therefore not a valid "nothing here" answer, and the
client retries instead of moving on. Several routes still answer that way by design (keepsakes off,
progressionEvents/active, roomkeys/v1/room); the retries are visible as 4x photon_access_token
and 5x experience/player per join.
Exception: Can't parse '' as a hex number! fires ~112 times per session and is not
PhotonAccessToken being empty, despite that field being "" in
GET /rooms/photon_access_token: the client fetches that route 7 times per session, and both
reference servers return "" there too (config/v1's UseNewPhotonAuthToken is false, so the
token is not used for the join). Whatever parses hex 112 times is elsewhere, and changing the token
to guess would risk the joins that currently work.
The failure at the end of a slow load
Unexpected Photon Disconnect due to cause: ServerTimeout is what the client reports as "Failed to
load room due to disconnect during load (Code 8706-3)": the load outlives the room's Photon
connection. The room-load stage timeouts are ours to set and are generous
(RoomLoad.DownloadAssetBundleTimeoutSeconds = 900, PreDownloadTimeoutSeconds = 120,
PostDownloadTimeoutSeconds = 240); the Photon peer timeout is not in that table, so the lever there
is client-side, not a config key.
Verified locally against a dev server, then the server was stopped:
GET /econ/api/roomcurrencies/v1/getBalance?currencyId=...&accountId=2 -> 200 {"CurrencyId":...,"AccountId":2,"RoomId":0,"Balance":0}
GET /cdn/data/{blob}.room -> 200 Cache-Control: public, max-age=86400
GET /cdn/data/{id}.assetbundle -> 200 Cache-Control: public, max-age=31536000, immutable
GET /cdn/data/{id}.assetbundle (-r 0-1) -> 206, Content-Range: bytes 0-0/1, same directive
Tests: the two CDN routes and getBalance are pinned in CdnRangeRequestTests and
RoomEconomyPersistenceTests (8/8), and the full suite is 688 passed / 1 failed - the pre-existing
AvatarCatalogSourceTests.ShippedLookupCoversTheCatalogServedToClients.
Run 30 fixes (2026-10-01): the ambiguous routes, the club category rows, the carousel envelope
One landed batch from a single player report. Each item is the client's own URL, taken from
BepInEx/LogOutput.log / Player.log in the deployed client rather than from the templates.
AmbiguousMatchException (HTTP 500) on the club member actions
ClubsAPI carries [Route("clubs/club")] (plus the api/clubs/v{1,2}/club copies), so an action
with BOTH the class-relative template {clubId}/minlevel and an absolute
~/clubs/club/{clubId}/minlevel registers the same path twice. ASP.NET reports that as ambiguity,
so every request to it threw before any handler ran - PUT minlevel, POST|DELETE members/remove,
PUT members/invite and the accept/decline actions were all dead. The absolute ~/clubs/club/...
copies are removed (the ~/club/... aliases are a different path and stay); the same duplication in
ClubAnnouncementsAPI and api/playerevents/v1/report (EventsAPI owns v1, ContentReportAPI keeps
v2) went with them, and PATCH admin/panel/{table}/{id} vs {id:long} is now one action that
parses the id. ServerTests/RouteAmbiguityTests.cs reflects over every controller and asserts no
(method, path) pair has two owners, so this cannot come back silently.
POST clubs/club/{id}/members/acceptinvite -> 405
The string literals (and the log) show the client accepts an invitation at .../members/acceptinvite;
the server only ever registered .../members/accept, so the invite notification led nowhere. Both
verbs are registered now, with declineinvite and changetype beside them, and invitemembers
joins invite. The invite reader also accepts the field spellings the reference DTO allows
(accountId/accountIds/playerId(s)/targetPlayerIds, single or repeated) instead of only
accountId, which is what silently invited nobody.
POST econ/api/influencerpartnerprogram/remove -> 415
The economy client does not post JSON, and [FromBody] only accepts JSON content types. The handler
reads the fields from the query string, a urlencoded form or a JSON body (content type or not), the
same way redeem does.
DELETE api/customAvatarItems/v1/{guid} -> 405 and POST .../report -> 404
The item's editor addresses it by id: PUT (metadata edit) and DELETE (remove) are registered
alongside the existing GET, both owner-gated, and the item report route resolves the item's creator
as the reported player before funnelling into the shared SubmitReport.
GET matchmaking/room/{roomId}/instances -> 204
The instance browser is the watch menu every player opens for the room they are in. It required
RoleType.Moderator and answered null to everyone else - null serialises to 204 No Content,
which the client renders as "Failed to get room instance browser: Response was empty". It now
answers an array (empty when the room has no live instance) to any signed-in caller.
"Clubs are appearing in every single category"
The Clubs screen asks clubs/club/categoryTags for its categories and then fetches each row on its
own: GET clubs/club/search?sort=0&category=Social&count=32, one call per tag. The server ignored
category and returned the whole catalogue, so the single club was drawn in every row.
ClubStore.CategoryFilter/MatchesCategory now filter on primaryTag or customTags
(case-insensitive, #tag tolerated) for search, mostactivetoday, mostactivenow and
search/mostactivenow; a request with no category still gets everything.
Error refreshing rooms from source CarouselEndpoint
Still one per Play-menu open, and now explained: the client's template for the row is
rooms/carousel/{0}&skip={1}&take={2} (a paged read), and the client logged our answer as
Malformed Response: '[ { "RoomId": 167, "IsDorm": false, ... } ]' - a bare array where it reads
{ Results, TotalResults }, the envelope every sibling *_rooms/{source}&skip=&take= feed already
answers with. rooms/carousel/{*source} answers that envelope now. The earlier "a bare array works"
conclusion came from a run whose Play menu was fed by lists/algorithmiclists + rooms/bulk
instead, so the route had never actually been read by the client.
Tests: RouteAmbiguityTests, ClubCategoryDiscoveryTests (9), CarouselFeedShapeTests (3) and
three new instance-browser cases in MatchmakeInstanceRouteTests; full suite 798 passed / 9 failed,
all nine the pre-existing catalog set (AvatarCatalogSourceTests, DorkNetCatalogTests,
EquipmentInventoryTests, StorefrontRewardConfigTests).
Run 34 (2026-10-15, live): every room's bundle answered 400, so no room loaded its content
What the player saw, and what it was not
"Error downloading room assets! Try again later or on a better connection." on every room, for every
player. In Player.log the exception surfaces inside RoomLoad.RoomManager and the client reports
FailedToSpawnPlayer / (Code 8715) to matchmaking; the Zenject
InvalidOperationException: You should not try to retrieve bindings from a disposed container! that
follows is that failed load task being torn down, not a second fault. Nothing in the client's own log
names the request, which is what made this look like a connection problem.
The request the client was actually making
BepInEx/LogOutput.log logs every call with its status, and it has the answer:
[HttpStatus] #322 Get /rooms/unity_assets/Showdown_0_1.assetbundle/0/1 -> 400 bytes=39 reason=Bad Request
[HttpStatus] #543 Get /rooms/unity_assets/2n8bbqwnk6zs6k2blzgcdhcnc.assetbundle/0/1 -> 400 bytes=39 reason=Bad Request
Room 247 is the MAKER_ROOM:247/313 in the error: subroom 313 advertises
UnityAssetId: 2n8bbqwnk6zs6k2blzgcdhcnc.assetbundle, and step two of the download is healthy -
GET /data/2n8bbqwnk6zs6k2blzgcdhcnc.assetbundle -> 200 68,673,320 b
GET /rooms/rooms/247?include=5421&unityAssetTarget=0&unityAssetVersion=1
-> 200 SubRooms[0].UnityAssetId = 2n8bbqwnk6zs6k2blzgcdhcnc.assetbundle
- so step one, the metadata call, was the only thing broken, and it was broken for every id of that shape on the server rather than for one room.
The regression, in one line
b33d5b3 ("Checkpoint the in-flight work before merging origin/2023") reworked UnityAssetsCDN for
the Studio pipeline and put this in front of the resolution:
if (!StudioUnityAssetRegistry.IsValidAssetId(assetId) || target < 0 || ...) return BadRequest();
IsValidAssetId is value is { Length: 40 } && value.All(char.IsAsciiLetterOrDigit) - the shape of a
Studio manifest id, and of nothing else that is stored as a bundle. Every legacy id fails it:
Showdown_0_1.assetbundle (23 characters, _ and .), the 24-character RecNet ids
(2n8bbqwnk6zs6k2blzgcdhcnc, 8aas7b5ivtaggn9swhe3wkgdj), the 36-character dashed id the importer
binds (32cafb11-c51b-485c-8f1f-73b73159feff), and the built-in rooms' *_0_1.assetbundle names.
ResolveLegacyFilename was still underneath, unreachable for exactly the ids it exists for, so every
room whose subroom points at a bundle - which is every room with custom content, for every player -
answered 400 {"success":false,"error":"bad request"}. The pre-Studio file had no such gate:
ResolveFilename(assetId), the id with no extension first, then the id plus .assetbundle.
What changed
Server/CDN/UnityAssetsCDN.cs - 40 alphanumerics with a manifest behind them is still the Studio
path, and a target/version its metadata does not name is a 404 (the file stored under a manifest id
is the manifest JSON, so it must never be handed over as a bundle). Everything else resolves as a
stored file: the name as given first, then <name>.assetbundle.
Server/CDN/CdnStorageNames.cs (new) holds the two read-side rules the blob routes share - one bare
file name in that one directory (no separator, no ..), never a name ending in the admin upload's
.uploading suffix, and <id> resolving to <id>.assetbundle - and DataCDN, RoomsCDN (both the
/cdn/room + /unity blob route and the raw {assetId}/{target}/{version} route) and
InventionCDN all go through it. Two things fall out of that: /data/<name>.uploading no longer
serves a half-imported bundle by name, and the bare /unity/{id} alias now answers for a bundle
stored only as <id>.assetbundle - the hard link per bundle that Run 31 needed is no longer
required.
Verified
ServerTests/UnityAssetMetadataTests.cs (21 cases: both spellings of three id shapes, the Studio
manifest with and without a bake, the manifest never served as a bundle, an id that only looks like a
manifest, unknown names, separators and .uploading, out-of-contract target/version, and the blob
routes) - 21/21. CdnRangeRequestTests, StudioUploadTests, AdminRoomImportTests,
AdminRoomPartUploadTests, AdminBundleBindingTests together with it: 59/59.
Live on a local instance (Development, cwd Server, http://127.0.0.1:3031, storage
Server/RadicadeServer-Data/CDN/Storage), the URL from the player's log:
GET /rooms/unity_assets/Showdown_0_1.assetbundle/0/1 -> 200
{"UnityAssetId":"Showdown_0_1.assetbundle","Target":0,"Version":1,
"Filename":"Showdown_0_1.assetbundle","RejectedDependenciesFilename":null}
GET /rooms/unity_assets/Showdown_0_1/0/1 -> 200 Filename="Showdown_0_1.assetbundle"
GET /rooms/unity_assets/Showdown_0_1.assetbundle -> 200 (the short form)
GET /rooms/unity_assets/not-on-disk.assetbundle/0/1 -> 404 {"success":false,"error":"not found"}
GET /data/Showdown_0_1.assetbundle (Range 0-15) -> 206 16 b, Accept-Ranges: bytes
GET /unity/Showdown_0_1 -> 200 76,719,774 b (was 404 without the hard link)
GET /data/Showdown.room -> 200 1,222,161 b
and the refusals, with the file present on disk so the rule is what refuses:
GET /data/probe-upload.assetbundle.uploading -> 400 (was 200, a truncated bundle)
GET /rooms/unity_assets/probe-upload.assetbundle.uploading/0/1 -> 400
GET /rooms/unity_assets/..%2F..%2Fappsettings.Development.json/0/1 -> 400
GET /data/../appsettings.Development.json -> 404
The probe file was removed afterwards and the instance stopped; nothing left listening on 3030/3031.
Full suite: 1556 passed / 12 failed. Eight of the twelve fail identically with this change stashed
out (the catalog set the Run 30 note records: AdminGiftSkinTests, ConnectionRecoveryTests,
DorkNetCatalogTests x2, EquipmentInventoryTests x2, StorefrontRewardConfigTests,
WeeklyChallengeTests), and the other four (GameRewardFlowTests x2, RecNetShapeConformanceTests
x2) pass when their classes run alone and mutate the same two statics -
DBConfig.DefaultAvatarCatalog / DefaultAvatarUnlocks - that AvatarCatalogSourceTests,
AvatarWireFormatTests and EquipmentInventoryTests write from parallel collections. That race is
pre-existing and is not this change's.
Not verified: nothing was deployed, so production still answers the 400 until the server is
rebuilt and restarted, and the client has to rejoin room 247 to confirm the load.
Run 35 (2026-10-15, live): the join box had no sender, so the client had no name to put on it
The box already carried a sender - no room could set one
RecNet's Gift.FromPlayerId is the field the client reads to name a box's sender, and the panel's own
Gift Player page already lands there (AdminGiftFromTests). A room's join gift built its Gift in
RoomJoinGiftUtils.TryDeliverAsync and left FromPlayerId at its default of zero, so every join box
arrived from nobody, and no room setting could change that.
What changed
RoomData.JoinGiftFromPlayerId(bigint,0= no sender), added both by20261015120000_AddRoomJoinGiftFromPlayerIdand byEnsureRoomJoinGiftColumnsfor a database whose migration history is already past that point.RoomJoinGiftUtils.TryDeliverAsyncputs it on the box it hands out on join.- The admin panel's Room Details modal carries a FromPlayerId inside the Gift Box On Join block
(
assets/js/rooms.jsinRadicade-Admin), andAdminPanelAPI.UpdateRoomreadsjoin_gift_from_player_id, refusing anything that is not digits rather than dropping it quietly.
The id travels as digits in a string, never as a JSON number: an account id is eighteen digits, a JSON number is read back as a double, and its last digits are rounded away - so the panel would save a different account than the one it was shown. The room row the panel reads and the value it posts are both text, the same way the Gift Player page already posts ids.
Only a sender that actually changes bumps JoinGiftVersion, so a save that leaves the field alone does
not hand every player a fresh box.
Verified
ServerTests/RoomJoinGiftFromTests.cs, 8/8, driving the real routes and then reading the box the way a
client does (GET api/avatar/v2/gifts, asserted on the wire JSON): the saved sender becomes the box's
sender; an eighteen-digit id survives the panel both ways; a changed sender re-issues the box while a
save that leaves it alone does not; a blank clears it back to no sender; a payload without the field
keeps the stored sender; a non-digit value is a 400; and a room that names no sender still hands out
its box.
Live on a local instance (Development, cwd Server, http://127.0.0.1:3031, SQL Server
.\SQLEXPRESS) the migration applied on startup and the route behaved:
Applying migration '20261015120000_AddRoomJoinGiftFromPlayerId'.
GET /admin/panel/rooms -> 200 all 45 rooms carry
"join_gift_from_player_id": "0"
PATCH /admin/panel/rooms/58 {"join_gift_from_player_id":"114903603133502227"}
-> 200 {"success":true}
GET /admin/panel/rooms/58 -> 200 "join_gift_from_player_id": "114903603133502227"
"join_gift_version": 2 (was 1)
PATCH /admin/panel/rooms/58 {"join_gift_from_player_id":"not-a-player"}
-> 400 'join_gift_from_player_id' must be
an account id in digits, or blank
for no sender. (room untouched)
PATCH /admin/panel/rooms/58 {"join_gift_from_player_id":"","join_gift_version":1}
-> 200 "join_gift_from_player_id": "0"
"join_gift_version": 1
Room 58 was left exactly as it started (no sender, version 1) and the instance was stopped; nothing left listening on 3031.
Not verified: the modal itself was not clicked in a browser - the panel sits behind SSO, so the served
page redirects before the room list loads. rooms.js was parsed as a classic script in the browser
(new Function over the served file, no SyntaxError) and its field markup and save handler read;
nothing was deployed, so production needs a rebuild and a restart, at which point the column arrives
with the rest of the startup migrations.
Full suite: 1566 passed / 10 failed / 1 skipped, and every failure is one the Run 34 note already
records as pre-existing: AdminGiftSkinTests (x1), ConnectionRecoveryTests (x1),
DorkNetCatalogTests (x2), EquipmentInventoryTests (x2), StorefrontRewardConfigTests (x1),
WeeklyChallengeTests (x1), and the two order-dependent RecNetShapeConformanceTests avatar-catalog
cases that mutate DBConfig.DefaultAvatarCatalog / DefaultAvatarUnlocks from parallel collections.
The two GameRewardFlowTests scheduled into that same race passed this run.