How to fix the HuC/PCEAS 384KB ROM size problem?

[elmer]

Well, the Dark Kobold & Gredler have just managed to trigger a new little PCE development issue that I didn't know existed, and it seem like a problem that could be fixed in a variety of different ways.

So, time to ask for opinions! 

Catastrophy has just hit exactly $30 banks long ... that's 384KB.

Now to HuC and PCEAS, that's no major difference to being 296KB long, or 512KB long ... all the banks are located one-after-another.

But to Mednafen, and presumably a Turbo Everdrive, 384KB ROMs were actually manufactured back-in-the-day using 2 separate ROM chips,  1 256KB ROM chip mirrored at banks $00-1F, and then 1 128KB ROM chip at bank $40-$4F.

So when Mednafen loads a 384KB ROM, it puts things into different banks than HuC/PCEAS expected, and that leaves a hole in the normal memory map (well it actually mirrors the data), and the program that you're developing suddenly crashes for no apparent reason.

That's not good.

Now, to me, the simplest "fix" for this would be to add an optional flag to PCEAS to make it automatically pad HuCard ROMs to the nearest power-of-2, i.e. 64KB/128KB/256KB/512KB/1024KB.

Those are the sizes that match modern hardware and HuCards anyway, so I can't see what problem it could cause.

Then HuC would always pass that flag to PCEAS, and HuC programmers would never care (or know) about the 384KB issue, ever again.

I think that it should be an "optional" flag in PCEAS, because folks that hack/translate 384KB ROMs need to be able to avoid corrupting/destroying an existing game image that actually uses/understands the split banks and mirroring.

Does anyone have a different view of the best way to handle this? 

[Arkhan Asylum]

manually bank some data beyond that point to pad the rom to 512kb.

I did something like that for Atlantean to get around the same sort of issue.

It won't require fixing the compiler/assembler, and is a thing you can do with basically no effort.

[elmer]

manually bank some data beyond that point to pad the rom to 512kb.

Yep, I already did that and it's going to be the quickest solution for Catastrophy ... but I'd *like* to contemplate a better fix so that developers in the future don't just suddenly find things blowing-up with absolutely no warning.

That's the problem from my POV ... it's less that this situation exists, and more that it is something that isn't "obvious" and so will catch people unawares.

One cheap-n-nasty solution would be just to print out a warning message whenever a HuCard is exactly 384KB.

[Arkhan Asylum]

a message with a headsup would work.  I know of emulators that do similar things and won't load weird ROMs.

[OldMan]

But to Mednafen, and presumably a Turbo Everdrive, 384KB ROMs were actually manufactured back-in-the-day using 2 separate ROM chips,

Correct, got that.

1 256KB ROM chip mirrored at banks $00-1F, and then 1 128KB ROM chip at bank $40-$4F.

Okay, so there's a hole in the address space.

So when Mednafen loads a 384KB ROM, it puts things into different banks than HuC/PCEAS expected

And mednafen is mapping the upper ROM chip where? Does it duplicate it (so its mirrored) or what? Any more info?
.........................................................................................

Now, to me, the simplest "fix" for this would be to add an optional flag to PCEAS to make it automatically pad HuCard ROMs to the nearest power-of-2, i.e. 64KB/128KB/256KB/512KB/1024KB.

Lol. <snort>. Changing HuC/PCEAS is probably going to be a pain. There's a better way, actually.

First, is your ROM actually 384K, or is it 384K+header (512 bytes)? That's important; I seem to remember mednafen use the header bytes if they were there.
So, try building your rom without the header if its present. (-h, iirc). If that works, let us know.
If it doesn't, PM Arkhan and get his permission. We have things to fix that

[touko]

What sort of problems catastrophy has ??
Because F.L's barbarian rom is 720KB and have no particular problems, except some datas have a wrong label address,and must be included directly after the "huc.h"

[OldRover]

What sort of problems catastrophy has ??
Because F.L's barbarian rom is 720KB and have no particular problems, except some datas have a wrong label address,and must be included directly after the "huc.h"

A 720KB ROM image wouldn't have the issue. Catastrophy is at that odd size where emulators expect a split ROM.

[Pokun]

So when Mednafen loads a 384KB ROM, it puts things into different banks than HuC/PCEAS expected

And mednafen is mapping the upper ROM chip where? Does it duplicate it (so its mirrored) or what? Any more info?

Yeah I don't get it either. Are there other special cases than a 384K ROM?

PC Engine really needs a good development/hardware wiki, like the nesdev wiki. The one that exists is good, but outdated and has lots of missing pages.
If all 384K games are using this kind of special mapping so that emulators expects it, it should be written in a wiki or somewhere so that developers are aware of it, and can pad the hole in the ROM.

[elmer]

What sort of problems catastrophy has ??

Code and data that PCEAS expects to be loaded into banks $20..$2F gets loaded into banks $40..$4F instead.

Yeah I don't get it either. Are there other special cases than a 384K ROM?

I seriously doubt it.

AFAIK, the problem only occurs when the ROM is exactly 384KB.

It's a historical issue from obsolete 30-year-old manufacturing technology that emulators have implemented  special-case code to handle.

PC Engine really needs a good development/hardware wiki, like the nesdev wiki. The one that exists is good, but outdated and has lots of missing pages.
If all 384K games are using this kind of special mapping so that emulators expects it, it should be written in a wiki or somewhere so that developers are aware of it, and can pad the hole in the ROM.

Yes, this kind of community information is sadly lacking, but that's what we get from being a small development community.

It would be great if there was someone technical here that loved to document stuff ... but that's a rare combination.

The original hardware issue isn't relevant moving forward, because modern HuCards just use 512KB/1MB flash chips, and it is trivial to expand an old 384KB ROM image into 512KB once you know about the problem.

But translation hackers certainly need to know this kind of stuff.

a message with a headsup would work.  I know of emulators that do similar things and won't load weird ROMs.

If there are other emulators that freak out at the non-power-of-2 ROM sizes that PCEAS allows, then I'd think that it would probably be best to just (optionally) pad out the ROM images that PCEAS creates.

And mednafen is mapping the upper ROM chip where? Does it duplicate it (so its mirrored) or what? Any more info?

For a 384KB ROM (and probably only for a 384KB ROM with *exactly* $30 banks).

The first 256KB of the ROM gets mirrored in banks $00..$1F and banks $20-$3F.

The next 128KB of the ROM gets mirrored in banks $40..$4F, $50..$5F, $60..$6F, $70..$7F.

So, try building your rom without the header if its present. (-h, iirc). If that works, let us know.

Do you mean the "-raw" flag to PCEAS? I can't see any way to pass that flag to PCEAS when using HuC.

But it was easy to strip off the header in a hex editor, so I did that.

It has nothing to do with the header, as far as I can tell, at least not the one that PCEAS generates.

It is completely an artefact of Mednafen (and other emulators) dealing with how stuff was manufactured back-in-the-day.

There is no *bug* here, both Mednafen and PCEAS are doing sensible things, it's just that there's a bad interaction here between their different views of how the world works.

It only seems to happen when the ROM is *exactly* $30 banks long.

As Arkhan suggested, you can work around it just by adding some extra data into your ROM to make it $31 banks long.

If it doesn't, PM Arkhan and get his permission. We have things to fix that

???

The problem isn't in just adding another "#incbin" to bump the ROM from $30 banks long to $31 banks long (which works).

Nor is it in supplying a utility to change a 384KB ROM image into a 512KB ROM image ... that's 20-minutes of C coding, most of which is taken up in writing the readme.

The point is to avoid new developers tripping over some black-magic problem and getting stuck by it.

This is presumably the same issue that Dark Kobold hit on Catastrophy a few months back with the old HuC, and was totally stuck on for a while, and IIRC, with nobody here even thinking to suggest that he might have hit "the well-known 384KB problem".

That must have been a frustrating issue for him, that IMHO should never have been hit in a sensible development environment.

[Arkhan Asylum]

I believe OldMan is referring to a padding utility that was written awhile back that may solve this problem.

What we need is a proper wiki site where we can all contribute information and such.   

We don't need one person to take on the burden of documentation, because they will only get sidetracked, bored, or burned out.

or be wrong on accident and then who fixes it?

the only other thing would be available PDF downloads, with appropriate mirrors incase one place goes down. 

[Gredler]

I would love to have access to a more fleshed out pce dev wiki- maybe it could lead me down the coding path a bit more!

I have some modern development techniques that have proved useful for making stuff on PCE that I could probably share too, if there were an art / tech art section to said wiki. I just saw the other thread and will continue my thoughts there.

I sent a message in private channels about this to elmer, but arkhan has asked me to try to make an effort towards revealing our dialog to allow for the tribal knowledge to spread;

Is this going to resolve itself as I dump art onto DK? That is, as I give him art to add and implement, will it bump the rom size over the 384 "awkward tween years"?


Should we know that "if it's 384, it wont build"? Otherwise when it stops building we stop adding more because it's not working and we try to resolve what broke the build - I HATE BREAKING THE BUILD!! HATE IT! Maybe some sort of compiler error spam along the lines of " ROM Size = 384 --- Illegal ROM size --- Please add or remove data to increase or decrease rom size to 256 or 512!" if the rom were built at 384?

As someone who prefers to test at every art completion/implementation, it does super suck for the build to stop working - especially with no error/assert/warning and no breakpoints/debugging tools. Seeing a broken build stops progress simply because we don't want to continue working on what might be faulty code or content.

I'll say this, if we are establishing this as a "best practice, should know" rule (like power of 2:8x8 images..) can we call the rule:

"If it's 384 you must add more!"

;)

Thanks for the help duders, nice to have this aired out and see a path forward - this has indeed blocked us in the past.

[elmer]

What we need is a proper wiki site where we can all contribute information and such.   

We don't need one person to take on the burden of documentation, because they will only get sidetracked, bored, or burned out.

or be wrong on accident and then who fixes it?

the only other thing would be available PDF downloads, with appropriate mirrors incase one place goes down.

Absolutely! 

Sticky threads and things like that are a terrible band-aid solution, when a wiki *that people can sign up on* is the right way to do things.

I once tried to sign up on Archaic Pixels, but couldn't see a way. 

Is this going to resolve itself as I dump art onto DK? That is, as I give him art to add and implement, will it bump the rom size over the 384 "awkward tween years"?

Yes, the problem goes away (in Mednafen, anyway) as soon as you add one more bank and bump up to a 392KB ROM.

For Catastrophy, I just duplicated one of the #incbin lines, and changed the label name ...

#incbin(obeylogomap2,"maps/obeylogo.fmp");

Adding new art should do the same thing ... it's just a case of making sure that the last bank in the build isn't bank $2F.

Otherwise when it stops building we stop adding more because it's not working and we try to resolve what broke the build - I HATE BREAKING THE BUILD!! HATE IT!

Yep ... those folks that keep on checking-in stuff without testing it, and so breaking the main build, quickly get told to find a different job/career. 

[OldRover]

The original obeybrew was in wiki format but some douchebag hacked it so I just settled for a static site. I wouldn't mind setting it up as a wiki again on a third-party server where I don't have to worry about updating the scripting.

[Arkhan Asylum]

where I don't have to worry about updating the scripting.

hell yeah.

https://www.pcengine-fx.com/forums/index.php?topic=22004.0

lets just migrate it here.   

[DildoKKKobold]

The point is to avoid new developers tripping over some black-magic problem and getting stuck by it.

This is presumably the same issue that Dark Kobold hit on Catastrophy a few months back with the old HuC, and was totally stuck on for a while, and IIRC, with nobody here even thinking to suggest that he might have hit "the well-known 384KB problem".

That must have been a frustrating issue for him, that IMHO should never have been hit in a sensible development environment.

This is exactly it, and I think it needs a special warning coming of PCEAS, at the end - "WARNING: Unstable ROM size - add data to 392k to fix corrupted mirroring " for exactly this fixed number of banks. Even if this info were in a wiki, I wouldn't have even had a clue where to start looking, as ROMs just go from working to not. It really appears to be black magic.

Also, as gredler pointed out - development just stops, and could potentially just kill a project. If adding stuff broke the ROM, my last fix in the entire world would be to add even more.

Don't get me wrong, I'm glad the Wiki exists, as there is plenty of info that devs need access to. However, this specific random error definitely needs a basic warning with suggested solution, just so future devs who inevitably encounter the error don't have projects killed by a weird hardware quirk.

[elmer]

However, this specific random error definitely needs a basic warning with suggested solution, just so future devs who inevitably encounter the error don't have projects killed by a weird hardware quirk.

There's now a fix for this in the "New HuC" thread. 

If you use the new build, you won't need to add/remove data from your project in order to get around the problem.

[touko]

What sort of problems catastrophy has ??
Because F.L's barbarian rom is 720KB and have no particular problems, except some datas have a wrong label address,and must be included directly after the "huc.h"

A 720KB ROM image wouldn't have the issue. Catastrophy is at that odd size where emulators expect a split ROM.

Ah ok i see, But with some dummy data this should solve the problem .
Or adding:

#asm
 .bank $31
#endasm

before including huc.h force huc to create a larger ROM than 384KB.