Xanadu II Translation Development Blog

[SamIAm]

Ahhhh ... not too dark, then? Phase's anti-aliasing *should* look really nice on a blurry composite signal. 

Definitely not too dark. It's basically fine as it is, but if we wanted to, we could probably get away with bumping down the brightness one step.

I was using RGB, by the way, and the letters were still totally smooth. I suppose I can check it on an HD-CRT just to see if anything really stands out, but I bet it will be fine.

EDIT: Yep, totally fine. 

[esteban]

This is shaping up to be BETTER than the original presentation.

Insane.

[elmer]

We thought that all traces of this CD had been lost in the mists-of-time, but ...

The Legend of Xanadu II - Someday with my Love

[pixeljunkie]

We thought that all traces of this CD had been lost in the mists-of-time, but ...

The Legend of Xanadu II - Someday with my Love

sweet! thanks for posting this!

[esteban]

We thought that all traces of this CD had been lost in the mists-of-time, but ...

The Legend of Xanadu II - Someday with my Love

sweet! thanks for posting this!

Ditto!

[elmer]

One final (hopefully) "blog" post on the terminally-dull subject of data compression.

As we're rapidly approaching the end of the project, it's time to concentrate on a few of the little details that I've been ignoring for some time.

That includes the following warning message that both SamIAm and I see every time we run the script-insterter to convert the English scripts for LoX2.

  META_BLOCK "ZAD", was   76KB, now   78KB.
  META_BLOCK is bigger than original! Aborting!


Whoops ... that's a "fatal" error that can't be recovered from, and should abort the whole conversion process.

That's what the message is for.

It's a problem ... that's why I put the test in there and made it print a scary message.

Then, because it was the only file in the two games with the problem, and it's not a huge show-stopper, I just disabled the "abort", and let the English conversion continue anyway.

I still have no idea what long-term bug we're creating by having that one file overwrite the start of the next file in the game ... but we've lived with it for over a year now.

No user-visible bug, no problem, right? 

Well ... not really. 

It's time to finally fix things properly before the release.


This all goes back to a whole bunch of blog posts from a long, long time ago.

We've talked here, and in many other threads, about how CD-based translations are a huge PITA because of the limited amount of free-memory available to change things (like the language), and how a Super-Duper System Card would make life a lot easier for translators.

In the case of the LoX games, I got around the need for that by figuring-out exactly where Falcom loaded every single asset in the game from, and then re-compressing every single graphic element or text script or code with a different and more-efficient compression algorithm, that made everything smaller in memory, and so freed-up the extra space that I needed to fit in SamIAm's wonderful-but-larger English translations.

Which was great ... a significant amount of work, but great ... except that I had a problem re-compressing a few files with my not-quite-as-brilliant-as-I'd-like compression algorithm.

Anyway ... we messed-around with this, messed-around with that, and eventually got to the point where there was only one file causing a problem, and we couldn't actually see that problem on-screen ... so we ignored it. 


The problem comes from the statistical-encoding. Let me explain in real English ...

Hudson, and Falcom, and everyone else in the 16-bit console era used some form of LZSS compression (except for the SNES's SDD-1 chip in Star Ocean).

My not-quite-as-brilliant-as-I'd-like compression algorithm is just a variant of that (like Falcom's enhanced "Falcom2" compression), that takes advantage of the fact that shorter repeat-lengths occur more-often than longer ones, and that "repeat" sequences are more-often closer to the current decompression-pointer than farther away.

I'm sorry if that sounds like gobbledygook ... but it's just-about as simply as I can explain it.


The huge stupid-problem with my compression tools was that they weren't originally designed to search for both of those cases, because the code that does the search-for-a-match was originally copied from the LZSS compressor in Mark Nelson's Data Compression book.

You'd always get the longest-possible match for a repeat ... but you had no idea if it was the longest-and-closest-possible match.

That threw away a number of opportunities for data compression ... and made that one file larger than it had been when Falcom compressed it.


I finally (after approx 20 years) fixed that, and now the compression code checks for both the longest match, and when it finds that, checks for the closest match of that exact size (no, it's not as trivial as it sounds).


I've avoided making that change for approx 20 years, because I was afraid of how it was going to slow-down the compressor, and make it unusable. At the end-of-the-day, I figured-out a reasonable way of doing it, and it only adds a few % to the compression time. Silly me. 


Anyway ... where does that leave us? 


Well, that one bad file ...

  META_BLOCK "ZAD", was   76KB, now   78KB.
  META_BLOCK is bigger than original! Aborting!


Now shows ...

  META_BLOCK "ZAD", was   76KB, now   74KB.


That's much better! 


And when it comes to the level data, which includes the scripts, which include the English text ...


// LoX1 GAME META-BLOCKS 176KB (22 8KB banks)

META_BLOCK "Z10",  Falcom 160KB,  simple 142KB,  optimal 136KB.
META_BLOCK "Z11",  Falcom 148KB,  simple 132KB,  optimal 128KB.
META_BLOCK "Z12",  Falcom 176KB,  simple 156KB,  optimal 150KB.
META_BLOCK "Z13",  Falcom 174KB,  simple 154KB,  optimal 150KB.
META_BLOCK "Z14",  Falcom 166KB,  simple 150KB,  optimal 146KB.
META_BLOCK "Z15",  Falcom 172KB,  simple 156KB,  optimal 152KB.
META_BLOCK "Z16",  Falcom 168KB,  simple 152KB,  optimal 146KB.
META_BLOCK "Z17",  Falcom 176KB,  simple 156KB,  optimal 150KB.
META_BLOCK "Z18",  Falcom 158KB,  simple 142KB,  optimal 138KB.
META_BLOCK "Z19",  Falcom 176KB,  simple 154KB,  optimal 150KB.
META_BLOCK "Z1A",  Falcom 132KB,  simple 116KB,  optimal 112KB.
META_BLOCK "Z1B",  Falcom 174KB,  simple 142KB,  optimal 136KB.


// LoX2 GAME META-BLOCKS 128KB (16 8KB banks) (only the largest blocks are shown)

META_BLOCK "Z25", Falcom  128KB, simple  124KB, optimal  122KB.
META_BLOCK "Z63", Falcom  128KB, simple  124KB, optimal  120KB.
META_BLOCK "Z22", Falcom  128KB, simple  122KB, optimal  118KB.
META_BLOCK "Z10", Falcom  128KB, simple  118KB, optimal  116KB.
META_BLOCK "Z40", Falcom  128KB, simple  118KB, optimal  116KB.
META_BLOCK "Z41", Falcom  128KB, simple  118KB, optimal  114KB.
META_BLOCK "Z42", Falcom  126KB, simple  120KB, optimal  116KB.
META_BLOCK "Z54", Falcom  126KB, simple  120KB, optimal  116KB.
META_BLOCK "Z84", Falcom  126KB, simple  120KB, optimal  116KB.
META_BLOCK "Z51", Falcom  124KB, simple  118KB, optimal  114KB.
META_BLOCK "Z74", Falcom  122KB, simple  118KB, optimal  114KB.
META_BLOCK "Z73", Falcom  120KB, simple  114KB, optimal  112KB.


In summary ...

// LoX1 Summary

Uncompressed 49 files, 12562926 bytes.
Falcom       49 files,  6062815 bytes.
SWD5 simple  49 files,  5235179 bytes.
SWD5 optimal 49 files,  5057690 bytes.

// LoX2 Summary

Uncompressed 103 files, 23278006 bytes.
Falcom       103 files,  9026275 bytes.
SWD5 simple  103 files,  8381998 bytes.
SWD5 optimal 103 files,  8140408 bytes.


So, again, we got a huge "win" with Lox1 ... but jeez ... we only-just won enough space in LoX2 in order to add the English translations. 

We were really, really close to needing one of those Super-Duper System Cards that we've talked about for a while!

[seieienbu]

So the translation patches for both games work (presumably) bug free now?  The only thing that's left ( only) is the voice acting?

[elmer]

So the translation patches for both games work (presumably) bug free now?  The only thing that's left ( only) is the voice acting?

Whatever was going on with that one file, it didn't cause any visible on-screen problems, and certainly won't now.

Both games have been completely played-through with no known bugs or glitches.

The only thing left undone on my side is the credits screens for both games (LoX2 has been hacked and is waiting for the final text, LoX1 hasn't been touched yet). They both waiting for the names of the English VO actors.

After the actual VO-insertion, I'll still have to fix up the lip-sync. It's hacked, and I know what to do, but it's going to be a huge, slow PITA to actually do it.

Then there's re-syncing the LoX2 introduction after we've got the new VO in there (that's one of SamIAm's jobs, since that's all in script-code).

Then a final play-through to make sure that it's all working ... either before or during a formal beta-test involving a few volunteers.

Then it's a case of putting together the final patch package.

Then it gets released!

So, lots of little "details" left to do, but both games could basically be released in a week or two if it wasn't for the VO and credits.

[seieienbu]

I hadn't even thought of matching voices up with the character animations.  That sounds quite tedious.  Will you include credits for the voice actors?

[elmer]

I hadn't even thought of matching voices up with the character animations.  That sounds quite tedious.

Hopefully the VO actors will hit the mark on their timings, which will significantly reduce the work, but there are also quite a few cases where we've deliberately focused on the quality of the dialog in the knowledge that the timings *can* be fixed.

It's not like the Japanese original is perfect, but I really want to do that best that I can. 

Will you include credits for the voice actors?

That's still up for discussion.

We've tried to avoid intruding ourselves as much as possible into Falcom's game, and there will be the technical problems if we try to extend the credits passed the end of the music ... but I'm sure that we can come up with some happy-compromise.

[NightWolve]

Impressive work on the codec. David S. also mentioned that he found an optimization for the LZ codec Ys IV and Emerald Dragon use (the decompression algorithm differs by 1 line of code, technically), and gained slightly a little more compression power than the version Hudson used in development.

I actually used his Hudson exact matching version because of a bug I introduced and initially blamed on his optimization work, so he made another version to behave exactly the same as Hudson's, which is what the Ys IV English patch ultimately went live with.

I never had to trim anything AFAIK, all text blocks fit by the end of it, so I didn't bother using it until Emerald Dragon. The needed trimming solution here is to stop repeating the name of the character in subsequent messages (like you have done for the Xanadus I noticed), and that hopefully satisfies the needs for most text blocks, if not minor rewrites to shorten things here and there worst case. Ys IV never printed the name of a character per text box, so guess that made fitting things easier too versus ED.

[elmer]

David S. also mentioned that he found an optimization for the LZ codec Ys IV and Emerald Dragon use (the decompression algorithm differs by 1 line of code, technically), and gained slightly a little more compression power than the version Hudson used in development.

I actually used his Hudson exact matching version because of a bug I introduced and initially blamed on his optimization work, so he made another version to behave exactly the same as Hudson's, which is what the Ys IV English patch ultimately went live with.

Yep, we talked about this a while back! 

https://www.pcengine-fx.com/forums/index.php?topic=19674.msg425790#msg425790

Unless you've fixed the search-window offsets since then, or switched to the code that I posted, then you're going to be throwing-away some compression opportunities in ED! 

Actually, when I was looking at the fast-CD-read routines in the Anearth Fantasy Stories code that bonknuts was talking about, I had a look at their AFS compression codec.

It looks like it's the same one that he talked about from Gate of Thunder with some weird pre-loading of the compression window.

Jeez ... it's one of the slowest decompressors that I've seen in ages! 

It would be amusing to code-up compressors for the Ys IV, ED & AFS methods and let them loose on the LoX data and see how they do.

I'm definitely curious to see if Hudson's pre-loading has any noticeable effect on real-world data.

The needed trimming solution here is to stop repeating the name of the character in subsequent messages (like you have done for the Xanadus I noticed), and that hopefully satisfies the needs for most text blocks, if not minor rewrites to shorten things here and there worst case.

That wasn't us ... that was in the original game. No memory savings for us in LoX by omitting those names, I'm afraid.

[NightWolve]

Yep, we talked about this a while back! 

https://www.pcengine-fx.com/forums/index.php?topic=19674.msg425790#msg425790

Unless you've fixed the search-window offsets since then, or switched to the code that I posted, then you're going to be throwing-away some compression opportunities in ED!  :-

Ah yeah, I shared our YsIV/ED text codec there. Yeah, maybe I should look at your alternate code and give it a whirl some time. My past inclinations were more to wanna upgrade, improve, mess around even when it isn't broken which got me in trouble in real jobs and fan projects... So I'm hesitant in older age to do so, you know how it is. ;) When I was in my late 20's and could still pull all nighter debugging sessions, I was more eager... Can't do that anymore, heh.

The needed trimming solution here is to stop repeating the name of the character in subsequent messages (like you have done for the Xanadus I noticed), and that hopefully satisfies the needs for most text blocks, if not minor rewrites to shorten things here and there worst case.

That wasn't us ... that was in the original game. No memory savings for us in LoX by omitting those names, I'm afraid.

Ah, so you gained nothing as I will by copying that style for ED.



David was pretty sure this [above] represented the biggest text block, and the elimination of repetition of character names is how I got it to fit! My system has a check as well: the size of the original Japanese text block is known, and if a new English compressed text block exceeds it, insertion is aborted. Same as I had worked out for Ys IV. I pre-null the whole block as well, don't want trailing bytes leftover from when it was Japanese. But yeah, that's what did it, I didn't edit anything away of Sam's (if memory serves).

[elmer]

My past inclinations were more to wanna upgrade, improve, mess around even when it isn't broken which got me in trouble in real jobs and fan projects... So I'm hesitant in older age to do so, you know how it is. ;)

Absolutely! If it ain't broken, then there's no need to change. 

What I'm worried by, after looking at the code that you posted, is not really that you're missing out on a small amount of compression, especially when you don't seem to need it.

It's that it looks like you've got a bug in there. 

From what I'm seeing in your code, it looks like the only difference between the YsIV compression and Emerald Dragon compression, is that minimum offset/distance for a match has changed from "-1" to "-18".

The thing is ... as far as I can see ... you haven't actually stopped [tt]check_match()[/tt] from returning matches that are closer than -18 away.

If so, when you're inserting more of the scripts, that's going to blow-up on you at some point, .

David was pretty sure this [above] represented the biggest text block, and the elimination of repetition of character names is how I got it to fit!

Those are looking really good! 

Just sayin' ... this 30th Anniversary Year would be a great time to release the translation.  :- 


***********************

It would be amusing to code-up compressors for the Ys IV, ED & AFS methods and let them loose on the LoX data and see how they do.

I'm definitely curious to see if Hudson's pre-loading has any noticeable effect on real-world data.

OK, curiosity got the better of me, so I tested the different compressors on the LoX2 data ...
[tt]
Total uncompressed data size            : 23,278,006
Total YsIV compression                  : 10,347,187
Total Emerald Dragon compression        : 10,280,860
Total Anearth compression (with preload) :  9,716,445
Total Anearth compression (no preload)  :  9,706,570
Total Legend of Xanadu 1 compression    :  8,937,897
Total Legend of Xanadu 2 compression    :  8,324,785
Total Elmer's SWD4 compression          :  8,114,319
Total Elmer's SWD5 compression          :  8,035,016[/tt]

Looking at that, I'm pretty unimpressed with the Hudson's "preload" trick ... it costs a bunch of extra time to set up, it forces the use of a slow window-buffer during decompression, and it doesn't seem to save much space (in this case it actually increases the size by a tiny amount).

[dshadoff]

It's that it looks like you've got a bug in there. 

From what I'm seeing in your code, it looks like the only difference between the YsIV compression and Emerald Dragon compression, is that minimum offset/distance for a match has changed from "-1" to "-18".

The thing is ... as far as I can see ... you haven't actually stopped check_match() from returning matches that are closer than -18 away.

If so, when you're inserting more of the scripts, that's going to blow-up on you at some point, .

As I recall, this was just the difference in how the algorithm worked - same algorithm as Ys IV, but with a different offset.  0x12 also sounds familiar.  It's a circular buffer, so doing an "overflow" into the back end of the buffer should also be valid, in the extremely remote chance that a match is actually found.

But the best proof of not blowing up is to encode and decode and verify that the strings match 100%.  As I recall, we also did that.

...I just hope that my memory isn't playing tricks on me as I grow older.  And you might indeed have found a bug, but it likely wouldn't yield a measurably different compression rate (just the chance of garble as you mention)

Anyway, if we didn't encode/decode/validate back then, it's easy enough to do now.

Just sayin' ... this 30th Anniversary Year would be a great time to release the translation.    

Yes it would.
NightWolve, have you decided to start taking another look at digging into this ?

-Dave

[elmer]

As I recall, this was just the difference in how the algorithm worked - same algorithm as Ys IV, but with a different offset.  0x12 also sounds familiar.  It's a circular buffer, so doing an "overflow" into the back end of the buffer should also be valid, in the extremely remote chance that a match is actually found.

But the best proof of not blowing up is to encode and decode and verify that the strings match 100%.  As I recall, we also did that.

...I just hope that my memory isn't playing tricks on me as I grow older.  And you might indeed have found a bug, but it likely wouldn't yield a measurably different compression rate (just the chance of garble as you mention)

Yep, no chance of a crash, just the possibility of garbled data on decompression, depending upon how the ED code actually works on the PCE.

If they're using a 256-byte window buffer, just like you are in the compression, then it probably wouldn't make any difference ... but if they're doing that, then there would be little point in using the 18-minimum-offset in the first place.

The devil-is-in-the-details, and I don't know ... I have no idea where the decompression code is located in ED.

It's just a worry, and not a certainty. As you say, more tests might be a good idea if you're not 100% sure.

Just sayin' ... this 30th Anniversary Year would be a great time to release the translation.    

Yes it would.
NightWolve, have you decided to start taking another look at digging into this ?

Now's your time to get into the Sam's queue. 

When/while the dub is recorded, he's likely to have some downtime while waiting for things to happen, particularly when I get to the stage of fixing the lipsync.

I know that he'd like to get the Spriggan2 dub luanched, but there may be limits on how far that can go if bonknuts continues to be MIA.

This is a perfect time to push ED forward and get all of his current-finished work inserted and tested and verified so that he'll want to finish-off the rest of the translation.

If other-folks don't keep him busy finishing off these old projects, then I'll have to find some way to keep him busy so that he doesn't get bored! 

BTW ... it was interesting to see that person on the NeoGaf forum wax lyrical on some of the PCE RPGs, including Emerald Dragon, and "Angel's Poem".

I had absolutely no idea that that was the English translation of "Tenshi No Uta"!

[dshadoff]

BTW ... it was interesting to see that person on the NeoGaf forum wax lyrical on some of the PCE RPGs, including Emerald Dragon, and "Angel's Poem".

I had absolutely no idea that that was the English translation of "Tenshi No Uta"!

Actually, I've always thought of it as "(The) Song(s) of (the) Angel(s)" - "Uta" could mean poem or song, depending on context.  I don't know which is more correct in this particular case (because I haven't played enough of the game), but I personally would associate angles with music rather than poetry...

[ccovell]

Euclid and Pythagoras are what come to mind for me, but whatever floats your boat, Dave.  ;D

[elmer]

Euclid and Pythagoras are what come to mind for me, but whatever floats your boat, Dave.  ;D

I'd read right past the typo ... hahaha ... good catch!   

[Mathius]

I can't tell you guys how excited I am! I've owned this game twice attempting to muddle my way through to no avail. Now that may become a reality. Thank you all how are/will be involved.

[elmer]

I can't tell you guys how excited I am! I've owned this game twice attempting to muddle my way through to no avail. Now that may become a reality. Thank you all how are/will be involved.

I think you might have meant to post that in the "auditions" thread ... this one only gets the "boring" posts! 

I find it hard to fathom that some folks have actually played though LoX1 without understanding Japanese, even with a walkthrough.

But it was pretty-amazing when I played it in English!   

[Mathius]

I can't tell you guys how excited I am! I've owned this game twice attempting to muddle my way through to no avail. Now that may become a reality. Thank you all how are/will be involved.

I think you might have meant to post that in the "auditions" thread ... this one only gets the "boring" posts! 

I find it hard to fathom that some folks have actually played though LoX1 without understanding Japanese, even with a walkthrough.

But it was pretty-amazing when I played it in English!   

No I meant it for here though I almost did post it over there.

 If I had the equipment I'd definitely give a performance a good attempt.

[elmer]

While I've enjoyed improving my compression code, it's always humbling to be reminded that you're not the smartest cookie on the block! 

Here's how 3 other popular (and free) compression algorithms do with the LoX2 data ...

Total uncompressed data size             : 23,278,006
Total YsIV compression                   : 10,347,187
Total Emerald Dragon compression         : 10,280,860
Total Anearth compression (with preload) :  9,716,445
Total Anearth compression (no preload)   :  9,706,570
Total Legend of Xanadu 1 compression     :  8,937,897
Total Legend of Xanadu 2 compression     :  8,324,785
Total Elmer's SWD4 compression           :  8,114,319
Total Elmer's SWD5 compression           :  8,035,016
Total Jørgen Ibsen's aPLib compression   :  7,782,457
Total Pasi Ojala's puCrunch compression  :  7,780,261
Total Magnus Lind's Exomizer             :  7,377,469

[ccovell]

If you don't count the continentals, you win!! 

[elmer]

If you don't count the continentals, you win!! 

Haha ... those darned Nordic programmers with too-much-time-indoors during their long Winter! 

Exomizer and puCrunch have too much decompression overhead for my comfort, and we're generally not *that* desperate for space on the PCE, IMHO.

But aPLib is fascinating to me ... it's a pure, fast, LZSS-variant, just like my SWD code ... but Jørgen Ibsen came up with a better encoding scheme than mine. 

<edit>

I can foresee an SWD6 in the near future!

I came, I saw, Jørgen conquered! 

It's not just the encoding, which I can recreate, I think that he's also doing some output-optimization of the emitted codes.

He's beaten me! 

So, if I need a good compressor in the future, I'll probably end up using aPLib!

[elmer]

Well ... whoops! We forgot to translate the maps that you pick up in LoX2! 

They've been extracted now, and we're looking at getting the text translated, and maybe see about improving the contrast and color choices at the same time.

Falcom were using some very strange colors in those maps that just don't make a lot of sense.

When they're done, it'll be time to share another pretty picture and keep this thread alive! 


***************************

In other news, a post on romhacking by Tengai-Makyou-translator Tom made me think about the text printing in the LoX games, and why the "slow" text speed seems like such a waste-of-time, and I always immediately switch to the "fast" text speed.

He's got a good point, and looking at the text printing in the LoX games, it doesn't flow like speech, it just gets printed out, one character after another.

So, there's no reason to look at it in "slow" mode, and I always change it to "fast" instead.

I took a look at Anearth for comparison, and they're definitely adding pauses in their printing, especially when you get both text and voice at once. It looks a lot better.


So ... I've added a couple of script-codes to Falcom's script language, and made some changes to my script-compiler, and now both LoX games actually feature natural speech delays at the end of sentences and for other punctuation.

The message boxes just look a whole lot nicer now, IMHO, and I actually prefer the "slow" text mode, with the delays. It just "reads" better. 

The delays are removed when you select "fast" text mode, so anyone that just wants to button-through stuff can still get the same quick response as before.

[ccovell]

Wow, that's attention to detail. 

[incrediblehark]

I really appreciate this addition! While I am a fairly quick reader, I prefer the text to scroll as i'm reading it as opposed to quickly appearing on screen. I find the dialogue sounds better in my head as I read it and the pauses in speech are a nice touch.

[elmer]

While I am a fairly quick reader, I prefer the text to scroll as i'm reading it as opposed to quickly appearing on screen. I find the dialogue sounds better in my head as I read it and the pauses in speech are a nice touch.

It's definitely a big help in making the text interesting to read, IMHO. You can actually imagine someone speaking it as you read.

It also adds that immediate extra "perception-of-quality" when you first start playing the games, even if you turn it off later on.

I can't remember how common an effect it was BITD, but IIRC I don't really remembering it being used until later on in the 4th generation, or maybe even early on in the 5th generation.

But it's not always a good idea, and it doesn't suit every game. It would have driven you mad in the Ys games, with their bump-into-someone-to-talk, because you'd be constantly waiting for text to appear as you accidentally run into people.

Wow, that's attention to detail. 

Thanks! 

Yeah, part of the idea here is to show just how darned-good the PCE is, which means putting in all the UI tricks that you wouldn't normally see on the TG-16, because neither NEC nor TTi nor Working Designs could afford to spend the time & money to change the original Japanese game too much.


The question is ... when to stop messing around with stuff?

I could probably backport LoX2's nice textured message boxes back into LoX1 ... but is that a step too far?

LoX1 doesn't display text in as many colors as LoX2 does, but it has most of them available.

That's another possible change that we could make to the scripts. But should we?


Now that we might be in the position, amazingly enough, to have a small fighting-chance of actually putting together a dub that could rival, or even beat, Ys I & II ... where should these little "upgrades" stop? 

[Michirin9801]

I can't remember how common an effect it was BITD, but IIRC I don't really remembering it being used until later on in the 4th generation, or maybe even early on in the 5th generation.

EarthBound did that on the SNES! It had a fairly robust text display system where some words or phrases would appear slower or faster than normal depending on not only who was talking but what they were saying, sometimes they'd also space out the characters in a word in order to 'make a big deal' of it, the small pauses between phrases were also a thing in that game and they made the game's text feel much better paced than usual for 16 bit RPGs, and that's not to mention the Mr. Saturns who had their own font...
I don't like reading a whole lot in video games, but I always found that reading was a lot more fun in EarthBound than in most other games!

[Gredler]

was a lot more fun in EarthBound than in most other games!

SMMMMAASSHHHHH

[Michirin9801]

was a lot more fun in EarthBound than in most other games!

SMMMMAASSHHHHH

I've actually played EarthBound before Smash, so when I unlocked Ness I knew exactly where he was from ;3

[Gredler]

I've actually played EarthBound before Smash, so when I unlocked Ness I knew exactly where he was from ;3

I played it at launch, loved it and kept my copy since then. It's a great game, and does great stuff with the way text is displayed - I agree - and figured the Smaaash during the battles when you get a critical is a great simple example.

I was referring to the way that when you get a critical hit a special font with a special text-reveal displaying the text SMMMAAAASSHHHH occurs, which I thought was the most apparent example of the way they create emotion in the player with how the text is displayed and revealed/animated.

The screen flashing helps (I feel like there was screen shake too), but I always got excited when I got a critical hit more than any other text/turnbased rpg because of that SMAAASSHHH. I cant find a youtube video of what I am talking about

I've never played a whole lot of smash brothers, that's after my time in the local multiplayer age range hahah

[elmer]

EarthBound did that on the SNES!
...
I don't like reading a whole lot in video games, but I always found that reading was a lot more fun in EarthBound than in most other games!

Hmmmm ... I just took a look at a YouTube video of Earthbound.

So they using a small (5 pixel wide) VWF font on a solid background, and they're printing the text pretty darned fast, but with pauses, sometimes.

Both LoX games print a lot slower than that, at a much more comfortable reading speed.

SamIAm posted this pcefx-forum private teaser (i.e. please do not share this) a long time ago that shows the game's slow text speed (with the beeps), and its fast text speed later on (no beeps).

A lot has changed since this was made, but the printing speeds are the same.


wayback.youtu.be/C2Z7fagH5Yw


*************

Here's an animated mockup to see what LoX1 would look like with the LoX2 textured background and drop-shadow.

Remember ... LoX1 lets you select whether to use a transparent box and outlined bold text, or a solid box and narrow text (like LoX2).




Does anyone have an opinion?

[Michirin9801]

Transparent boxes? How are they printing transparent dialogue boxes?
Also, I can't stress enough how beautiful LoX2 looks~
The text speed seems fine too...

Oh and, the textured BG for text in LoX1 would be a pretty damn good addition! I think it makes the dialogue boxes less boring-looking (not that they look boring or anything)

I've actually played EarthBound before Smash, so when I unlocked Ness I knew exactly where he was from ;3

I played it at launch, loved it and kept my copy since then. It's a great game, and does great stuff with the way text is displayed - I agree - and figured the Smaaash during the battles when you get a critical is a great simple example.

I was referring to the way that when you get a critical hit a special font with a special text-reveal displaying the text SMMMAAAASSHHHH occurs, which I thought was the most apparent example of the way they create emotion in the player with how the text is displayed and revealed/animated.

The screen flashing helps (I feel like there was screen shake too), but I always got excited when I got a critical hit more than any other text/turnbased rpg because of that SMAAASSHHH. I cant find a youtube video of what I am talking about

I've never played a whole lot of smash brothers, that's after my time in the local multiplayer age range hahah

Oh yeah, I had kinda forgotten that... I mean most people I talk to only know EarthBound because of Smash Bros, and when they say "Smash" they're more often than not referring to Smash Bros, so I kinda just went with it...
You're right about the "SMAAAASH" critical hits btw...

I played a LOT of Super Smash Bros. on the N64 as well as Melee, but the main driving force of my enjoyment of the series had been the "Nintendo Fangasm" you know what I mean? And by the time I played Brawl that feeling had already died down, so I just played the Subspace Emissary story mode and then pretty much never again... Smash just isn't my kind of fighting game, it sure is fun, but it's no Street Fighter...

[elmer]

Transparent boxes? How are they printing transparent dialogue boxes?

As you might guess ... the dialogue boxes are actually sprites.

Also, I can't stress enough how beautiful LoX2 looks~
The text speed seems fine too...

Thanks! We haven't made any changes to the game window itself ... Falcom's graphics are beautiful.

All the "improvements" that we've made (since that video) have been purely in the UI side of things.

The slow text speed is a nice "reading" speed for conversations that you haven't seen. But I could certainly understand some people wanting to switch to fast text if they're less interested in the dialog.

Oh and, the textured BG for text in LoX1 would be a pretty damn good addition! I think it makes the dialogue boxes less boring-looking (not that they look boring or anything)

I know what you mean. Large single-color boxes often just look dull and rather 8-bit (IMHO).

But ... it was still pretty common in the 16-bit era.

[ParanoiaDragon]

I definitely like the textured boxes more, looks nicer that way!

[Michirin9801]

Transparent boxes? How are they printing transparent dialogue boxes?

As you might guess ... the dialogue boxes are actually sprites.

Actually, that makes it even more mind-boggling to me, I mean, I'd think they'd use part of the BG layer to render them and the text due to how much easier I think that would be, I mean, that way they could just swap out the empty tiles for the tiles with letters and numbers, but they're doing that with sprites? Are they changing parts of the sprite on-the-fly with code or what? Each letter can't possibly be its own sprite, that would cause a whole lot of flickering...

[elmer]

Actually, that makes it even more mind-boggling to me, I mean, I'd think they'd use part of the BG layer to render them and the text due to how much easier I think that would be, I mean, that way they could just swap out the empty tiles for the tiles with letters and numbers, but they're doing that with sprites? Are they changing parts of the sprite on-the-fly with code or what? Each letter can't possibly be its own sprite, that would cause a whole lot of flickering...

It's just a design-choice. 

When you're dealing with 12x12 kanji text, instead of simple 8x8 or 8x16 English/Katakana/Hiragana, then you've got to deal with packing 2 glyphs into 3 tiles-wide.

You can either do that with dynamic-drawn 8x8 tiles, or dynamically-drawn 16x16 sprites.

Either way is a PITA, and neither is that much more difficult than the other.

With the LoX games, they're basically creating a 256x64 bitmapped sprite that they can draw the text into.

Other games may use a 256x64 bitmapped set of tiles to draw into ... or one of the many little variations on that.


Generally, the Japanese code will deal with text coordinates as either on 4-pixel boundaries, or on 8-pixel tile boundaries plus an even/odd flag.

Both of those are easy(ish) to hack in a bi-width English font, since there's minimal changes to the underlying Japanese code. But you need spare RAM for the English font, and the table of character widths.

Going to a full VWF is a much more tricky proposition, because you've got to change the game's whole text coordinate system to 1-pixel boundaries.


But, whatever you do, you still end up dynamically uploading text glyphs into sprites/tiles as the message box is drawn.

It's not really any different to uploading new data for animated backgrounds or animated sprites.


In some ways, the big & huge sprite-based text box is easier to deal with, because you don't have to make sure that the screen scroll is always tile-aligned, you don't have to worry about masking out any other sprites that may be underneath the message box, and you don't have to fixup the background data after the box is removed.

[Michirin9801]

It's just a design-choice. 

When you're dealing with 12x12 kanji text, instead of simple 8x8 or 8x16 English/Katakana/Hiragana, then you've got to deal with packing 2 glyphs into 3 tiles-wide.

You can either do that with dynamic-drawn 8x8 tiles, or dynamically-drawn 16x16 sprites.

Either way is a PITA, and neither is that much more difficult than the other.

With the LoX games, they're basically creating a 256x64 bitmapped sprite that they can draw the text into.

Other games may use a 256x64 bitmapped set of tiles to draw into ... or one of the many little variations on that.


Generally, the Japanese code will deal with text coordinates as either on 4-pixel boundaries, or on 8-pixel tile boundaries plus an even/odd flag.

Both of those are easy(ish) to hack in a bi-width English font, since there's minimal changes to the underlying Japanese code. But you need spare RAM for the English font, and the table of character widths.

Going to a full VWF is a much more tricky proposition, because you've got to change the game's whole text coordinate system to 1-pixel boundaries.


But, whatever you do, you still end up dynamically uploading text glyphs into sprites/tiles as the message box is drawn.

It's not really any different to uploading new data for animated backgrounds or animated sprites.


In some ways, the big & huge sprite-based text box is easier to deal with, because you don't have to make sure that the screen scroll is always tile-aligned, you don't have to worry about masking out any other sprites that may be underneath the message box, and you don't have to fixup the background data after the box is removed.

I see... This is all some pretty complicated stuff, but I think I can better get the picture now...
I always thought that drawing dialogue boxes was better with the BG layer than tiles because then you'd free up sprites that you can use to do something else and prevent flickering, and also because I wasn't exactly aware that you could update a sprite's data dynamically other than animating it replacing it in its entirety with another sprite that you have to have stored either in ROM or in RAM, but apparently sprites are more flexible than I thought...

[elmer]

I always thought that drawing dialogue boxes was better with the BG layer than tiles because then you'd free up sprites that you can use to do something else and prevent flickering, and also because I wasn't exactly aware that you could update a sprite's data dynamically other than animating it replacing it in its entirety with another sprite that you have to have stored either in ROM or in RAM, but apparently sprites are more flexible than I thought...

Overlap and flickering are always things to be concerned about.

It's one good reason to use a fullscreen wide solid box if you're going to use sprites.

If you look at LoX1, you'll see that they never bring up a text box when there are huge amounts of enemy sprites on the screen.

But they wanted the transparent status bar on the side (which has to be sprites), so they were already allowing VRAM for a large set of status-bar sprites.

Other games like to draw multiple cascaded text boxes on the screen. That pretty-much has to be done in BG layer on the PCE.

It's all about making a careful decision between the tradeoffs. Everything has a cost.

And yep, you can modify sprite data on the fly, just like you can background tiles ... it's all just VRAM.

The only thing that you can't change during the frame is the sprite-attributes themselves.

The question is ... when to stop messing around with stuff?

I could probably backport LoX2's nice textured message boxes back into LoX1 ... but is that a step too far?

LoX1 doesn't display text in as many colors as LoX2 does, but it has most of them available.

That's another possible change that we could make to the scripts. But should we?

Hmmmm ... even though using LoX2's textured message box backgrounds in LoX1 looks like it's probably technically possible, and it looks quite pretty ... there are other concerns that make it look like it really is a "step too far".

In particular, the changes snowball down to the Weapon Shop text boxes, and then the Weapon Shop status bar as well.

You can't really do one box without doing them all, and doing them all would be making too many graphical change changes to the LoX1 game. Neither SamIAm nor I feel that it is our job to just override Falcom's original game in that way.

I'm OK with tweaking palettes so that stuff looks a little better on modern output devices; and anything to do with dialog text itself is fair-game, but we really don't want to make major changes to the in-game graphics.

[Michirin9801]

The only thing that you can't change during the frame is the sprite-attributes themselves.

Umm, what attributes? >w>

The question is ... when to stop messing around with stuff?

I could probably backport LoX2's nice textured message boxes back into LoX1 ... but is that a step too far?

LoX1 doesn't display text in as many colors as LoX2 does, but it has most of them available.

That's another possible change that we could make to the scripts. But should we?

Hmmmm ... even though using LoX2's textured message box backgrounds in LoX1 looks like it's probably technically possible, and it looks quite pretty ... there are other concerns that make it look like it really is a "step too far".

In particular, the changes snowball down to the Weapon Shop text boxes, and then the Weapon Shop status bar as well.

You can't really do one box without doing them all, and doing them all would be making too many graphical change changes to the LoX1 game. Neither SamIAm nor I feel that it is our job to just override Falcom's original game in that way.

I'm OK with tweaking palettes so that stuff looks a little better on modern output devices; and anything to do with dialog text itself is fair-game, but we really don't want to make major changes to the in-game graphics.

That's fair, if you don't feel like you should be altering the game's graphics like that then don't! The flat text boxes look fine as they are...

[elmer]

The only thing that you can't change during the frame is the sprite-attributes themselves.

Umm, what attributes? >w>

The "sprites" that you see on the screen are made up of two parts ... the graphical data that sits in VRAM, and makes up the different pixel values, and the attribute data that is internal to the VDC chip, that says where to put the sprite on the screen, how big it is, what palette to use, and where in VRAM to read the pixel data from.

You can change the graphics data in VRAM at any time, but you can't modify the attribute data directly.

You've got to upload copies of the attribute data into VRAM (which you can do at any time), and then tell the VDC chip to load up a new set of attribute data from VRAM at the start of the next display frame.

That's pretty-much how sprites work on nearly-all of the 4th-generation and earlier machines.

There's usually a very clear separation between the sprite graphics and the sprite attributes.

[Michirin9801]

The only thing that you can't change during the frame is the sprite-attributes themselves.

Umm, what attributes? >w>

The "sprites" that you see on the screen are made up of two parts ... the graphical data that sits in VRAM, and makes up the different pixel values, and the attribute data that is internal to the VDC chip, that says where to put the sprite on the screen, how big it is, what palette to use, and where in VRAM to read the pixel data from.

You can change the graphics data in VRAM at any time, but you can't modify the attribute data directly.

You've got to upload copies of the attribute data into VRAM (which you can do at any time), and then tell the VDC chip to load up a new set of attribute data from VRAM at the start of the next display frame.

That's pretty-much how sprites work on nearly-all of the 4th-generation and earlier machines.

There's usually a very clear separation between the sprite graphics and the sprite attributes.

Oh I see... I didn't know that, that explains a lot!

[elmer]

This is shaping up to be BETTER than the original presentation.

Insane.

Well ... we are certainly trying to polish some of the UI elements that deal with English text so that they're both a bit clearer and also work better on modern display technologies (without compromising their display on an old TV).

The maps are a case-in-point of that.

They're also a great example of trying to figure out just where the line is in terms of what we're willing to change, and what is us taking that "step too far".


In LoX2, you get given a map for the current level at some point, and you can look at it from the Inventory section of the Pause Menu.

Now, it's easy to miss that Inventory section itself, because there's no graphical hint that you can scroll in that direction on the Pause Menu, but that's a battle for a different day. 

Here's what it looks like on the first level, with me about to select the icon for the classic rolled-up crinkly old faded parchment map ...




And here's what you see next ...




Eeek! That looks a bit strange! 

We've still got the dark grey border from the previous screen, but now there is a razor-sharp (not crinkly) scanline transition to a brown map, with an interior crinkly border, and some strange-looking pinky-purple hues for "fading", plus some even stranger-looking green hues for the "fading" on the town name.

Not only that, but you've got blue-stippled shading on the inland part of the island that looks perilously close to the color of the rivers and ocean.


Hmmm ... well, after some thought, I guess that the pinky-purple and green bits are probably an attempt to take advantage of the limited color bandwidth of a composite TV signal, and produce more of a subtle brightness change. I've seen that trick used a few times before ... but mostly in the 8-bit era.

But these days, when using a modern high-resolution display with an RGB-modded PCE, or when running on an LCD display with an emulator, those color choices just look, errrr ... "not good" (IMHO).


So, with only some palette changes, and without changing Falcom's main map graphics (except for the English names), here is a version that still tries to keep the faded-out and yellowing look, but fix both the strange color-changes, and the inland vegetation color.



Now, there is still the question of whether this is too much of a change from the original graphics, but I currently believe that this both honors Falcom's original intent and style, and also gives us something that will look good on both old composite tube TVs, and modern high-resolution displays.

[CrackTiger]

That map looks great and doesn't feel out of place. After seeing the original next to it, only the original feels like it's from another game.

[Michirin9801]

The map looks perfectly fine to me!

[Gredler]

Ship it!

[ParanoiaDragon]

Wow, offhand, I don't recall ever knowing there was a map, though, it's been, maybe 10 years or so since I beat LoX2?

[esteban]

Now, there is still the question of whether this is too much of a change from the original graphics, but I currently believe that this both honors Falcom's original intent and style, and also gives us something that will look good on both old composite tube TVs, and modern high-resolution displays.

There are some art techniques that are directly tied to the medium, and it is crazy that we are at point where we can decide whether "purity" of experience should be in (1) original canvas CRT or (2) modern canvas.

This modified map (which looks fantastic, by the way), really is appropriate in 2017, since it is true to the *original intentions* of the designers (they, obviously, wanted it to look nice on the displays mainstream consumers would be using).

Preserving every pixel for historical accuracy is one thing (and should be done).

But the current LoX project is about MOAR than that


DISCLAIMER:
**There will always be debate about how to interpret the original intentions/aspirations of the original designers**