Moral Recordings

Making the Plagiarism Machine Write Code Still Counts as Plagiarism

2026-05-27T00:00:00+08:00

I have been struggling to put into words my frustration with the open source community in failing to meet the present moment.

In case you've missed the last three years: AI labs have strip-mined the internet, robbing creators of their work and selling it back to the public as a machine designed to crush labour. Thanks to their relentless lobbying of businesses and governments, there is immense pressure on workers to offload their cognition onto AI products backed by a Large Language Model, on the false pretense that the model is a reliable source of knowledge.

(For avoidance of doubt, a reliable source of knowledge doesn't habitually make fraudulent citations, and doesn't invent answers then spout them off with 100% confidence. These behaviours are referred to as hallucination; a problem which researchers from OpenAI and Anthropic admit is intrinsic of LLMs and essentially not fixable.)

It is no surprise that AI's biggest success story to date has been in computer programming, a field where a lot of people have no professional standards and are allergic to solidarity. A lot of attention has been paid to new projects that have embraced vibe coding; usually for their oafish levels of complexity, repetition and serious design flaws. Less attention has been paid to established open-source projects, where in an attempt to avoid corporate pressure or alienating key contributors, some level of LLM-driven development has been accepted.

Everywhere I look, there are big projects opening the door to LLM-generated code submissions. The Linux kernel. systemd. Kubernetes. Firefox. Python. Rust. Even ScummVM, a project I contribute to, has added such a policy. All perhaps motivated by different reasons, but collectively they provide a broad social license to ignore all ethical concerns and normalise the presence of LLM code generation.

I wish I was shocked at the lack of a visible resistance movement in this space as there has been with writing and visual arts, but that would be in denial of the extreme free-market libertarianism that has dominated software/the tech sector for the last 50 years. There has always been a level of dry rot visible from space, and now we get to watch part of the house collapse from its own weight.

There are plenty of reasons to despise and oppose the frontier AI labs. For example, the cartoonish level of waste found at every stage of LLM technology. From the billions of investor cash set alight for training and inference and warehouses of unused GPUs, all the way to unhinged multi-agent cack factories like Steve Yegge's Gas Town, a project which openly boasts "you won't like [it] if you ever have to think, even for a moment, about where money comes from".

Other good reasons: the constant AI lab doom-mongering about sentience and mass unemployment to goose their valuation and land more marks, the worldwide breakdown of trust and consensus reality as their slop permeates everything, the end of discoverability as the internet drowns, and their willingness to sell their tech for accelerating mass surveillance and genocide.

In addition, the AI labs have been so good at subsidising demand that the major cloud providers have committed hundreds of billions toward new datacenter spending. These datacenter projects are best known for siphoning massive amounts of water and energy away from communities, creating atmospheric and noise pollution, inflating the worldwide cost of essential parts like memory chips, and adding a destabilising amount of construction debt to the global financial system.

And of course, the legal question is still open on whether training an LLM on copyright-infringing works is itself an act of infringement, or whether LLM-generated works are copyrightable at all. Early rulings by the US Copyright Office have stated that works where "the expressive elements are determined by a machine" are not eligible for copyright, and lobbyists are desperately trying to get new laws to firm up their position before the wave of lawsuits fully breaks.

All of these are important and justified grounds for opposing the onslaught of AI companies into every facet of life, and have been written about by people more eloquent and better-read than myself.

To keep things brief, let's talk about the ethics of using any large language model trained on all published open-source code to generate new code. Just that case! Not provide advice on punctuation and spelling, not work as a version of American Fuzzy Lop that costs 20 grand a throw, not the mostly-transformative use cases that are wheeled out to justify all LLM usage being good. New code.

We know on a fundamental level that frontier models are a laundromat for existing content, and AI labs have focused on randomising the output enough to make direct plagiarism claims harder to prove. We know that the frontier model companies go to extreme lengths to take everything, including material clearly licensed or marked by the author as not for use in LLM training. Even the so-called "open weight" LLMs, the golden boy of bikeshedded ethical LLM theory, have opaque training sets stuffed to the gunnels with scraped content that fails the same ethical test. We know that training a personal model with just stuff you wrote is physically not enough data to get a workable result, and plenty of charlatans will sell you this false promise and fake it with a frontier model.

My view is that generating code wholesale with a frontier LLM, then sticking your own name and copyright notice on it, is a massive slap in the face and should be treated with contempt, regardless of how easy or expected it is. Hundreds of thousands of developers contribute to the commons, often on their own dime, in exchange for attribution and making the world a better place. Working for exposure! And you can't even do that all of a sudden? That's now too hard? Spare me.

Having the LLM generate a merge request wholesale for an existing project combines the above with a second slap in the face. There's plenty of debate about how many lines of unvetted changes still counts as polite to throw at a maintainer. I would start at 0. Why should a maintainer bother to read a changeset that you couldn't be bothered to write, and most likely won't be able to answer follow-up questions on?

As a maintainer of ScummVM's Macromedia Director support, where the aim is to recreate the precise ordering and edge-cases of the original engine, the number of lines changed doesn't even matter. It could be one line! The work is in the bench-testing that has been done to confirm how the original engine does it, and incorporating it in a way that doesn't break other games or other versions of Director. In this context, LLM generation would be a pretty strong signifier that no thought was put into the manner or consequence of these changes outside of getting a specific game to work, which is against the project's goal of universal compatibility.

For what it's worth, it is possible to both loathe LLM-driven development, and not get mad at non-developers using them in a personal capacity to solve a code problem. They are a tiny fraction of the overall usage. My ire is firmly directed at the self-described software engineers talking up their love of scabwork.

But here we are. A lot of prominent developers decided they want to slam on the treat button ad nauseam and still call it engineering, and the united front is dead before it ever had a chance to form. That's life. My fault for expecting better.

I feel for every single person who has had their future ripped out from under them, and I want no part of it. The overwhelming urge has been to wind up all my coding projects and move to a hobby which is harder to ruin with slop.

Still, we've been here before. This isn't the first time a technology has been hyped to the world with the promise of improved productivity, then prove to be a disappointing (and now load-bearing!) albatross that drinks money. Nor will it be the last! Just look at the number of businesses still welded to Oracle Forms; management paid top dollar for that asbestos and by God it's going to stay in those walls until they collapse.

Maybe this is a flash in the pan and developers will regain their senses once LLM pricing is dragged kicking and screaming toward what it costs to run. Maybe the trauma of the oil crisis smashing into the impending debt collapse will be enough to spark a new renaissance, and we will collectively rediscover the joy of what is only possible through human labour. But until then, all we can do is look at the comments.

"I have never been so productive" they say, drunk on an indecipherable colossus they've spent $5000 to build, marveling at all the time they used to waste writing their own words of intimacy or comfort. "Claude writes much better code than I ever did" they say, face still notionally human but with a deadness behind the eyes, long ago having smothered any dreams of self-improvement or professional responsibility. "You obviously haven't tried the latest model" they say, turgid with excitement for the heat death of everything, celebrating as the sky is blotted out by their poisonous exhaust.

Can you complete the Oregon Trail if you wait at a river for 14272 years: A study

2025-01-11T00:00:00+08:00

If you're into retro computing, you probably know about Oregon Trail; a simulation of the hardships faced by a group of colonists in 1848 as they travel by covered wagon from Independence Missouri to the Willamette Valley in Oregon. The game was wildly successful in the US education market, with the various editions selling 65 million copies. What you probably don't know is the game's great untold secret.

Two years ago, Twitch streamer albrot discovered a bug in the code for crossing rivers. One of the options is to "wait to see if conditions improve"; waiting a day will consume food but not recalculate any health conditions, granting your party immortality.

From this conceit the Oregon Trail Time Machine was born; a multiday livestream of the game as the party waits for conditions to improve at the final Snake River crossing until the year 10000, to see if the withered travellers can make it to the ruins of ancient Oregon. The first attempt ended in tragedy; no matter what albrot tried, the party would succumb to disease and die almost immediately.

A couple of days before New Years Eve 2025, albrot reached out and asked if I knew anything about Apple II hacking. In reality the answer was no, I knew three things about the Apple II:

It has a MOS 6502 processor
It was popular in the US educational market
Something something Carmen Sandiego something Prince of Persia?

But all old computers are basically the same right? Specialist knowledge is for cowards.

Where to start

First I loaded Oregon Trail into MAME's Apple II emulator. MAME is an emulator that was originally targeted at arcade hardware, however it has support for hundreds of game consoles and home computer systems thanks to merging with the MESS project. It has one of the nicest debuggers I've ever used, with plenty of documentation, which is why I strongly recommend it for reversing work. You will need a copy of the Apple II ROMs, listed in the MAME ROM index as apple2e.zip.

We are using the most common disk image of the game (aka "peperony and chease" edition). To open the game:

mame apple2e -debug -flop1 oregonmod_a.do

This will open a separate Debug window and pause execution until you start running the machine (F5). In the game window you can activate the MAME menu keys by pressing Scroll Lock; useful ones to know are TAB for the main menu, F6 for saving a state, F7 for loading a state.

This was my thinking for how to get a first toehold into what was going on. As you travel, the game gives your party one of four health ratings:

good
fair
poor
very poor

It follows that the game code must read the text string "poor" from somewhere in memory so it can be printed on the screen, and this code would be attached to the routine that determines how sick everyone is. So I sent the party off from Independence, Missouri with 1 ox, no food and no clothes. As soon as we were on the road, I made a save state. I then called "Step Into" in the debugger (F11) which paused execution, and made a dump of the Apple II's memory to a file with the following command:

save /tmp/mem_before,0,0x10000

I loaded this dump into a hex editor and searched for the word "poor", which I found at address 0xA6AE. Then in the debugger I added a watchpoint for any code attempting to read the first byte of "poor":

wpset 0xA6AE,1,r

As expected, once my party became emaciated enough the debugger halted execution at the following instruction:

DB47  LDA ($5E), Y

The Apple II CPU is a 6502, which has a nice instruction reference here. Looking it up, we can see that LDA basically means "load a byte from the address stored at 0x005E into register A". Because the 6502 is a very basic 8-bit CPU, you are limited in how you access the 16-bit address space. Generally speaking you have the choice of using a fixed offset (up to 16-bit, baked in the instruction) plus whatever 8-bit value is in a register, or using a Zero Page pointer. The above instruction is using a Zero Page address; for this mode you have to write a 16-bit pointer containing the location you want to somewhere in the first 0x100 bytes of memory, then use an instruction which references that pointer.

So we know that the pointer to the text describing the player's health is at 0x005E, which we can confirm by opening a memory window in the debugger and see AE A6 at address 0x005E (the 6502 is little endian, so addresses are stored backwards). It's fair to say that this is just for updating the display, and not for checking how ill everyone is.

We need to find the code that sets this status, so I added another watchpoint for any writes to the pointer at 0x005E:

wpset 0x005E,1,w,wpdata == 0xAE

After reload the save state, the game stopped in another area:

E630  STX $5E       ; mem[0x005E] = X

I started using the debugger's Step Out feature (Shift+F11) to move up the call stack and see what was calling what. From further poking around it became apparent that we were in some sort of generic high level text printing function. Then a penny dropped, and I started wondering about the memory layout of the Apple II. A memory map shows the D000–F7FF range to be the part of the ROM containing the BASIC interpreter. Oh.

In fact there's a very helpful annotated Apple II ROM disassembly, showing that function is indeed a high-level print function for text.

Well that just got complicated

So 1985 Oregon Trail is written in Applesoft BASIC, and the program is stored as some sort of bytecode. My heart sank a little at this news; 6502 assembly is cumbersome to understand at the best of times, and throwing a BASIC virtual machine on top of that makes live debugging even worse. I didn't have time to hack together a way to trace the BASIC code as it executed.

However this did open up a new angle of attack. A BASIC program uses variables, which are stored in a known location in a known format. What if we let the game play, then kept an eye on the debugger's live memory view to see what variables change over time?

The good news is that Applesoft BASIC keeps the original names for each variable; a critical clue. The downside is that those names are (at most) two letters long. Also the memory is managed dynamically, so variables will move around as the program chugs on.

From poking around the ROM disassembly, the pointer to the variable store is at address 0x0069 of the Zero Page (usually 0x9902), followed by a pointer to the array store, and a pointer to all of the string data. Each variable is 7 bytes long; 2 bytes for the name (using the high bits of each to encode type), and 5 bytes data. The default numeric type in Applesoft BASIC is a cursed 40-bit floating point format, with an 8-bit exponent and a 31-bit mantissa.

After a bunch of test runs I found a variable called H, which increased in line with my health status getting worse. Even better, setting the H data to all zeros reset the status back to "good", after which it seemed to decay back to "very poor" at the previous rate. There was also a H1 array that seemed to keep track of party members that had died (0 for alive, -1 for dead), and a H2 array which seemed to keep track of whatever exciting disease each party member had. Easy! Open and shut!

Did we make it?

New Years Eve 2025 had arrived, and with it attempt 2 to reach the barren wastelands of future Oregon. This time, the time machine stopped at 16120 AD.

A dismal failure. Even when zeroing out the memory, every day H would reset to 139, dooming the party to a short-but-agonizing fate. Cheating death is harder than expected.

We can still save this

Several days later, I tried writing a scrappy decompiler for the Applesoft BASIC bytecode. From past experience I was worried this would be real complex, but in the mother of all lucky breaks the "bytecode" is the original program text with certain keywords replaced with 1-byte tokens. After nicking the list of tokens from the Apple II ROM disassembly I had a half-decent decompiler after a few goes.

(This ultimately turned out to be unnecessary. If I had looked more closely at the options for CiderPress II, I would've seen that you can use the "import" and "export" commands to convert Applesoft BASIC files to and from plain text).

So of course we had to look for the code that calculates the player's health. Here it is, from OREGON TRAIL:

...
# 3200
LET H0 = C0
FOR L = C0 TO C4
IF H1(L) > C0 THEN H2(L) = H2(L) - C1
LET H0 = H0 + C1
IF H2(L) < C1 THEN H1(L) = C0

# 3205
NEXT
IF  RND (C1) < P5 THEN TM = QT +  INT ( RND (C1) * 41)
LET PP = ( RND (C1) < QP)
LET W = INT ((TM + 10) / 20)
LET TM = W

# 3206
LET TR = C0
LET TS = C0
IF PP THEN Z = ( RND (C1) < .3)
LET W = 6 + Z + Z
LET TR = .2 + .6 * Z
IF TM < C2 THEN W = W + C1
LET TS = 8 * TR
LET TR = C0

# 3207
LET ZT = TM - C3
IF ZT < C0 THEN ZT = C2 - TM

# 3210
LET ZC = 5 - TM - TM - OP
IF ZC < C0 THEN ZC = C0

# 3215
LET X = (ZC > P5)
LET Y = (PF = C0)
LET ZF = F0
IF Y THEN ZF = 8

# 3220
LET ZP = (W > 5) + (W > 7) + P + P

# 3225
LET Z = FS * P5
IF X OR Y THEN Z = FS + .8

# 3230
LET FS = Z
LET H = .9 * H + ZT + ZC + ZF + ZP + FS + H0 + HR
LET PF = PF - FC
IF PF < C0 THEN PF = C0

# 3235
IF  NOT W1 AND H > 139 THEN  GOSUB 10300
INVERSE
...

Constants:

C0: 0.0
C1: 1.0
C2: 2.0
C3: 3.0
C4: 4.0
P5: 0.5

Inputs:

NP: number of party members alive.
P: pace. 1: steady, 2: strenuous, 3: gruelling
PF: pounds of food left. Copied over from I(8).
R: rations. 1: filling, 2: meagre, 3: bare-bones
W: weather. 0: very cold, 1: cold, 2: cool, 3: warm, 4: hot, 5: very hot, 6: rainy, 7: snowy, 8: very rainy, 9: very snowy

Calculated factors:

FC: pounds of food consumed. FC = NP * (4 - R)
ZC: clothing misery factor. ZC = 5 - W * 2 - (clothes/NP)
ZF: food misery factor. ZF = 2 * (R - 1), or = 8 if you have no food left
ZP: pace misery factor. ZP = (W > 5) + (W > 7) + P + P
ZT: temperature misery factor. ZT = 0 if W is warm or cool, otherwise the number of steps away from warm or cool.
FS: food starvation factor. FS = FS + 0.8 if there's no food or if ZC > 0.5, otherwise FS = FS * 0.5.
H0: seems to be H0 = 5.0 at calc time?
HR: hardship factor. HR = 10 if the trail is rough, HR = 20 if someone just died, 20% chance of HR = 20 if the water is bad, 10% chance of HR = 10 if there's very little water

Output:

H: party health score. 0-34 is good, 35-69 is fair, 70-104 is poor, 105-139 is very poor. Capped at 139. H = .9 * H + ZT + ZC + ZF + ZP + FS + H0 + HR

The inputs were figured out by checking what text the game prints on the screen for those variables; e.g. W is used to fetch a string from an array W$ that looks like this:

[0xa0ee]  5780250001000a  W$[10] =
[0xa0f8]  09efa6  - b'very cold'
[0xa0fb]  04eba6  - b'cold'
[0xa0fe]  04e7a6  - b'cool'
[0xa101]  04e3a6  - b'warm'
[0xa104]  03e0a6  - b'hot'
[0xa107]  08d8a6  - b'very hot'
[0xa10a]  05d3a6  - b'rainy'
[0xa10d]  05cea6  - b'snowy'
[0xa110]  0ac4a6  - b'very rainy'
[0xa113]  0abaa6  - b'very snowy'

The health factors are all conjecture based on the inputs, but I think they make sense. We have the formula for H, and there's only one misery factor which would be affected by waiting 14272 years: the food starvation factor. In case you're wondering what that looks like:

[0x9bb0]  4653967e4f4234  FS = 4166608.55078125

Yikes. We can see that every day after crossing Snake River, H would increase by at least 4.1 million, and then be rounded down to the worst possible health score of 139. It would take 15 days of doing nothing but sitting and eating to bring the food starvation factor down to 127, by which point everyone in the party would be dead.

But finally, FINALLY we know how to distort reality enough to get this train back on the rails. The party is back to full health, all that's left is to get past the final fork at The Dalles and...

Did we make it to Oregon???

Piss. I couldn't find a technical contact for MECC, so I dug around and saw line 50050 was in TRADE.LIB; namely the code responsible for drawing the inventory. Error 53 is the Apple II error code for GS/OS: parameter out of range. But what could possibly be wrong with the inventory? It was fine on the road!

Oh. Wait.

[0x9d29]  49802200010009  I$[9] =
[0x9d33]  000000  - b''
[0x9d36]  0512a9  - b'Wagon'
[0x9d39]  040ea9  - b'oxen'
[0x9d3c]  10fea8  - b'sets of clothing'
[0x9d3f]  07f7a8  - b'bullets'
[0x9d42]  0ceba8  - b'wagon wheels'
[0x9d45]  0be0a8  - b'wagon axles'
[0x9d48]  0dd3a8  - b'wagon tongues'
[0x9d4b]  0ec5a8  - b'pounds of food'
[0x9d4b]  49003400010009  I[9] =
[0x9d57]  ffffffffff  - -1.7014118342085515e+38
[0x9d5c]  ffffffffff  - -1.7014118342085515e+38
[0x9d61]  8310000000  - 4.5
[0x9d66]  8528000000  - 21.0
[0x9d6b]  8a09000000  - 548.0
[0x9d70]  8100000000  - 1.0
[0x9d75]  8100000000  - 1.0
[0x9d7a]  8100000000  - 1.0
[0x9d7f]  8200000000  - 2.0

The game was kind enough to gift me -170 billion billion billion billion wagons.

Actually this was a red herring. Line 50050 is also on Side B inside END.LIB, aka. the screen before the crash (choosing between floating down the Columbia River or taking the Barlow Toll Road).

And guess what.

It reads the date:

# 50050
[0x023d]  POKE 900,NP
[0x0245]  POKE 901,AY - 1800
[0x0252]  POKE 902,AM
[0x025a]  POKE 903,AD

AY is the variable containing the year, and the POKE instruction tries to shove the value of AY - 1800 into a single byte of the regular Apple II memory, causing a crash as it is much larger than the maximum of 255.

These POKEs seem to be how Oregon Trail saves key information when it switches between BASIC programs. Most of the game is handled by the main program "OREGON TRAIL" (written by John Krenz), which loads in modules like "RIVER.LIB" and keeps access to all the variables. Whenever a new main program is loaded in, all the BASIC variables are lost except for whatever data you happened to copy to regular memory with POKE. At the end of the game you have the option of floating down the river, which uses a new main program "FLOAT" (written by Steven Splinter) before switching again to the "WIN" program, so it follows that the game would need to use the POKEd numbers.

What happens if we mod the BASIC code in memory to say something else?

...
POKE 901,255
...

Finally we have a definitive answer about whether this whole ordeal is possible. Unlike every other progress screen in the game, the ending screen has the year hardcoded to start with "18", followed by the printed text of whatever PEEK 901 is. The maximum number of years you can take without crashing the game is 207, and any attempts longer than 51 years will end in the wrong century.

Really, there wasn't much point going on; it felt like further trickery went against the spirit of the challenge. Maybe there's closure enough in knowing that if you somehow avoid starvation for 142 centuries, the game dicks you at the last possible moment by expecting the year to be sensible.

Then again we're 98% of the way there

I lied. Here's a improved version of Oregon Trail side B with the following quality-of-life changes:

Waiting by a river for conditions to improve resets the food starvation factor to 0.
The final sequence uses two bytes to store the year.

Many thanks to the CiderPress II team for making it easy to load replacement code onto the floppy.

Conclusions

Applesoft BASIC is majestically slow, but thanks to 40-bit floating point and standard commands for drawing and text manipulation, Oregon Trail works much better than I could have expected after running the simulation for 14272 years. The final screen being broken is a bit of a letdown, but understandable given how unlikely it is that you'd take 14272 years to finish the game.

Honestly, one of my aims has been to create a reverse engineering approach that will work for any system, regardless of knowledge or experience, and give you the basic steps for how to understand and modify it. This was a bit of a test run of this, and I think it could be considered a success. Eventually I'll write it down properly, but hopefully this effort will give you some ideas for your next reversing project.

And albrot? Well, he was able to fulfill his dream and become the first person to survive for 15000 years on the Oregon Trail. A well deserved victory.

If you're interested in the tools I wrote for this work, the Applesoft BASIC decompiler and variable scraper I wrote are available here. Happy trails!

Show Your Working: Making Powerline Ethernet faster

2024-02-25T00:00:00+08:00

Having wired networking is nice! The current iterations of Wi-Fi are admittedly very fast, but I feel a bit iffy trusting that the signal will always work at a good latency, when there's rogue slabs of concrete and surprise microwave sources to deal with.

My house has fiber internet which terminates at the wireless router I keep under my TV. To enable mission-critical house activities (e.g. streaming from my studio to Twitch/Diode Zone, using Steam Link from the couch), I have some Powerline Ethernet adapters which link the router to my studio PC upstairs. And for the past several years, they have been indistinguishable from magic; somehow they turn 1970s home electrical wiring into Cat 5e!

Until the rubber finally met the road; I tried to watch a Blu-Ray on my Home Theatre PC, streamed from my Studio PC, and it had big buffering pauses every 60 seconds. An early speed test I did showed the link speed for streaming the Blu-Ray was only 24 Mbps! But for some reason, downloading things from the studio PC was absolutely fine! At this point I realised I knew very little about how the magic boxes worked, or how I would go about isolating the problem.

What is Powerline Ethernet?

The Powerline Ethernet adapters I have are a pair of Netgear PL1000 extenders, which (amusingly) offer "up to Gigabit speeds". These adapters are based on the HomePlug AV2 standard (later rolled into IEEE 1901), which uses the live-neutral and live-ground pairs of your house's mains wiring as communication channels. Which is to say, the transmitter in the adapter generates a radio-band AC signal and sends it into the wiring with a coupling capacitor, allowing it to overlay the thumping great 50 Hz wave of the mains current. The receiver in the adapter filters out all the non-radio AC frequencies and tries to decipher a working signal from what's left. As for other mains devices, they should be filtering their power and ignore this as a little bit of line noise.

The trick that makes this possible is our friend Orthogonal Frequency-Division Multiplexing; basically if you have an analogue medium as a channel, you can cram multiple digital signals into that channel using carefully overlapped frequency bands and fancy error correction. For HomePlug AV2 the frequency bands in use range from 1.6 MHz to 86 MHz; you might recognise these numbers as "the end of AM radio" and "the start of FM radio", or if you are an Australian RF engineer "the end of getting sued by Macquarie Media" and "the start of getting sued by Southern Cross Austereo".

All well and good, but how do we test performance? Ethernet is a protocol running on top of this physical channel layer. None of the losses or error correction are exposed as packet level messages; all we get is the apparent link speed being slower or faster.

Testing Powerline Ethernet

After a bit of trial and error, I used a little tool called iperf3 to measure the maximum bandwidth on the link, with various combinations of things plugged in. Early on it became apparent that the speed in one direction was determined by the amount of noise at the receiver end; so in this case, something on my router's powerboard was dumping out enough RF slop to slow down the Blu-Ray data streaming from my studio PC. But which thing, and by how much? Let's find out!

Is it the surge protector?

One of the many axioms about Powerline Ethernet that frequently comes up is "for the love of God never run it through a surge protector". I assume this advice comes from a place where single-plug wall outlets are banned and lightning doesn't exist. Given that there's still a signal getting through, my gut feeling was to downgrade this advice to "never run it through a crap surge protector". But we can do some tests to prove the actual impact, even if we intend to ignore it. Here are speed tests for the bare outlets with/without a surge protector inline; both surge protectors are recent model HPM 12-socket powerboards, with the powerline adapter in the socket closest to the cable.

Speed tests with a surge protector
Router surge protector	Studio surge protector	Router -> Studio speed	Studio -> Router speed
No	No	196 Mbits/s	141 Mbits/s
No	Yes	186 Mbits/s	135 Mbits/s
Yes	No	186 Mbits/s	144 Mbits/s
Yes	Yes	168 Mbits/s	131 Mbits/s

So it's about a 10% speed hit. Sounds good to me! It's almost as if the designers knew that powerline communication was an electrical standard and designed their filtering circuits to cope.

Is it one of my appliances?

Now to test the impact of individual devices. My house is wired entirely with single-plug outlets (why would you do this???), so the intention is to keep using powerboards on both ends. The baseline setup is:

my full studio setup online and unchanged
the Powerline Ethernet adapter, an ASUS router and a fiber NTD plugged into the router-side powerboard

As the issue was happening at the router-side receiver, I decided to try plugging in combinations of the devices that make up my home theatre system.

Speed tests with combinations of appliances
Amp	TV	MiSTer	PC	Router -> Studio speed	Studio -> Router speed
No	No	No	No	108 Mbit/s	147 Mbits/s
No	No	No	Yes	108 Mbit/s	114 Mbits/s standby, 93.1 Mbits/s on
No	No	Yes	No	107 Mbit/s	122 Mbits/s standby, 108 Mbits/s on
No	Yes	No	No	109 Mbit/s	142 Mbits/s standby, 112 Mbits/s on
Yes	No	No	No	103 Mbit/s	104 Mbits/s standby, 38 Mbit/s on
Yes	Yes	Yes	Yes	105 Mbit/s	91 Mbit/s standby, 40 Mbit/s on

There it is! My problem was that I wanted some sound to go with my Blu-Ray, unaware that my digital amplifier has a side-gig of spewing out RF pollution all day.

The class-D amp in question came with this unbranded 24V 4A DC power brick. Like all mystery power bricks it is adorned with The Runes: CE certification, the C-Tick mark for ACMA approval, and an approval number that suggests TUV certified it. I could find none of these numbers in the public certification database, but we can at least be at ease that whoever sold this power brick owned a sticker printer.

Does replacing this with a legit Australian-certified power supply fix the issue? I got a replacement from Altronics; notably it is heavier, has bigger capacitors and features a ferrite choke on the lead. My enthusiasm for knowledge did not extend to cracking open the sealed cases to compare circuit boards, but filtering components are usually the first things that get stripped for cost reasons as end-users tend not to notice.

Connecting to host 192.168.1.4, port 5201
Reverse mode, remote host 192.168.1.4 is sending
[  5] local 192.168.1.8 port 53286 connected to 192.168.1.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.25 MBytes  52.4 Mbits/sec
[  5]   1.00-2.00   sec  7.50 MBytes  62.9 Mbits/sec
[  5]   2.00-3.00   sec  8.75 MBytes  73.4 Mbits/sec
[  5]   3.00-4.00   sec  9.12 MBytes  76.5 Mbits/sec
[  5]   4.00-5.00   sec  7.38 MBytes  61.9 Mbits/sec
[  5]   5.00-6.00   sec  6.88 MBytes  57.7 Mbits/sec
[  5]   6.00-7.00   sec  7.12 MBytes  59.8 Mbits/sec
[  5]   7.00-8.00   sec  6.25 MBytes  52.4 Mbits/sec
[  5]   8.00-9.00   sec  7.12 MBytes  59.8 Mbits/sec
[  5]   9.00-10.00  sec  7.75 MBytes  65.0 Mbits/sec
[  5]  10.00-11.00  sec  7.25 MBytes  60.8 Mbits/sec
[  5]  11.00-12.00  sec  7.75 MBytes  65.0 Mbits/sec
[  5]  12.00-13.00  sec  9.12 MBytes  76.5 Mbits/sec
[  5]  13.00-14.00  sec  9.38 MBytes  78.6 Mbits/sec
[  5]  14.00-15.00  sec  9.12 MBytes  76.5 Mbits/sec
[  5]  15.00-16.00  sec  8.88 MBytes  74.5 Mbits/sec
[  5]  16.00-17.00  sec  9.25 MBytes  77.6 Mbits/sec
[  5]  17.00-18.00  sec  8.62 MBytes  72.3 Mbits/sec
[  5]  18.00-19.00  sec  8.75 MBytes  73.4 Mbits/sec
[  5]  19.00-20.00  sec  8.62 MBytes  72.3 Mbits/sec
[  5]  20.00-21.00  sec  8.50 MBytes  71.3 Mbits/sec

Much better. Interestingly you can see what looks to be the adaptive error correction optimising for the new noise profile.

Conclusions

So, lessons learned:

The receiving speed of Powerline Ethernet is determined by the amount of RF noise at the receiving end.
Recent surge protectors and powerboards are fine.
Plugging in devices at the receiving end will nibble away at the speed but not by much, IF they use a certified power supply that doesn't cut corners!
If you're experiencing weird or asymmetrical drops in link speed, try unplugging all of the power supplies to isolate the problem.
Consider replacing no-name DC power bricks with certified ones, or adding a ferrite choke to the lead.

Very special thanks to @adonm for identifying the source of the problem in one guess.

Show Your Working: Forcing Linux to print at actual size

2023-05-19T00:00:00+08:00

One day I decided I've had enough of computers and wanted to do literally anything else. Specifically, in that moment I wanted to print out some sewing patterns. Here is a 100mm test square I printed on my printer. Except it's only 94mm. 94mm!!!

Not going to lie, searching the internet for useful information has become real bleak lately. There's plenty of advice floating around to "Open the PDF in Acrobat and select "Actual size" when printing", but no advice about the Linux equivalent. To be honest the coward's way out would be to dig out the sacrificial Windows laptop, find a print driver and use Acrobat Reader, but I figured other people might like to know how to fix it, so here you go!

In order to figure out what's going on, let's have a bit of a recap of the Linux printing stack and the problem space in general. Printing under Linux is handled by CUPS, a userspace daemon responsible for connecting to printer hardware and feeding it the right slurry of bits. In this context, a printer "driver" is a PrinterName.ppd file (PostScript Printer Description), containing a bunch of information about the hardware, what features are supported, what slurry types to feed it, and so forth. Seperate to that, there is a printers.conf file listing each active printer and how it is connected; locally (LPT, USB) or via the network (IPP, JetDirect, SMB).

Now for the longest time, all of this configuration was manual. There were large collections of PPDs (such as foomatic) containing definitions for thousands of different printer models, usually not the one in front of you. More often than not you'd have to settle for one of the generic drivers, which produce a slurry that a lot of printers eat such as PCL6. If you're really unlucky the printer would need a proprietary driver; this would take the form of a PPD file plus a binary blob executable responsible for generating special Brand-Name Printer Slurry.

But here we are, with the Dark Age of Printing long behind us. In 2013 the Printer Working Group developed the IPP Everywhere standard, which Apple (amongst others) decided was the future for printing from mobile devices, and so has become a standard for consumer printers. Finally, network printers could enunciate exactly the type of slurry they would like and remove the need for drivers once and for all! You no longer had to suffer like your printing ancestors did.

Or so the theory goes. The printer I'm using is a Fuji Xerox DocuPrint P265 over IPP. For printing I'm using lpr, the CUPS-provided tool for sending files to the print spooler, with print-scaling set to none. In this scaling mode, content should be centered to fit on the page, with no resizing. This means e.g. an A4 sized document printed on A4 paper should be perfectly to scale and have about 5mm chopped off by the hardware page margins. When printing from an application such as Evince, the correct setting for "actual size" printing is Page Handling -> Page Scaling -> None.

First I tried printing a PDF with a document size of exactly 100mm x 100mm.

$ lpr 100mm.pdf -o print-scaling=none

The result was a square in the center of the page with a 94mm edge.

I tried printing another PDF in A4 with a box at the edge of the page, along with a 100mm square in the center. This produced an interesting tell; the square printed with a 94mm edge, but the page box showed up too! This implies that something was flat-out ignoring the page margin of the printer hardware. And yet surely the IPP standard would account for that!

After a bit more digging I discovered the driverless command bundled with CUPS, which connects to an IPP printer and generates a PPD file:

$ driverless ipp://192.168.1.2/ipp/print
*PPD-Adobe: "4.3"
*FormatVersion: "4.3"
*FileVersion: "1.28.17"
*LanguageVersion: English
*LanguageEncoding: ISOLatin1
*PSVersion: "(3010.000) 0"
*LanguageLevel: "3"
*FileSystem: False
*PCFileName: "drvless.ppd"
*Manufacturer: "FX"
*ModelName: "FX DocuPrint P265 dw"
*Product: "(FX DocuPrint P265 dw)"
*NickName: "FX DocuPrint P265 dw, driverless, cups-filters 1.28.17"
...

This all looks very sensible, let's see what it has to say about paper dimensions.

*DefaultPageSize: A4
*PageSize A4/A4: "<</PageSize[595.275590551181 841.889763779528]>>setpagedevice"
*DefaultPageRegion: A4
*PageRegion A4/A4: "<</PageSize[595.275590551181 841.889763779528]>>setpagedevice"
*DefaultImageableArea: A4
*ImageableArea A4: "12.245669291339 12.245669291339 583.029921259842 829.644094488189"
*DefaultPaperDimension: A4
*PaperDimension A4: "595.275590551181 841.889763779528"

Yep, there's a margin. It's in weird printer units but there's definitely about 5mm carved out from the edge of the full page. What gives?

Oh wait, CUPS makes its own cache of the PPD doesn't it. Maybe mine is slightly di

$ cat /etc/cups/ppd/FujiXerox.ppd
*PPD-Adobe: "4.3"
*%%%% PPD file for Generic IPP Everywhere Printer with CUPS.
*%%%% Created by the CUPS PPD Compiler CUPS v2.3.3op2.
*% (c) 2014 OpenPrinting
*FormatVersion: "4.3"
*FileVersion: "1.0"
*LanguageVersion: English
*LanguageEncoding: ISOLatin1
*PCFileName: "pwgrast.ppd"
*Product: "(Generic IPP Everywhere Printer)"
*Manufacturer: "Generic"
*ModelName: "Generic IPP Everywhere Printer"
*ShortNickName: "Generic IPP Everywhere Printer"
*NickName: "Generic IPP Everywhere Printer"
*PSVersion: "(3010.000) 0"

Oh. Oh no.

*DefaultPageSize: A4
*PageSize A4/A4: "<</PageSize[595.28 841.89]/ImagingBBox null>>setpagedevice"
*DefaultPageRegion: A4
*PageRegion A4/A4: "<</PageSize[595.28 841.89]/ImagingBBox null>>setpagedevice"
*DefaultImageableArea: A4
*ImageableArea A4/A4: "0 0 595.280029296875 841.890014648438"
*DefaultPaperDimension: A4
*PaperDimension A4/A4: "595.280029296875 841.890014648438"

The short version is that (CUPS/GNOME Settings/whatever I used to set it up) has gone down some other, more sinister route to create a PPD file for my driverless printer. One which doesn't involve page margins. But it does have about four times as many exotic media types, on the offchance that I might need to print a swan.

$ sudo su
$ cd /etc/cups/ppd
$ systemctl stop cups
$ mv FujiXerox.ppd FujiXerox.ppd.bak
$ driverless ipp://192.168.1.2/ipp/print > FujiXerox.ppd
$ systemctl start cups

After swapping out the generic PPD for the one created by driverless, the 100mm square is now 100mm. We did it!

In closing, there was a fantastic bit the late John Clarke did in his show The Games, where a bumbling contractor has to justify to the Sydney Olympic Committee that it's fine the 100 metres track he built is only 94 metres long. I'd include a clip, except the ABC decided to copyright strike it. Oh well! Now to give up computers for good this time.

Intro to ROM Hacking: Ghost House

2020-06-09T00:00:00+08:00

A long time between updates! I've been distracted by working on ScummVM's Macromedia Director engine, but it's mostly bugfixing and I couldn't find a good focal point. So on a whim I made this! The previous video took maybe a week, this was a good test of writing/filming/editing in two days.

In this video Scott takes apart "Ghost House" by Sega, and figures out how to remove the obnoxious screen flashing effects by reverse engineering the Z80 code. This contains a full step-by-step explanation of how to use the MAME debugger for hacking ROMs, even if you don't know assembly language!

Let's Reverse: Adventures of Lomax graphics

2019-09-25T00:00:00+08:00

I guess I'm a YouTuber now. Wonder how long I can fight this newfound urge to tase a dead rat.

In this video Scott takes apart "Adventures of Lomax" by Psygnosis and reverse engineers the graphics data. This didn't require taking apart the code, and contains a step-by-step explanation of the black-box analysis process. A good introduction for anyone getting started in reversing games!

DDD 2018: Reverse Engineering is Good and also For Everyone

2019-04-20T00:00:00+08:00

Last year I gave my first talk at DDD Perth 2018! It was about reverse engineering, how I think everyone interested should be encouraged to try it, and some useful tips for people starting out. Here is the slide deck.

Apologies that the audio is a bit muffled; the microphone had a recording issue, so instead enjoy some lo-fi camcorder goodness. Thanks again to DDD for having me, and for going the extra mile to accomodate first-time presenters. Extra special thanks to David for salvaging these black box recordings (on a long weekend, no less!).

Using your car as a giant joystick for $20

2017-12-18T00:00:00+08:00

DISCLAIMER: Cars can be dangerous. The electrical and mechanical systems of a car are not designed for this use case, and you run the risk of damaging them. This project worked fine for this particular vehicle, but could damage critical systems such as steering and braking in a different model. Do not attempt this with a vehicle that isn't yours, or a vehicle that would leave you with no contingency if it were to break. The author will not be held accountable for any damage caused from following these instructions.

This project is about how to rig the controls of nearly any recent car (in my case a 2007 edition Mazda 3) to act as a giant game controller. In a nutshell; input from the car will be scraped as CANbus messages from a cheap OBD-II adapter, then converted to standard joystick and keyboard events, which will be used to drive a video game projected onto a screen in front of the windshield. No physical modifications to the car are required; the only one I made was to pull the fuses for the headlights so as not to blind the projector screen.

The full source code used for the above demo is available at GitHub. With luck, anyone with entry-level Python experience should be able to adapt this for their car. I think this is a nice practical introduction to car hacking and reverse engineering, without the need to spend a lot on exotic debugging hardware.

Requirements

A car with a CANbus based internal network and an OBD-II port. Most cars manufactured 2005 and later have this port in either the driver or passenger side footwell under the dashboard. Once you find the port, you should do a search for your car's model, see what internal control networks it has, and how they map to pins on the port.
A bootleg ELM327 OBD-II adapter. These things are everywhere and dirt cheap, you can pick up one on eBay for about 20 dollarydoos. Make sure you get one with a USB cable and not a WiFi/BT one; honking great security risk aside, the chip used in the wireless variant is a different architecture with worse firmware than a stock bootleg ELM327, and troubleshooting will make you go mad/start smashing things up.
A computer running Linux, a projector, and some HDMI/3.5mm audio cables.

Your friend the CANbus

I think it's important that we touch briefly on what CAN is.

Controller Area Network (CAN) is a standard for a twisted wire pair bus designed for the very noisy conditions inside a vehicle.
There's a few different variants in terms of speed and message size, but communication over the bus is done with messages made from a message ID (either 11 or 29 bits) and a payload (usually 2-12 bytes). In general the message ID denotes the type of message.
Unlike OBD-II PID requests, CANbus messages are non-standard; every vehicle model has its own proprietary message formats, usually with multiple sensor readings packed into each message.
Being a multi-master bus, there is no source or destination for messages! Electronic Control Units in the vehicle are constantly listening for/sending out messages thousands of times a second. A pretty ideal medium for sniffing and injecting traffic!

As mentioned above, every communications bus in your vehicle should be accessible through the OBD-II port. If your vehicle's CANbus is exposed on a different set of pins than the standard pair, be prepared to crack the adapter open and resolder some wires, or to buy a premodified adapter. My Mazda 3 has two CANbus networks; a boring medium-speed bus on the standard OBD-II pins, and a high-speed bus where the cool stuff happens on two of the vendor pins. The adapter I bought has a throw switch to toggle which bus to use; these ones are sold as "modified for Ford".

The ELM327 supports autodetection of several common message formats and CAN baud rates, plus has two slots for custom settings.

Protocol mapping (taken from ELM327 datasheet)
'0': Autodetect (everything except User1 and User2)
'1': SAE J1850 PWM (41.6 kbaud)
'2': SAE J1850 VPW (10.4 kbaud)
'3': ISO 9141-2 (5 baud init)
'4': ISO 14230-4 KWP (5 baud init)
'5': ISO 14230-4 KWP (fast init)
'6': ISO 15765-4 CAN (11 bit ID, 500 kbaud)
'7': ISO 15765-4 CAN (29 bit ID, 500 kbaud)
'8': ISO 15765-4 CAN (11 bit ID, 250 kbaud)
'9': ISO 15765-4 CAN (29 bit ID, 250 kbaud)
'A': SAE J1939 CAN (29 bit ID, 250* kbaud)
'B': User1 CAN (11* bit ID, 125* kbaud)
'C': User2 CAN (11* bit ID, 50* kbaud)

And yes, there are some vehicle models which have an OBD-II port but don't use (ISO 15765-4) CAN. If you have one of these, I would suggest searching to check if anyone has had success using the ELM327 with your car model, and from there seeing if you can dump bus messages through the "Monitor All" feature (described below).

Speaking of searching, check to see if anyone on the internet has mapped out the CAN message format for your vehicle. We will be reversing the message format ourselves for kicks, but it's always nice to have reference material and build off the research of others. I was lucky that someone had mapped most of the CANbus format for the Mazda 3, but in the end I didn't really use the information other than to confirm my own findings.

So how's this going to work then?

We are going to make a software bridge in Python that connects between sensors on the car's CANbus, and a virtual joystick made with Linux uinput. uinput is amazing; writing a driver for is literally just calling a function every time you want to press/unpress a button or change the readong on an axis. Better still, in Python you can use pyserial and python-uinput to skip nearly all of the messy boilerplate.

If you're not running Linux that's ok, a live USB of Linux with persistent storage will work just as well for running games and emulators. I imagine a lot of readers right now are internally screaming "yes yes but what about the only OS that ever truly matters which is Microsoft Windows?!?!?". Things there are... considerably harder, as the OS does not come with a easy-to-script userspace input driver. You would need to install a third party driver like vJoy, write a Python shim for the C API it offers or maybe chain it through another input manager like AutoHotkey, then replace all of the calls to uinput with that. I should mention that nearly all of the joystick configurations I made generated keyboard presses for some buttons, which is a seperate Windows API to joystick events but you could use a cross-platform wrapper like PyUserInput. And the other software option would be to write a Windows kernel driver! So that's nice.

(I'm starting to think that the massive popularity of microcontroller boards such as the Teensy, pretending to be standard USB devices so as to piggyback generic driver support, might be related to Windows' userspace driver options being so bad?)

Limitations of cheap hardware

Real talk: the ELM327 is not a proper CAN interface, it's a cheapo OBD-II diagnostic unit with CAN sniffing bolted on as an afterthought. ELM327 based devices are not very good at capturing all of the packets off the CANbus. The read buffer on the chip is a paltry 256 bytes, so in the default configuration the "Monitor All" command (which prints incoming CAN messages until the buffer overflows) will vomit blood and give up after a few seconds.

By contrast, a real CAN interface will allow duplex communication (aka. near-simultaneous read and write) with the CANbus; you can sort of write messages to the CANbus with the ELM327, but it's not duplex and you have to switch modes first and the process is really painful. In Linux there is a whole subsystem devoted to CAN support (SocketCAN), which makes CANbus interfaces show up just like network interfaces (i.e. can0 instead of eth0), and all Linux CAN software (e.g. the can-utils collection) builds on top of this generic layer. A proper CAN interface will have native kernel support for SocketCAN, or provide a serial interface compatible with the barebones SLCAN/LAWICEL protocol.

Anyway. So those downsides make the ELM327 not the greatest adapter ever for CAN reverse engineering, BUT! Doesn't mean we can't do it.

Connecting to the ELM327

I purchased a bootleg ELM327 adapter with a beautifully soldered "Ford" mod switch on eBay for $30. Straight out of the gate, I could not get my adapter to respond in either Linux or Windows on any of the standard serial baud rates, even with the (cracked, commercial) software included on the complimentary CD-R! But the little TX light did flash momentarily after turning it on, which suggested that the adapter was sending out the startup message, and I could make the TX/RX lights flash by sending garbage over the serial connection. Evidently the chip was fine and I was trying to communicate over serial at the wrong baud rate (aka. speed in bits/second), but none of the standard RS232 rates seemed to work!

After a few hours I gave up poking it, stripped the adapter down, shoved some tiny wires down the vias on the main chip's TX and RX lines, and ran them through an oscilloscope. Lo and behold it was communicating at exactly 500000 bit/s! (AKA a completely non-standard speed for RS232 that I never would have guessed in a million years).

Non-standard speeds aren't a bad thing; the datasheet suggested that stock ELM327s run at 38400 bit/s or 9600 bit/s, which are unbearably slow for data capture given our high-speed CANbus is rated for 500kbit/s. But that's still some expert trolling by whoever rigged this, as it rendered the adapter incompatible with pretty much EVERY OBD-II/ELM327 program ever. Maybe they picked 500000 because the HS CANbus was rated for "exactly" that much... except the ELM327 dumps CAN messages in human-readable hexadecimal, which cuts the effective data rate by more than half. Oops.

Once you've figured out the baud rate, you can connect to the device using whatever serial terminal you want. Make sure your user has permission to access the /dev/ttyUSBx interfaces!

Device: Run "dmesg" just after you plug in the USB to see what it comes up as (most likely /dev/ttyUSB0)
Baud rate: As discussed (possibly 38400, 9600, or 500000)
Data bits: 8
Parity bits: None
Stop bit: 1
Hardware flow control: No
Software flow control: No
Extra: Add line feeds

One thing I learnt when doing this project; there are hundreds of open source serial terminals and nearly all of them are bad, or have the same dumb problems. To save time I recommend grabbing ssterm (a.k.a. the only good serial terminal) and using it like so.

scott@hoagie:~$ ssterm /dev/ttyUSB0 -b 500000 --rx-nl cr --tx-nl cr

?

>AT I
ELM327 v1.4

Hitting the enter key once should give you a > prompt. Typing "AT I" and hitting enter should give you the version message. At any time, Ctrl-] stops the terminal. "AT I" is an example of an AT command supported by the ELM327, which you can find in the official chip datasheet/AT Commands list.

Changing the comms speed

The very first thing we're going to do is tempt fate and force the ELM327 to run at the fastest speed it can. I know from the FT232RL datasheet that the USB serial interface is capable of up to 2Mbit/second. According to the ELM327 datasheet, baud rate is stored as a divisor for 4Mbit/second in Programmable Parameter 0C. I confirmed this by looking at the PP summary:

>AT PPS
00:FF F  01:FF F  02:FF F  03:32 F
04:01 F  05:FF F  06:F1 F  07:09 F
08:FF F  09:00 F  0A:0A F  0B:FF F
0C:08 N  0D:0D F  0E:9A F  0F:FF F
10:0D F  11:00 F  12:FF F  13:32 F
14:FF F  15:0A F  16:FF F  17:92 F
18:00 F  19:28 F  1A:FF F  1B:FF F
1C:FF F  1D:FF F  1E:FF F  1F:FF F
20:FF F  21:FF F  22:FF F  23:FF F
24:00 F  25:00 F  26:00 F  27:FF F
28:FF F  29:FF F  2A:38 F  2B:02 F
2C:E0 F  2D:04 F  2E:80 F  2F:0A F

Seems legit. PP 0C is on (N means oN!) and set to 0x08, meaning 4Mbit/sec / 8 = 500kbit/sec.

By default, PP 0C is set to off, which sets the baud rate to the default of 38400. If your ELM327 is running at 38400, you can boost it to 500000 by running the following commands:

>AT PP 0C ON
OK

>AT PP 0C SV 08
OK

>AT Z

Disconnect your serial terminal and you should be able to reconnect at 500000. To revert it back to the default (and make all your ELM327/OBD scanning apps work again):

>AT PP 0C OFF
OK

>AT Z

On my bootleg chip 500000 baud is the fastest you can go; setting the baud rate divisor any lower than 08 will return '?'. That kinda sucks, but whatever.

Teaching Python ELM327ese

I wrote a little Python harness for bootstrapping and communicating over serial with the ELM327. By default, the ELM327 will echo the user's input back to them, and seperate new lines with a carriage return ('\r'). Prompts for input are made with a >. Because it's all terminal-styled plaintext, we have no idea how long to expect a serial message to be; we just know it'll be terminated with a '>' (when ready to receive input again) or a '\r' (for a stream of CAN messages).

As we are both cheap and lazy, we will use pyserial and synchronous IO. If you call read( n ), pyserial will block until there's enough data in the receive buffer to fulfil your request. So the easiest way around that is to repeatedly ask for 1 byte (oh man I die a little on the inside when I think how inefficient this is), then stop when you hit the terminator character.

class ELM327:

    def __init__( self, device='/dev/ttyUSB0', baud_rate='500000', protocol='0' ):
        self.elm = serial.Serial( device, baud_rate )
        self.protocol = protocol


    def send( self, cmd ):
        self.elm.reset_input_buffer()
        self.elm.reset_output_buffer()
        self.elm.write( cmd )
        self.elm.write( b'\r' )


    def recv( self ):
        output = bytearray()
        while True:
            b = self.elm.read( 1 )
            if b == b'>':
                break
            output.append( b[0] )
        return output


    def recv_line( self ):
        output = bytearray()
        while True:
            b = self.elm.read( 1 )
            if b == b'\r':
                break
            elif b == b'>':
                raise EOFError
            output.append( b[0] )
        return output

Now that we have serial comms, we'll have to write a small set of bootstrap commands to get the adapter into a receive-ready state.

def get_prompt( self ):
    # send garbage
    self.send( b'this will reset the prompt' )
    self.recv()


def reset( self ):
    # reset interface
    self.get_prompt()
    self.send( b'AT Z' )
    self.recv()

    # return to defaults
    self.send( b'AT D' )
    self.recv()

    # turn echo off
    self.send( b'AT E0' )
    self.recv()

    # check that information string works
    # (my bootleg responds with 'ELM327 v1.4')
    self.send( b'AT I' )
    ack = self.recv()
    print( ack )
    assert ack.startswith( b'ELM' )

    # set the CANbus protocol used by the car
    self.send( 'AT SP {}'.format( self.protocol ).encode('ascii') )
    self.recv()

    # initialize the CANbus interface
    self.send( b'0100' )
    self.recv()

    # get protocol
    self.send( b'AT DPN' )
    print( self.recv() )

    # next 3 are about disabling the "helpful" autoformatting of CAN messages
    # enable messages longer than 7 bytes (standards are for chumps!)
    self.send( b'AT AL' )
    self.recv()

    # turn on headers (why would you not want those)
    self.send( b'AT H1' )
    self.recv()

    # disable formatting (boo)
    self.send( b'AT CAF0' )
    self.recv()

    # disable whitespace in CAN messages
    # ELM327 has only 256 bytes, so this nets you ~33% extra buffer space!
    self.send( b'AT S0' )
    self.recv()

    # set huuuuge timeout before giving up
    self.send( b'AT STFF' )
    self.recv()


def start_can( self ):
    self.get_prompt()
    self.send( b'AT MA' )

And yes we need the recv() after each command; the ELM327 only buffers input when the prompt is displayed and Python will execute faster than the chip has time to react, so we need to block Python until we get another '>'. All of the above was figured out by trial and error; playing with the ELM327 through a serial terminal helped a lot, plus reading all the AT commands in the ELM327 datasheet and the source code for some ELM327-targeted software (e.g. pyOBD). The ELM327 communicates everything in human-readable text, so I just kept poking at it in the terminal until it produced CAN output in a nice compact format.

Sniffing the CANbus

I mentioned earlier the ELM327 isn't the best for reverse engineering. If you don't mind writing your own sniffer, it's actually alright for basic things like this!

The Python harness will set everything up into "Monitor All" mode. Once we ask the ELM327 for messages, it will begin transmitting them over the serial connection encoded as plaintext hexadecimal; first three characters are the message ID (the CANbus in the Mazda 3 uses 11-bit message IDs; 000 to 7FF), the rest is message body. We will need to quickly decode this back into bytes (groan) so we can scrape the sensor information.

CAN_RE = re.compile( b'([0-9A-F]{3})\\W*([0-9A-F\\W]+)' )
...
    def recv_can( self ):
        msg_raw = self.recv_line()
        msg_m = CAN_RE.match( msg_raw )
        if msg_m:
            msg_id = int( msg_m.group( 1 ), 16 )
            msg_b = bytes.fromhex( msg_m.group( 2 ).decode( 'ascii' ) )
            return (msg_id, msg_b)
        return (-1, b'')

There is a CAN filtering feature in the ELM327 which sounded ideal for cutting down traffic, but I struggled a bit figuring out how it worked. There's two components: a filter (CF) and a mask (CM). Like an idiot I thought it would work like any other mask, as in the test would pass any message ID that ANDs with it and remains the same. It's kind of the opposite: you set the filter to any message ID that passes the test, and set the mask to be 1s for the bits in the filter (1 or 0) that must match with the incoming message ID. So you could implement an AND-style filter by setting CF to your mask and CM to (0x7ff ^ mask).

As it happens, cranking the baud rate and disabling whitespace gave juuuuust enough buffer space for the ELM327 to deal with every CAN message produced by my car. Which is good, because the message IDs I needed were so sprawled I couldn't use filtering.

With that working I rigged up a simple tool that read every CAN message and printed only the changes for each message ID. This is available in the source code repository as elm_scan.py.

There was a little bit of noise; two CAN message types had a clock signal of some kind, and there was some in-place oscillation from (I think) the level sensor in the fuel tank, so I just ignored those specific message IDs. I then performed the highly-advanced reverse engineering technique of wiggling a thing in the car while staring at the screen. If a message showed up, it meant the wiggling had made it in some way to the CANbus, and the changed bits were what I needed to check for in the joystick driver.

Designing a joystick

We're going to be using the car with the ignition set to ON, the engine OFF (duh), transmission in park and handbrake on, so there are physical limits on what controls we can use. In the Mazda 3 power steering isn't available with the engine switched off, but you can still tug on the fixed steering wheel and get a decent force reading left and right. I read some research online which suggested you could get the brake position as a force axis measurement, but I only saw it as an on-off switch with my sniffer. (maybe the ECU only reports force when the engine is running and the pedal isn't locked?) Also to my chagrin, most of the buttons on the steering wheel etc. are wired up directly to an ECU, so those signals never even reach the CANbus! So sad.

UPDATE: someone pointed out correctly that a lot of cars will enable power steering and turn the wheels even if the engine is off. If this is your vehicle, I would strongly advise placing something slippy like a folded tarp underneath each front tyre, else you might end up with some nice bald spots where the rubber meets the bitumen.

Regardless I found some good controls after a bit of sniffing, all on the HS CANbus. I was worried I'd need to rig a second adapter in tandem to collect traffic from the car's second MS CANbus, but there weren't any usable inputs I could find (other than the indicators and the volume control) so I didn't bother.

STEERING ANGLE (range while locked):
4da: 8000c00000000000 - neutral
4da: 80d2c00000000000 - hard right
4da: 7f2ec00000000000 - hard left

ACCELERATOR (range):
201: 000040000000c800 - floored
201: 0000400000000000 - depressed

BRAKE (on/off):
205: 0000400000000000 - on
205: 0000000000000000 - off

CRUISE CONTROL (on/off):
4ec: 8000000000000000 - on
4ec: 0000000000000000 - off

DRIVER'S SIDE DOOR (open/shut):
433: 800450010004f000 - open
433: 000450010004f000 - shut

HIGH BEAMS (on/off):
433: 000450410004f000 - on
433: 000450010004f000 - off

Each CAN message has a bunch of sensors bitpacked into it, so we need some fast routines to extract only the data we care about. I used the struct unpacker from my library Mr. Crowbar (plug, plug!) to do the dirty work:

class Steering( mrc.Block ):
    RANGE = 0x00D2

    axis_raw = mrc.UInt16_BE( 0x00 )

    @property
    def axis( self ):
        return min( max( (255*(self.axis_raw - 0x8000)//self.RANGE), -255 ), 255 )


class Accelerator( mrc.Block ):
    RANGE = 0xC8

    axis_raw = mrc.UInt8( 0x06 )

    @property
    def axis( self ):
        return min( max( (255*(self.axis_raw)//self.RANGE), 0 ), 255 )


class Brake( mrc.Block ):
    button = mrc.Bits( 0x02, 0b01000000 )


class Cruise( mrc.Block ):
    button = mrc.Bits( 0x00, 0b10000000 )


class Controls( mrc.Block ):
    driver_door = mrc.Bits( 0x00, 0b10000000 )
    high_beams = mrc.Bits( 0x03, 0b01000000 )

Of course you could use Python's built in struct module to do the same thing, or hack something together with mask and shift operations. But look how clean and pure the message parser is now!

if msg_id == 0x4da:
    self.steering = Steering( msg_b ).axis
elif msg_id == 0x201:
    self.accelerator = Accelerator( msg_b ).axis
elif msg_id == 0x205:
    self.brake = Brake( msg_b ).button
elif msg_id == 0x4ec:
    self.cruise = Cruise( msg_b ).button
elif msg_id == 0x433:
    obj = Controls( msg_b )
    self.high_beams = obj.high_beams
    self.driver_door = obj.driver_door

Cruise control is a latched on/off switch, and the driver's side door is a bit unwieldy to use as a pressable button, so I had to wrap those to emit a single button press when the state changed. Bit annoying you can't register them as a long press, but whatever.

You can see the results in mazda3_joystick.py. It's pretty basic; there's a base class called Mazda3 storing all the state, with a method update( msg_id, msg_b ) that tries to parse each incoming CAN message, and if there's a match the state gets changed and an update of the uinput controls is triggered.

Obligatory live action demo

Showing is better than telling, so I taped myself testing a pile of games on my car joystick with a projector. Apologies for the quality; I am not into streaming and don't have a convenient arsenal of broadcast cameras and studio mics laying around, so you get to savour the lo-fi goodness of a single unlit 720p camcorder gaffer-taped to a car headrest.

In order, the games I tested were:

I had to make almost as many joystick configurations as there were games. A common thing I had to tweak was the range of the accelerator pedal; even if you define a uinput axis with a range of (0, 255), nearly all games expecting an analogue stick will renormalize that to something like (-255, 255). The solution was to size the uinput axis as (-255, 255) but create an enormous dead zone from (-255, 0). I was glad the event processing loop was Python code as it meant I could do stupid things like overload a control to act as two buttons, giving me the misplaced confidence to think playing Descent on camera was a good idea.

Conclusions

Doom is objectively the best game to play on a car.

I was kind of taken aback by how few people had attempted something like this already, given how accessible it turned out to be. As of the time of this writeup I found a few similar projects:

This team from a Finnish university rigged a VW Scirocco to play Rally Trophy. But they never wrote it up and their website is long gone :(
A father and son team hacked a Chevrolet Volt to play Mario Kart 64 (code). Their approach was similar to mine; main difference is they used a Raspberry PI + native CANbus module to capture the vehicle messages, then shunted the raw data via UDP to a Windows laptop which converted them to keyboard presses.
wirklichkeitssteuerungsgeraet (code), a wildcard demo at Revision 2017 by Akronyme Analogiker which involved connecting up a live motorcycle to drive an Amiga joystick?! This was passed to UAE through an Amiga-to-USB adapter in order to play The Cycles: International Grand Prix Racing while coasting around a car park.
After publishing, Stan "P1kachu" Lejay let me know about his project from earlier this year to rig a Fiat 500c to play Dirt Showdown and VDrift (code, slides). His approach uses a RasPi native CAN module + uinput. Go check out his talk at 34C3!

If you want to lay down 20 clams for an ELM327 and rig your own car to play games, nice! Hopefully there should be enough information in this writeup and the source code repositories to help you along. I am always available through email or Twitter for advice.

If you're interested in more general car hacking resources I can thoroughly recommend the following:

Craig Smith's Car Hacker's Handbook published by the fantastic No Starch Press.
The reverse engineering writeups done by Charlie Miller and Chris Valasek (a.k.a. the duo that remotely hacked the controls of a Jeep Cherokee while a WIRED journalist was driving it).

Show Your Working: Fixing a pile of bugs in MAME

2017-12-03T00:00:00+08:00

Show Your Working is a brand new segment where I write up things I have attempted to fix in open source software. Sometimes it's interesting to debug a problem yourself from first principles, even if the codebase is huge and you don't know anything going in. I will try and explain my thought process as I venture out into the weeds armed only with a butter knife.

The software

If you haven't heard of it, MAME is an emulation project with the aim of supporting basically every arcade machine ever made. To do this, thousands of circuit boards and custom chips have been probed, analysed, decapped with acid, and ultimately written out as code. By adding board ROMs (whole dumps of the memory chips containing the copyrighted program data), MAME can emulate a system by connecting together drivers for all of the chipsets as per the layout of the original board. Recently, the sister project MESS has been merged back into the MAME main codebase, adding ~2200 consoles and home computer systems to the lineup. It is an incredible feat of volunteer-driven engineering.

Unlike other emulators, MAME places accuracy and preservation above all else. All CPU code is interpreted (with some IL speedups via the UML architecture) and runs time-shared in a single thread. Instead of tightly coupling the hardware code with the framework for performance, MAME has an infinitely-rewirable generic module architecture, to encourage reuse of chip drivers across platforms. Imagine MAME as a big box of all the different types of chips, plus some templates that explain what chips to take from the box and how to wire them together to make a system.

Also I was not exaggerating when I said every arcade machine ever made. The MAME source tree has slightly north of 6900 .cpp files, and a princely 2.9 million lines of code (4 million if you count headers!). This is a Big Codebase.

The problem

The reason I started looking at MAME was because of this tweet:

<@bfod> I am beyond sad that MAME doesn’t support save states for Daisu Kiss, so I can’t put it in Multibowl

Multibowl is a pretty amazing project; Bennett Foddy and AP Thomson have turned hundreds of MAME ROMs into this weird tournament of two-player minigames, thanks to judicious abuse of memory scraping and savestates. Similar to Joe Dante's The Movie Orgy, Multibowl is only available as a touring art installation due to the staggering amount of copyright infringement involved. Nevertheless I have a soft spot for illegal art, and I was looking for an opportunity to get my feet wet with emulators, and this seemed like it! I mean the hard part is already done, the game is playable, all that's left is to make sure all the bits of state are being saved... right?

The preparation

The only software we need installed to debug this is the Git VCS and the standard Linux command line tools, plus whatever build dependencies MAME has. Start by grabbing the latest MAME source code

$ git clone https://github.com/mamedev/mame
$ cd mame

The MAME build system is sort of nutso, however there is a nice way to build just the machine you want to debug. (And you will want to do this, as a full symbols build of all the arcade machines is a 2.2gb executable.)

First, find the .cpp file for the driver of the machine named on the MAME splash screen when you try and emulate it (in my case it was src/mame/drivers/konamigx.cpp). Then, run the following command from the project root:

make REGENIE=1 SUBTARGET=arcade OPTIMIZE=0 SYMBOLS=1 SYMLEVEL=2 NOWERROR=1 PYTHON_EXECUTABLE=/usr/bin/python2 TOOLS=1 ARCHOPTS=-flifetime-dse=1 VERBOSE=1 SOURCES=src/mame/drivers/konamigx.cpp -j3

(For further recompiles, remove the REGENIE=1)

This will build a copy of MAME with symbols, but without the features of the debug build level, like asserts! I did it this way because one of the asserts in the audio CPU was killing the emulator after loading a save state, and I wasn't quite feeling brave enough to debug the whole stack to find out why. (Alternatively you can add DEBUG=1 and comment out all the things)

The other modification I recommend for save state hacking:

diff --git a/src/emu/save.cpp b/src/emu/save.cpp
index c85f0d1..5e33895 100644
--- a/src/emu/save.cpp
+++ b/src/emu/save.cpp
@@ -291,6 +291,8 @@ save_error save_manager::write_file(emu_file &file)
        if (m_illegal_regs > 0)
                return STATERR_ILLEGAL_REGISTRATIONS;

+    dump_registry();
+
        // generate the header
        u8 header[HEADER_SIZE];
        memcpy(&header[0], STATE_MAGIC_NUM, 8);
@@ -353,7 +355,7 @@ u32 save_manager::signature() const
void save_manager::dump_registry() const
{
        for (auto &entry : m_entry_list)
-               LOG(("%s: %d x %d\n", entry->m_name.c_str(), entry->m_typesize, entry->m_typecount));
+               osd_printf_verbose("%s: %d x %d\n", entry->m_name.c_str(), entry->m_typesize, entry->m_typecount);
}

This will modify the save state command to print out a big list of all the saved properties. Apparently you can get the logging mode to trigger the list, but I gave up trying. Make sure not to commit this change :P

For testing I ran my single-machine build inside GDB, with MAME's built-in debugger switched on. The MAME debugger is great for inspecting the code and memory usage of the emulated CPUs, but if you want to see what's happening inside those chips you need to break out into C++ land with GDB.

gdb --args ~/Development/mame/arcade64 -debug -window -verbose daiskiss.zip

The diagnosis

Start by doing the obvious thing: open Daisu Kiss up in MAME and check out the machine info:

Daisu-Kiss (ver JAA)
1996 Konami
Driver: konamigx.cpp

CPU:
M68EC020 24.000000MHz
M68000 8.000000MHz
TMS57002 12.000000MHz

Sound:
TMS57002 12.000000MHz
2xSpeaker
2xK054539 ADPC 18.432000MHz

Video:
288 x 224 (H) 59.185606 Hz

So it's a game using the Konami GX model of arcade boards, with a couple of trusty old Motorola 68000 processors doing the computing spadework, some Texas Instruments DSP thingo, and... what looks like an avalanche of custom Konami chips doing the graphics and sound! Lovely.

As expected, loading a saved state from a cold boot didn't work; the game would either hang with the music looping in the background, or hang with the music stuck on a slur. However save states usually DID work if you loaded them after the machine has finished booting, which indicates there's some bit of state which gets set on boot that isn't accounted for.

Wrapping your head around save states

My initial assumption about how save states would work: MAME dumps a copy of the memory, CPU registers, and anything else which holds machine state to a file. Loading does the reverse: updates the memory and sets all the CPU registers. The documentation mostly confirms this (ok fine so I missed timing information, bank-switching and perhipherals from my list), and reassures that most of the stock hardware has state support coded in already! So that's something. There was a rather tantalising page about save states in the MAME wiki, but it's maybe two codebase refactoring sprees behind and none of the names matched up anymore.

Anyway, the way it works is there's a small window at the end of the driver init cycle where a component can flag the parts of itself which contain state. MAME provides two helper methods for this:

void save_item(_ItemType &value, const char *valname, int index = 0)
void save_pointer(_ItemType *value, const char *valname, u32 count, int index = 0)

save_item is pretty malleable, and works for scalar values, 1D and 2D static arrays, and 1D and 2D std::vectors. For malloc'd memory you can use save_pointer, but you need to provide a count of the number of elements. Most of the time you'll see them used in the form save_item(NAME(state_object)), where NAME(state_object) is a macro that expands to 'state_object, "state_object"', to save on repetition.

(I am sad that save_item can't be used on structs. I mean, it totally can, but every compiler aligns items inside structs differently, meaning that save states from different compilers would break in terrible ways).

And that's it. Once an item has been flagged, the contents will be written to and loaded from save states. The simplicity of this system is that because each component tracks its own state, reuse between different arcade boards is quite high. For instance the Motorola 68000 has a fairly well-defined set of internal state, so every game which uses the M68000 driver will have that chip's state saved automatically.

Bonus fact: there's no actual seperation of concerns in the save state file. There's a game ID, a CRC signature for the set of fields, then an unlabelled stream of raw data. If the CRC in the file matches the one generated by MAME from the field list, it will load said data into the fields sequentially.

In which we consider the majestic Konami GX arcade board in all its proprietary splendour

First place to look would be in /mame/src/mame/drivers/konamigx.cpp, which defines the hardware layout for all Konami GX games. Search around for "daiskiss":

GAME( 1996, daiskiss, konamigx, konamigx, gokuparo, konamigx_state, konamigx, ROT0, "Konami", "Daisu-Kiss (ver JAA)", MACHINE_IMPERFECT_GRAPHICS )

"gokuparo" is the name of another Konami GX game, so presumably this piggy-backs off the hardware config. The actual definition of how the hardware in the konamigx driver is wired together is in a big macro block MACHINE_CONFIG_START:

static MACHINE_CONFIG_START( konamigx, konamigx_state )
    /* basic machine hardware */
    MCFG_CPU_ADD("maincpu", M68EC020, MASTER_CLOCK)
    MCFG_CPU_PROGRAM_MAP(gx_type2_map)
    MCFG_CPU_VBLANK_INT_DRIVER("screen", konamigx_state, konamigx_type2_vblank_irq)

    MCFG_CPU_ADD("soundcpu", M68000, SUB_CLOCK/2)
    MCFG_CPU_PROGRAM_MAP(gxsndmap)

    MCFG_CPU_ADD("dasp", TMS57002, MASTER_CLOCK/2)
    MCFG_CPU_DATA_MAP(gxtmsmap)

    MCFG_DEVICE_ADD("k053252", K053252, MASTER_CLOCK/4)
    MCFG_K053252_OFFSETS(24, 16)
    MCFG_K053252_INT1_ACK_CB(WRITELINE(konamigx_state, vblank_irq_ack_w))
    MCFG_K053252_INT2_ACK_CB(WRITELINE(konamigx_state, hblank_irq_ack_w))
    MCFG_VIDEO_SET_SCREEN("screen")

    MCFG_QUANTUM_TIME(attotime::from_hz(6000))

    MCFG_MACHINE_START_OVERRIDE(konamigx_state,konamigx)
    MCFG_MACHINE_RESET_OVERRIDE(konamigx_state,konamigx)

    MCFG_EEPROM_SERIAL_93C46_ADD("eeprom")

    /* video hardware */
    MCFG_SCREEN_ADD("screen", RASTER)
    MCFG_SCREEN_VIDEO_ATTRIBUTES(VIDEO_UPDATE_AFTER_VBLANK)
    MCFG_SCREEN_RAW_PARAMS(8000000, 384+24+64+40, 0, 383, 224+16+8+16, 0, 223)
    /* These parameters are actual value written to the CCU.
    tbyahhoo attract mode desync is caused by another matter. */

    //MCFG_SCREEN_VBLANK_TIME(ATTOSECONDS_IN_USEC(600))
    // TODO: WTF, without these most games crashes? Some legacy call in video code???
    MCFG_SCREEN_SIZE(1024, 1024)
    MCFG_SCREEN_VISIBLE_AREA(24, 24+288-1, 16, 16+224-1)
    MCFG_SCREEN_UPDATE_DRIVER(konamigx_state, screen_update_konamigx)

    MCFG_PALETTE_ADD("palette", 8192)
    MCFG_PALETTE_FORMAT(XRGB)
    MCFG_PALETTE_ENABLE_SHADOWS()
    MCFG_PALETTE_ENABLE_HILIGHTS()

    MCFG_DEVICE_ADD("k056832", K056832, 0)
    MCFG_K056832_CB(konamigx_state, type2_tile_callback)
    MCFG_K056832_CONFIG("gfx1", K056832_BPP_5, 0, 0, "none")
    MCFG_K056832_PALETTE("palette")

    MCFG_K055555_ADD("k055555")

    MCFG_DEVICE_ADD("k054338", K054338, 0)
    MCFG_K054338_MIXER("k055555")
    MCFG_K054338_SET_SCREEN("screen")
    MCFG_K054338_ALPHAINV(1)

    MCFG_DEVICE_ADD("k055673", K055673, 0)
    MCFG_K055673_CB(konamigx_state, type2_sprite_callback)
    MCFG_K055673_CONFIG("gfx2", K055673_LAYOUT_GX, -26, -23)
    MCFG_K055673_SET_SCREEN("screen")
    MCFG_K055673_PALETTE("palette")

    MCFG_VIDEO_START_OVERRIDE(konamigx_state, konamigx_5bpp)

    /* sound hardware */
    MCFG_SPEAKER_STANDARD_STEREO("lspeaker", "rspeaker")

    MCFG_DEVICE_MODIFY("dasp")
    MCFG_SOUND_ROUTE(0, "lspeaker", 0.3)
    MCFG_SOUND_ROUTE(1, "rspeaker", 0.3)

    MCFG_K056800_ADD("k056800", XTAL_18_432MHz)
    MCFG_K056800_INT_HANDLER(INPUTLINE("soundcpu", M68K_IRQ_1))

    MCFG_DEVICE_ADD("k054539_1", K054539, XTAL_18_432MHz)
    MCFG_K054539_REGION_OVERRRIDE("shared")
    MCFG_K054539_TIMER_HANDLER(WRITELINE(konamigx_state, k054539_irq_gen))
    MCFG_SOUND_ROUTE_EX(0, "dasp", 0.5, 0)
    MCFG_SOUND_ROUTE_EX(1, "dasp", 0.5, 1)
    MCFG_SOUND_ROUTE(0, "lspeaker", 1.0)
    MCFG_SOUND_ROUTE(1, "rspeaker", 1.0)

    MCFG_DEVICE_ADD("k054539_2", K054539, XTAL_18_432MHz)
    MCFG_K054539_REGION_OVERRRIDE("shared")
    MCFG_SOUND_ROUTE_EX(0, "dasp", 0.5, 2)
    MCFG_SOUND_ROUTE_EX(1, "dasp", 0.5, 3)
    MCFG_SOUND_ROUTE(0, "lspeaker", 1.0)
    MCFG_SOUND_ROUTE(1, "rspeaker", 1.0)
MACHINE_CONFIG_END

static MACHINE_CONFIG_DERIVED( gokuparo, konamigx )
    MCFG_DEVICE_MODIFY("k055673")
        MCFG_K055673_CONFIG("gfx2", K055673_LAYOUT_GX, -46, -23)
MACHINE_CONFIG_END
**

That was long, but we have our full list of chips that the board uses, which we can cross-reference against the MAME driver source to determine what they do.

M68EC020 (Motorola 68020 - main CPU)
M68000 (Motorola 68000 - sound CPU)
TMS57002 (Texas Instruments- sound DASP)
K053252 (Konami - display timing/interrupt)
K056832 (Konami - tilemap)
K055555 (Konami - video mixer/priority encoder)
K054338 (Konami - video mixer/alpha)
K055673 (Konami - sprite generator)
K056800 (Konami - sound controller)
K054539_1, K054539_2 (Konami - PCM sound)

Yep. That's a lot. Uhhh.

Finding where the lockup is

Okay, we can reproduce the issue and we have access to a fancy debugger. My plan is to open two debug copies of MAME: one pre-boot, one post-boot, load the state in each, then step through them both line by line until execution diverges. Then that line shall magically disclose what the problem is. No, it will! Shush.

Well. Well well well. The main CPU activity on the hung session is basically 4 lines of assembly.

# The code which is responsible for the hang loop (A1: 0x00c08400):

28707C   tst.w    (A1)       # check that the uint16_be at address 0x00c08400 == 0x0000
28707E   beq      $287184    # if so, jump to 287184!
...
287184   tst.w    ($646,A1)  # check that the uint16_be at address (0x00c08400 + 0x646) == 0x0000
287188   beq      $28707c    # if so, jump to 28707C!

Now, at the time of execution, register A1 is set to 0x00c08400. What part of the memory map is this?

static ADDRESS_MAP_START( gx_base_memmap, AS_PROGRAM, 32, konamigx_state )
    AM_RANGE(0x000000, 0x01ffff) AM_ROM // BIOS ROM
    AM_RANGE(0x200000, 0x3fffff) AM_ROM // main program ROM
    AM_RANGE(0x400000, 0x7fffff) AM_ROM // data ROM
    AM_RANGE(0xc00000, 0xc1ffff) AM_RAM AM_SHARE("workram")
    AM_RANGE(0xd00000, 0xd01fff) AM_DEVREAD("k056832", k056832_device, k_5bpp_rom_long_r)
    AM_RANGE(0xd20000, 0xd20fff) AM_DEVREADWRITE16("k055673", k055673_device, k053247_word_r, k053247_word_w, 0xffffffff)
    AM_RANGE(0xd21000, 0xd21fff) AM_RAM // second bank of sprite RAM, accessed thru ESC
    AM_RANGE(0xd22000, 0xd23fff) AM_RAM // extra bank checked at least by sexyparo, pending further investigation.
    AM_RANGE(0xd40000, 0xd4003f) AM_DEVWRITE("k056832", k056832_device, long_w)
    AM_RANGE(0xd44000, 0xd4400f) AM_WRITE(konamigx_tilebank_w)
    AM_RANGE(0xd48000, 0xd48007) AM_DEVWRITE16("k055673", k055673_device, k053246_word_w, 0xffffffff)
    AM_RANGE(0xd4a000, 0xd4a00f) AM_DEVREAD16("k055673", k055673_device, k055673_rom_word_r, 0xffffffff)
    AM_RANGE(0xd4a010, 0xd4a01f) AM_DEVWRITE16("k055673", k055673_device, k055673_reg_word_w, 0xffffffff)
    AM_RANGE(0xd4c000, 0xd4c01f) AM_DEVREADWRITE8("k053252", k053252_device, read, write, 0xff00ff00)
    AM_RANGE(0xd4e000, 0xd4e01f) AM_WRITENOP // left-over for "secondary" CCU, apparently (used by type 3/4 for slave screen?)
    AM_RANGE(0xd50000, 0xd500ff) AM_DEVWRITE("k055555", k055555_device, K055555_long_w)
    AM_RANGE(0xd52000, 0xd5201f) AM_DEVREADWRITE8("k056800", k056800_device, host_r, host_w, 0xff00ff00)
    AM_RANGE(0xd56000, 0xd56003) AM_WRITE(eeprom_w)
    AM_RANGE(0xd58000, 0xd58003) AM_WRITE(control_w)
    AM_RANGE(0xd5a000, 0xd5a003) AM_READ_PORT("SYSTEM_DSW")
    AM_RANGE(0xd5c000, 0xd5c003) AM_READ_PORT("INPUTS")
    AM_RANGE(0xd5e000, 0xd5e003) AM_READ_PORT("SERVICE")
    AM_RANGE(0xd80000, 0xd8001f) AM_DEVWRITE("k054338", k054338_device, long_w)
    AM_RANGE(0xda0000, 0xda1fff) AM_DEVREADWRITE("k056832", k056832_device, ram_long_r, ram_long_w)
    AM_RANGE(0xda2000, 0xda3fff) AM_DEVREADWRITE("k056832", k056832_device, ram_long_r, ram_long_w)
ADDRESS_MAP_END

Why, it's smack in the middle of the work RAM! Let's dump it from the save state and do a diff with one dumped from a clean run.

dump /tmp/work.dmp,0xc00000,0x20000

Eh, the RAM contents match. Seems legit, MAME will add any part of the address range flagged as RAM to the save state.

(taken from my dump_registry printout)
...
memory/:maincpu/0/00c00000-00c1ffff: 4 x 32768
...
memory/:maincpu/0/00d20000-00d2ffff: 4 x 16384

Still, we can confirm by putting breakpoints immediately after those two jumps that no amount of waiting will break the main CPU out of this hang. Contrast that with the full run, where the breakout happens instantaneously!

After a more thorough examination, it doesn't look like that section of work RAM has a direct memory access (DMA) arrangement, or is touched by anything other than the CPU. Which means the CPU has to be the one to change it, despite being stuck in an infinite loop. Perhaps there's an interrupt that causes it to jump to another chunk of code...

Oh look! Using the debugger's "Run to the next Interrupt on this CPU" command on the working copy, we can see there are two interrupts that get called every frame in the working copy: IRQ 1 and IRQ 4. And neither of those are being called in the busted save state copy! Dun dun duuuuuuuuunnnn.

/*
 * IRQs:
 * 1: VBL (at 60 Hz)
 * 2: HBL
 * 3: Sprite DMA complete
 * 4: from protection device, indicates command completion for "ESC" chip
 */

As far as I can tell it's the vertical blanking interrupt (IRQ 1) that breaks the main CPU out of the loop and gives it more things to do, so let's inspect that. In real life, you'd interrupt the M68020C by pulling the voltage up on one of the chip legs or something. In MAME this happens in the driver C++ code with the handy command:

m_maincpu->set_input_line(1, HOLD_LINE);

There are two spots in the code (?!) which call IRQ 1, and only one of them is called every frame for the hung session:

INTERRUPT_GEN_MEMBER(konamigx_state::konamigx_type2_vblank_irq)
{
    // lift idle suspension
    if (m_resume_trigger && m_suspension_active)
    {
        m_suspension_active = 0;
        machine().scheduler().trigger(m_resume_trigger);
    }

    // IRQ 1 is the main 60hz vblank interrupt
    if (m_gx_syncen & 0x20)
    {
        m_gx_syncen &= ~0x20;

        if ((m_gx_wrport1_1 & 0x81) == 0x81 || (m_gx_syncen & 1))
        {
            m_gx_syncen &= ~1;
            // TODO: enabling ASSERT_LINE breaks opengolf, annoying.
            device.execute().set_input_line(1, HOLD_LINE);
        }
    }

    dmastart_callback(0);
}

There it is! If either of those two if statements fail, it doesn't call IRQ 1! In the hung session, m_gx_syncen is always 0, and that's not saved in the state. Oh hoh hoh hoh hoh.

MACHINE_START_MEMBER(konamigx_state,konamigx)
{
    save_item(NAME(m_gx_wrport1_0));
    save_item(NAME(m_gx_wrport1_1));
    save_item(NAME(m_gx_wrport2));

    save_item(NAME(m_gx_rdport1_3));
    save_item(NAME(m_gx_syncen));
    save_item(NAME(m_suspension_active));
    save_item(NAME(m_prev_pixel_clock));
}

I added all the registers that were nearby m_gx_syncen in MACHINE_RESET_MEMBER, because why the hell not. AND IT WORKS! Man what an adventure that was, ah well let's hammer out a pull request and wrap this up nicely.

Hang on, what about the dodgy sound you mentioned earlier? Don't be cheap

You got me, it mostly works. The main CPU execution saunters through without a hitch, but I could still break the audio by loading a save state on a cold boot. There must be some more state variables we haven't captured yet.

Good thing we have a canary to help us out; the crashy assert in the debug build mentioned earlier.

Thread 1 "arcade64d" received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff3e3e367 in kill () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff3e3e367 in kill () at /usr/lib/libc.so.6
#1  0x0000555555fc7475 in osd_break_into_debugger(char const*) (message=message@entry=0x55555a530448 "assert: ../../../../../src/devices/cpu/tms57002/tms57002.cpp:790: samples == 1") at ../../../../../src/osd/modules/lib/osdlib_unix.cpp:97
#2  0x0000555555acb405 in emu_fatalerror::emu_fatalerror(char const*, ...) (this=0x55555a530440, format=<optimized out>) at ../../../../../src/emu/emucore.cpp:29
#3  0x00005555559ba7c2 in tms57002_device::sound_stream_update(sound_stream&, int**, int**, int) (this=<optimized out>, stream=..., inputs=<optimized out>, outputs=<optimized out>, samples=<optimized out>) at ../../../../../src/devices/cpu/tms57002/tms57002.cpp:790
#4  0x0000555555c006c4 in delegate_base<void, sound_stream&, int**, int**, int>::operator()(sound_stream&, int**, int**, int) const (args#3=721, args#2=<optimized out>, args#1=<optimized out>, args#0=..., this=0x555559687cb0) at ../../../../../src/lib/util/delegate.h:544
#5  0x0000555555c006c4 in sound_stream::generate_samples(int) (this=this@entry=0x555559687c00, samples=721) at ../../../../../src/emu/sound.cpp:640
#6  0x0000555555c01759 in sound_stream::update() (this=0x555559687c00) at ../../../../../src/emu/sound.cpp:289
#7  0x0000555555c01759 in sound_stream::sync_update(void*, int) (this=0x555559687c00) at ../../../../../src/emu/sound.cpp:299
#8  0x0000555555be64ae in delegate_base<void, void*, int>::operator()(void*, int) const (args#1=<optimized out>, args#0=<optimized out>, this=<optimized out>) at ../../../../../src/lib/util/delegate.h:544
#9  0x0000555555be64ae in device_scheduler::execute_timers() (this=<optimized out>) at ../../../../../src/emu/schedule.cpp:908
#10 0x0000555555be64ae in device_scheduler::timeslice() (this=this@entry=0x7fffffffce68) at ../../../../../src/emu/schedule.cpp:519
#11 0x0000555555b99398 in running_machine::run(bool) (this=this@entry=0x7fffffff6820, quiet=quiet@entry=false) at ../../../../../src/emu/machine.cpp:348
#12 0x0000555555743942 in mame_machine_manager::execute() (this=this@entry=0x555556afb0e0) at ../../../../../src/frontend/mame/mame.cpp:223
#13 0x00005555557cb8c4 in cli_frontend::start_execution(mame_machine_manager*, int, char**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (this=this@entry=0x7fffffffda50, manager=manager@entry=0x555556afb0e0, argc=argc@entry=5, argv=argv@entry=0x7fffffffe128, option_errors="") at ../../../../../src/frontend/mame/clifront.cpp:287
#14 0x00005555557cbdc9 in cli_frontend::execute(int, char**) (this=this@entry=0x7fffffffda50, argc=argc@entry=5, argv=argv@entry=0x7fffffffe128) at ../../../../../src/frontend/mame/clifront.cpp:314
#15 0x00005555557416b1 in emulator_info::start_frontend(emu_options&, osd_interface&, int, char**) (options=..., osd=..., argc=5, argv=0x7fffffffe128) at ../../../../../src/frontend/mame/mame.cpp:319
#16 0x000055555561271c in main(int, char**) (argc=5, argv=0x7fffffffe128) at ../../../../../src/osd/sdl/sdlmain.cpp:214

Actually I think this is a red herring. The issue is that upon loading a save state, generate_samples is getting called with a large sample count (~720) which gets passed on to sound_stream_update, which has an assert(samples == 1) at the beginning. (Under normal operation the function gets called like a billion times a second with a sample count of 1.) This doesn't have any reflection on whether or not sound is broken, as it always happens when you load a state.

So, back to zero clues! I revisted src/mame/drivers/konamigx.cpp to try and get more of an idea of how the audio system is wired together.

These two K054539 chips interest me. K054539 is listed as a "PCM Sound Chip", and a quick peek at the registers makes it clear that it's designed for wavetable music playback.

/* Registers:
00..ff: 20 bytes/channel, 8 channels
  00..02: pitch (lsb, mid, msb)
      03: volume (0=max, 0x40=-36dB)
      04: reverb volume (idem)
  05: pan (1-f right, 10 middle, 11-1f left)
  06..07: reverb delay (0=max, current computation non-trusted)
  08..0a: loop (lsb, mid, msb)
  0c..0e: start (lsb, mid, msb) (and current position ?)

100.1ff: effects?
  13f: pan of the analog input (1-1f)

200..20f: 2 bytes/channel, 8 channels
  00: type (b2-3), reverse (b5)
  01: loop (b0)

214: Key on (b0-7 = channel 0-7)
215: Key off          ""
225: ?
227: Timer frequency
228: ?
229: ?
22a: ?
22b: ?
22c: Channel active? (b0-7 = channel 0-7)
22d: Data read/write port
22e: ROM/RAM select (00..7f == ROM banks, 80 = Reverb RAM)
22f: Global control:
     .......x - Enable PCM
     ......x. - Timer related?
     ...x.... - Enable ROM/RAM readback from 0x22d
     ..x..... - Timer output enable?
     x....... - Disable register RAM updates

 The chip has an optional 0x8000 byte reverb buffer.
 The reverb delay is actually an offset in this buffer.
 */

At first I assumed the two chips were for some form of stereo playback, but the MCFG_SOUND_ROUTE stuff showed that each chip was outputting stereo and sending to both speakers. The K054539 has 8 sound channels with adjustable panning and effects, including a luxury 8kb buffer just for reverb! That means 16 audio channels all up. For kicks I tried turning down the gain on each chip to hear the changes; both chips are handling a mixture of PCM sample and music playback.

Sound programming is handled by the M68000 audio CPU, which I'm guessing keeps the two K054539s fed with frame-by-frame playback instructions, and meaning the main CPU only has to worry about firing off the odd cue every now and then. Communication between the two is managed with another beloved K chip, the K056800! Mercifully there's not too much to this one, it's basically a set of registers that the main CPU and sound CPU can read/write to.

Because clues are so thin on the ground, we need to make our own luck. The K054539 doesn't sound like it's playing back the music properly, so let's gets some eyes on those registers and check if that's indeed the case.

diff --git a/src/devices/sound/k054539.cpp b/src/devices/sound/k054539.cpp
index c275494..15e4b10 100644
--- a/src/devices/sound/k054539.cpp
+++ b/src/devices/sound/k054539.cpp
@@ -94,12 +94,14 @@ bool k054539_device::regupdate()

 void k054539_device::keyon(int channel)
 {
+    osd_printf_verbose("k054539:%p Ch%d ON\n", this, channel);
         if(regupdate())
                 regs[0x22c] |= 1 << channel;
 }

 void k054539_device::keyoff(int channel)
 {
+    osd_printf_verbose("k054539:%p Ch%d OFF\n", this, channel);
         if(regupdate())
                 regs[0x22c] &= ~(1 << channel);
 }

The above code prints a message every time one of the sound chips starts or ends playing a note. Here's a text dump from a working copy:

Samples: 871
k054539:0x55f4aa874710 Ch0 OFF
k054539:0x55f4aa874710 Ch0 ON
k054539:0x55f4aa874710 Ch1 OFF
k054539:0x55f4aa874710 Ch1 ON
k054539:0x55f4aa874710 Ch2 OFF
k054539:0x55f4aa874710 Ch2 OFF
k054539:0x55f4aa874710 Ch2 ON
k054539:0x55f4aa874710 Ch3 OFF
k054539:0x55f4aa874710 Ch3 ON
k054539:0x55f4aa874710 Ch4 OFF
k054539:0x55f4aa874710 Ch4 ON
k054539:0x55f4aa874710 Ch5 OFF
k054539:0x55f4aa874710 Ch5 ON
k054539:0x55f4aa874710 Ch0 OFF
k054539:0x55f4aa874710 Ch0 ON
k054539:0x55f4aa874710 Ch1 OFF
k054539:0x55f4aa874710 Ch1 ON
k054539:0x55f4aa874710 Ch0 OFF
k054539:0x55f4aa874710 Ch0 ON
k054539:0x55f4aa874710 Ch1 OFF
k054539:0x55f4aa874710 Ch1 ON
k054539:0x55f4aa874710 Ch2 OFF
k054539:0x55f4aa874710 Ch2 ON
...

And here's a dump from a busted save state copy:

Samples: 871
k054539:0x5600870a8680 Ch3 OFF
k054539:0x5600870a8680 Ch4 OFF
[slight pause]
k054539:0x5600870a8680 Ch0 OFF
k054539:0x5600870a8680 Ch1 OFF
k054539:0x5600870a8680 Ch2 OFF
k054539:0x5600870a8680 Ch3 OFF
k054539:0x5600870a8680 Ch4 OFF
k054539:0x5600870a8680 Ch5 OFF
k054539:0x5600870a8680 Ch6 OFF
k054539:0x5600870a8680 Ch7 OFF
k054539:0x5600870a9a40 Ch0 OFF
k054539:0x5600870a9a40 Ch1 OFF
k054539:0x5600870a9a40 Ch2 OFF
k054539:0x5600870a9a40 Ch3 OFF
k054539:0x5600870a9a40 Ch4 OFF
k054539:0x5600870a9a40 Ch5 OFF
k054539:0x5600870a9a40 Ch6 OFF
k054539:0x5600870a9a40 Ch7 OFF
[nothing]

To my eyes that looks like a CPU somewhere tripping over and resetting, so the K054539 is mostly absolved of blame. Reloading the save state after the reset provides working sound. Checking the IRQs, on the busted save state the sound CPU infiniloops and doesn't receive any signals on IRQ 2 until juuust before the reset dance, which is very unusual. Normal behaviour is lots and lots and lots of IRQ 2 all of the time.

Oh man. Interest is fading fast, but we're so close! SO CLOSE! Let's find out what is responsible for smashing IRQ 2.

Wait, what?! There's a whole other chunk of address map that I missed?!?

WRITE16_MEMBER(konamigx_state::tms57002_control_word_w)
 {
     if (ACCESSING_BITS_0_7)
     {
         if (!(data & 1))
             m_soundcpu->set_input_line(M68K_IRQ_2, CLEAR_LINE);

         m_dasp->pload_w(data & 4);
         m_dasp->cload_w(data & 8);
         m_dasp->set_input_line(INPUT_LINE_RESET, data & 0x10 ? CLEAR_LINE : ASSERT_LINE);

         m_sound_ctrl = data;
     }
 }

 /* 68000 memory handling */
 static ADDRESS_MAP_START( gxsndmap, AS_PROGRAM, 16, konamigx_state )
     AM_RANGE(0x000000, 0x03ffff) AM_ROM
     AM_RANGE(0x100000, 0x10ffff) AM_RAM
     AM_RANGE(0x200000, 0x2004ff) AM_DEVREADWRITE8("k054539_1", k054539_device, read, write, 0xff00)
     AM_RANGE(0x200000, 0x2004ff) AM_DEVREADWRITE8("k054539_2", k054539_device, read, write, 0x00ff)
     AM_RANGE(0x300000, 0x300001) AM_READWRITE(tms57002_data_word_r, tms57002_data_word_w)
     AM_RANGE(0x400000, 0x40001f) AM_DEVREADWRITE8("k056800", k056800_device, sound_r, sound_w, 0x00ff)
     AM_RANGE(0x500000, 0x500001) AM_READWRITE(tms57002_status_word_r, tms57002_control_word_w)
     AM_RANGE(0x580000, 0x580001) AM_WRITENOP // 'NRES' - D2: K056602 /RESET
 ADDRESS_MAP_END

 static ADDRESS_MAP_START( gxtmsmap, AS_DATA, 8, konamigx_state )
     AM_RANGE(0x00000, 0x3ffff) AM_RAM
 ADDRESS_MAP_END


 WRITE_LINE_MEMBER(konamigx_state::k054539_irq_gen)
 {
     if (m_sound_ctrl & 1)
     {
         // Trigger an interrupt on the rising edge
         if (!m_sound_intck && state)
             m_soundcpu->set_input_line(M68K_IRQ_2, ASSERT_LINE);
     }

     m_sound_intck = state;
 }

 **

So there's two spots inside the address mapper that change the state of IRQ 2 on the sound CPU, which rely on (surprise!) some untracked state. Now keep cool, but I think I've got this. Calm blue ocean, calm blue ocean, calm blue ocean...

diff --git a/src/mame/drivers/konamigx.cpp b/src/mame/drivers/konamigx.cpp
index e200d1c..e58a552 100644
--- a/src/mame/drivers/konamigx.cpp
+++ b/src/mame/drivers/konamigx.cpp
@@ -3707,6 +3707,9 @@ ROM_END

 MACHINE_START_MEMBER(konamigx_state,konamigx)
 {
+       save_item(NAME(m_sound_ctrl));
+       save_item(NAME(m_sound_intck));
+
        save_item(NAME(m_gx_wrport1_0));
        save_item(NAME(m_gx_wrport1_1));
        save_item(NAME(m_gx_wrport2));

In true monkey's paw fashion, the music plays back now but never stops, even as the game continues on. Ah well, like before it's only happening from a cold boot, so it's the same problem.

Backtracking a little: how does the main CPU tell the sound CPU to cue up a different thing? If you guessed "some ridiculous one-off chip starting with a K", you're right! It's the K056800.

Oh look. There's one variable that wasn't saved out of the four that makes up the chip.

Surely it couldn't be that.

diff --git a/src/devices/sound/k056800.cpp b/src/devices/sound/k056800.cpp
index 19e572d..fe42ed0 100644
--- a/src/devices/sound/k056800.cpp
+++ b/src/devices/sound/k056800.cpp
@@ -36,6 +36,7 @@ void k056800_device::device_start()
        m_int_handler.resolve_safe();

        save_item(NAME(m_int_pending));
+       save_item(NAME(m_int_enabled));
        save_item(NAME(m_host_to_snd_regs));
        save_item(NAME(m_snd_to_host_regs));
 }

YEAH BOIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

The solution

Keep adding save_item() calls to chunks of untracked state until the machine can resurrect itself.

(MAME pull request #1857)

Bonus content!

After getting in contact with Bennett Foddy, I foolishly asked if there were any more MAME games he wanted to include in Multibowl but couldn't for technical reasons. And, uh, there were some! So in my hubris I did some more debugging.

pc_vga/cs4031: fix save state support

(MAME pull request #2127)

With the VGA driver I could see what was wrong immediately (registers not being saved), but I had to comb through the huuuuuge commit history for the file to figure out what was going on. Quite often the problems are easy to read, but the line of reasoning for the current design choices are much harder to guess.

In short, the registers for the VGA card were being stored twice; once as a flat array called vga.crtc.data, and again as a whole bunch of unpacked discrete variables (e.g. vga.crtc.horz_total, vga.crtc.horz_disp_end, ...). All of the business logic was checking the discrete variables, but vga.crtc.data was the only thing being save_pointer()'d. Some repo dredging revealed the mystery: vga.crtc.data was originally the only thing, but 5 years ago there was a refactor that added the discrete variables and ported all the business logic to use them. vga.crtc.data hung around I guess for ease of testing (when you save_pointer() a thing, you can watch it in the debugger) and still gets updated by MAME, but it remains functionally useless.

So far so good. But what I couldn't figure out was why all the Protected Mode DOS games I ran would crash when loading the state from a cold boot? And at points which seemed to have nothing at all to do with the code?

Unfortunately, I couldn't find a nice way of trapping the MAME debugger when the crash happened, so I had to do the old standby of manually stepping through a bunch of times and taking lots of notes about e.g. which iteration of an inner loop to step into rather than over. Here are my scrawly notes from debugging the game Boppin':

Find that the function called at 0x002a326f causes the crash and severs control to the debugger!
Step into the function and keep hitting "step over" until the crash instruction is reached
It was on the 4th call to 0x002ac752, but why?!?
0x002ac9ca, step into function, loop ahead until DL = 0x16
0x002b2a2a, step 6 times
it's this iteration of the inner loop which is triggering it

0x002B2A18      mov ecx,[ebp+20h]
0x002B2A1B      lodsb               // load byte at address DS:ESI into AL, increment ESI
0x002B2A1C      or al, al
0x002B2A1E      je 2B2A30h          // jump if AL == 0
0x002B2A20      stosb               // store AL at address ES:EDI, increment EDI
0x002B2A21      loop 2B2A1Bh        // decrement ECX, continue if == 0

...

0x002B2A30      inc edi
0x002B2A31      loop 2B2A1Bh

The value of ESI is loaded in as garbage! Or rather, the 32-bit int from 0x0030248c is loaded in. And it's wrong. Very wrong.

Before my eyes in the debugger, on that stosb command for loop iteration CL=0D, the whole stack memory segment gets overwritten by garbage, from 0x300000 to 0x310000. Oh lord it looks like a reeaaaaaallll MAAAAAAAME buuuuuggggg. I'm sooooooooo saaaaaaaaddddddd.

Showing off my high pain tolerance, I ran MESS through Valgrind to see if the garbage came from uninitialized memory. It was very slow, but thankfully by now I knew what to type and could afford to wait 10 seconds after each action. No luck, the garbage was perfectly reproducible each time and didn't trigger any warnings.

INTERESTINGLY, when I tried to hot reload the save state, that region of memory remained as garbage! This suggests this region wasn't part of the saved CPU RAM map, which is definitely problematic. Most of the PC address range above 0x100000 is just free extended memory.

Now we're walking the knife edge of sanity that is not trusting your tools, or rather not trusting the MAME debugger. The only way to bring back certainty is to start proving individual facts. First up, is what we're seeing here really the allocated memory used for the RAM getting trashed?

We can do this in GDB with the following steps:

Put a breakpoint in the constructor of ram_device so you have eyes on all new RAM objects created
Grab the last buffer that appears before the emulator initializes

&((ram_device * const) 0x555557257380)->m_pointer[0]

For the default PC configuration it should be 4M in size. This maps perfectly to the address space.
Confirm that this is the right buffer by finding a string in the memory viewer and searching for it in GDB with find

find /b &((ram_device * const) 0x555557257380)->m_pointer[0], +0x400000, '\\' ,'B', 'O', 'P', 'P', 'I', 'N', '.', 'E', 'X', 'E'

Add a memory write watch

p &((ram_device * const) 0x555557257380)->m_pointer[0x300000]

// $39 = (unsigned char * ) 0x7fffacab2010

watch * 0x7fffacab2010

No change. Even after the debugger shows the address space getting sprayed, the RAM pointer is fine! Does this mean the bug is in MAME's memory mapper?

We need to go up one more level. The memory mapper is run inside the i386_device, so grab the device pointer from a breakpoint (I did it in i386_common_init), and try running READ8

p ((i386_device * ) 0x5555572ae3f0)->READ8(0x300000)
$47 = 0 '\000'

As expected. Now we can fast forward until after the memory has been sprayed.

p ((i386_device * ) 0x5555572ae3f0)->READ8(0x300000)
$53 = 65 'A'

FINALLY. FINALLY WE CAN REPRODUCE THE PROBLEM IN GDB SPACE. AFTER MANY GORMLESS HOURS OF PISSING IN THE WIND. I AM CRYING.

So, next question is: what in God's name is going on in READ8 to mess up like this?

b i386_device::READ8
Breakpoint 7 at 0x555555d4412c: file ../../../../../src/devices/cpu/i386/i386priv.h, line 596.
p ((i386_device * ) 0x5555572ae3f0)->READ8(0x300000)

From there we step through things a bit. The address is ANDed with m_a20_mask (uh?) which in the post-spray scenario reduces it from 0x300000 to 0x200000. Then it gets sent through a wall of templatese to read the single byte, which ends up in address_space::read_native. From HERE it does a lookup for the right read handler for the job, which itself has a pointer to some memory!

The difference seems to be the offset passed to ramptr. After spray, the byteaddress is 0x200000 and offset returned is 0x100000. Before spray, the byteaddress is 0x300000 and offset returned is 0x200000. Why, that's the same amount that m_a20_mask thing removed. Maybe I should read up a bit on this A20 thing and we might get som

OH NO

OH SWEET MOTHER OF GOD NO

aaaaaaaaaaaAA AAAAAAAA̧̩̖A͔̱̖AAA̘̖̗̠͟A̶͕͙͍̪̣ AA̛̺̱̗͇̳̬̺̟͕͢À͕͍̀Ą͍̺̜̬͍͟A̻̠̦͞A͇̦̻͔͕̤̪̤͢͜A͓̭̫̺̪̜A̯̮A͉̠̕͡Ḁ͙̺͡A̡̛͉̞̪̕Ą͙A̮͚̞͇͇͕͢͠ Ą͚̯̭̰̹̮̫̬̥Ą̡͇̙̫̤̟̲̖͚̞̖͔͎̫͈̯̕͟A̢̮̪͈̳̝̖̘̤̟̜͔̲͕͓̱̘͢Ą͍̺̜̬͍͟A̻̠̦͞A̵̴͈̱͔̳̝̠̙̘̗̹͟͟A̴̛͚̠̰̻̻̣̟͔̭̯̪̮̬Ą̸̝̹͖̻́͞͠A͏̵҉̨̗̝̖̜̻̟̲̺̥̪̯A̸̛̝̼͍̹͚̦̤͉̠͙͉͖͜A̘͚̤̤̺̜̗͘͜͠͝ͅḀ̶̹̜͖̠̝̟̮͉̟̖̣̙̲̻̞̣͡A̡͙̞̫̭̳͙͈͖͢ͅ

Okay I'm back now. This whole experience in learning Old Testament x86 has been quite troubling.

x86 nerds probably know what went wrong. So say you're IBM, carving up a storm with your Intel 8088 range of business computers. After a while you decide to upgrade to Intel's fancier 80286 chip, which features a luxurious 24 line address bus (16MB) instead of 20 lines (1MB). But what happened on the old chip when you tried to access memory addresses greater than 1MB? Well the upper bits weren't used, so it wrapped around. And because segment math in x86 is so bad and adding things is fast, smart programmers everywhere started to use this "feature".

No problem, says Intel! The 80286 will boot in Real Mode to satisfy people who wrote bad apps between 1981 and 1984, and have the new Protected Mode which allows addressing all those new megabytes!

The 80286 ships, and Intel completely forgot to add 8086-style addressing to Real Mode. 0x100000 still accesses Extended Memory. At this point the IBM engineers started to get clever/desperate; nearly all of the bad apps only need to wrap around once (from 0xfffff to 0x00000), so technically we only need to be able to disable a single bit of the address space (0x100000, bit 20). That's easy; chip exposes address line 20, we AND-gate it against a switch, bish bash bosh. Better yet, force it to off by default, because it's 1984 and our future selves definitely won't mind having to do this dumb workaround again and again for eternity! Now how to control this switch from userspace... saaaayyyy, is that a spare line I see on the 8042?

What's that? You don't think the absence of a bit in the IBM AT keyboard controller should make the memory map repeat itself every alternate 1MB? Pat yourself on the back, that one took Intel 29 years.

The one silver lining to all this is that countless people have written low-level x86 and survived the freaky design cruft before; once your monster has a name, it's easy to track.

One mystery left; where is the state for the A20 switch? The trail goes cold from the i386 side: m_a20_mask is set by an interrupt fired by lord knows what code, and isn't read back anywhere. Better check out the ct486.cpp driver, as that defines the routing between components.

MCFG_DEVICE_ADD("keybc", AT_KEYBOARD_CONTROLLER, XTAL_12MHz)
MCFG_AT_KEYBOARD_CONTROLLER_SYSTEM_RESET_CB(DEVWRITELINE("cs4031", cs4031_device, kbrst_w))
MCFG_AT_KEYBOARD_CONTROLLER_GATE_A20_CB(DEVWRITELINE("cs4031", cs4031_device, gatea20_w))
MCFG_AT_KEYBOARD_CONTROLLER_INPUT_BUFFER_FULL_CB(DEVWRITELINE("cs4031", cs4031_device, irq01_w))
MCFG_AT_KEYBOARD_CONTROLLER_KEYBOARD_CLOCK_CB(DEVWRITELINE("pc_kbdc", pc_kbdc_device, clock_write_from_mb))
MCFG_AT_KEYBOARD_CONTROLLER_KEYBOARD_DATA_CB(DEVWRITELINE("pc_kbdc", pc_kbdc_device, data_write_from_mb))
MCFG_DEVICE_ADD("pc_kbdc", PC_KBDC, 0)
MCFG_PC_KBDC_OUT_CLOCK_CB(DEVWRITELINE("keybc", at_keyboard_controller_device, keyboard_clock_w))
MCFG_PC_KBDC_OUT_DATA_CB(DEVWRITELINE("keybc", at_keyboard_controller_device, keyboard_data_w))
MCFG_PC_KBDC_SLOT_ADD("pc_kbdc", "kbd", pc_at_keyboards, STR_KBD_MICROSOFT_NATURAL)

Oh yeah, the chipset! I forgot that the PC architecture uses those.

OH! OH! cs4031.cpp has 3 registers for A20 state (to deal with the three different ways of triggering it that existed in 1993), and only two of them had the state saved.

So there we go. A single missing register in the IBM keyboard controller was causing the PC's whole memory map to fail explosively. What day is it again?

i386/pc_vga: save state fixes

(MAME pull request #2178)

More PC fun! Liero would run for about 4 frames after cold-loading a state, then the screen would go black. By watching the the vga.memory buffer in the debugger as the frames went by, it was clear the animation continued on well after the screen disappeared, suggesting that maybe the VGA palette was getting trashed. And lo, it was: I forgot to add vga.dac to the save state. The MAME palette device acts as a cache which gets the real palette in vga.dac pulled into it every so often. I guess I didn't run into it with the test game I first tried because it only forced an update after the palette had physically changed? Weird.

Jump 'n Bump, on the other hand, would crash after a single frame somewhere in the VGA redraw wait loop. More precisely, all CPU instructions to pull info from the VGA port about redraw (0x3da) were getting cut off at the knees by a macro CRTC_PORT_ADDR which queried vga.miscellaneous_output (?!?)? I would've thought that getting as far as the READ8_MEMBER called "port_03d0_r" would be enough proof, but maybe this fixes some other crappy edge case.

Pressing further would make DOS4GW throw a page fault; the mini stack trace it printed showed the instruction pointer (EIP) was being set to some huge address like 0x3DF80000. That looks really suspicious; we know for a fact there's no memory mapped there, and the low word being all blank suggests that it was actually a 16-bit address being loaded. Jump 'n Bump is a proper 32-bit game, perhaps the issue is some "am I 16 or 32 bit" flag inside the CPU isn't being saved? I'm chuffed, this is officially the first time I've used the mini stack trace given by a DOS program to fix something.

Of course, vanilla DOS doesn't care about page faults! By loading a 16-bit savestate of the DOS command prompt from 32-bit mode, I could fill the screen with garbage as soon as e.g. a RET instruction hit and it tried to read the wrong size address off the stack. And it would chug along fine; one time it even ran AUTOEXEC.BAT again without asking.

Okay now I've probed some more, it appears to be crashing in the system timer IRQ 0 interrupt. Is it because that code is 16-bit and the stack segment is 32-bit? A working session changes SS, SSBASE, SSFLAGS et al. to be 16-bit, but the save state run doesn't do that. Let's throw a gdb breakpoint into the code which changes the segments and see how often it gets called right up until the interrupt handover.

BROKEN:
#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x7fffffff6690, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d08ec9 in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:845

#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x55555e0024ac, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d06b42 in i386_device::i386_load_segment_descriptor(int) (this=0x55555dfffda0, segment=1) at ../../../../../src/devices/cpu/i386/i386.cpp:244
#2  0x0000555555d09f8b in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:1089

WORKING:
#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x7fffffff6690, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d08ec9 in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:845

#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x7fffffff66b0, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d0929d in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:895

#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x55555e0024c0, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d0991f in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:971

#0  0x0000555555d06841 in i386_device::i386_load_protected_mode_segment(i386_device::I386_SREG*, unsigned long*) (this=0x55555dfffda0, seg=0x55555e0024ac, desc=0x0) at ../../../../../src/devices/cpu/i386/i386.cpp:180
#1  0x0000555555d06b42 in i386_device::i386_load_segment_descriptor(int) (this=0x55555dfffda0, segment=1) at ../../../../../src/devices/cpu/i386/i386.cpp:244
#2  0x0000555555d09f8b in i386_device::i386_trap(int, int, int) (this=0x55555dfffda0, irq=136, irq_gate=1, trap_level=0) at ../../../../../src/devices/cpu/i386/i386.cpp:1089

This looks promising, whole chunks of i386_trap aren't being reached!

                    CPL = m_CPL;  // current privilege level
                    DPL = (desc.flags >> 5) & 0x03;  // descriptor privilege level
...
                    if((desc.flags & 0x0004) == 0 && (DPL < CPL))
                    {

There it is. m_CPL isn't saved. Ughhhhh. So far I've tried to be austere about which registers I track, but that is proving to be a worse idea than just tracking everything. Shortly after finding that, a whole slew of new bugs pop up from not having certain two-letter CPU flags in the state, so I add all of them (after a cursory check that they were all persistent and not clobbered each cycle or whatnot). Surprise! No more crashes.

taito_f3: fix missing graphics on state load

(MAME pull request #1903)

This was pretty straightforward! CPU and sound were both fine, only problem was large chunks of graphics were missing. The one big catch was this editable set of tiles (called the "pixel layer") which wasn't being updated in MAME's gfx_element when the state loaded, even when the backing RAM for it was! I didn't really want to mess with the behaviour of gfx_element and friends, so after a lot of hemming and hawing I added a device_post_load hook on taito_t3_state to mark all the tiles in the gfx_element as "dirty", which hints to MAME to reload all the tiles from the source (i.e. the backing RAM) on the next access. I'm still not sure why this doesn't happen by default. And I probably never will! Because it's done.

sblaster/ymf262: save state fixes

(MAME pull request #2193)

Another boring one. The DSP and YMF262 chips on a Sound Blaster card have like a billion registers that needed to be tracked for sound playback to survive a save state. Actually while testing this I got really irritated by another bug; every time MAME dropped a frame my speakers would make this horrendous popping noise, but this behaviour would immediately stop once the soundcard had played some music. I recorded my computer's audio output with Audacity; on the signed 16-bit scale audio output was being held at -1 by default, but during the pops it would step back to 0. The fix was to explicitly set the twin DACs to 0 (or rather 0x8000, as the DACs are unsigned 16-bit) in sb_device::device_reset().

Show Your Working: Patching a bug in a 25 year old Sierra game

2017-09-23T00:00:00+08:00

The software

ScummVM is a heroic effort to make a cross-platform engine for hundreds of classic adventure games, making them natively playable on modern systems. Since 2010, ScummVM contains an interpreter originally from the FreeSCI project, and by now does a pretty good job playing games based on the Sierra Creative Interpreter. If you haven't heard of ScummVM, I can't recommend it enough.

It's gotten to the point where ScummVM provides a much better experience than emulating the games in DOS/Windows. Still, given that the engine is reverse engineered without the original code, there's all manner of edge cases and quirks that have to be emulated in order for the games to work correctly, as we'll find out!

The problem

The old Sierra mantra was "save early, save often"; and not just because a game could be rendered unwinnable by a moon-logic puzzle requiring a 1px * 1px item from the first screen you didn't pick up. Or a bad arcade sequence tied to the CPU clock, playing at insane speeds on anything faster than a 386. Or the frequent risk of clicking the wrong spot and watching your character have all their skin eaten away. No, there are bugs and race conditions in the scripting code that makes up these games! Quite a lot of these bugs are overridden/ignored by the build of SCI the game was shipped with, but other times you aren't that lucky and the whole game tanks, wiping out whatever progress you've made.

In case you're not familiar, the original Sierra crash screen looked like this:

Oh man do not get me started on this message. Protip for designers: if your software crashes, don't be condescending ("of course doing that thing would break the game, what are you taking stupid pills???") and don't lie to save face (plenty of crashes can be triggered by doing the correct thing and you damn well know it Sierra). 7 year old me felt worse than he should have because of that.

Okay, some specifics. I felt compelled to replay Island of Dr. Brain, an edutainment game for ages 12 and up, to rack up a large score. (Don't judge.) I got as far as that accursed puzzle with the colour changing flamingos, when ScummVM threw open the debug console.

The preparation

The only software we need installed to debug this is the Git VCS and the standard Linux command line tools, plus whatever build dependencies ScummVM has. Start by grabbing the latest ScummVM source code.

$ git clone https://github.com/scummvm/scummvm
$ cd scummvm

The diagnosis

The ScummVM source tree is very modular, with each engine compartmentalised from the rest of the cross-platform code. All the stuff we're going to want is in the SCI engine folder /engines/sci.

Let's start with the autodetection. ScummVM does some very basic game auto-detection by MD5ing the first 5000 bytes of files in the game directory. The detection tables are in /engines/sci/detection_tables.h

// The Island of Dr. Brain - English DOS (from Quietust)
// Executable scanning reports "1.001.053", VERSION file reports "1.1 2.3.93"
{"islandbrain", "", {
    {"resource.map", 0, "3c07da06bdd1689f9d07af78fb94d0ec", 3101},
    {"resource.000", 0, "ecc686e0034fb4d41de077ac7167b3cf", 1947866},
    AD_LISTEND},
    Common::EN_ANY, Common::kPlatformDOS, 0, GUIO4(GUIO_NOSPEECH, GAMEOPTION_PREFER_DIGITAL_SFX, GAMEOPTION_ORIGINAL_SAVELOAD, GAMEOPTION_FB01_MIDI)    },

This is the entry that matches my copy. The 5000 byte hashes for resource.map and resource.000 are correct, and date in the version file is correct. Now, let's see if there's any specific modifications made for the game:

scott@hoagie:~/Development/scummvm/engines/sci$ grep -rni islandbrain
engine/workarounds.cpp:305: { GID_ISLANDBRAIN,   100,   937,  0,            "IconBar", "dispatchEvent",                   NULL,    58, { WORKAROUND_FAKE,   0 } }, // when using ENTER at the startup menu - bug #5241
engine/workarounds.cpp:306: { GID_ISLANDBRAIN,   140,   140,  0,              "piece", "init",                            NULL,     3, { WORKAROUND_FAKE,   1 } }, // first puzzle right at the start, some initialization variable. bnt is done on it, and it should be non-0
engine/workarounds.cpp:307: { GID_ISLANDBRAIN,   200,   268,  0,          "anElement", "select",                          NULL,     0, { WORKAROUND_FAKE,   0 } }, // elements puzzle, gets used before super TextIcon
engine/workarounds.cpp:550: { GID_ISLANDBRAIN,   300,   300,  0,           "geneDude", "show",                       NULL,     0, { WORKAROUND_IGNORE,    0 } }, // when looking at the gene explanation chart - a parameter is an object
engine/workarounds.cpp:648: { GID_ISLANDBRAIN,   300,   300,  0,         "dudeViewer", "show",                       NULL,     0, { WORKAROUND_STILLCALL, 0 } }, // when looking at the gene explanation chart, gets called with 1 extra parameter
engine/workarounds.cpp:670: { GID_ISLANDBRAIN,   290,   291,  0,         "upElevator", "changeState", sig_kGraphSaveBox_ibrain_1,     0, { WORKAROUND_STILLCALL, 0 } }, // when testing in the elevator puzzle, gets called with 1 argument less - 15 is on stack - bug #4943
engine/workarounds.cpp:671: { GID_ISLANDBRAIN,   290,   291,  0,       "downElevator", "changeState", sig_kGraphSaveBox_ibrain_1,     0, { WORKAROUND_STILLCALL, 0 } }, // see above
engine/workarounds.cpp:672: { GID_ISLANDBRAIN,   290,   291,  0,    "correctElevator", "changeState", sig_kGraphSaveBox_ibrain_1,     0, { WORKAROUND_STILLCALL, 0 } }, // see above (when testing the correct solution)
engine/workarounds.cpp:741: { GID_ISLANDBRAIN,   -1,   999,  0,                "List", "eachElementDo",                NULL,     0, { WORKAROUND_FAKE, 0 } }, // when going to the game options, choosing "Info" and selecting anything from the list, gets called with an invalid parameter (type "error") - bug #4989
engine/workarounds.cpp:869: { GID_ISLANDBRAIN,   300,   310,  0,         "childBreed", "changeState", sig_kStrAt_ibrain_1,     0, { WORKAROUND_FAKE,      0 } }, // when clicking Breed to get the second-generation cyborg hybrid (Standard difficulty), the two parameters are swapped - bug #5088
...

Yup. There's a few workarounds on file for this. Here's the C struct format for each workaround, taken from /sci/engine/workarounds.h:

enum SciWorkaroundType {
    WORKAROUND_NONE,      // only used by terminator or when no workaround was found
    WORKAROUND_IGNORE,    // ignore kernel call
    WORKAROUND_STILLCALL, // still do kernel call
    WORKAROUND_FAKE       // fake kernel call / replace temp value / fake opcode
};

struct SciWorkaroundSolution {
    SciWorkaroundType type;
    uint16 value;
};

/**
* A structure describing a 'workaround' for a SCI script bug.
*
* Arrays of SciWorkaroundEntry instances are terminated by
* a fake entry in which "objectName" is NULL.
*/
struct SciWorkaroundEntry {
    SciGameId gameId;
    int roomNr;
    int scriptNr;
    int16 inheritanceLevel;
    const char *objectName;
    const char *methodName;
    const uint16 *localCallSignature;
    int index;
    SciWorkaroundSolution newValue;
};

Seems pretty straightforward. Our game is GID_ISLANDBRAIN, and from that stack trace we know the room number is 180 and the script number is 185. Inheritance level I assume is 0, and again from the stack trace objectName is "flamingo" and methodName is "showHelp". index I assume for now is 0. localCallSignature is more interesting... in SCI code there are subroutines that can get called without names, and we need a way of identifying them without just saying "the code starts from this offset", as that isn't robust for multiple versions of the same game. Our stack trace was blaming localCall 5d6 for doing an invalid command, so we'll possibly need to identify it. newValue will be the workaround action we'll ask ScummVM to shim in, which we'll find out once we troubleshoot.

As this is an emulator, we are going to be using the internal debugger a lot. Internal debuggers are pretty daunting, as tradition dictates they have barely any documentation, plus you're grappling with some weirdo niche instruction set maybe 10 people know properly. That's ok! Most internal debuggers list all the allowed commands with "help", and as long as you have a very basic understanding of how assembled programs work (i.e. registers, pushing and popping things on the stack, jumps, conditionals), you can do pretty much anything you want in a debugger as soon as you find out how to do 3 things:

Look at the current state. For ScummVM's SCI engine, this is "registers" to see all the registers, "stack 10" to see the first 10 things pushed on the stack, "vm_vars [local|global|temp|param] id" to view a variable.
Run/step in/step over code. For ScummVM's SCI engine, this is "go" to continue execution, "s" to step in and "p" to step over.
Set code breakpoints. For ScummVM's SCI engine, the two I used were "bp_method object::methodName" to break at the start of a named method, and "bp_address [segment]:[offset]" to stop at a section of code.

# in ScummVM debug console
> disasm flamingos showHelp bc
...
0036:01b5: b5 01          sati      01
0036:01b7: 76             push0
0036:01b8: 85 00          lat       00
0036:01ba: b3 1c          sali      1c              ; 28
0036:01bc: c5 00          +at       00
0036:01be: 33 ea          jmp       ea  [01aa]
0036:01c0: 76             push0
0036:01c1: 40 11 04 00    call      0411  [05d5]    00
0036:01c5: 78             push1
0036:01c6: 39 14          pushi     14              ; 20, name
0036:01c8: 40 48 03 02    call      0348  [0513]    02
0036:01cc: 35 00          ldi       00
0036:01ce: a5 00          sat       00
0036:01d0: 8d 00          lst       00
0036:01d2: 39 04          pushi     04              ; cel
0036:01d4: 81 72          lag       72              ; 114, 'r'
0036:01d6: 02             add
0036:01d7: 22             lt?
0036:01d8: 31 1c          bnt       1c  [01f6]
...

In the stack trace, the second last line said the program counter for the bad call was 0036:01c5. To make sense of this, I recommend having a look in /engines/sci/engine/vm.cpp, which contains the opcode list.

Annoyingly I couldn't figure out how to make ScummVM disassemble the local method, so I had to rely on a third-party tool SCICompanion to do the dirty work and disassemble all of script 185.

05d6:3f 02             link 2 // (var $2)
05d8:35 00              ldi 0
05da:a5 00              sat temp0

        code_05dc
05dc:8d 00              lst temp0
05de:35 09              ldi 9
05e0:22                 lt?
05e1:31 22              bnt code_0605
05e3:39 03            pushi 3 // $3 loop
05e5:78               push1
05e6:85 00              lat temp0
05e8:9b 1c             lsli local28
05ea:39 03            pushi 3 // $3 loop
05ec:76               push0
05ed:93 00             lali local0
05ef:4a 04             send 4

05f1:36                push
05f2:35 01              ldi 1
05f4:12                 and
05f5:02                 add
05f6:36                push

05f7:38 00d9          pushi d9 // $d9 show
05fa:76               push0
05fb:85 00              lat temp0
05fd:93 00             lali local0
05ff:4a 0a             send a

0601:c5 00              +at temp0
0603:33 d7              jmp code_05dc

        code_0605
0605:48                 ret

05ef is where the last line of the stack trace claims the code stops executing. From my very tenuous understanding of the error, send is freaking out because the argument passed in through the accumulator register (0000:0000) isn't an object. And the lali instruction sets the accumulator to the value of local variable local0.

# in ScummVM console
> acc_object
Information on the currently active object or class at the address indexed by the accumulator:
[0000:0000]: Not an object.
> vm_vars local 0
local var 0 = 0027:0eb9 (object 'bird')

Uhhhhh. What. If local0 is not a null, why did that break?!

Okay. Reloading the game up to before the crash, I opened the console with CTRL + LSHIFT + D, and added a breakpoint in the broken method. I then triggered the puzzle and set a breakpoint for the code just before the crash, using the segment from the program counter.

# in ScummVM console
> bp_method flamingos::showHelp
  #0: Execute flamingos::showHelp
> exit

# open the puzzle
Break on flamingos::showHelp (in [0036:075c])
flamingos::showHelp(0000:0000) at 0036:019b
> registers
Current register values:
acc=0000:0001 prev=0000:0001 &rest=0
pc=0036:019b obj=0036:075c fp=ST:0036 sp=ST:0036
> bp_address 0036:05ec
> go

# click through
> acc_object
Information on the currently active object or class at the address indexed by the accumulator:
[0000:0000]: Not an object.
> vm_vars local 0
local var 0 == 0027:0eb8 (object 'bird')
# skip to 05ef
> p
> p
> acc_object
Information on the currently active object or class at the address indexed by the accumulator:
[0027:0eb8] bird :  34 vars,    2 methods
...

So the first time around we're entering the send instruction exactly as we should, with the bird object in the accumulator. This subroutine loops a few times, and every test run it crashed at a different iteration of the loop. Which I 100% do not get, as there is nothing in the subroutine that would set acc to the wrong thing! Could... could it be that the bug was in ScummVM all along?

Looks like we have to go a level deeper. If we run ScummVM inside gdb and stop it just before we execute the lali instruction, we can add a breakpoint to the code responsible for lali.

(gdb) b engines/sci/engine/vm.cpp:1284
Breakpoint 1 at 0x5555567b9750: file engines/sci/engine/vm.cpp, line 1284.

case op_lag: // 0x40 (64)
case op_lal: // 0x41 (65)
case op_lat: // 0x42 (66)
case op_lap: // 0x43 (67)
        // Load global, local, temp or param variable into the accumulator
case op_lagi: // 0x48 (72)
case op_lali: // 0x49 (73)
case op_lati: // 0x4a (74)
case op_lapi: // 0x4b (75)
        // Same as the 4 ones above, except that the accumulator is used as
        // an additional index
        var_type = opcode & 0x3; // Gets the variable type: g, l, t or p
        var_number = opparams[0] + (opcode >= op_lagi ? s->r_acc.requireSint16() : 0);
        s->r_acc = read_var(s, var_type, var_number);
        break;

Probing the variables with gdb, I realised I was mistaken. The "lali [arg]" instruction doesn't load local variable [arg] into acc, it loads local variable [arg+acc] into acc. And at that part of the code acc (stored on the stack) increases by 1 on each loop. Like an idiot I forgot the puzzle has 9 birds in it.

# in ScummVM console
> vm_vars local 0
local var 0 = 0027:0e09 (object 'bird')
> vm_vars local 1
local var 1 = 0027:0e07 (object 'bird')
> vm_vars local 2
local var 2 = 0027:0e04 (object 'bird')
> vm_vars local 3
local var 3 = 0027:0e03 (object 'bird')
> vm_vars local 4
local var 4 = 0027:0e0a (object 'bird')
> vm_vars local 5
local var 5 = 0027:0e09 (object 'bird')
> vm_vars local 6
local var 6 = 0000:0000
> vm_vars local 7
local var 7 = 0027:0e06 (object 'bird')
> vm_vars local 8
local var 8 = 0027:0e0b (object 'bird')

Well there's your problem. But where did that 7th bird go? Better yet, on other runs there are multiple birds missing from the list at very different spots. In a twist the showHelp command seems to be innocent, and it's really a setup process somewhere else that's botching up! Thankfully we only have one short disassembly to look through for the culprit (script 185), and from vm.cpp there's only a few opcodes which can set local variables.

Sure enough, in the flamingos::init method, there's an instruction "sali local0". I throw a breakpoint on it, then quizzed the VM for information before and after.

# in ScummVM console
> bp_method flamingos::init
...
> bp_address 0036:00ac
> go
...
> registers
Current register values:
acc=0000:0006 prev=0000:0009 &rest=0
pc=0036:00ac obj=0036:075c fp=ST:002b sp=ST:0042
> stack 5
ST:003d = 0000:0001
ST:003e = 0000:0006
ST:003f = 0000:0075
ST:0040 = 0000:0000
ST:0041 = 0027:0ec0
> p
> registers
Current register values:
acc=0027:0ec0 prev=0000:0009 &rest=0
pc=0036:00ae obj=0036:075c fp=ST:002b sp=ST:0041
> stack 5
ST:003c = 0000:003d
ST:003d = 0000:0001
ST:003e = 0000:0006
ST:003f = 0000:0075
ST:0040 = 0000:0000

As expected (after reading vm.cpp), "sali local0" pops the bird object from the top of the stack, and stores it at variable local[0+acc]. I did this for each iteration of the loop, and found something strange. I was expecting acc to randomly cycle through the numbers from 0 to 8 with each bird getting a slot, but sometimes the same number was coming up multiple times, which would overwrite a variable and leave a gap. Which leads to my next question; how is acc chosen?

00a1:8b 1b              lsl local27
00a3:8d 00              lst temp0
00a5:43 62 04         callk StrAt 4     // acc = StrAt(local[27], temp[0])

00a8:36                push             //
00a9:35 61              ldi 61          //
00ab:04                 sub             // acc -= 0x61
00ac:b3 00             sali local0      // index = 0+acc; acc = pop(); local[index] = acc

I really didn't get this at first. What the hell is StrAt? Why the arbitrary 0x61 offset?

It's not that bad. "callk" is the opcode for a so-called kernel call, which is basically predefined C code in the SCI VM. It took me way too long to realise, but acc = StrAt(string, index) is the equivalent of acc = string[index], which seems ridiculous but okay. And what was the string, I hear you ask?

Thread 1 "scummvm" hit Breakpoint 1, Sci::kStrAt (s=0x555558dff180, argc=2, argv=0x555558e7292c) at engines/sci/engine/kstring.cpp:88
88          if (argv[0] == SIGNAL_REG) {
(gdb) l
83          return argv[0];
84  }
85
86
87  reg_t kStrAt(EngineState *s, int argc, reg_t *argv) {
88          if (argv[0] == SIGNAL_REG) {
89                  warning("Attempt to perform kStrAt() on a signal reg");
90                  return NULL_REG;
91          }
92
(gdb) p argv
$1 = (Sci::reg_t *) 0x555558e7292c
(gdb) p argv[0]
$2 = {
_segment = 54,
_offset = 2100
}
(gdb) p argv[1]
$3 = {
_segment = 0,
_offset = 0
}
(gdb) p argc
$4 = 2
(gdb) n
93          SegmentRef dest_r = s->_segMan->dereference(argv[0]);
(gdb) n
94          if (!dest_r.isValid()) {
(gdb) p dest_r
$5 = {
isRaw = true,
{
    raw = 0x555558fa70a4 "cdhfagbbi",
    reg = 0x555558fa70a4
},
maxSize = 54,
skipByte = false
}

Oh. Ohhhhhh. So there's ANOTHER method that's generating a 9 character string with the letters from a to i to indicate the bird position in the variables list, and storing the pointer to it in local[27]. 0x61 is the ASCII character 'a', so subtracting that from each letter gives an index from 0 to 8. Does that seriously mean the SCI VM doesn't have any array types other than byte strings? If so it's pretty wild they made all these math based puzzles.

# in ScummVM console
> bp_method flamingos::init
  #0: Execute flamingos::init
> exit

# open the puzzle
Break on flamingos::init (in [0036:075c])
flamingos::init() at 0036:002c
> vm_vars local 27
local var 27 == 0036:0834 (reference)
> view_reference 0036:0834
0036:0834 is of type 0x10: raw data
000000: 61 62 63 64  65 66 67 68  69 00 52 65  73 65 74 74   |abcdefghi.Resett|
000010: 69 6e 67 00  66 6c 61 6d  69 6e 67 6f  73 00 66 6c   |ing.flamingos.fl|
000020: 61 6d 57 69  6e 00 62 69  72 64 00 00  04 00 3a 00   |amWin.bird....:.|
000030: 18 01 72 01  aa 01                                   |..r...          |

flamingos::init() starts with local[27] pointing to the pre-baked string 'abcdefghi', then does some operations to shuffle the contents around. This process fails and overwrites parts of the string 10-20% of the time. In fact, the 6th instruction in flamingos::init() throws to a subroutine, and checking before/after we know that subroutine is responsible for doing the shuffle.

Because I don't know the instruction set that well, usually I go through and translate each instruction to the C/Python equivalent, in this case using the interpreter in vm.cpp as a guide, consolidating lines as I go. Here's the copy with my annotations:

(procedure proc_04c3                      // param[1] = local[27]
04c3:3f 03             link 3             // link(3)
04c5:78               push1               // # 1 arg
04c6:8f 01              lsp param1        //
04c8:43 46 02         callk StrLen 2      // acc = StrLen(param[1])

04cb:a5 00              sat temp0         // temp[0] = acc

        code_04cd
04cd:85 00              lat temp0         // acc = temp[0]
04cf:31 42              bnt code_0513     // if (acc == 0) goto code_0513
04d1:7a               push2               // # 2 args
04d2:78               push1
04d3:36                push
04d4:43 3c 04         callk Random 4      // acc = Random(1, acc)

04d7:36                push               //
04d8:35 01              ldi 1             //
04da:04                 sub               // acc -= 1
04db:a5 02              sat temp2         // temp[2] = acc
04dd:7a               push2               // # 2 args
04de:8f 01              lsp param1        //
04e0:36                push               //
04e1:43 62 04         callk StrAt 4       // acc = StrAt(param[1], acc)

04e4:a5 01              sat temp1         // temp[1] = acc
04e6:7a               push2               // # 2 args
04e7:8f 01              lsp param1        //
04e9:85 02              lat temp2         //
04eb:02                 add               // acc = param[1] + temp[2]
04ec:36                push               // dest = acc
04ed:8f 01              lsp param1        //
04ef:85 02              lat temp2         //
04f1:02                 add               // acc = param[1] + temp[2]
04f2:36                push               //
04f3:35 01              ldi 1             //
04f5:02                 add               // acc += 1
04f6:36                push               // src = acc
04f7:43 47 04         callk StrCpy 4      // strcpy(dest, src); acc = src

04fa:39 03            pushi 3             // # 3 args
04fc:8f 01              lsp param1        //
04fe:39 08            pushi 8             //
0500:8d 01              lst temp1         //
0502:43 62 06         callk StrAt 6       // acc = StrAt(param[1], 8, temp[1])

0505:39 03            pushi 3             // # 3 args
0507:8f 01              lsp param1        //
0509:39 09            pushi 9             //
050b:76               push0               //
050c:43 62 06         callk StrAt 6       // acc = StrAt(param[1], 9, 0)

050f:e5 00              -at temp0         // temp0 -= 1
0511:33 ba              jmp code_04cd     // if (temp0 != 0) goto code_04cd

        code_0513
0513:48                 ret               // return
)

Easy. For 9 iterations, the shuffler picks a random point in the string, copies all the characters to the right of it one space left, then re-adds the missing character and a null to the end.

Right I think I can guess the twist, it's going to be the Random function doing something stupid. If I was a gambling man, probably Random(1, 1) will do something idiotic like return 0 and balls up the shuffling math.

# in ScummVM console
> bp_address 0036:04d4
> bp_address 0036:04f7
...
Break at 0036:4d4
> p
> registers
Current register values:
acc=0000:0001 prev=0000:0001 &rest=0
pc=0036:04d7 obj=0036:075c fp=ST:0030 sp=ST:0033
...
0036:04f7: 43 47 04       callk     StrCpy[47],     04
Kernel params: (0036:0834, 0036:0835)
000000: 61 62 64 65  66 67 68 69  63 00 52 65  73 65 74 74   |abdefghic.Resett|
000010: 69 6e 67 00  66 6c 61 6d  69 6e 67 6f  73 00 66 6c   |ing.flamingos.fl|
000020: 61 6d 57 69  6e 00 62 69  72 64 00 00  04 00 3a 00   |amWin.bird....:.|
000030: 18 01 72 01  aa 01                                   |..r...          |
Step #144526
0036:04fa: 39 03          pushi     03              ; loop
000000: 62 64 65 66  67 69 63 63  00 00 52 65  73 65 74 74   |bdefgicc..Resett|
000010: 69 6e 67 00  66 6c 61 6d  69 6e 67 6f  73 00 66 6c   |ing.flamingos.fl|
000020: 61 6d 57 69  6e 00 62 69  72 64 00 00  04 00 3a 00   |amWin.bird....:.|
000030: 18 01 72 01  aa 01                                   |..r...          |

I lose. Random is working fine. It was StrCpy; the one in ScummVM's SCI engine doesn't react well when the source and destination strings overlap!

And by "the one in ScummVM's SCI engine", I mean glibc. The C spec says the behaviour in the case of overlapping is undefined, and some individuals have taken that to mean that it's 100% A-OK to change that behaviour and (why not) break every closed-source program that relies on it. In fact, this legendary bug report has those two titans of community outreach and civil discourse, Linus Torvalds and Ulrich Drepper, arguing whether or not it was fine to break Adobe Flash for an alleged infintessimal speed boost you got on x86 by copying things in reverse.

The solution

As the bug was in one of the SCI kernel functions, we didn't need to add anything to the workaround table after all. The fix was to replace the SCI kernel's underlying calls to strcpy/strncpy with strlen + memmove, which allow an overlapping source and destination buffer. Voila, awful flamingo puzzle works.

diff --git a/engines/sci/engine/seg_manager.cpp b/engines/sci/engine/seg_manager.cpp
index aa70d5f838..3b65e78d52 100644
--- a/engines/sci/engine/seg_manager.cpp
+++ b/engines/sci/engine/seg_manager.cpp
@@ -624,11 +624,18 @@ void SegManager::strncpy(reg_t dest, const char* src, size_t n) {


        if (dest_r.isRaw) {
+               size_t len = ::strlen(src);
                // raw -> raw
-               if (n == 0xFFFFFFFFU)
-                       ::strcpy((char *)dest_r.raw, src);
-               else
-                       ::strncpy((char *)dest_r.raw, src, n);
+               if (n == 0xFFFFFFFFU) {
+                       ::memmove((char *)dest_r.raw, src, len+1);
+               } else {
+                       if (n < len) {
+                               ::memmove((char *)dest_r.raw, src, n);
+                       } else {
+                               ::memmove((char *)dest_r.raw, src, len);
+                               ::memset((char *)dest_r.raw+len, 0, n-len);
+                       }
+               }
        } else {
                // raw -> non-raw
                for (uint i = 0; i < n; i++) {

(ScummVM pull request #1030)

Show Your Working: Turning off PulseAudio "flat volumes"

2017-07-23T00:00:00+08:00

The software

PulseAudio is the software audio mixer for most desktop Linux distributions. Let's not mince words; PulseAudio has an awful reputation, mostly earned from the botched rollout by Ubuntu and other distributions back in 2008. Long-time Linux users remember these as the dark years where games were unplayable, and everything was punctuated by a loud crackling from the constant buffer underruns. The most egregious problems were caused by the aforementioned shonky buffering, plus a reliance on timing-related ALSA features which had never been used/tested properly for many sound drivers. It didn't help that barely any applications had direct support for the PulseAudio API; there were compatibility shims for ALSA/OSS applications, but they were a crapshoot at best.

But that was a decade ago! We've moved from version 0.9 to version 11.1, a metric ass-ton of effort has gone into making the mixer first rate, most Linux sound libraries are fully Pulse compatible, etc. etc. Surely everything is perfect now, right?

The problem

Ahahahaha not quite. Five years ago PulseAudio added a feature called "flat volumes"; wherein by default, every per-application volume control is welded in some byzantine way to the master volume control. Completely ignoring that, you know, everyone assumed that per-application volume controls actually meant PER-APPLICATION, and quite a lot of apps had a knack for setting the volume to 100% on startup (or just at random intervals)! Cue newly-deafened users coming out in droves to complain, and package maintainers grumping about one more non-upstream setting they'll have to override.

Anyway. Disabling flat volumes reverts things back to per-application volumes scaled (post-fader) by an independent master volume, and all is right in the land once again. As of... recently? Disabling flat-volumes in /etc/pulse/daemon.conf stopped working on my system.

The preparation

The only software we need installed for tracking this down is the Git VCS and the standard Linux command line tools, plus whatever build dependencies PulseAudio has. Start by grabbing the PulseAudio source code of the release we know is bad.

$ git clone https://anongit.freedesktop.org/git/pulseaudio/pulseaudio
$ cd pulseaudio
# commit taken from https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/pulseaudio&id=f9dd2850e5eff2d2a7fde2af6e932d785a521912
$ git checkout 84952e6a092b6a0c5b153bd7a4f6e490810681c8

The diagnosis

Let's track down what went wrong. We know one thing for sure: the name of the config switch which is meant to fix this ("flat-volumes"). Let's find where it's used.

$ grep -rn flat-volumes *
man/pulse-daemon.conf.5.xml.in:220:      <p><opt>flat-volumes=</opt> Enable 'flat' volumes, i.e. where
src/daemon/daemon-conf.c:531:        { "flat-volumes",               pa_config_parse_bool,     &c->flat_volumes, NULL },
src/daemon/daemon-conf.c:740:    pa_strbuf_printf(s, "flat-volumes = %s\n", pa_yes_no(c->flat_volumes));
src/daemon/daemon.conf.in:61:; flat-volumes = yes

If I had to pick the single most important tool for troubleshooting something you've never seen before, it would be grep. It does two things extremely well: search a bunch of files or piped input for a text/regex match, and adjust the amount of context you get back for the findings. grepping is the de-facto method to scrub through a ton of source code quickly to establish causality; the chain of events linking the thing we change (i.e. the config file) to the expected action (i.e. the volume adjusting as it damn well should).

Now we follow the trail of breadcrumbs and zero in on the code path which does the useful work. We can see above that the "flat-volumes" config parser sets a variable called flat_volumes, so we search for that.

$ grep -rn flat_volumes *
src/daemon/daemon-conf.c:71:    .flat_volumes = true,
src/daemon/daemon-conf.c:531:        { "flat-volumes",               pa_config_parse_bool,     &c->flat_volumes, NULL },
src/daemon/daemon-conf.c:740:    pa_strbuf_printf(s, "flat-volumes = %s\n", pa_yes_no(c->flat_volumes));
src/daemon/daemon-conf.h:76:        flat_volumes,
src/daemon/main.c:1044:    c->flat_volumes = conf->flat_volumes;
src/pulsecore/core.c:138:    c->flat_volumes = true;
src/pulsecore/core.h:197:    bool flat_volumes:1;
src/pulsecore/sink.c:541:    enable = enable && s->core->flat_volumes;
src/pulsecore/source.c:492:    enable = enable && s->core->flat_volumes;

Okay, the only plased where flat_volumes is used to drive a decision is in src/pulseaudio/sink.c and src/pulseaudio/source.c. We'll focus in on sink.c as "sink" is PulseAudioese for "sound output".

$ grep -rn -B8 -A16 flat_volumes src/pulsecore/sink.c
533-}
534-
535-static void enable_flat_volume(pa_sink *s, bool enable) {
536-    pa_sink_flags_t flags;
537-
538-    pa_assert(s);
539-
540-    /* Always follow the overall user preference here */
541:    enable = enable && s->core->flat_volumes;
542-
543-    /* Save the current flags so we can tell if they've changed */
544-    flags = s->flags;
545-
546-    if (enable)
547-        s->flags |= PA_SINK_FLAT_VOLUME;
548-    else
549-        s->flags &= ~PA_SINK_FLAT_VOLUME;
550-
551-    /* If the flags have changed after init, let any clients know via a change event */
552-    if (s->state != PA_SINK_INIT && flags != s->flags)
553-        pa_subscription_post(s->core, PA_SUBSCRIPTION_EVENT_SINK|PA_SUBSCRIPTION_EVENT_CHANGE, s->index);
554-}
555-
556-void pa_sink_enable_decibel_volume(pa_sink *s, bool enable) {
557-    pa_sink_flags_t flags;

We can see that in terms of honouring the setting, nothing has changed. Enabling flat volume is always overridden by what the user has set, and the actual state is stored in a variable flags with the bitflag name PA_SINK_FLAT_VOLUME. Guess what we're searching for next???

$ grep -rn PA_SINK_FLAT_VOLUME *
src/modules/dbus/iface-device.c:461:    has_flat_volume = (d->type == PA_DEVICE_TYPE_SINK) ? !!(d->sink->flags & PA_SINK_FLAT_VOLUME) : FALSE;
src/modules/dbus/iface-device.c:841:        has_flat_volume = !!(d->sink->flags & PA_SINK_FLAT_VOLUME);
src/modules/module-virtual-sink.c:564:        u->sink->flags |= PA_SINK_FLAT_VOLUME;
src/modules/module-virtual-surround-sink.c:702:        u->sink->flags |= PA_SINK_FLAT_VOLUME;
src/pulse/def.h:801:    PA_SINK_FLAT_VOLUME = 0x0040U,
src/pulse/def.h:838:#define PA_SINK_FLAT_VOLUME PA_SINK_FLAT_VOLUME
src/pulsecore/cli-text.c:286:            sink->flags & PA_SINK_FLAT_VOLUME ? "FLAT_VOLUME " : "",
src/pulsecore/sink.c:547:        s->flags |= PA_SINK_FLAT_VOLUME;
src/pulsecore/sink.c:549:        s->flags &= ~PA_SINK_FLAT_VOLUME;
src/pulsecore/sink.c:1554: * When a sink uses volume sharing, it never has the PA_SINK_FLAT_VOLUME flag
src/pulsecore/sink.c:1563:        return (s->flags & PA_SINK_FLAT_VOLUME);
src/pulsecore/sink.h:441:/* Use this instead of checking s->flags & PA_SINK_FLAT_VOLUME directly. */

Now THIS is a great example of why you want to grep as much source code as possible. We could have continued our investigation just in sink.c, but searching everywhere revealed something super useful! Namely src/pulsecore/cli-text.c; there is a command line thing somewhere which prints if PA_SINK_FLAT_VOLUME is set or not. This just potentially saved us having to make a debug build to find this out.

A little bit of poking around revealed that the string is printed when you invoke pacmd list-sinks (not pactl list sinks! that of course produces output that's near identical, but missing the one thing we want.), so we did that.

$ pacmd list-sinks | grep -B8 -A8 FLAT
6 sink(s) available.
    index: 0
    name: <alsa_output.pci-0000_2a_00.3.iec958-stereo>
    driver: <module-alsa-card.c>
    flags: HARDWARE HW_MUTE_CTRL DECIBEL_VOLUME LATENCY FLAT_VOLUME DYNAMIC_LATENCY
    state: IDLE
    suspend cause:
    priority: 9058
    volume: front-left: 0 /   0% / -inf dB,   front-right: 0 /   0% / -inf dB
            balance 0.00
    base volume: 65536 / 100% / 0.00 dB
    volume steps: 65537
    muted: no
--
    ports:
        iec958-stereo-output: Digital Output (S/PDIF) (priority 0, latency offset 0 usec, available: unknown)
            properties:

    active port: <iec958-stereo-output>
* index: 1
    name: <jack_out>
    driver: <module-jack-sink.c>
    flags: DECIBEL_VOLUME LATENCY FLAT_VOLUME
    state: IDLE
    suspend cause:
    priority: 0
    volume: front-left: 22925 /  35% / -27.37 dB,   front-right: 22925 /  35% / -27.37 dB
            balance 0.00
    base volume: 65536 / 100% / 0.00 dB
    volume steps: 65537
    muted: no

Okay well there's the problem; the sink I'm using (JACK to my Scarlett box) has FLAT_VOLUME enabled. But why. Whyyyyyyyyy. I mean I told it not to!

No getting around it, we need to have a look at the program while it's running. Let's make a test build.

# copying the build steps from the PKGCONFIG
# by default CFLAGS should include -g, which adds debug symbols
$ ./bootstrap.sh
$ ./configure --prefix=/usr\
     --sysconfdir=/etc \
     --libexecdir=/usr/lib \
     --localstatedir=/var \
     --with-udev-rules-dir=/usr/lib/udev/rules.d \
     --with-database=tdb \
     --disable-tcpwrap \
     --disable-bluez4 \
     --disable-samplerate \
     --disable-rpath \
     --disable-default-build-tests \
     DATADIRNAME=share
$ make

There's now a freshly built pulseaudio executable in the src directory. Except it's not, it's some libtool shell spew that passes enough lies to the built copy (in .libs) to make it run. Which is a problem, because you can't run gdb against shell spew.

Because they're not all heartless bastards, the libtool developers made a wrapper for the wrapper which allows pass-through execution. Before we start the session, make sure PulseAudio isn't set to autospawn, kill the current daemom, and then fire up the debugger.

$ pulseaudio -k
$ libtool --mode=execute gdb --args pulseaudio
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
...
(gdb)

Smashing. All of the functions we want to test are in shared libraries, which we can only see after execution starts, so lets breakpoint immediately after starting.

(gdb) b main
Breakpoint 1 at 0x52e0: file daemon/main.c, line 369.
(gdb) r
Starting program: /home/scott/Development/pulseaudio/src/.libs/lt-pulseaudio
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at daemon/main.c:369
369 int main(int argc, char *argv[]) {

We'll start by adding a breakpoint in enable_flat_volume() and seeing what's going down there.

(gdb) b pulsecore/sink.c:536
Breakpoint 2 at 0x7ffff7b75690: pulsecore/sink.c:536. (2 locations)
(gdb) c
Continuing.
W: [lt-pulseaudio] main.c: /proc/self/exe does not point to /usr/bin/pulseaudio, cannot self execute. Are you playing games?
N: [lt-pulseaudio] daemon-conf.c: Detected that we are run from the build tree, fixing search path.
E: [lt-pulseaudio] module-alsa-card.c: Failed to find a working profile.
E: [lt-pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="3" name="usb-Clavia_DMI_AB_Nord_Electro_5-01" card_name="alsa_card.usb-Clavia_DMI_AB_Nord_Electro_5-01" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes card_properties="module-udev-detect.discovered=1""): initialization failed.

Breakpoint 2, enable_flat_volume (s=0x555555877570, enable=true)
    at pulsecore/sink.c:538
538     pa_assert(s);

Let's do a backtrace to see where the method got called from.

(gdb) bt
#0  enable_flat_volume (s=0x5555558734f0, enable=true) at pulsecore/sink.c:541
#1  0x00007ffff7b77110 in pa_sink_enable_decibel_volume (s=0x5555558734f0,
    enable=<optimized out>) at pulsecore/sink.c:566
...

The default behaviour seems to be to enable flat volumes if the sink identifies itself as having decibel-based volume control? I... you know what I won't even pretend to understand why. So what happened to the config file? Isn't it meant to override this?

(gdb) p s->core->flat_volumes
$3 = true

I knew it! You betrayed me, config parser!

(gdb) b daemon/daemon-conf.c:607
Breakpoint 6 at 0x55555555f3a9: file daemon/daemon-conf.c, line 610.
(gdb) c
Continuing.

Breakpoint 6, pa_daemon_conf_load (c=0x55555578d990, filename=0x0)
    at daemon/daemon-conf.c:607
...
612         pa_open_config_file(DEFAULT_CONFIG_FILE, DEFAULT_CONFIG_FILE_USER, ENV_CONFIG_FILE, &c->config_file);
...
(gdb) b pa_open_config_file
Breakpoint 7 at 0x7ffff6d5d580: file pulsecore/core-util.c, line 1993.
(gdb) c
Continuing.

Breakpoint 7, pa_open_config_file (
    global=global@entry=0x5555555638e1 "/etc/pulse/daemon.conf",
    local=local@entry=0x5555555638eb "/daemon.conf",
    env=env@entry=0x5555555638d4 "PULSE_CONFIG",
    result=result@entry=0x55555578d9e8) at pulsecore/core-util.c:1993
1993        FILE *pa_open_config_file(const char *global, const char *local, const char *env, char **result) {

Okay mea culpa time. I stupidly thought it would load /etc/pulse/daemon.conf first, which has the line 'flat-volumes = no' in it, then any changes made in the local config file (~/.pulse/daemon.conf) would override those base settings. We can see the truth by reading the code for pa_open_config_file; what ACTUALLY happens is that PulseAudio only reads the first config file it can find (first it tries the path in environment variable PULSE_CONFIG, then ~/.pulse/daemon.conf, then /etc/pulse/daemon.conf).

So because I had a near-empty ~/.pulse/daemon.conf which didn't define flat-volumes, the flat-volumes setting was forced on. I don't even know where the file came from, it has a last-modified date of two days ago. Urghhhh.

The solution

Remove ~/.pulse/daemon.conf. Restart PulseAudio.

Balls to learning how to animate, let's film some parkour!

2016-08-27T00:00:00+08:00

Hallå! Welcome to my first game devlog.

I figured I'd start out talking about graphics, as they are a large source of anxiety for first-time developers. And not without reason! Everyone is a judgy bastard when it comes to how your low-budget game is presented. Oh sure, it's become easier for a solo developer to mask the familiar, rancid stench of programmer art with the equivalent of some pine-scented air freshener, but in practice this is a huge creative tradeoff. It really pays off to have a unique and consistent visual style in your work, even if you're not a proper artist.

Which I'm not. Lord no. What little visual talent I have is spread thinly between drafting/technical drawing (a survival skill picked up working as an engineer) and sandwich-grade graphic design. Oh and I can maybe do a scratchy pencil test of a background at 1/8th the speed of a regular artist. Drawing actual characters and animating them? That's just not going to happen.

But I wanted to do both of those things! So I figured the best way to fly closer to the sun would be to create an art and asset pipeline that played to my strengths, making the process for churning out Fine Art™ as close as possible to drafting. My weapon of choice for Fine Art™ is the vector editor Inkscape; very easy to pick up, and with a little practice can produce some polished results. I especially like the CAD-like pixel-perfect control you can use to make things like schematics or icons. Saved my bacon loads of times.

The only downside is that Inkscape, while wonderful, is not an animation editor. And even if it was, there was still the matter of sprouting talent from somewhere to draw realistic human movement. And I wasn't satisfied with the options of buying pre-fabbed animation packs from an asset store (artistic license strongly urges me to include a diverse range of player movements) or even commissioning someone competent to do it all (artistic license strongly urges me to be a cheap bastard). If only there was a way for dunce programmers like myself to cheat their way across this insurmountable art gap...

Rotoscoping

Let's wind the clock back to Prince of Persia (yes that one, from 1989); critically acclaimed, and one of the best-known examples in computer game history of programmer art surpassing regular art. Along with its predecessor Karateka, POP was one of the first home video games to use rotoscoped animation. And just look at it! Even on the Apple II with only 4 colours it's mind-blowingly detailed and smooth!

But how was it done? The ingenious process is detailed in Jordan Mechner's "Making of Prince of Persia" diary:

Mechner would film his brother David performing a movement (e.g. climbing onto a wall) freehand with a VHS camcorder
The cassette was taken to a professional edit suite with a space age VHS deck that supported freeze-frame, and for every frame of animation at 15fps, Mechner would take a film photograph of the screen.
After the roll of film was developed Mechner would hand-draw a high contrast monochrome silhouette on each photo; blacking out the background with permanent marker and using white paint to add in shape detail rendered blurry by the video.
Finally, the finished sheet of silhouettes was mounted in front of the camcorder. The analogue video out was fed into an Apple II fitted with a GenLock device; no fancy editing software here, all it did was replace the black background of the Apple II screen with the live video, whereupon Mechner painted over the image again but digitally!

Another similar use of rotoscoping was by Eric Chahi in his masterpiece Another World. The setup was similar to Mechner's in that Chahi would film himself with a tripod-mounted VHS camcorder. The main difference is he owned a VHS deck with freeze-frame capability; he had this hooked up to his Amiga GenLock and would draw polygons over each frame of the frozen recording from the VHS cassette. In his notes Chahi describes this process as "a time attack race"; a paused VHS deck stops showing the picture after a minute or so to avoid ruining the playback heads. This setup was mostly used to animate closeups and cutscenes; by this point Chahi was already an accomplished game artist, and from what I gather drew a lot of the player movement animations freehand.

Instead of pixel-based bitmap graphics, most of the art was made with solid-filled polygons. At the time it was an innovative way to fit cinematic visuals into 2 floppy disks worth of space. Indeed, publisher Delphine Software found massive success using the same style in their next game Flashback, shortly before dooming the company with Shaquille O'Neal fighting simulator Shaq-Fu.

In the Another World engine, animation is stored as a fixed-FPS series of static keyframes. That is to say, every frame is a dumb list of pre-baked polygons. This can be rendered extremely quickly; no preprocessing or transformation was required, which is really good news on pride-of-the-90s computer hardware with no floating point processor. But this format has the same limitation as sprites: each animation has one native speed (~15fps), which looks jarring if the refresh rate of the device is higher (e.g. mobile devices run at 60fps). Also this can be a gruelling process for the animator, as they have to spend much of their time redrawing frame after frame of largely the same collection of body parts.

Regardless, I really liked this visual style of simple filled polygons. Most projects nowdays use flat-shaded 3D models to achieve this look (e.g. Toryansé, Kentucky Route Zero). In fact, if you have any measurable skill in the fields of modelling and rigging, this approach is considerably cheaper than creating 2D animation frames. (I'm unburdened by either, so pure expensive 2D it is!)

Skeletal animation

3D games have moved away from static animation techniques and instead rely on skeletal animation: that is, the skin of the character is created in 3D from vertices joined up as triangles, then rigged around a series of bones and joints which define the ranges of movement for each appendage. Bones and joints are just a convenient analogy to something we are already familiar with; the musculoskeletal system of an animal. In reality, they are a stack of mathematical operations used by the renderer to warp the vertices of the skin in real time.

What does this mean? Well, with Chahi's approach, for each animation keyframe you'd be setting the position of every vertex in the model. It's cheap to compute, but if you have lots of geometry you will waste a ton of time streaming the data in for every single frame. Also as mentioned before, a fixed FPS animation will look stuttery if your engine is drawing the rest of the world at a faster framerate.

Skeletal animation solves both these problems; you upload a single copy of the character model into memory, and your renderer uses a tiny list of bones to transform said model on the fly. Better still, you can make the animation buttery smooth by interpolating between frames. The traditional cheapo way doing this was linear interpolation (storing each bone frame as two points in (x, y) space); nowdays 2D animators prefer polar interpolation (storing each bone frame as a rotation angle and a length). Here's a demo of how both styles look from the same low-FPS source material:

At first it doesn't look like much of a difference, but play close attention to the length of the character's legs around the knee joint as they swing over the bar. The linear tween looks weird and telescopy, because the lengths of the bones aren't preserved during the diagonal point-to-point movement. And as rotations/scaling are both fairly cheap and easy to do on today's graphics hardware, going for polar is a no-brainer.

Writing your own tools

If you haven't guessed from all that foreshadowing, my plan is to use rotoscoping, 2D vector graphics and skeletal animation. To my advantage I have the luxury of writing my engine for today's graphics hardware, and the technology to capture and edit HD video is super cheap.

There was one missing piece: tooling. When I was starting out in 2012, what I really needed was an animation tool that could do the following:

Construct a 2D skeleton with named bones
Use a feed of video frames as a background
Allow binding vector assets to each bone and editing them frame-by-frame
Provide some simple controls for adding metadata (e.g. set the relative scale, define a fixed point in space like a ledge, define the distance covered by a walk cycle)
Output all of the above in some format I can use in my asset pipeline

You'll be shocked to learn that I couldn't find a tool that did these things! The closest was Flash, which was a fairly ugly choice to contemplate, or a 3D animation tool like Maya or Blender, which (after a few hours of YouTube tutorials encouraging me to unlearn all of my Inkscape know-how) I had zero interest in learning how to use.

If I were starting out again I would probably consider Spine for skeletal animation; they seem to have fixed nearly all of the grievences people had with Flash, and it supports a lot of game engines out of the box. No idea if it supports rotoscoping, but the workflow looks pretty solid. Also worth checking out is the rotoscoping paint tool Paint of Persia, if pixel art is your forte.

In the end I wrote my own animation tool, jaiwhite. It's spartan and has the above features and nothing else. It's not the best tool, but by writing it I learned a fair amount about animation formats and making the data useable in a game engine.

The first thing I ever animated with it was a clip from the film Caddyshack: the iconic "Start a Party With a Radio in Your Golf Bag" scene. I rotoscoped it at the full 24fps of the source material, here's a playback capture at 48fps:

That went quite well, if a little jittery, so I moved onto phase 2: filming parkour.

Le Parkour

I have been training as a traceur for 4 years. At some stage I intend to write a full piece about how parkour has changed my life for the better, and how much I love the philosophy behind it, and how the wider parkour community is one of the most wonderfully diverse, amazing, inclusive groups of humans you'll ever come across. For now I'll leave you with this: if you've ever thought about trying parkour, go for it. Find a local parkour association and see if they do classes. You absolutely do not have to be a tank to start out, but if you persist you will become a tank. And for the love of God ignore everything on YouTube.

Where was I? Oh yeah. Filming. I used the following:

1x computer (Toshiba US build-a-PC)
1x Canon Vixia HF M500 NTSC camcorder + cheap SD card (Amazon US)
1x Fancier FT-6222A tripod (mystery HK seller on eBay)
1x Lufkin Autolock 8m tape measure (no idea, maybe Bunnings?)

The camera I have is not the best. It films in 720p30 and has lots of motion blur, but it's more than acceptable for low-rent mocap. If you have sunk all your money nursing an expensive photography habit, a DSLR with a rapid-fire capture feature makes nice crisp frames to animate with. I wouldn't recommend using a GoPro/other fixed-focus action camera, as the fish eye lens would distort the footage too much.

After a few practice runs I found a good rhythm for recording. I'd go to a place with an unobstructed view of an obstacle, set the camera up to frame the obstacle square-on with plenty of side margins to capture the movement's entrance and exit, mark out 1 metre on the ground with the tape measure, and then go ahead and perform the same action multiple times. After each take I would run past the camera and double-tap the record button to switch to a new file, then immediately go back for another shot.

This worked out pretty well. When you drill a movement in parkour there's a sort of bell curve for smoothness and control; the first attempts are a little rusty as you size up the obstacle, but after a few tries muscle memory will outgrow hesitation and you can focus entirely on form. Until you fatigue, at which point form is the first thing to get thrown under the bus.

Editing

Great! We have an SD card full of parkour footage. What comes next?

If your answer was "some light filing", have a biscuit. Nothing fancy, just renaming all of the takes from today to have the name of the movement. Now out of the footage collected, pick a movement and go find the least silly-looking take. Eventually you'll realise that there aren't any (because you're an adult) and just guess which one'll animate the cleanest. I use ffmpeg to convert the video to a series of still images to use as keyframes.

For this runthrough I've picked a dash vault. On a practical level these are pretty useless; a speed vault or cat pass will do just as well for clearing an obstacle, with a much lower risk of catching both your feet on the edge and faceplanting. But they do wonders for your self-esteem and I'm about as shallow as they come! On a more technical level this is a good animation to run as a test because all of the limb movements happen exactly (if you squint) along the viewing plane, meaning a single copy of the limbs drawn in at a side-facing angle should be enough for the whole animation. Other more complicated movements (e.g. lazy vaults, rolls) will have limbs overtly pointing towards/away from the screen, which might be better handled with multiple angles.

Let's run through the rotoscoping process. I start out by marking some useful points on the ground: the length of the 1 metre tape so that each animation has the same scale, and the start and finish point of the animation to create a walk cycle. There are other markers for e.g. ledges, to indicate a climbing animation around a fixed point, but we don't need them. Next, the bones are drawn in. My model has 19 bones; 4 for each limb, 2 for the back, one for the head. The bones are stored as a flat tree structure, with each keyframe storing relative polar coordinates plus depth for all the bones. I set the base of the neck as the "anchor" point from which all of the bones are defined, and ground level between the feet as the (x, y) origin of the model to match the physics engine.

There's a common naming scheme for the bones (e.g. head, forearm-l) so that multiple animations can use the same skeleton tree. We can make a reverse copy of a directional animation for free by replacing "-l" with "-r" and inverting all the x coordinates. If the animation needs to span two depth planes (e.g. that lazy vault animation I showed earlier), there's a hack where you can add a non-functional "plane" bone to represent the barrier.

After that, you pose the skeleton to match each frame of footage. Unlike Spine, jaiwhite has zero safeguards. You drag the nodes around until the pose looks okay. Plus there's no undo! I'm guessing this is very unhelpful for consistency if you're animating out of thin air? But we aren't, so... eat that, nerds! For my 30fps source video, I found that sampling at 10fps kept a lot of detail while smoothing out the jitter.

Rigging

And now the hardest part by far: drawing a vector representation of the body in Inkscape, and mapping it to the skeleton. By "drawing" I of course mean "noob tracing". All of the above was based on stuff I did over the previous couple of years. This? This is brand new. I have put it off for ages.

It's important to expand on why have I put it off for ages. I kept thinking about where I wanted to be (quickly-prototypable easy 2D rigging for my skeletons that I can churn out) in terms of what I already had (an animation format without the ability to store hints for the vector graphics format, an animation tool with semi-working support for drawing vector graphics, some dinky Python scripts), and kept getting hung up on how it was technically possible to move forward without changing anything! Just very, very unreasonably hard. And hey, it's not like there's a time limit to any of this...

That loud screeching sound was the whole project grinding to a dead stop. To get things back on track, I wrote down all of the roadblocks and a workaround strategy for each. Here's that list in full:

No animation file format support for binding graphics to bones
- Add another int32 index to each bone movement frame, denoting which graphics frame to use for each limb. Even if we start out with only one perspective (i.e. set the index to 0 for all frames), there's enough wiggle room to scale up later.
- Once a standard set of perspectives was picked, all a skin would have to do is make frames for each orientation.
jaiwhite support for editing graphics on the skeleton is fiddly and weak
- Shelve jaiwhite graphics editing for now; we don't want it to be easy to add a lot of extra graphics frames, otherwise making new character skins will be a nightmare
- Instead, pick a number of reference poses (e.g. front facing, profile, 45deg) and create graphics for these in Inkscape, because it's a good tool and you're good at that
- To whit, figure out a way of storing the reference poses bone information + graphics information in a SVG file, make a template pose in Inkscape, then bodge up another Python script to export this to the engine graphics format
- Don't worry about getting multiple poses to tesselate with the same mesh, I'm sure Future Me won't mind cleaning up that particular mess
No game engine support for binding graphics to bones
- Add engine stubs to load a skin, bind a graphics file containing all limbs (i.e. a complete skin) to an animation object, and render the correct graphics frame based on the bone frame
No vector graphics for player character yet
- Well trace something, you munch!

Extending the animation file format was by far the easiest part; all my asset formats were done in Protocol Buffers, where you can write the schema once and get lovely type-checked C++/Python/whatever bindings for free. Adding an extra int32 for the graphics frame took one additional line to the .proto file.

The rest I bodged together with scripts. I wrote one to dump a single frame from my animation format into a SVG template, with one layer per bone and the bones drawn in as arrowed lines, and another to convert it back to my graphics format. Once that was set up I traced over each limb with low-res polygons. I stress that this is pure drafting; choosing a handful of points along the curves and connecting the dots. If I did not have the photo as a base I would be reeeeally struggling. If you were feeling lazy you could make a single arm and leg then copy them across, but as all my bone lengths were non-fixed I opted to draw them in twice.

There we have it, a perfect likeness! Now we just go through the motions of binding those shapes to the bones we did earlier, load it all into the engine, and we should have a finished product that's ready to ship!

Well, maybe if you work for Activision! Hi and welcome to game development.

The thickness was a bit of a surprise. I did a thorough recheck of the source material to make sure I hadn't merely eaten too many pies that day, but it turned out I forgot to add the metre-autoscaling feature to the skin compiler. As in, the parallel axis of each bone was saved okay, but the perpendicular axis was saved in whatever weirdo non-metric moon units the SVG was drawn in, not the engine's 1 metre:100px.

After fixing up the scaling, colour palette, playback speed, missing arm, deformed limbs, joint seam ordering and edge lighting, it looked more like this!

Conclusions

I think this is an okay result, coming from a programmer with coarse drawing ability and zero prior experience of animating things. Rotoscoping is a powerful technique that hasn't aged a bit, and is incredibly cheap to produce compared to proper motion capture. I am at a loss as to why it has dropped in popularity, maybe it's the aforementioned lack of good tools?

Okay, so perhaps this exact method for producing animation is hard to recommend. You do waste a lot of time dicking about with tooling and engine support; time that could be spent rotoscoping statically! In essence, for a tool-and-engine driven approach you need to be sure that that [estimated time spent dicking about] < [estimated time agonizing over animating everything by hand].

Also you absolutely don't need to learn parkour for the filming part! [1] Rotoscoping is useful for capturing any physical object or realistic human movement you care to name.

Anyway, the actual conclusion I wanted to bring up was that you should always try and take advantage of whatever tools and skillsets you have to hand. After all, part of the fun in amateur gamedev is to try a bit of everything yourself, even the stuff you suck at.

(TIGSource thread for this post)

[1]	Seriously you should though it's super fun and useful

Dead KORG microKEY, Part 2: Noobface McHamfists Desolders His First SMD

2016-06-17T00:00:00+08:00

Oh God. Oh God. I've been putting this one off, but the stuff has arrived. I can't back down now!

A brief résumé of what happened in Part 1: this great little KORG microKEY MIDI controller stopped powering up for reasons unknown. By wiring it up to a J-Link debug probe, we found that the CPU was still alive and kicking, yet somehow the computer was receiving garbled USB messages. As this keyboard has a built-in USB hub, the stab-in-the-dark diagnosis was that the chip responsible (a Genesys GL850G) had gone bad and needed replacing. Unfortunately, the chip is a 28-pin 10mm*8mm surface mounted design, and our bumbling hero has never before soldered an SMD component, much less reworked a board!

To properly capture the fear of someone doing surface-mount rework for the first time, everything in this piece was written minutes after happening.

Tools used:

Brymen BM867s multimeter; possibly overkill for the job? Should only need the voltmeter and continuity tester.
Tube of 5 replacement GL850Gs from eBay, snipped off the end of someone's pick-and-place reel.
Pack of ChipQuik SMD Removal Alloy. This stuff has a super low melting point (85°C), meaning you can heat it up once with your iron and it will blob around like soft-serve icecream. Just the thing for pulling miniature chips off a board crowded with things that can be ruined.
Solder, 0.5mm and 40% lead. Mmmmm, that's good breathing!
Wick. I decided to go all high-falutin' and buy a new reel which has flux in the braid.
Stainless steel chip pincers. I hope these things don't get welded to my hand.
Flux pen. The miracle ingredient that makes solder do impossible things, like stick uniquely to pads that are less than 0.4mm apart. Uh.
Electronics-grade isopropyl alcohol. I've used up about a third of this bottle from zapping parkour wounds.
Flux cleaner. I just read the warning label on the side and I am terrified the exposure will make me vomit up a kidney or grow a second anus.

Let's begin by snapping the ChipQuik reel in a futile attempt to unwind it. Surprisingly the solid form is really brittle, but all we care about is that it remains super molten after you've heated it. Hell, I had enough time to take 4 crappy photos of the blob with my lousy camera and it was still pliable with tweezers. Welp, nothing for it but to caaaarefully daub it all over the chip legs!

oh god oh god oh god oh god what am i doing

My hands are shaking like crazy; I'm cooking the ChipQuik with my wrong hand and tugging at the chip with my tweezers in the other, but it's not moving! Still, gotta be patient, after all the solder melts at 80 and my iron is 340...

YES! I DID IT! THE CHIPQUIK FREAKING WORKS! And by some miracle, the blobbing is confined to the chip pads and not the rest of the PCB!

Oh. It would appear that ChipQuik's amazing surface tension and blobbing ability means that it's damn near impossible to absorb with solder braid. The best you can do is smoosh it down. Reading the instructions, what you're supposed to do is flux up a cotton bud and wipe it off while applying heat. After using my pen to create a little puddle of flux on the workbench, I get to work.

The cotton swab is still smooshing, but at least you can scoop it in one go. It dawns on me that it's probably a good idea to swipe away from all the exciting SMD parts. Also you need to be frigging ambidextrous in this game; with the red hot piece of metal in one hand, and the fiddly cotton bud in the other...

Right, an hour has passed. I have switched to pumping flux from the pen straight onto the cotton buds, and employing a lot more rubbing force to scrape the last of the ChipQuik off. This, of course, makes a lot of angel-hair bridges. Now that I've finished polishing the board with isopropyl I can still see two left. Booooo.

Next day! Right, let's put the new chip on there. Start off by blobbing some solder on one of the legs to make a tack point.

DAMN YOU TO HELL, SURFACE MOUNT TECHNOLOGY

Okay. It's on the right spot. There's zero room for error as the pads are slightly fatter than the chip. How the shit are you supposed to tack down one corner of the chip without touching it!?

The answer to this question is apparently "try a whole pile of times with the chip cupped with tweezers, instead of pre-placed". Brillo, time to flux up the pin opposite and get this shit tacked!

Sigh. There's a bridge already.

Right; because the definition of madness is to keep trying what isn't working, I'm swapping the iron tip. All the source material on YouTube shows a large chisel tip; originally I used the (pack-in) tiny chisel tip because I doubted I'd be any better at microsurgery with a shovel. This apparently was a mistake; the larger tip has a much bigger thermal mass and is supposedly easier to rake over the chip legs.

Sigh. There's 5 bridges already. (I didn't take a photo, but imagine a scene similar to the mess made with the ChipQuik).

I give up. Every attempt to use the sucker or the desolder braid mushes the solder deeper under the chip legs, making the bridges even harder to remove. Okay. Time out. We'll return to this soon.

Since I wrote that last bit, 6 months have passed. Many things have changed: I'm in a brand new apartment, the sun now goes down at 5pm, I can do almost proper climb-ups if the wall is grippy and there's a strong tail wind, and I've gotten some practice in by soldering up a fuzz factory pedal with the whacking great chisel tip.

I can hardly believe it. It took about 50 goes but I found a way of using solder braid on the SMD chip legs.

Drop some flux on the bridge area with your flux pen. (This encourages all the solder in the nooks and crannies to flow out). Crank your iron to 320-350degC, clean your tip well on a sponge. If the solder is all oxidised and tough, preheat the bridge area with the chisel vertical (maximum surface area over a minimum no. of pins) until the solder is molten. As soon as it starts becoming liquid, apply the end of the braid vertically (as shown) and wait for 5-10 seconds. Raising two middle fingers to logic, the solder will rise up through the braid like a sticky silver phoenix. Once the braid is soaked, cut off the end and continue sucking with a fresh piece.

Afterwards I went and checked every adjacent pair of chip legs with my multimeter's continuity tester, then repeated cleaning and wicking until all the shorts had gone. It took 3 passes to clean it all up; for one really stubborn pair I had to melt a tiny bit more solder on before the braid would wick all of it.

That was definitely near the worst case scenario, and I've proven I can recover from it. Time to solder down the other side!

Done, that went a lot better. See the thing they don't tell you in SMD soldering guides is when raking with your leviathan chisel tip, you should put a miserly amount of solder on the flat part. An amount that would leave a church mouse angry and resentful because of the false hope. An amount so small even Scrooge McDuck wouldn't think twice about giving it away unconditionally to shut you up. The literal smallest dab possible.

This plus the flux will send it where it needs to go.

There's one pair of pins which was bugging me; the pair from the 5V-to-3.3V regulator (5th and 6th from the right, with the meaty traces underneath). No matter how much I wicked it, the continuity tester insisted there was a 1 ohm connection from Vout to the +5V power rail. The datasheet added to the confusion by insisting all of the supply voltage points on the chip were 3.3V, yet I know for a fact the board ran them all on 5V. I'm guessing that's okay then.

This is all fine and dandy, but two big questions remain unanswered.

How sturdy is this board against trained apes like me reworking it?
Did I wreck the replacement chip by heating the crap out of it with my iron?

Whatever there is a time for checks and a time for action and THIS IS A TIME FOR TEST RIG 2: THE SECRET OF THE OOZE

PLUGGED IN! No initial response from the J-Link (probably I attached the probes wrong), but no magic smoke from the board or my PC! Wait does this mean... could... could it be?!

scott@dangerfield:~/Tools/JLink_Linux_V502_x86_64$ dmesg -T
...
[Tue Jun 14 20:02:10 2016] usb 3-1: new full-speed USB device number 2 using xhci_hcd
[Tue Jun 14 20:19:33 2016] usb 3-3: new high-speed USB device number 3 using xhci_hcd
[Tue Jun 14 20:19:34 2016] hub 3-3:1.0: USB hub found
[Tue Jun 14 20:19:34 2016] hub 3-3:1.0: 4 ports detected
[Tue Jun 14 20:19:34 2016] usb 3-3.4: new full-speed USB device number 4 using xhci_hcd
[Tue Jun 14 20:19:34 2016] usbcore: registered new interface driver snd-usb-audio

I DID IT

scott@dangerfield:~/Tools/JLink_Linux_V502_x86_64$ lsusb
...
Bus 003 Device 004: ID 0944:0111 KORG, Inc.
Bus 003 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 003 Device 002: ID 1366:0101 SEGGER J-Link PLUS
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I FIXED THE KEYBOARD

scott@dangerfield:~/Tools/JLink_Linux_V502_x86_64$ ./JLinkExe
SEGGER J-Link Commander V5.02 ('?' for help)
Compiled Aug 28 2015 19:48:33
DLL version V5.02, compiled Aug 28 2015 19:48:30
Firmware: J-Link V9 compiled Aug 28 2015 17:49:48
Hardware: V9.30
S/N: [redacted]
OEM: SEGGER-EDU
Feature(s): FlashBP, GDB
Emulator has Trace capability
VTarget = 5.022V
Info: TotalIRLen = ?, IRPrint = 0x..FFFFFFFFFFFFFFFFFFFFFFF1
Info: TotalIRLen = ?, IRPrint = 0x..FFFFFFFFFFFFFFFFFFFFFFF1
No devices found on JTAG chain. Trying to find device on SWD.
Info: Found SWD-DP with ID 0x2BA01477
Info: Found Cortex-M3 r2p1, Little endian.
Info: FPUnit: 6 code (BP) slots and 2 literal slots
Info: CoreSight components:
Info: ROMTbl 0 @ E00FF000
Info: ROMTbl 0 [0]: FFF0F000, CID: B105E00D, PID: 000BB000 SCS
Info: ROMTbl 0 [1]: FFF02000, CID: B105E00D, PID: 003BB002 DWT
Info: ROMTbl 0 [2]: FFF03000, CID: B105E00D, PID: 002BB003 FPB
Info: ROMTbl 0 [3]: FFF01000, CID: B105E00D, PID: 003BB001 ITM
Info: ROMTbl 0 [4]: FFF41000, CID: B105900D, PID: 003BB923 TPIU-Lite
Info: ROMTbl 0 [5]: FFF42000, CID: B105900D, PID: 003BB924 ETM-M3
Cortex-M3 identified.
Target interface speed: 100 kHz
J-Link>

The little bugger has risen from the dead, working as well as it ever has. It's great when a crazy guess turns out to be vindicated like that. Plus I definitely levelled up on my soldering from this experience.

I suppose we should finish with some Lessons Learned:

Soldering is a game of mastering surface tension to your own ends. You've probably noticed that surface tension causes large bulbous drops of water to form on dry surfaces and edges, whereas adding water to already-damp surfaces produces tinier thinner droplets. The clown-sized chisel tip is far better at carrying small amounts of solder because of this same principle; because there's a large pre-wetted tip area, the solder won't accumulate in a ball but instead hug closer to the tip surface. As a bonus, you don't have to overcome the large natural surface tension of a ball to get it to flow onto another surface.
For that reason, if your iron doesn't come supplied with a 2mm chisel tip, buy one on eBay and make it your standard.
ChipQuik really takes the misery out of desoldering and saved having to buy an expensive board heating/hot air station, but it costs a fortune.
The one piece of advice I took from YouTube that paid off was to use a smaller gauge of braid (1.5mm) and solder wire (0.5mm), both multicore flux.
Oh and cover everything in flux too, it gives you more cheat points to beat surface tension.
Solder suckers are useless. You get a gap of maybe 30 milliseconds between your iron leaving the joint and the solder hardening. Save your money.
I probably could have bought 3 new microKEYs with the amount I ultimately dropped on tools and consumables. Save your money.

I might return later to explore the rest of this Cortex M3 hardware; all of the interesting code isn't blocked off by the broken USB comms anymore, the key input and wheel ADC routines are up for grabs. Right now I'm just happy to finally clear away this mess of tools that have invaded my workbench, and to have my keyboard back. I knew you would pull through, old chum!

AFTERWORD: The more clued-up people might've noticed that I missed a crucial step; resoldering the first set of legs properly after wicking away the bridges. I didn't realise this would be an issue until after a couple of days of running, when the keyboard would lose USB sync every hour or so:

Jun 17 15:27:12 dangerfield kernel: usb 1-2-port4: disabled by hub (EMI?), re-enabling...
Jun 17 15:27:12 dangerfield kernel: usb 1-2.4: USB disconnect, device number 16
Jun 17 15:27:13 dangerfield kernel: usb 1-2.4: new full-speed USB device number 17 using ehci-pci

Port 4 is the Cortex M3, so I guess DM4 and DP4 were hanging on by nothing but the whims of thermal expansion. I re-raked those pins and the dropouts have gone.

Audio Analysis, Part 1: Digital Audio

2015-11-16T00:00:00+08:00

Today I'm going to try and explain how digital audio works. Most people have a vague idea about how sound works. "Sound is a wave!", they might say. "Sound bounces off things!", adds another. "Sound increases as you get closer to the stacks!", relates one subject matter expert. "What?", asks someone else? "I SAID SOUND INCREASES AS YOU GET CLOSER TO THE STACKS!" "Sorry I couldn't hear you over my crippling tinnitus! Dear Jesus if only I had known more about sound!"

This will be a really simple primer about digital sound waves and how they contain musical notes. To follow along at home, you'll need the excellent free audio editor Audacity. Download it if you don't have it already, then open it up to an empty window.

Right, so what does a single note "look" like? Let's start with a sine wave. Sine waves have an easily recognisable and pure-sounding tone. In our Audacity window, we'll click Generate -> Tone.

Oh look, they've helpfully picked a Sine wave as the default, with a frequency of 440 Hz and an amplitude of 0.8! (I changed the duration to only 5 seconds, as the default of 30 is a bit too long.) Let's hit OK.

That's a sine wave? Looks more like a giant blue slab to me. Let's zoom in a little bit.

Flashbacks of early high-school maths should be running through your head. Let's figure out what's going on in this picture. The graph you see here is basically tracing the path that the speaker cone will take to reproduce the sound, with +1.0 meaning pushed all the way forward and -1.0 pulled all the way back. If we think of sound output (vertical axis) as y, and time (horizontal axis) as t, here's the formula:

y = A*sin( w*t+ph )

A is the amplitude of the wave; in our case, we picked 0.8, and sure enough we can see on the graph that the wave has peaks and troughs at +/- 0.8.
w is a little tricky, but it corresponds to the frequency we picked. We picked 440Hz as the frequency f. Basically, we want sin() to repeat itself every 1/f seconds (also known as the period of the wave), and the function sin() takes radians as an input (so one full repetition of sin() happens after every 2*pi radians). Combining these, we get w = 2*pi/(1/f) = 2*pi*f. This means that the wave should repeat itself every (1/440) = 0.002272 seconds, which matches up with what we see on the graph.
ph is the phase of the wave; basically, at what point along the wave pattern do we want to start our signal. Right now it's 0.

If you also suffer flashbacks to high-school music lessons, playing back the 440 Hz sine wave might also remind you of "concert A", aka. the favourite tuning note for string musicians and orchestras. Musical notes from instruments (as we'll find out later) are a mix of a whole pile of different frequency waves, but usually there's one frequency that's loudest and that's the one we assign the note value to. 440 Hz (a.ka. the A wot comes after middle C on a keyboard) is referred to as A4 in scientific pitch notation, or 69 on the MIDI scale. There are 12 notes in an octave (great choice of name, guys), and interestingly an octave jump between notes is a multiple of 2 in the frequency domain. So that means that 57 (A3) is 220 Hz, while 81 (A5) is 880 Hz! Try these out in the tone generator if you like, don't they sound real similar?

Anyway, you're probably wondering if there's a nice and easy formula for converting note values in the MIDI domain to tones in the frequency domain for your tone generator. To which my response would be "define easy":

#define FREQ_TO_MIDI( freq ) (69.0f+12.0f*log( freq/440.0f )/log( 2.0f ))
#define MIDI_TO_FREQ( midi ) (exp( log( 2.0f )*(midi-69.0f)/12.0f )*440.0f)

Sometimes it's easier to use a lookup table instead. Like this one! (thanks J. Wolfe of UNSW)

One final thing; digital audio is based on the principle of sampling; unlike an analogue audio signal, which is a continuous sound output from an electrical or mechanical process (e.g. piezo cartridge running through the grooves of a record, electric guitar plugged into a tube amp), the sound output of a digital audio signal operates at a fixed number of "samples" per second. You see that number 44100 Hz down in the bottom left corner? That's known as the sampling rate. Every second of audio can be thought of as a big two-column list with 44100 timecodes down the left hand side and 44100 speaker positions on the right. 44100 Hz is the standard sampling rate of audio CDs and most downloadable music.

Wouldn't having a fixed number of samples per second limit the quality of sound you would get? Absolutely! If you've wondered why voices on a landline telephone sound so lousy, it's because Telstra can't maintain their copper for beans and the pit outside is probably full of water. It's also because the sound is processed with G.711, an audio codec from 1972 that has an abysmal sampling rate of 8000 Hz.

In digital audio, the highest frequency sound you can physically reproduce is half the sampling rate, known as the Nyquist frequency. Which makes sense, because the tightest you'd be able to pack sound into your digital signal is with a +1.0, followed by a -1.0, followed by another +1.0… and so on. Now 44100 Hz has a Nyquist frequency of 22050 Hz, which conveniently is juuuust above the range of human hearing. Telephones have a Nyquist frequency of 4000 Hz, which is not. It's extremely audible. Try loading some music into Audacity and crank the Project Rate down to 8000 Hz; notice how it's sounding all muddy and not crisp? That's what happens when you can't play frequencies higher than 4000 Hz. It stinks, right?

Alright, enough minutiae. Time to get to the most mind-bending part about how sound works. So we're looking at our sine wave here, thinking about how it corresponds to the movement of the speaker cone, but also how nice and symmetrical it is. One question that springs to mind then, how is it possible for a good set of loudspeakers to reproduce… well, any damn sound you can think of? How do you combine multiple sounds? Wouldn't the movement of the cone look all jagged and weird?

Let's generate another tone in our project; a complimentary one to A4. Let's go for C#4 (61), as that produces a nice-sounding A major third. Add a new tone of a C#4 sine wave at 277.18Hz.

When you play that back, there's going to be a sharp edge to it. This is because the two tones we picked have a very high amplitude, enough so that combining them produces a result outside the range of -1.0 to +1.0, which you can hear as clipping. Lower the volume of the two waves by setting the little -/+ slider on the left of both tracks down to -10 dB. Listen again, you'll have a much cleaner sound.

But they're still both separate! To see the resulting signal that drives the speaker, we need to merge them. Press Ctrl-A to select both tracks, then pick Tracks -> Mix and Render to New Track

Will you look at that? The samples from each of the two waves are just added together. The resulting signal, being simple, has a few obvious features which match up with the two parent sine waves. Still, looking at it by itself it's difficult to say what frequencies are in this new combination, and it gets even harder to examine by eye as more tones are combined. And yet, if you listen closely it's obvious it contains the two tones we made earlier! How do we separate tones back out again? If our ears can do it, why can't maths?

If you haven't guessed already, a large part of audio analysis is going backwards from a complete digital audio signal to its component parts. We'll go into that more in the next thrilling installment!

J-Link Detective Squad: Dead KORG microKEY-37

2015-11-08T00:00:00+08:00

I love the KORG microKEY 37. It's an excellent entry-level MIDI controller that fits unobtrusively on your desk, great for impromptu jamming when you should be focused on something boring like "tax" or "finding a new house before eviction day". The keys feel pretty soft, as they use rubber domes instead of metal springs, but the velocity response is excellent. I highly recommend it, as it's one of those products which basically lives forever and delivers year after year of good service.

I bring this up because back in about February, my KORG microKEY 37 stopped turning on. The unit would no longer connect via USB; each time it would give up at different points during the initial handshake, with dmesg spewing a number of unhappy messages. Here's one attempt at plugging the device in.

[Sun Sep 27 06:00:56 2015] usb 3-1: new full-speed USB device number 17 using xhci_hcd
[Sun Sep 27 06:00:56 2015] usb 3-1: device descriptor read/64, error -71
[Sun Sep 27 06:00:57 2015] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 16.
[Sun Sep 27 06:00:57 2015] usb 3-1: hub failed to enable device, error -22
[Sun Sep 27 06:00:57 2015] usb 3-1: new full-speed USB device number 18 using xhci_hcd
[Sun Sep 27 06:00:57 2015] usb 3-1: device descriptor read/64, error -71
[Sun Sep 27 06:00:57 2015] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 17.
[Sun Sep 27 06:00:57 2015] usb 3-1: hub failed to enable device, error -22
[Sun Sep 27 06:00:57 2015] usb 3-1: new high-speed USB device number 19 using xhci_hcd
[Sun Sep 27 06:00:57 2015] usb 3-1: Device not responding to setup address.
[Sun Sep 27 06:00:57 2015] usb 3-1: Device not responding to setup address.
[Sun Sep 27 06:00:58 2015] usb 3-1: device not accepting address 19, error -71
[Sun Sep 27 06:00:58 2015] usb 3-1: new full-speed USB device number 20 using xhci_hcd
[Sun Sep 27 06:00:58 2015] usb 3-1: Device not responding to setup address.
[Sun Sep 27 06:00:58 2015] usb 3-1: Device not responding to setup address.
[Sun Sep 27 06:00:58 2015] usb 3-1: device not accepting address 20, error -71
[Sun Sep 27 06:00:58 2015] usb usb3-port1: unable to enumerate USB device

Oh snap! I'm torn on what to do... buying a replacement is doable, but this one was limited edition and a replacement wouldn't be the same cool colour scheme! Maybe there's a clue to what went wrong inside the unit?

Here's the view of the main KLM-3130-B board. As you can see, there's nothing obviously charred or melted. I tried following the golden rule of electronics repair (check for busted capacitors), but all of the capacitors in this are tiny ceramic surface-mount ones. Which hopefully aren't going to fail because they don't have liquid in them.

The big square chip I identified as a Fujitsu (now Cypress) MB9A310A series ARM Cortex M3 processor (MB9AF312L), and the smaller rectangular chip is a GL850G USB hub controller (as this keyboard has a two port expansion USB hub). Checking the datasheet for the CPU, and employing my patented multimeter stabbing technique on the teeny tiny chip legs, I'm sure the CPU is getting +5V at the Vcc (a.k.a. supply voltage) pins and ground at the GND pins. The two clock crystals I believe are 12MHz (X1) and 4MHz (X2). There. That's my electronics knowledge exhausted.

But wait! What are these intriguing pads there on the board marked "For Debug"? SWO? SWDIO? You mean, there's a chance that this board could be fixed in the software realm?

Ah well, in for a penny... after checking in with a Subject Matter Expert I purchased a Segger J-Link debug probe for $89. I then went ahead and hacked open a sturdy quadruple-shielded USB type-B cable (TIP: if you need to strip a USB cable go for the ultra-cheapass thin ones; the only "shielding" they have is a few microns of aluminium foil. Also cutting it ensures it will never be used again), soldered some hookup wire to the pads, grafted some prototype wires on the ends, and plugged everything into a breadboard.

For Serial Wire Debug there are four pins on the J-Link which correspond to our pads (SWCLK, SWDIO, SWO, RESET), plus VTref (spliced off our supply voltage, aka. the +5V pin of the keyboard's USB in) and GND (spliced off ground pin of USB in). D- and D+ of the USB cable are passed through without any splicing. To keep things slightly less failure prone I'm running the J-Link off my desktop's USB controller, and the keyboard's spliced cable will be attached to my laptop.

So... it was at this point that I thought about reading the J-Link manual. According to the Getting Started section, I should switch on the J-Link, then the keyboard, and it should identify the hardware. Hah. As if it'd be that easy.

scott@dangerfield:~/Tools/JLink_Linux_V502_x86_64$ ./JLinkExe
SEGGER J-Link Commander V5.02 ('?' for help)
Compiled Aug 28 2015 19:48:33
DLL version V5.02, compiled Aug 28 2015 19:48:30
Firmware: J-Link V9 compiled Aug 28 2015 17:49:48
Hardware: V9.30
S/N: [redacted]
OEM: SEGGER-EDU
Feature(s): FlashBP, GDB
Emulator has Trace capability
VTarget = 4.950V
Info: TotalIRLen = ?, IRPrint = 0x..FFFFFFFFFFFFFFFFFFFFFFF1
Info: TotalIRLen = ?, IRPrint = 0x..FFFFFFFFFFFFFFFFFFFFFFF1
No devices found on JTAG chain. Trying to find device on SWD.
Info: Found SWD-DP with ID 0x2BA01477
Info: Found Cortex-M3 r2p1, Little endian.
Info: FPUnit: 6 code (BP) slots and 2 literal slots
Info: CoreSight components:
Info: ROMTbl 0 @ E00FF000
Info: ROMTbl 0 [0]: FFF0F000, CID: B105E00D, PID: 000BB000 SCS
Info: ROMTbl 0 [1]: FFF02000, CID: B105E00D, PID: 003BB002 DWT
Info: ROMTbl 0 [2]: FFF03000, CID: B105E00D, PID: 002BB003 FPB
Info: ROMTbl 0 [3]: FFF01000, CID: B105E00D, PID: 003BB001 ITM
Info: ROMTbl 0 [4]: FFF41000, CID: B105900D, PID: 003BB923 TPIU-Lite
Info: ROMTbl 0 [5]: FFF42000, CID: B105900D, PID: 003BB924 ETM-M3
Cortex-M3 identified.
Target interface speed: 100 kHz
J-Link>

AWW YISS. The chip is alive! Let's read through the command list and see what things we can do. Oh look! CPU registers! They contain things that are important. Can we see those?

J-Link>regs
CPU is not halted !

J-Link>halt
PC = 00009C56, CycleCnt = E3B49CD7
R0 = 00000000, R1 = 00000001, R2 = 1FFFE444, R3 = 1FFFEF04
R4 = 1FFFEF04, R5 = 1FFFE306, R6 = 00000000, R7 = 00000000
R8 = 00000000, R9 = 00000000, R10= 00000000, R11= 00000000
R12= 4266609C
SP(R13)= 20001FD0, MSP= 20001FD0, PSP= 00000000, R14(LR) = 00009E29
XPSR = 81000000: APSR = Nzcvq, EPSR = 01000000, IPSR = 000 (NoException)
CFBP = 00000000, CONTROL = 00, FAULTMASK = 00, BASEPRI = 00, PRIMASK = 00

Yep. Those are certainly ARM registers. Piece of piss, this electronics lark.

Okay. What next? We don't have the source code of what's running on this M3, so we want to whip out a debugger. But first, what about the stuff stored on the flash memory? It'd be a good idea to take a copy of that to safeguard against us screwing up. More importantly, where IS the flash memory? The Cortex M3 is a 32-bit microcontroller, so there's 4GB of address space!

Back to the CPU datasheet, there's a memory map for the chip on page 55; of course, only a small amount of the 4GB address space is mapped to anything useful. We can see that the chip's princely 128kb of flash (a.k.a. stored code) is mapped to the range 0x00000000-0x00020000, and the generous 16kb of SRAM can be found at 0x1FFFE000-0x20002000. Notice how the values stored in several of those registers above are within the SRAM range? I think we can say with confidence that these are pointers to stuff in memory!

But what's the deal with R12? According to our memory map that address sits in somewhere called the "Bit band alias". The ARM manual for the M3 describes this feature pretty well; this is basically another way of reading and writing to the SRAM, except every bit is given a full processor word. Meaning the 1MB SRAM range gets stretched out to 1MB*(8 bits in a byte)*(4 bytes in a word) = 32MB! This may sound like the definition of wastefulness, but it means you can do bit-level operations with one instruction. As in, normally if you wanted to set bit 3 in a byte of memory to 1 you'd have to do something like [load byte into register] + [OR register with 0x08] + [write register to byte] = 3 instructions; with a bit band alias you can just write 1 to the address that corresponds to the bit. Snazzy!

Okay, enough fun registers talk, let's get ourselves a dump of the flash memory range from our datasheet.

J-Link>savebin microkey.bin, 0x00000000, 0x20000
Opening binary file for writing... [microkey.bin]
Reading 131072 bytes from addr 0x00000000 into file...O.K.

Poking around inside the file with a hex editor, there's a definite rhythmic quality to the bytes, interspersed with large blocks of nulls and the occasional plaintext referencing KORG. Code!

The first few subroutines are probing around the peripheral map at 0x40000000, but what does any of it map to? The memory map in that datasheet was a start, but pretty thin on detail. I found a page containing a copy of the C header file for the chip (mb9a310l.h, for those playing at home), which has a more detailed listing of all of the peripheral memory addresses. (Lesson learned: more detail doesn't help) This crap shouldn't be so hard, all we want is a rough idea what addresses match output from the keys and wheels. Let's take advantage of some street-grade debugging!

Right, step one: halt the CPU and get a dump of the full peripheral range named in the datasheet.

J-Link>halt
PC = 000090FA, CycleCnt = 5314F092
R0 = 00000002, R1 = 00000008, R2 = 00000008, R3 = E000E100
R4 = 000000FF, R5 = 1FFFE306, R6 = 00000000, R7 = 00000000
R8 = 00000000, R9 = 00000000, R10= 00000000, R11= 00000000
R12= 4266609C
SP(R13)= 20001FC0, MSP= 20001FC0, PSP= 00000000, R14(LR) = 00009E0F
XPSR = 21000000: APSR = nzCvq, EPSR = 01000000, IPSR = 000 (NoException)
CFBP = 00000000, CONTROL = 00, FAULTMASK = 00, BASEPRI = 00, PRIMASK = 00

J-Link>savebin per_clean.bin 0x40000000 0x61000
Opening binary file for writing... [per_clean.bin]
Reading 397312 bytes from addr 0x40000000 into file...Could not read memory.

Whuuuuuuuut. You can't talk to me like that, I'm the debugger! I pay your salary!

Fine. Okay, clearly "reserved" is EE speak for "get off my land", so let's try just the GPIO block. Lots of buttons sounds like General-Purpose Input/Output to me.

J-Link>savebin per_gpio.bin 0x40033000 0x1000
Opening binary file for writing... [per_gpio.bin]
Reading 4096 bytes from addr 0x40033000 into file...O.K.

$ hexdump per_gpio.bin
0000000 001f 0000 0300 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 0003 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0000 0000 000c 0000 0000 0000
0000040 0000 0000 0000 0000 0000 0000 0000 0000
*
0000100 001f 0000 0000 0000 0000 0000 020f 0000
0000110 00c0 0000 0007 0000 0000 0000 0000 0000
0000120 0000 0000 0000 0000 0000 0000 0000 0000
*
0000200 9000 0000 00ff 0000 000a 0000 fc00 0000
0000210 7e00 0000 0000 0000 0006 0000 0000 0000
0000220 0000 0000 0000 0000 0000 0000 0000 0000
*
0000300 0407 0000 0000 0000 0000 0000 320f 0000
0000310 00c0 0000 0007 0000 0001 0000 0000 0000
0000320 0000 0000 0000 0000 0000 0000 0000 0000
*
0000400 0000 0000 0000 0000 0000 0000 3000 0000
0000410 0000 0000 0000 0000 0000 0000 0000 0000
*
0000500 ff00 ffff 0000 0000 0000 0000 0000 0000
0000510 0000 0000 0000 0000 0000 0000 0000 0000
*
0000580 0014 0000 0000 0000 0000 0000 0000 0000
0000590 0000 0000 0000 0000 0000 0000 0000 0000
*
0000600 0000 0003 0000 0000 0000 0000 0000 0000
0000610 0000 0000 0000 0000 0000 0000 0000 0000
*
0000800 0000 0000 000d 0000 0000 0000 0000 0000
0000810 0000 0000 0000 0000 0000 0000 0000 0000
*
0001000

Cracking it open in a hex editor, it's mostly 0s but a few 1s! Pulling a second dump shows it to be identical to the first.

Now! The next step is to pull another dump of the GPIO range, but this time whilst holding down C3 on the keyboard. Then, compare it with our first clean dump.

$ hexdump per_gpio.bin > per_gpio.hex
$ hexdump per_gpioC3.bin > per_gpioC3.hex
$ diff per_gpio.hex per_gpioC3.hex
16c16
< 0000310 00c0 0000 0007 0000 0001 0000 0000 0000
---
> 0000310 00c0 0000 0004 0000 0001 0000 0000 0000

Well whoda thunk it. We can see that bits 0 and 1 of offset 0x315 are zeroed when we hold down C3. How about C#3?

$ diff per_gpio.hex per_gpioCsh3.hex
15,16c15,16
< 0000300 0407 0000 0000 0000 0000 0000 320f 0000
< 0000310 00c0 0000 0007 0000 0001 0000 0000 0000
---
> 0000300 0407 0000 0000 0000 0000 0000 320e 0000
> 0000310 00c0 0000 0003 0000 0001 0000 0000 0000

Interesting. Bit 2 of 0x315 and bit 0 of 0x30d got zeroed that time. Strangely, this only worked for keys C3 to D#3, and the two octave buttons. Where's the rest of the keys and the two wheels? Or the key velocity? Given that the wheels are just potentiometers wired to the main board, I have a good feeling that they'd use the M3's built in analog-to-digital converter, but at a guess I'd say the block of code responsible for activating the ADC sampling wasn't being hit, so no dice.

Okay, so far this seems like a distraction. The Cortex M3 is alive and well; at a guess from stepping through the code with the J-Link I'd say it was stuck in a holding pattern waiting for the USB interface to come up. That leaves the GL850G USB hub-on-a-chip, which has the M3 wired up to one of the downstream inputs, along with the two female USB connectors. Everything we care about is getting 5V; the M3, the hub, and the two USB ports. What's not happening is a proper USB handshake!

And sadly, this is where the current line of investigation stops. The GL850G doesn't have a JTAG/SWD interface; the datasheet does mention a TEST mode and an I2C interface, but buggered if I know how those work (the datasheet certainly doesn't say), and I'm not keen on soldering more stuff to the microscopic chip legs. Dumping the USB traffic using the Linux usbmon driver doesn't reveal anything new; just like the dmesg output says, the device fails to identify itself at seemingly random points during the initial handshake. I guess the next level down from there would be to try a USB logic analyser like the Beagle 12, or maybe one of those fancy digital oscilloscopes that support USB TX, but those cost more than 500 dollars! I think the new plan will be to buy a totally-not-bootleg replacement GL850G on eBay (hello China!), then have a stab at desoldering the current one without ruining everything. Plan C is buy a used microKEY 37 on eBay and steal the main board.

Still, using the J-Link was a fun experience. For anyone looking to brush up their reverse engineering know-how, I highly recommend finding some hardware with one of these Cortex M3 SOCs inside and having a go debugging it with a J-Link. As a beginner to assembly language, I was quite impressed with how easy to read the Cortex M3 instructions were. Then again my last experience was reading 16-bit x86, which even the fans will tell you is a horrific architecture with no redeeming qualities, except maybe the power to induce vomiting. If the GL850G transplant works, we'll be sure to pick up where we left off and continue exploring the many nooks and crannies of the microKEY hardware.

Introducing Mr. Crowbar! Untold minutes of reverse engineering fun for ages 26-29!

2015-10-22T00:00:00+08:00

I believe that one of the best ways of allowing creativity to overcome limited technical skill is to build upon stuff that already exists. Take game modding; just try and count the number of successful games that started out as a modification of an existing one. Would these developers have gotten anywhere as far with their idea if they couldn't build on an existing engine and assets, not to mention community support? WOULD THEY?

Indeed, game modding is an important springboard for new developers to cut their teeth in a familiar setting and form a good understanding of how games work in the real world. There is, however, one rather large barrier of entry for people to start playing with the innards of their favourite game: tools. Occasionally the engine author will encourage custom content and give their developer tools out for free. But most of the time games are 100% not designed with modding in mind, leaving it up to a tiny number of skilled reverse engineers to write their own tools and share them with the community.

And these tools? Sometimes they'll let you edit stuff, sometimes only view. Sometimes they will be open source, sometimes not. Sometimes they'll be actively maintained, sometimes a rando will post an EXE on a forum then vanish off the face of the planet. Sometimes there won't even be a tool; just some old C code, or a meticulously-put-together yet scabby text file from the bulletin board age describing the file format. There's no standard!

It occured to me that most of the tools that are out there share a ton of common ground, in that they are mostly just fancy GUIs that implement a CRUD (Create Read Update Delete) interface. Sometimes it's just R! Well, what if you didn't have a fancy GUI per se, but instead had a nice understandable CRUD model bolted onto a scripting language with the option of a fancy GUI later on?

So after getting this dumb idea after unearthing some horribly-written tools I made in high school, I reached for my favourite language for messy hacking times (Python), put together a test model targeted at a game that I mostly knew the file formats for (the 1991 classic "Lemmings" by DMA Design), and named it Mr. Crowbar.

The object model steals a lot of ideas from Django and Schematic; each Block class is defined as a set of Field objects, but thanks to the mindbending power of metaclasses they can be get/set by the user at runtime with normal types like ints and strings, hiding all of that tedious validation and type-safety. In Mr. Crowbar, each Block class is strongly typed and defines the equivalent of a C struct, and gives you importing and exporting stuff as bytes for free. For non-trivial storage (e.g. compression algorithms) you can write manual import/export routines and wrap them in a Transform class, which can be used as a preprocessor for loading bytes into Blocks. Finally there's a Loader class, which does most of the spadework in loading the full spread of files from a game and linking up cross-references.

Here's an example of a Block object:

class Terrain( mrc.Block ):
    _block_size =       4

    x_raw =             mrc.UInt16_BE( 0x00, bitmask=b'\x0f\xff', range=range( 0, 1600 ) )
    draw_back =         mrc.Bits( 0x00, 0b10000000 )
    draw_upsidedown =   mrc.Bits( 0x00, 0b01000000 )
    draw_erase =        mrc.Bits( 0x00, 0b00100000 )
    y_raw_coarse =      mrc.Int8( 0x02, range=range( -17, 82 ) )
    y_raw_fine =        mrc.Bits( 0x03, 0b10000000 )
    obj_id =            mrc.UInt8( 0x03, bitmask=b'\x3f', range=range( 0, 64 ) )

    @property
    def x( self ):
        return (self.x_raw-16)

    @property
    def y( self ):
        return (self.y_raw_coarse*2 + self.y_raw_fine)-4

This was taken from mrcrowbar.lib.games.lemmings. Every Level object has a list of Terrain objects; basically, info on where to put selected background tiles to build the level up with. Each of these Terrain references is 4 bytes, and there's a lot packed in there! The x position of the background piece is stored in the first two bytes (unsigned big-endian) as the least significant 12 bits. The remaining most significant bits are used as flags which affect how the tile is rendered. The y position is stored in two parts: a signed coarse component in byte 2, and a fine modifier as the most-significant-bit of byte 3. The 6 least significant bits of byte 3 make up the ID of the terrain object.

What does this all mean? Well, if we open this class with 4 bytes as input, or it gets chainloaded by a larger struct (e.g. Level), all of the above field definitions will get replaced with a variable you can edit. Notice how we've exposed proper cartesian x and y as properties, maybe later we could add a setter which casts back to the original packed format. Then, after you're done changing stuff up in the class, you can run export_data() and get four bytes back. Or do it on the parent Level class and get a full level file back as bytes. What's not to love?

Also, we strongly recommend you use a terminal with support for ANSI 24-bit colour, such as GNOME Terminal for Linux or ConEmu for Windows. Why? Well...

That's right. Live bitmap previews in the terminal! At two pixels per letter! This is exactly what Guido van Rossum had in mind when he allowed people to override __str__ and balls to you if you think otherwise.

BONUS ARCHAEOLOGICAL FIND: I got sidetracked at one point and wrote a parser for those colour text screens you sometimes see at the end of shareware games, nagging you to buy the full thing. Take a look at the nag screen for the game "Boppin'" published by Apogee:

The top is what you would normally see, painstakingly recreated with Unicode box-drawing characters to mimic the ol' DOS text mode. Underneath is exactly the same text, but without any of the colours. Notice how there's a bunch of hidden text in the blank areas? Looks like the Nag Screen Creative Director was caught short and had to steal a graphic from the company BBS. After all, no-one would ever suspect that it wasn't an original work... UNTIL NOW!

Anyway, right now the project is just a babby; the only thing it can do is read some files from the 1991 classic "Lemmings" by DMA Design. Here's a list of features in the pipeline:

Export support! The whole idea of having a strongly-typed data model is so you can edit the content in Python, then reverse the import steps to get a binary again.
Foreign key support! Games love nothing more than to have multiple files that reference one another.
More standard models for common structures like lookup tables and streams!
More games!

We're not even up to a proper release yet; I just needed to write something down before I exploded. First release will be when enough of Lemmings is satisfactorily editable; as of this moment we are an R application, with CUD to follow. Thanks for reading!

Guide: Backing up crap DVDs

2015-10-10T00:00:00+08:00

Let's tangent for a moment and talk about DVDs. The DVD standard was designed for the low-powered embedded hardware of the late 1990s. The copy protection is primitive, and the DVD format is public knowledge by now, making it easy to backup or format shift your DVD collection as is your God-given first sale right (unless you live in the US or anywhere that just signed a trade agreement).

For reasons of Christ Knows Why, some publishers (ignorant of how top-down video piracy generally works) think that the best way to prevent this is by using third-party copy protection schemes. This so-called protection is of the "no True Scotsman" variety; it adds a lot of intentional errors to the DVD that would make most software players (and the authors of the spec) rightfully vomit. But that's ok! Because the discs are designed for a True Hardware DVD Player, one that is infinitely more robust than your PC-based counterparts and will soar over the corrupted blocks like a majestic eagle.

Just to repeat this again, these publishers sell a product with severe mastering defects on purpose and just hope the error correction is really good and you don't notice.

This article will walk you through the five categories of dirty tricks used to limit fair use. Normally you only have to worry about a couple of these; but I had the misfortune to buy a DVD (Chris Morris' excellent Brass Eye) which had ALL DAMN 5. And for the first time, none of the DVD playback software I tried would play the episodes when selected on the menu.

This is the official pitch video for the copy-protection found on this DVD.

I had been officially swindled out of £5.75, so for the record I did make the exact face shown here at 1:07. I mean, at least when a music CD is mastered with defects in a similar way (e.g. Copy Control), the Red Book standard forbids those idiots from stamping the CDDA logomark onto their flat plastic turd. But I guess the DVD Consortium has a much simpler set of rules to qualify for their logo, such as "anyone who pays us $5000".

The categories

CSS. The region coding protection built into the DVD standard. Each VOB file in VIDEO_TS has a 40-bit descrambling key, which is stored on a hidden area of the disc only accessible by querying the drive firmware. If you're lucky enough to have a RPC1 (region-free) drive, then libdvdcss will have no problem yanking the keys. Nearly all drives are RPC2 (a.k.a. you're allowed 5 region changes and you better like the last one because IT'S PERMANENT BUDDY), and will only let you access the disc keys if the drive region is one of the disc regions. Some DVD drives are manufactured by assholes; not only will they withold the keys, but they'll also throw a ton of IO errors if the region is incorrect!
Analogue copyprotection. Another pack-in option, this is a flag which causes the analogue video ports on a hardware DVD player to send crazy nonsense during the vertical blanking interval, which will either confuse the shit out of the image gain circuitry in an old VHS deck, or tell a new VHS deck/DVR to stop recording and instead display a patronising message about stealing.
Corrupted sectors. A popular technique for some protection schemes is to simply master the DVD with intentional cockups in the data sectors on the disc. The thinking behind this is that True Hardware DVD Players are meant to deal with misanthropes who leave their discs face-down on sandpaper and powdered glass, and should therefore glide smoothly (hah!) over any unreadable bits. If you try and copy the VOB files straight from the disc with a file manager, it'll die in the arse once it reaches the first 2048-byte sector that it can't read.
File table abuse. Like most filesystems, UDF can be dumbed down to the idea of a file table (i.e. the tree structure of directories and files), followed by a large blob containing all the binary data. Each entry in the table has a reference to exactly where, inside the huge blob, a file's data starts and finishes. If you're thinking like a copy-protection engineer, you will immediately see the opportunity to abuse this and make hundreds of decoy files with references to overlapping data, creating a mess that appears several times larger the 9GB limit of a dual-layer DVD.
Title index mauling. Because they just haven't done enough already, the final place to inflict pain on the user is in the title and chapter structure of the DVD. Hey, the standard lets us have up to 99 titles which can reference any chunk of the DVD video stream, why shouldn't we fill them all up with corrupted copies that terminate 15 minutes in! And those chapter breaks; that sounds like an amazing opportunity to randomly skip to different episodes!

How to get around it

I'm going to go through the steps (in order) I went through to surgically remove the damage. You will need:

A faulty piece of shit video DVD
A computer with a good DVD drive running Linux (a bootable Ubuntu USB will work in a pinch)
libdvdcss2
ddrescue (apt: gddrescue)
VLC
dvdbackup/vobcopy/Handbrake (depending on what you want to do with the DVD image)
Wine + VobBlanker

If you want instructions on how do this kind of thing using only Windows, you're insane. And you'll have to scour through someplace like Doom9.org because I don't know!

CSS

Try playing your disc in VLC.

If the picture looks fine, your drive is cooperating with libdvdcss.
If the picture looks green and corrupted and stuttery, then your drive can access the video data stored on the disc but can't get the CSS keys to descramble it; see the end of this section about how to copy the CSS title keys from a working computer.
If VLC is throwing errors or a blank screen, and dmesg is full of whining like "Read of scrambled sector without authentication", the firmware on your DVD drive has noticed the disc is the wrong region and is telling you at the IO level to piss off. At this point, you are stuck; you can't get around this without first switching the drive's region or flashing the firmware to be RPC1 (aka. region-free).

Region-free flashing requires Windows; try looking up your drive's model using DiscInfo, then check if either the RPC1 firmware database or the RPC1 forums have an exact match. You can properly ruin your drive if you don't know what you're doing; if you're just starting out, try it on an older drive which you can live without if things go wrong.

I can't remember if I'm just lucky, or if I flashed it once at 3am and can't remember, but my Pioneer BDR-206 is a region free drive 99% of the time. (so far it's failed exactly once; it couldn't read the CSS title keys for the brilliant Mr. Show with Bob and David. I had to grab the keys from a machine with a Region 1 drive)

Anyway, do whatever you have to do to get the disc playing in VLC under Linux. The next step is to open up the .dvdcss folder in your home directory; in here is a copy of the extracted CSS title keys of every disc you've played on the computer up until now. Mine were stored in a folder called BRASS_EYE-2010012910090800-0000000063. This is going to be important later on.

Analogue copyprotection

Not an issue, as all DVD and VOB backup tools disable this flag. Moving on!

Corrupted Sectors

I'm going to invoke the famous "duck test" here; if it looks like a duck, walks like a duck, swims like a duck, has feathers like a duck and quacks like a duck, then it is probably best served crispy with hoisin sauce. We're going to treat our DVD just like we would any other damaged, inches-away-from-death piece of media, and use a recovery tool designed to scrub around all of the bad sectors, which (if the copy-protection peddler is to be believed) aren't part of the video stream. It's a command-line tool called ddrescue, and works roughly like the following:

Create an image file exactly the same size as the source medium
Do a quick first pass, skipping over any read errors and rescuing as much good data as possible
Do a series of bisection passes over the holes, attempting to subdivide the corrupted blocks into smaller and smaller chunks until some data can be read
Repeat the above step over and over again

$ sudo ddrescue -v -b 2048 -r 5 /dev/sr0 brasseye.iso brasseye.log

2048 is the bytes-per-sector size on a standard DVD, so you can't subdivide smaller than this. /dev/sr0 is the block device for my DVD drive, and I want it to retry the bad parts 5 times before giving up.

Before long, I have a brand new ISO that's a near-perfect copy of the defective DVD. True, there's still 1.3mb worth of holes, but they're filled with zeros and won't throw any IO errors. Sweet, let's toss it at VLC!

But wait, what?! But I copied all of it! Well, most of it! It worked fine before!

Yeah, some of you have already got this. Copying the disc image doesn't copy the CSS keys! Let's take another look inside ~/.dvdcss

Notice how we've now got two sets of disc keys: the ones pulled when I originally played the DVD (timestamped 3:40pm), and a new incomplete set guessed by libdvdcss for that ISO I just tried (timestamped 11:41pm). Do the sensible thing and copy the real keys from the old folder into the new disc folder. Now let's try playing that ISO with VLC again.

Much better. This ISO will now work with any video player or tool that uses libdvdcss to handle descrambling.

File table abuse

If you are processing the DVD with a tool that supports loading ISOs through libdvdcss, then you don't have to worry about this.

Me, I wanted to have a try at scouring that broken title index with VobBlanker, which needs a fully decrypted VIDEO_TS folder to work on. There's a nice little tool called dvdbackup; just run it in mirror mode to decrypt the whole DVD structure to a folder.

$ dvdbackup --mirror --input=brasseye.iso --output=brasseye-decss

Depending on how creative the mastering engineer was, you're going to need a lot of hard-drive space for this. As you can see from the preview icons, this DVD had a lot of straight-out duplicated junk, inflating the size of the mirror to an impossible 18.7 GB. But it's still accurate to the file structure of the original DVD image, and we didn't intend to save it back into DVD format, so not a big deal.

Title index mauling

Popular DVD tools such as dvdbackup, handbrake and vobcopy are usually fine for getting the raw video data out of the VOBs; they just need to know what title number to rip.

The easy method of working around a huge list of broken decoy titles, is to navigate to an episode using the DVD menus, and then write down what title number is being played back (e.g. the "Playback -> Title" menu in VLC)

Unfortunately for me, the Brass Eye disc would only play one episode from the menu. It also had about 90 decoy titles in the index, each with a random length between 27 and 33 minutes, and wired up to random (sometimes unplayable) chunks of video.

I mentioned VobBlanker earlier. This is a Windows tool designed to pick and mix titles from the DVD index, and shows information about how they sit in the VOB filesystem structure.

Here we can take advantage of a few things:

Regardless of how ruined the indexing is, the video data for a title has to be stored in the VOBs consecutively. Jumping around on the disc too much will leave noticeable seek-related gaps in the footage
In this case, duplicates made by abusing the file table appeared in consecutive order

We can see looking here that the disc can be divided up into rough 1GB-sized chunks, which correspond to an episode's worth of footage. This image here shows one of the VOBs, with an index full of broken decoy titles as seen in the bottom pane. The 3rd column contains the title number, and the 7th column contains the "duration", as seen in VLC's playback controls (good luck guessing which titles are real by the length!). By contrast, the 8th contains the actual size of the content the title marker points to, with most of them being 0 bytes long! Yes, all of these titles are bollocks.

This image shows the correct title (number 46), which as you can see has a sensible duration and the largest filesize associated with it. Oddly enough these always seemed to appear at the end of the chunk.

Archival

After going through and writing down all the correct title numbers, I had something I could give to Handbrake. And there you have it; after 4 hours of slog, finally I can watch the content I paid for! Yep, can't imagine why anyone would be driven to piracy.