About Me

Curriculum Vitae

A brief list of my current skill set


More Benchmarking in Node.js and V8
Tuesday, 14th August 2012, 12:19

Working out the fastest way to route

MinnaHTML.js Benchmarking for Speed in Node.js
Monday, 13th August 2012, 17:55

Don't believe it, test it

Playing Around With HTML5's Canvas
Friday, 10th August 2012, 16:19

Speccy loading screens in a browser!

Scripting in Node.js AKA How to Watch for Olympic Tickets Using a Script
Tuesday, 7th August 2012, 23:37

Let Node refresh the webpage so you don't have to!

A Javascript Confirm/Alert Replacement jQuery Plugin
Monday, 6th August 2012, 23:06

Much prettier than the horrible alert and confirm dialogs

A Few Node.js Essential Modules
Friday, 3rd August 2012, 14:40

As essential as they can be anyway

Retro Coding Corner: Loading ZX Spectrum Snapshots off Microdrives - Part 2
Tuesday, 31st July 2012, 18:20

It's like juggling 8 balls with one hand behind your back

One Layout To Rule Them All
Monday, 30th July 2012, 16:03

Designing a layout to fit every screen

Retro Coding Corner: Loading ZX Spectrum Snapshots off Microdrives - Part 1
Thursday, 26th July 2012, 16:57

From emulation to super fast tape

Upgrading Libxml2 on CentOS 5.8 fixes Libxmljs
Wednesday, 25th July 2012, 11:33

Fixes strange compile errors and everyfin

Installing Node.js 0.8.x under CentOS 5.8
Tuesday, 24th July 2012, 14:13

It's just an upgrade of Python away

Node.js and Object Oriented Programming
Sunday, 22nd July 2012, 23:46

Make pages inherit pages

MinnaHTML.js - A HTML Web Framework For Node.js
Saturday, 14th July 2012, 22:53

Make webpages the OOP way

Everyone Should Learn to Debug Without a Debugger
Monday, 11th June 2012, 22:42

Because one day they'll all get eaten

Geeky Things What I Wrote Elsewhere
Wednesday, 2nd May 2012, 17:51

A few links to geeky articles I posted on another website

Projects and Sillyness

MAME Cabinet Diary

How I built my own arcade cabinet

Loading Screen Simulator

I don't miss the ZX Spectrum, I still use it!

The Little Guy Chat Room

It's a Pitfall inspired chat room


A fully featured MP3 player what I wrote

GP Space Invaders

My first little emulator

GP32 Development Page

Some info and links about this cute little handheld

Disney Nasties

Uncensored images, you must be 18 to view them

Diary of a Hamster

Learn about how hamsters think, first hand


Time Calculator

A simple little online utility for working out how many hours to bill a client

A Few Links

Retro Coding Corner: Loading ZX Spectrum Snapshots off Microdrives - Part 1
Thursday, 26th July 2012, 16:57

The older, less American amongst you, will probably be at the very least familiar with the ZX Spectrum microcomputer. During it's long life, before Amstrad swallowed it up and redesigned the hardware to make it cheaper and give it a real keyboard, original makers Sinclair Research created their own fast storage medium.

You can skip the next bit if you know what they are...

When I say fast storage, things are of course relative. With a capacity ranging between about 70k-120k, depending on how much the tape had stretched over time (yes really), it could load a full 48k worth of memory in about 5 seconds. That might sound a lot these days, but the alternative back then was cassette tape, and a few minutes worth of it.

Far from perfect as a medium, the stretching of the tape would render the data unreadable over time, and inside it was actually a long tape loop which meant to read a bit of info that had ran past the head required the full go-around to get there again. Random access it was not.

But this legendary doomed 8-track inspired system has a special place in my heart, not because I could afford it during the original short lived release cycle, but because it's just so unique as solutions to problems go. It went on to appear in the even more ill-fated computer system, the Sinclair QL, something worth reading about if you like tech disaster stories.

Saving Games to Microdrives

This was my goal, save first 48k and then 128k games so they can be loaded off a microdrive cartridge, complete with a loading menu to allow multiple ones per cartridge, room permitting.

Not as easy as it sounds, firstly games have copy protection in the form of fancy loaders, and secondly when it comes to 128k games, if a typical cartridge can store say 90k, how do you save 128k worth of it? The answer to the latter is of course compress it, but that in itself leads to interesting challenges.

A solution to the copy protection issue is to use emulator snapshots as the starting point. These are a saved state of the computer, with every byte of memory and CPU register all recorded at a moment in time. It's very easy to launch an emulator, start loading a game and then pause it at the point just before the loading screen disappears, and save a snapshot.

Doing things this way also has another added advantage, you can write a quick command line program to read in the snapshot, compress it, then write it out with a loader.

Now if only it all turned out to be this easy. :)

Compressing the Data

Because my C was a tiny bit rusty, I caved straight away and installed Visual C++ Studio Express. Even though I don't like Microsoft stuff, it had a built in debugger that would speed up finding silly mistakes I would no doubt make due to being out of practise with all things C.

I did however decide to intentionally keep things as portable as possible, and limit myself to non-Windows specific libraries, which also meant command line only. Still, whilst UIs are nice and all, anyone who is taking snapshots in emulators and trying to convert them to run on the original machines is probably already more than capable of coping with a command line tool.

So with fopen and friends, details of the Z80 file format from various places to cross reference, reading in the limited amount of data you need to support standard retail 48K and 128K Speccies is pretty straight forward. Even the Z80 compression side of things is trivial. However the reason it is trivial, is because it has a terrible ratio, so we need to use something else.

Options for compression when it comes to Z80 is actually quite varied, with some great ratios. However selecting one requires a few things to think about, the first being the decompressor must be small, ideally under 150 bytes. The second is there must be a compressor option written in C.

This lead me ultimately to MegaLZ, written by Russian coder mayHem. It not only includes source code for a C compressor and decompressor, but the Z80 code is pretty fast, fully relocatable and just 110 bytes in length. With that whacked in, I now had a method for compressing the memory data, all that I needed to add was the bits at the other end.

The BASIC Loader

After a hard reset, a ZX Spectrum can launch a file on a microdrive cartridge by doing nothing more than typing the RUN command. The moment you issue any other command though, and this no longer works, it defaults to running the program you've typed or loaded in already, which is obviously not there. But it's a great shortcut, and after doing some simple tests I soon realised that when it came to 48K based games, more than one was going to fit on a cartridge.

So the best thing to do would be write a simple BASIC loader, since that is the only thing that RUN will launch automatically anyway. This loader would catalogue the microdrive and present a list if files in a simple menu. When you select the game to load, it would load that into memory and then call the decompression code.

Easy enough in theory, in practise I came across two annoying issues. The first was, BASIC has only one way to catalogue the files on a microdrive, the CAT command, and that directs output to the screen which means there is no easy way to grab it into an array for presenting as a list to load.

However, wandering through an old book called Make the Most of Your ZX Microdrive, I found a machine code routine with a matching type in that redirected output of the CAT command to a variable. From here you just have to work through the variable and build an array of files, excluding of course the one called "run" and anything beginning with an underscore, for reasons I'll explain later.

Making the loader as small as possible, whilst still leaving room for variables it used, resulted in the ability to lower the top of BASIC RAM (RAMTOP) to 26999. That gives us a maximum total of 37.6K for our compressed data, obviously not enough for a 128K snapshot but we'll deal with the workaround for that later.

The 48K Loader

As previously stated, with roughly 37K to play with, and that includes the 250 odd bytes we'd need to store our restoration code, if MegaLZ cannot save us 11K (a compression ratio of 78.3%) then whatever game fails to be reduced that much just won't work. So far though, most games do a lot better than that, the worst I've found is Sabre Wulf which reduces to 30K. Small games like Cookie are a mere 10K, and Manic Miner is only 15K.

Now, I like things pretty, so I decided to do restore the screen area first, then the remaining memory so you had something to look at whilst it went off on one. This did mean compressing the screen area (16384-23295) separately, and then the remaining block (23296-65535) after, but there is method to my madness. The ultimate plan here would be 128K games, and that is a lot more loading time to await so having a loading screen to view is more aesthetically pleasing.

Restoring a ZX Spectrum to an identical state as possible to that of the machine in a different point of time snapshot requires a number of things.

Set All Memory Contents to Match the Snapshot

Setting all the memory to match a snapshot is, firstly, impossible on a real machine without special external hardware. This is because the program that restores everything has to sit in memory, and clearly whatever area it sits in overwrites data in the snapshot.

So we stick it in screen memory, because whilst it will corrupt that a bit and make it look a tiny bit messy, all this will effect is us looking at it, 99.99999% of programs won't care one jot, especially straight off a loading screen. Screen memory is treated very much as an output device, not a place to store important data or code.

Set Any Important Ports to Their Correct State

Port wise, for 48K snapshots the only one we care about really is 0xFE, because it sets the colour of the border and we want that to be correct. However, because 128K users could be loading in 48K snapshots, we will also write to 0x77FD and disable RAM paging. A lot of early games don't run in 128K mode for the simple reason that they unintentionally output to ports which switch RAM banks, and nothing screws a program flow up more than half of its code being swapped out for something completely different without warning.

Setup All the CPU Registers to Match Those in the Snapshot

Speccies have a number of registers, and then a duplicate set of most of them that you can swap between. The general ones BC, DE, HL, IX, IY, AF', BC', DE', HL' are trivial to reset to their old values. The interrupt register I isn't too bad either, but PC, SP, AF and R require more thought.

PC, being the program counter, points to the current instruction. The last thing our loader code will do is jump to wherever PC pointed to in the snapshot. SP is the stack pointer, and we'll need to position it somewhere safe for the decompression code to use, not to mention later on when we need to load more things from microdrive for 128k snapshots and are calling ROM routines.

But just before we set PC, we'll set SP (stack pointer) to wherever it used to be, so again quite simple. AF is more of an issue, we'll set it just before we set the SP, because whilst A (the accumulator) is only affected directly, F is the flags register and all sorts of commands can cause the Carry and Zero flags to get set/unset. Changing PC and setting the SP shouldn't affect F though so again providing we do it at the end, we are all good.

Now the R register, a mystical beast, this is the refresh register and is connected to memory. Trying to understand how this 7 bit wonder behaves is, er, fun. So why bother? Well some games actually use the R register to detect attempts to circumvent copy protection, and in the interests of completeness, if we can set it why the hell not?

A reason that makes it difficult is... it's the refresh register! If we set it to the same as the snapshot, then the moment we execute a single other instruction it changes, so by the time we set PC it will be wrong. So what we do is calculate backwards from what it should be after JP call, to where we set it, so by the time PC is set correctly, R is what it was in the snapshot.

Relocation, Relocation, Relocation

All 251 bytes of our 48K loader code is prefixed before the compressed screen, which in turn is followed by the remainder of compressed memory. Our BASIC loader will load this in at 27000, then execute it. The code itself will decompress the screen, and then copy the absolute minimal remaining code to the start of the screen display and execute that. Because MegaLZ can potentially overwite the compressed code with the decompressed code before it is finished, we also need to move that as high as possible in memory first.

So the source code for all of this becomes the following...

   ORG 27000
IM 1

So we disable interrupts and set IM1, although if IM2 is active our C program will change this code as appropriate. It's very important to disable interrupts as we'll be changing not only the interrupt register but also the memory it may point to, which would result in a crash until the full snapshot is restored.

LD A,0x30
LD BC,0x77FD

Here we make sure the 48K ROM is paged in, which does nothing on a 48K Spectrum, just a 128K version. And we also disable further paging on 128K models, this is what essentially happens if you select the 48K BASIC option in the 128K menu.


Since we aren't actually going to use anything like all of the Z80 registers during our decompression routine, we can safely load their original values early. We abuse the stack pointer for this purpose, to save a few bytes.

LD DE,16384

Next we call the MegaLZ decompression routine to show us that nice pretty loading screen we compressed.

LD DE,65535
LD BC,0x1010 ; Set to data length

Now we copy the remaining compressed data to the top of memory with a backwards copy so as not to overwrite itself. Note that BC and HL are set to the correct values by our C program, since otherwise this generic loader could not possibly know the correct values.

LD DE,16384
LD HL,65536 - 0x1010 ; Set to new start of data
LD DE,23296 ; Destination
JP 16384

This section is initially a bit complicated without an explanation. Essentially what we do is move all of the remaining loader code to the start of the screen memory, then we fix the stack pointer so it also sits in screen memory. Then we load up HL and DE for the source/destination for the MegaLZ decompressor (which we've also relocated), but then what we do is call the start of screen memory where our remaining code now resides.

Doing it this way minimises the screen memory required, and corruption we cause.

CALL 16384 + (DEC40 - NEWDEPACK)

This part and all code following now resides at 16384, and not 27000, so we can safely decompress all the data after screen memory to the top of RAM.

LD DE,65536 - 10
LD BC,10

Now, due to the threat of overlaps with MegaLZ, it is entirely possible that when it reaches close to the end of memory it begins to overwrite the compressed data with the decompressed data. To help alleviate this, what we actually do with our C program is compress 23296-65525, and then copy the final 10 bytes by hand using the section above. Not perfect, but it gives us a little leeway which we didn't have before.

In testing, I did find some snapshots that ended up with the last few bytes corrupted, and that can effect a number of things not least custom interrupt routines. Doing the above really helped solve this, but is by no means a guaranteed flawless solution for every single snapshot.

EX AF,AF' ;'
POP BC ; R and Border
OUT (C),A ; set Border
LD R,A ; set R
EI ; NOP out if DI

Now everything is decompressed, we set up the border colour and remaining registers, re-enable interrupts (or if they shouldn't be the C program overwrites the command with a NOP instead), and then jump to the required value of PC.

   DEFW 0x0000 ; for depacker usage
DEFW 0x0000 ; for DE call
DEFW 0x0606 ; DE
DEFW 0x0707 ; HL
DEFW 0x0808 ; AF'
DEFB 0x09 ; R
DEFB 0x01 ; Border Colour
DEFW 0x1010 ; BC
DEFW 0x1111 ; AF
DEFB "0123456789"

Following the main chunk of our code is a chunk of data, this is used to set the final registers, the last 10 bytes of RAM and provide us with a tiny bit of space for stack usage.

LD A,0x80
blah blah blah blah blah blah

This next bit is a heavily summarised version of the MegaLZ decompression code. Suffice to say it is a wonderous thing, 110 bytes long and available as part of the MegaLZ package.

DEFW 0x0101 ; BC'
DEFW 0x0202 ; DE'
DEFW 0x0303 ; HL'
DEFW 0x0404 ; IX
DEFW 0x0505 ; IY

Lastly, we have a bit more data that we use to set the registers at the very beginning of our program, and then SCREENDATA marks the start of our compressed code.

The 128K Loader

The above is very simple stuff, alas things get a lot more complicated with the 128K Spectrum. Firstly, if we can only load in data as 37K at a time, there is no way we are going to be able to load a 128K compressed file in one go. We need to split it up into multiple files, and three is the minimum.

It is probably possible to make a funky custom microdrive loader that lets you read it all from one large file, but that is a very difficult proposition and also makes transfering data from your PC to the real thing harder. At the moment the output of the C loader is a TZX tape file, so you'd need a custom loader for that as well.

So now we need to do a lot more than just load in some code from our BASIC program and execute it, we need to then have that code load more code, twice. To complicate matters even further, the 128K and +2 models can only page in the extra RAM into two banks, and only one of those can have all RAM banks paged into it.

The solution to this, is a juggling act and an awful lot of copying. A story for another time.


posted by Salvatore on Tuesday, 1st December 2015, 16:07

Very compliment for this fantastic job!
Please, how can bypass the limit of 92160 bytes to obtain a snapshot?
Best regards.

posted by Robee on Tuesday, 1st December 2015, 16:23

Do you mean you have a snapshot which doesn't compress small enough for the utility? Atm there isn't a workaround for that, but the compression library I'm using has improved so that might increase this limit slightly but not I suspect by more than 1 or 2k.

Have you got a link to the snapshot you are trying to restore?

posted by Salvatore on Tuesday, 1st December 2015, 17:31




posted by Robee on Tuesday, 1st December 2015, 17:57

I don't think there is any good way to reduce the file size any further, the new version of Exomizer doesn't seem to increase the compression level. :/

I need to put the source on github, I'll do that soon, and when I do I'll recompile it to make that a warning rather than a stop error.

There is a chance it won't fit on your microdrive cartridge, but at least you can give it a go!

posted by Salvatore on Tuesday, 1st December 2015, 18:27

Ok, i'll wait.
Many thanks!

posted by Robee on Saturday, 5th December 2015, 19:58

Now on github with source:

Also new version makes the overall size limit a warning rather than an error. You'll still need a Microdrive cartridge big enough for the overspill though, no magic way around that I'm afraid!

posted by Salvatore on Wednesday, 9th December 2015, 19:01

I'm sorry but the new exe gives me always an error.

This is the error screenshot


posted by Robee on Wednesday, 9th December 2015, 21:13

Are you sure you have the right version?


posted by Salvatore on Wednesday, 9th December 2015, 21:16

Yes, i am.

posted by Robee on Thursday, 10th December 2015, 08:58

Can you translate the error into English? Maybe then I can help.

Also are you running 32 bit or 64 bit?

posted by Salvatore on Thursday, 10th December 2015, 17:29

I'm running a 32 bit (Xp)
The error message says:
"The NTVDM CPU has encountered an illegal instruction
Choose 'close' to terminate the application."

It's very strange because the 0.21 version works perfectly....

posted by Robee on Saturday, 19th December 2015, 13:48

I think your download might be corrupt. I just downloaded a fresh copy from github and ran it on my WinXP 32-bit machine, and it worked first time. :/

posted by Salvatore on Saturday, 24th December 2016, 13:34

Many thanks, Robee. Finally now it works!
It was my pc fault!

posted by Robee on Saturday, 24th December 2016, 16:31

Thanks for letting me know! Have fun with it, microdrives are great. :)

Add Your Own Comment