Support for extended speed zone feature in g64 spec?

[robertjmcintyre]

Hello! A friend tried to reach you via email, but was unable to, and asked me to raise an issue here…

There are several titles (Paperback Filer, Paperback Planner, Chessmaster 2100, The Black Hole, etc.) that contain multiple speed zones per track.

We’re wondering if you would be interested and able to add support to g64conv for the extended speed zone feature of the g64 spec?

Cheers!

[markusC64]

I'd like to discuss that feature request and exchange thoughts about that:

Let's start with the fundamentals:

Automatically detecting speed changes on a track is somewhere between very hard and impossible in the general case. Once you split a track into smaller segments, statistical speed detection becomes unreliable because there simply isn't enough data per segment.

Furthermore, virtually no emulator actually evaluates the extended speed zone information in a G64 image. In fact, I am not aware of any emulator that makes real use of that feature.

So the real question is not whether we can store multiple speed zones — the question is:

How do we determine where speed changes occur?

Without reliable detection, extended speed zone support becomes mostly theoretical.

So the real question is: How to get the speed changes?

Note that I am currently doing extensions in the FloppyDiskImageTool. That tool can already perform more and more tasks of g64conv — faster and scriptable.

In addition to a few bug fixes, I have added multispeed track support — but “only” in the form that you have to specify the speed changes manually:

$p64 | Convert-FloppyDiskImage -FluxToBitstream | Convert-G64resp71ToDxx | Find-ErrorsInDxx

This always provides the number of the sync in the textual output. Then you proceed and specify the switch times relative to the syncs – here using the example of Burstnibbler 1.3, which has copy protection in the form of a multispeed track on track 10:

$decider = New-FloppyDiskImageConfiguration -Multispeed -TracknumberingStyle Commodore1541 -TrackConfig @{
10.0 = @(
@($false, 4, 0, 0),
@($false, 5, 0, 3),
@($false, 12, 0, 0),
@($false, 13, 0, 3),
@($false, 20, 0, 0),
@($false, 21, 0, 3))

}

$false means: Count from begin of sync, $true means count from end of sync. The following number is the number of the sync. The i nteger that follows is the number of bits after the sync (counted from the point specified before). And the last number is the speed to switch to.

$false means: count from begin of sync, $true means count from end of sync.
The following number is the sync index.
The next integer is the number of bits after the sync (counted from the point specified before).
The last number is the speed to switch to.

Now you can call up the decoder and get a real multispeed track:

$g64 = $p64 | Convert-FloppyDiskImage -FluxToBitstream -MultispeedDecider $decider
Update-FloppyDiskImage -PadGCRBitstream -BitStreamImage $g64

An extension of G64, which remains compatible with software that ignores the speed data entirely or evaluates it only with byte accuracy, even allows storing speed information with bit accuracy:

$g64 | Export-FloppyDiskImage -Bitstream -Filename bn13.g64

So multispeed support is technically possible — but it currently requires explicit knowledge of where speed transitions happen.

Any suggestions on how to autodetect speed changes are welcome.

But please keep in mind:
In the general case, automatic detection is not solvable. At best, heuristics may work for specific known titles.

[robertjmcintyre]

Hi, Markus! Thanks for the response! I've actually put some time into thinking about to detects multiple speed zones in a track, and I think I have it (largely) worked out. Happy to share my thoughts, and hope they help.

For reference, I'm using a KF dump of track 10 with 5 rotations from Paperback Writer 128 which has speed zone 0 and speed zone 3 in it (but predominantly one, with just a small section of the other), unlike what BurstNibbler 1.3 has, it looks like.

There's two issues at play: first, determining whether there's multiple speed zones on a track in the first place, and the second is exactly where the boundaries are.

For the first, detecting whether there are any at all, my approach is to:

1. Take the revolutions in the file, and split them into some number of slices (on that track, 36 works really well to illustrate)
2. Filter the flux values to those in the 3x of base band with a bit of extra (~9.25 usec to ~12.5 usec), and create a histogram (I used a bucket size of 2 flux units, but tune as you see fit). Focusing on just the 3x band makes it easier to isolate and identify the peaks since they are further apart than in the 1x/2x bands.
3. Identify the (3x band) peak(s) in each slice, and accumulate the list across all slices, and go from those to speed zones

For the second, knowing exactly where each boundary is, I'm working with mixture fractions:

1. Slide a window around the revolution, and calculate the mixture fraction f:

f(theta) = number of fluxes in the window meeting zone a / total fluxes in the window

2. f(theta) will go from 1 -> 0 (or near it, depending on the window size and the zone size) -> 1. Where f(theta) hits 0.5 (or the maximum rate of change if it doesn't hit 0.5), the middle of the window is a boundary point.

I think that the process of identification of multiple zones is pretty spot-on. The identification of the boundary is pretty good, but can have some wiggle if the relative distribution of fluxes across the bands (1x/2x/3x) on either side of the boundary aren’t the same. For example, the track I'm using, right after the boundary there's a sync, so that will throw it off a small amount. But knowing that, it can be factored in.

But (!!!) as you point out, it does seem academic unless there's actually a use for it and that's where you come in. :) It's a bit of a "chicken and egg" problem: can't get emulators to support the full G64 spec if there's no images that use it. Can't get an image made that supports it if there's no use for it.

Anyway, I hope all that makes sense, and helps somewhat.

Cheers!

[robertjmcintyre]

I'm looking at creating a multi-zone g64 per your instructions/tools above, and my tooling, of pbw128. Is your index:

• Zero or 1-based?
• Starting with the first sync after the index?

Or something else?

[markusC64]

Well, for index counting, it starts at 1 for the first sync after the index. And increases the number at every sync. With one major exception:

• If the index is on a sync, it is counted as number 1 even if there are not enough ones triggering the sync when start reading at the index.
• If a sync is completed before the index, it's number is the number of syncs.

And the textual representation always contain the sync number as comment.

I haven't access to an pbw128 dump, so I cannot tell about that special image.

[markusC64]

Well, in rare cases, floppydiskimagetool counted wthe sync numbers wrong. I have commited a fix.

[robertjmcintyre]

Thanks, Markus. I've tried to make a multi-speed image of this disk dump and have some questions:

1. The track I'm working with is track 11, which is normally at speed zone 3, but in the protected track it is speed zone 0, except for the segment starting at the beginning of sync 16 and ending just before sync 17. I'm unsure how to represent that in the TrackConfig. I tried doing:

$decider = New-FloppyDiskImageConfiguration -Multispeed -TracknumberingStyle Commodore1541 -TrackConfig @{11.0 = @(@($false,1,0,0),@($false,16,0,3),@($false,17,0,0))}

But I'm not sure that's correct.

2. When I did the above, and the rest of the steps, I did get a disk image, but the custom speed zone section is... not what I expected. It is:

• 12650 bytes long, instead of the 1990 or so I would have expected
• Has 8 repeating sections of 1272x 00's, followed by a string of 86x FF's (I didn't do counts on all 8 segments, but they look similar/the same). I expected a string of 00's, followed by a string of FF's, followed by a much shorter string of 00's, and that's it.

Can you confirm the right TrackConfig syntax for a track that is NOT predominantly at its normal speed zone (so the whole track needs a mapping). And clarify that the custom speed zone is/is not as defined here: https://vice-emu.sourceforge.io/vice_17.html#SEC408

Thanks!

[markusC64]

Ok, a track is a circle on the disk. You have to set speed only where the speed changes, the warparound is automatically considered.

TrackConfig @{11.0 = @(@($false,1,0,0),@($false,16,0,3),@($false,17,0,0))}

is therefore equivalent to

TrackConfig @{11.0 = @(@($false,16,0,3),@($false,17,0,0))}

under the assumption that sync 17 exists.

This means: On detection of the 16th sync switch to speed 3 and at the 17th synmc switch back to speed 0.

You should know that official g64 format defines speed zones per byte. That is a major problem because bytes are not synced to what the 1541 sees since the 1541 adjusts the byte borders at every sync. So we need a speed per bit.

In order to make it compatible to software that reads the sync per byte, the speed for all bits i with (i mod 8) == 0 is first written to the stream, then all bits with (i mod 8) ==1 and so on up to (i mod 8 == 7).
So when just reading the first few speeds, you get them per byte.

This extension holds if and only if the "EXT\2" extension is found in the G64, so when reading, existing G64 are not misinterpreted.

And yes, Custom speed is following the VICE specificatioon, but at the end more bits are appended. Since G64 is specified by a table that has file positions, that should not break any reading software.

[robertjmcintyre]

Ok, thanks for the clarification on the TrackConfig. I imagined that you were doing something with the bits vs. bytes when I counted the 8 vs. 1.

I appreciate you taking the time for this experiment. I do fear that the bits vs. bytes will make a difference, not only in the handling of speed zone but the need for data to be byte aligned as well.

I constantly go back and forth between the value of making g64's work vs. embracing p64's. p64's have 1st-class support by VICE, and I have a request out to Gideon for adding support to 1541UII/C64U/U64. But, at least right now there's ONE emulator that supports p64.

(the p64 image we have is a direct result of your work with p64conv, btw, so thanks!!)

[markusC64]

I can tell you one more emulator that supports P64: Denise.

And you're right. having the track length to be a multiple of 8 bytes is yet another problem with g64. But the good news is that this powershell library can handle g64 that does not have tjhat limitation. But they're not really backward compatible in the sense that they work in tools not adjusted for that format. But they also should not crash anyway. Just leave the pading step out - and you might get such an image. It depends on the data, if there is nothing to pad then padding is of cause a noop and therefore leaving it out makes no difference.

BTW: I have implemented that because the powershell library can convert such a g64 to p64 obeying all the speed information.

[robertjmcintyre]

Thanks for the tip on Denise. I haven't looked at it in a while, but I will now. :)

Stepping back, one of the goals for this exercise is/was to see what's involved with getting multi-zone g64 support into an emulator, and into image creation tooling. The image creation tooling is, I believe, possible as you've shown, but looking at the emulator part and just being honest (with myself and anyone else reading this), I think that effort would be better spent getting p64 support included (for those emulators/devices that don't support it already).

The multi-speed-zone is one issue with g64's right now. It's unclear whether there will ever be another issue found with it that would need working around/fixing because of the byte-alignment requirement, etc., whereas p64 -should- support anything that a disk physically can.

So, I think that this exercise has run its course. I thank you so much for the engagement/discussion, and the tooling you've created!