RFC: Application Package Format

Alcoholics Anonymous
Mensajes: 8
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensajepor Alcoholics Anonymous » 07 Jul 2018, 08:33

It's been a while and I am about to implement a RIFF-like .exe format for the next. However, it is necessarily very next-centric because of the machine's feature set. The start-resume block might be a transferable idea that allows the program to run part way through the load. This would be for introductions, configuration, cut scenes, etc. Another idea is the obfuscation seed in the stored data; this is likely going to be a seed for a prng that generates bytes XORed with data after it is loaded. This is not a protection scheme but a means to hide plaintext as people on facebook asked that files not be readable in a text editor. Apparently the temptation to cheat in games is too great. Remark blocks could contain information like copyright, brief description and so on, that could be printed via the dot loader instead of loading the program. A special loading screen block is not defined yet but will be added. Having a separate loading screen block not only makes it easy to add one, but allows the dot loader to display it instead of running the program.

Most of the existing blocks are for loading into a logical memory space (one that is allocated from free memory pages) where the loader needs to create an allocation table and write it into the loaded program. Anyway because of the unique features of the next, the format is necessarily next focused.

Código: Seleccionar todo

**************************************************************************************

Header Format:

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'NX' ('N' in byte 0, 'X' in byte 1)
2        2     File format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        1     Size of page allocation table (0 if none)
7        1     Size of extra pages allocation table (0 if none)
8        1     Size of divmmc allocation table (0 if none)

9        ?     Page allocation table (0xff = skip, 0xfe = allocate, other = error)
?        ?     Divmmc allocation table (0xff = skip, 0xfe = allocate, other = error)


Block Format:

OFFSET   SIZE  DESCRIPTION

0        2     2 characters describing block format
2        2     Size, in bytes, of the data following these two bytes
4        ?     Block data


**************************************************************************************



-- Remark

OFFSET   SIZE  DESCRIPTION

0        2     Characters '--'
2        2     Block format version (should be 0 for printable comment block)
4        2     Size, in bytes, of the data following these two bytes

6        ?     Optional data that may be skipped by loader





PL Page Load

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'PL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Obfuscation seed (0 = none)

8        1     Page # to load to (range 0-223)
9        2     Offset into page (range 0-8191)
11       2     Number of bytes to load (can span several pages)

13       ?     The bytes to load into the page, length equal to value at offset 11




DL Divmmc Load

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'DL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Obfuscation seed (0 = none)

8        1     Page # to load to (range 0-15)
9        2     Offset into page (range 0-8191)
11       2     Number of bytes to load (checked to remain within 8k divmmc page)

13       ?     The bytes to load into the divmmc page, length equal to value at offset 11



;;; LS Load Screen
;;;
;;; OFFSET   SIZE  DESCRIPTION
;;;
;;; 0        2     Characters 'LS'
;;; 2        2     Size, in bytes, of the data following these two bytes
;;;
;;; Not specified yet
;;; Screen data already loaded, copy to destination for effects?
;;; Choose ula mode (spec, timex, lores) or layer 2
;;; set palette maybe different block to load nextreg state
;;; loading effect (eg border)



PB Pause Block

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'PB'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (4)

6        2     Wait in milliseconds (0xffff = wait for no keypress)
8        2     Pause in milliseconds (interrupt with keypress and maybe with joystick fire)




CL Write command line

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'CL'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (8)

6        2     Page # to write to
8        2     Offset into page (range 0-8191)

10       1     Command line type (0 = unprocessed esxdos, 1 = unprocessed nextos-esx, 2 = empty argc/argv, 3 = argc/argv)
11       1     Write direction (0 = forward else stack push)
12       2     Maximum length in bytes (can span pages)



WD Write current working directory

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'WD'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (4)

6        2     Page # to write to
8        2     Offset into page (range 0-8191)




AT Write allocation table

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'AT'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes

6        2     Page # to write to
8        2     Offset into page (range 0-8191)

; list follows terminated by zero byte

?        1     1 = Write sysvar bank state
?        1     2 = Write initial mmu state
?        1     3 = Write page allocation table
?        1     4 = Write extra pages allocation table
?        1     5 = Write combined page and extra pages allocation table
?        1     6 = Write divmmc allocation table
?        1     0 = End of list (any other byte)




ST Start Block

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'ST'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (13)

6        2     PC
8        2     SP (required if PC < 0x4000 and must be >= 0x4020)

10       1     Flags (bit 0 = 1 for DI, bit 7 = 1 to keep file open)
11       8     MMU state (mmu0...mmu7, 0xff = no change; ROM3 Banks 5,2,0 default)

; start with:
;
;   hl = argv / esxdos command line
;   bc = argc / esxdos command line length
;    e = file handle




SR Start-Resume

;;; Program runs with ".run" still in divmmc memory and returns to resume
;;; loading.  For introductions, cut-scenes, instructions, etc.

OFFSET   SIZE  DESCRIPTION

0        2     Characters 'SR'
2        2     Block format version (should be 0)
4        2     Size, in bytes, of the data following these two bytes (13)

6        2     PC
8        2     SP (required, return address on stack)

10       1     Flags (bit 0 = 1 for DI)
11       8     MMU state (mmu0...mmu7, 0xff = no change; ROM3 Banks 5,2,0 default; mmu0/1 ignored)


Matt also landed on the RIFF format idea and began this spec with the leading NX identifier block and a couple of blocks to load into physical memory and start. I've filled in the rest to do with logical memory mapping as I'm the only one working on that.

In the meantime, several big memory snapshot formats arrived. One is the big sna which is a standard 128k snapshot with data for more memory banks appended. This is compatible with esxdos. SNX is like a big sna but when it is loaded, the file handle for the SNX file is left open so the program can continue to load without having to know its filename (the big sna must have its filename hardcoded into its binary). These are mainly for development because one of the most important Next emulators is SNA only. Jim's big file format is called NEX and is memory state with an optional loading screen.

I did implement environment variables and temp files for esxdos and finished the .run dot command mentioned earlier.
https://github.com/z88dk/z88dk/tree/mas ... ommand/run

This dot command uses an environment PATH variable (the environment is just a text file holding "name = value" pairs) to find and load programs. It currently handles .tap, .sna, .snx, .z80, .p, .o, .nex. NextOS supplies the sna/snx/z80/p/o capability but I plan to build a version for esxdos as well using snapload so an esxdos version should be able to support tap, sna, z80 at least. For tap, I decided on exiting after the tap file is mounted with instructions to LOAD"" after the dot command completes.

Avatar de Usuario
chernandezba
Mensajes: 789
Registrado: 02 Oct 2015, 23:35

Re: RFC: Application Package Format

Mensajepor chernandezba » 09 Jul 2018, 14:16

Good idea. But I think there's a mistake on just using two bytes to tell the data lenght: you are limiting data blocks to 64kb.
Why not using 3 bytes, so having 16 MB data block maximum?
----

ZEsarUX
ZX Second-Emulator And Released for UniX
https://github.com/chernandezba/zesarux

Alcoholics Anonymous
Mensajes: 8
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensajepor Alcoholics Anonymous » 10 Jul 2018, 03:27

chernandezba escribió:Why not using 3 bytes, so having 16 MB data block maximum?


Yes I am thinking about that too. I may make it 4 bytes instead as that will allow skipping over large sections of the file. The format allows unloaded data to be appended to the file for things that are loaded on demand at runtime and the SR block (start and resume) may want to embed a large unformatted data block that is skipped by the loader. There are people doing video that might need segments > 16MB.

I may remove the obfuscation as well and let the program / generating tool handle that.

Avatar de Usuario
chernandezba
Mensajes: 789
Registrado: 02 Oct 2015, 23:35

Re: RFC: Application Package Format

Mensajepor chernandezba » 10 Jul 2018, 09:58

Alcoholics Anonymous escribió:
chernandezba escribió:Why not using 3 bytes, so having 16 MB data block maximum?


Yes I am thinking about that too. I may make it 4 bytes instead as that will allow skipping over large sections of the file. The format allows unloaded data to be appended to the file for things that are loaded on demand at runtime and the SR block (start and resume) may want to embed a large unformatted data block that is skipped by the loader. There are people doing video that might need segments > 16MB.

I may remove the obfuscation as well and let the program / generating tool handle that.


Yes, 4 bytes would be even better.
On my ZEsarUX snapshot file format (.ZSF) I use 32 bit values for data block lenght, and 16 bit values for data type
----

ZEsarUX
ZX Second-Emulator And Released for UniX
https://github.com/chernandezba/zesarux


Volver a “Core ZX Spectrum”

¿Quién está conectado?

Usuarios navegando por este Foro: No hay usuarios registrados visitando el Foro y 2 invitados