RFC: Application Package Format

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 11 Ago 2017, 21:13

Supporting types like sna & tap just offers a way to organize your programs through the .run command. For sna, I will try to load up and run .snapload to do the actual load in that case.

I've been thinking a little further. I think it wouldn't be too hard to have .run maintain a list of directories to search, like a PATH variable. Then:

.run name
will look for the program name to launch in the ordered list of directories in PATH.

.run -w name
will report which directory name is found in

.run -p
will print out PATH

.run -lp src.txt
will load PATH from a text file containing a list of directories.
PATH is saved inside the .run binary.

.run -ap pathname
pathname is added to the end of PATH.
PATH is saved inside the .run binary.

The default build of .run would only contain /programs in its path but the user can change that and have multiple directories searched if he wanted.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 12 Ago 2017, 01:20

Alcoholics Anonymous escribió:Supporting types like sna & tap just offers a way to organize your programs through the .run command. For sna, I will try to load up and run .snapload to do the actual load in that case.
That will be a pain to do. You'll have to get out of divMMC RAM into main RAM and force the BASIC command ".snapload <filename>". The way divMMC arbitrarily gets paged in and out is bad enough to begin with before you have to deal with dropping in and out of divMMC RAM.
I've been thinking a little further. I think it wouldn't be too hard to have .run maintain a list of directories to search, like a PATH variable.
Or you could just have a path file in the OS folder as a plain text file with a list of folders.
.run name
will look for the program name to launch in the ordered list of directories in PATH.
Seems reasonable, although I can't see a reason not to keep all the apps in an app folder.
.run -w name
will report which directory name is found in
Seems redundant, but ok.
.run -p
will print out PATH
.PATH would be a better command for viewing and setting the path file. Any other dot commands that are aware of the path can use it. I'd keep the file itself in the root of the disk though, because some versions of UnoDOS don't require the DOS folder.
.run -lp src.txt
will load PATH from a text file containing a list of directories.
PATH is saved inside the .run binary.
As I say, I think a generic path would be better than one specific to .RUN. If I was specifying a path with .RUN I'd expect it to take the form .RUN <path>/<filename>.
.run -ap pathname
pathname is added to the end of PATH.
PATH is saved inside the .run binary.
Again, I'd put that in a .PATH command.
The default build of .run would only contain /programs in its path but the user can change that and have multiple directories searched if he wanted.
I still can't see the need for more than one apps folder. Especially if one of the apps is a game browser. But there you go. Plenty of things to consider but my primary concern was getting a robust app format working, which it does, and coming up with a simple way to convert snapshots to apps, which I'm working on. Everything else can wait until later.

One thing I would like to avoid is competing standards. I've been working on this for nearly a year now and, like the rest of UnoDOS 3, it will be open source. So there's the potential to contribute somewhere down the line. Of course I can't stop anyone from doing it (which is why I haven't published full details on the format ... because a certain group of people are likely to help themselves to it and break compatibility).

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 12 Ago 2017, 17:05

aowen escribió:
Alcoholics Anonymous escribió:Supporting types like sna & tap just offers a way to organize your programs through the .run command. For sna, I will try to load up and run .snapload to do the actual load in that case.
That will be a pain to do. You'll have to get out of divMMC RAM into main RAM and force the BASIC command ".snapload <filename>". The way divMMC arbitrarily gets paged in and out is bad enough to begin with before you have to deal with dropping in and out of divMMC RAM.
Dot commands dont't have to be hard because you can load them to 0x2000, put a command line in HL and then jump to 0x2000. Because this is a snapshot loader you don't have to worry about trashing any ram.

But doing taps means you have to somehow do LOAD"" and keep basic happy. If pushing LOAD"" into the edit buffer is the way to go then maybe it would be just as easy to do ".snapload name" that way too.
Or you could just have a path file in the OS folder as a plain text file with a list of folders.
Yes that may be the way to go. Have a special "PATH" file that is easy to edit and is available to anything. Place it in the root dir of the system drive. If it is absent then just default to a built in default search path.

Paths would probably have to be able to accommodate a drive identifier:
hd0:/programs

If drive id is left out, maybe a default can be provided. Paths have to be absolute so if initial / is missing, that's an error. Paths can be separated by CR/LF (one per line). Leading and terminating whitespace in a path is ignored. Maybe accommodate both / and \ as directory separator.

So:

.run [n] name ["command line"]
will look for the program name to launch in the ordered list of directories in PATH.
name can be a relative pathname (path without leading / followed by name)
name can be an absolute pathname with optional drive (no PATH search is done)
the optional n is a number indicating the nth match should be used in a PATH search
optional command line enclosed in quotes

.run -w name
will report which directories name is found in

.run -p
will identify problems in PATH
(it's essentially free since checks will be done anyway)
One thing I would like to avoid is competing standards. I've been working on this for nearly a year now and, like the rest of UnoDOS 3, it will be open source. So there's the potential to contribute somewhere down the line. Of course I can't stop anyone from doing it (which is why I haven't published full details on the format ... because a certain group of people are likely to help themselves to it and break compatibility).
Yes things should stay compatible. I think the only point of incompatibility is how to load that first binary. In my view there has to be a header and it has to be a header that can change if new ideas come up.

At the moment I can see a header like this being useful:

<?> Signature
<2> Length of header from here
<2> Optional RAMTOP value (0 = no ramtop check) - behaviour is to exit RAMTOP no good if current RAMTOP is too high
<2> ORG of binary following header
<2> Optional length of binary following header (0 = load whole thing)
<1> Flags: bit 0 = 1 to pass command line, bit 1 = 1 to pass file handle

If such a header does not exist (signature is not there) then just load at 32768 and execute.

The file handle thing is there because I see the option of loading the first part of a file as a loader and then the loader continues with that file handle to load the rest of its bits into memory banks.

The command line is constructed on the stack which will be below RAMTOP.

So loading would mean reading the initial header block to learn ORG and all that and then load the rest to execute.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 14 Ago 2017, 08:12

Alcoholics Anonymous escribió:Dot commands dont't have to be hard because you can load them to 0x2000, put a command line in HL and then jump to 0x2000. Because this is a snapshot loader you don't have to worry about trashing any ram.

But doing taps means you have to somehow do LOAD"" and keep basic happy. If pushing LOAD"" into the edit buffer is the way to go then maybe it would be just as easy to do ".snapload name" that way too.
The problem is you can't call the BASIC ROM from $2000 and have it read data from there, so you do have to trash some main RAM. For most systems it would be better to just use or extend the current snapload command, but I plan on supporting systems that page RAM in the first 16K. You can't load data there because it will overwrite the OS, so you have to load it somewhere else and page it in and again, you can't do that while operating from $2000.
Paths would probably have to be able to accommodate a drive identifier:
hd0:/programs
I'm working on the assumption that there will be a single drive. With IDE it made sense to have multiple drives for reading CDs and so on, but there really isn't any point having more than one drive under MMC, even though it's supported.
Yes things should stay compatible.
Well that seems to be off the cards now as I gather Jim is writing something and will presumably do it his own way. So that will be yet another standard the Next has broken.
I think the only point of incompatibility is how to load that first binary. In my view there has to be a header and it has to be a header that can change if new ideas come up.

At the moment I can see a header like this being useful:

<?> Signature
<2> Length of header from here
<2> Optional RAMTOP value (0 = no ramtop check) - behaviour is to exit RAMTOP no good if current RAMTOP is too high
<2> ORG of binary following header
<2> Optional length of binary following header (0 = load whole thing)
<1> Flags: bit 0 = 1 to pass command line, bit 1 = 1 to pass file handle

If such a header does not exist (signature is not there) then just load at 32768 and execute.
The UnoDOS way is to stack any parameters following the program name at the top of RAM and leave it up to the app as to what to do with them. This is because SP might be overwritten by the binary so it's not guaranteed to be able to retrieve parameters from the stack.
I think an optional header would be useful. I'd suggest using $C7, $41, $50, $50 as the signature and using the RIFF format (no plain binary should start by doing a RST 0). The header should be:

dwMagic
Byte sequence of $C7, 'A', 'P', 'P' to identify the file as an app.

chMajorVersion
Major version number of the file format. Currently 1.

chMinorVersion
Minor version number of the file format. Currently 0.

chMachineId
The model of ZX Spectrum (or clone) the app is designed for (same as SZX format)

dwStart
32-bit start address (enables starting in a paged RAM chunk)

chFlags
any flags

The rest of the file can contain memory chunks and other settings in the same format as an SZX file with additional chunks added as required.

But as I say, it's all moot because Jim will do his own thing. Too bad.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 14 Ago 2017, 08:22

Actually scratch that. The app format should either be a headerless binary that runs at $8000 or an SZX file. That solves the issue of needing main RAM to set up all the registers and so on. New chunks can be added to SZX as required.

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 14 Ago 2017, 23:57

aowen escribió: Well that seems to be off the cards now as I gather Jim is writing something and will presumably do it his own way. So that will be yet another standard the Next has broken.
Although he seems to be writing some sort of program launcher, it will be much less useful than what is being discussed here so that part will not take off. However there is a second part where he is trying to define a single-file format for containing Next programs that will be automatically loaded. If it happens, you can just add another case for loading it or you can load a loader that does it. I personally prefer leaving the loading to the programmer so the task is just getting that loader into memory and running it but he wants to go a bit further and tools like z88dk will go further and your szx format is another one that goes further.
The UnoDOS way is to stack any parameters following the program name at the top of RAM and leave it up to the app as to what to do with them. This is because SP might be overwritten by the binary so it's not guaranteed to be able to retrieve parameters from the stack.
This is part of the reason I want to include a max ramtop value everywhere. With an allowed ramtop, you now have space below ramtop to hold the stack (set by basic), command line (I'm now thinking the command line should include the full path to the launched program so that no open handles are passed) or even code/data if something can't be done from 0x2000. With a defined ramtop variable, you're guaranteed that at least the loader part of the program can start without interfering with these things. Then the loader can ignore or copy things to where they should be.

The other part is you want to be able to load programs that can run and then exit to basic. A ramtop check will ensure that they don't interfere with basic.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 15 Ago 2017, 11:49

Alcoholics Anonymous escribió:Although he seems to be writing some sort of program launcher, it will be much less useful than what is being discussed here so that part will not take off. However there is a second part where he is trying to define a single-file format for containing Next programs that will be automatically loaded. If it happens, you can just add another case for loading it or you can load a loader that does it. I personally prefer leaving the loading to the programmer so the task is just getting that loader into memory and running it but he wants to go a bit further and tools like z88dk will go further and your szx format is another one that goes further.
Well I can't take any credit for the SZX format. But it does strike me that using RIFF is a sensible way of packing data into a single file and it should be possible to request additional blocks to be added to the SZX specification (or if necessary create a superset of SZX with additional blocks). It has the added advantage that the apps can be loaded by emulators that support the SZX format (as they will just ignore blocks they don't know about).
This is part of the reason I want to include a max ramtop value everywhere. With an allowed ramtop, you now have space below ramtop to hold the stack (set by basic), command line (I'm now thinking the command line should include the full path to the launched program so that no open handles are passed) or even code/data if something can't be done from 0x2000. With a defined ramtop variable, you're guaranteed that at least the loader part of the program can start without interfering with these things. Then the loader can ignore or copy things to where they should be.
I'm starting to think it might be simpler to just write out the command parameters to a temporary file in the application folder. The problem with keeping the existing RAMTOP is that either your program has to be relocatable or you're just going to quit and tell the user to clear the required RAMTOP. Not very user friendly. It would be better to force the correct RAMTOP but all my attempts to invoke the CLEAR command from code have failed thus far. I'm sure it's doable, but it's going to take more than a simple ROM call.
The other part is you want to be able to load programs that can run and then exit to basic. A ramtop check will ensure that they don't interfere with basic.
UnoDOS apps currently do this. The hardest bit is unraveling the stack after the MMC paging. Realistically, programs that return to BASIC are probably going to need to be relocatable and should be loaded above the current CLEAR location.

I think for now I'm just going to do a snapload command that supports SZX as a first step.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 13 Oct 2017, 19:38

Seems all this is moot because esxDOS support is being dropped from the Next.

Alcoholics Anonymous
Mensajes: 10
Registrado: 14 Nov 2016, 04:53

Re: RFC: Application Package Format

Mensaje por Alcoholics Anonymous » 19 Oct 2017, 07:54

NextOS supports the esxdos api and dot commands so that's not going away.

Because of the mmu, there is likely going to be an allocation map to show which 8k ram pages are available, which will mean loading programs into a scattered memory map could be a thing along with really big dot commands and terminate-and-stay-resident programs.

Avatar de Usuario
aowen
Mensajes: 178
Registrado: 07 Oct 2015, 13:32

Re: RFC: Application Package Format

Mensaje por aowen » 21 Oct 2017, 15:14

Alcoholics Anonymous escribió:NextOS supports the esxdos api and dot commands so that's not going away.

Because of the mmu, there is likely going to be an allocation map to show which 8k ram pages are available, which will mean loading programs into a scattered memory map could be a thing along with really big dot commands and terminate-and-stay-resident programs.
Now that the FrankenSpec is basically an extended +3 that imposes all sorts of additional restrictions. I'm not even sure how Garry is going to get +3DOS to use the MMC ROM page given that the only mechanism for paging the thing in is via traps.

Most of the existing dot commands won't work because they are not API clean. Also they are a very clumsy mechanism if you can actually add proper OS commands instead. And I presume the API compatibility will be with the private 0.8.7 API rather than the 0.9 release that was supposed to come out in January.

I'm not going to worry about maintaining compatibility with a short-run machine with specifications that are a moving target.

Responder