|
|
Post by spiff on Oct 13, 2008 17:19:03 GMT -5
Hello all. It has been more than a year (about a year and a half I think) since my last update of dtvmkfs, so I thought it was about time for a new version. There have been a few things that people have requested over and over again, and others that I have been wanting to put in myself. The two features that I think are most commonly requested, and which are now implemented are: - Import PRG-files directly, instead of needing to dtvpack first.
- Import files from the command line, instead of needing to put them in a ZIP.
There are a number of other changes, and I have done some testing of most functions. Please try it out, and if you find any bugs, I would like to hear about it. The latest version is available from: symlink.dk/nostalgia/dtv/dtvmkfs/Next up is finishing the new repository system and including all the files that I have received, but haven't gotten around to including yet. |
|
|
|
Post by nojoopa on Oct 14, 2008 14:50:57 GMT -5
I have done a few tests, no bugs seen so far. A handy fix that I didn't see mentioned is that now dtvmkfs actually exits if there's an invalid filename if the "@" list; would have saved some head stratching a few days ago  My feature request list (once again): - Import $1f8000-$1fffff from image (as an option)
...so one could use: dtvmkfs -Xki old.bin @list...instead of something like: dtvmkfs -ki old.bin @list && { dd if=flashfs.bin bs=32k count=63; dd if=old.bin bs=32k skip=63; } > new.binOther than that, dtvmkfs has everything I need. Great job!  |
|
|
|
Post by spiff on Oct 15, 2008 2:28:22 GMT -5
I have done a few tests, no bugs seen so far. Thanks. A handy fix that I didn't see mentioned is that now dtvmkfs actually exits if there's an invalid filename if the "@" list; would have saved some head stratching a few days ago OK, I guess I wasn't even aware of that. The input processing has been rewritten, because now the input lines (read from stdin, from @-file or directly from the parameters) are processed in the same way as the lines in index.txt in the archives. Import $1f8000-$1fffff from image (as an option) Sounds like a usable option. I have never done anything that requires fixed addresses, and I am not really sure if there should be special regions to reserve, or if it should be done in a more generic fashion. What you describe is importing from the binary image, but what I had thought of doing was to have en option on files to include them at fixed loactions. In other words something like: kernel.bin,,,RAW,FLASHADDR=$1F8000 The option RAW already exists (or the file can be named with a .raw-extension), and loading at a fixed position should not be difficult. My main problem here is making sure that files included in a fixed position are not overwritten by other imported data. Also, for this option it might be handy with a new option to not make an entry in the $10000-directory. I guess this could be the default if no C64name is specified and FLASHADDR is set. |
|
|
|
Post by 1570 on Oct 15, 2008 5:53:28 GMT -5
Also, for this option it might be handy with a new option to not make an entry in the $10000-directory. I guess this could be the default if no C64name is specified and FLASHADDR is set. ...adding this hack to the "do not add entry to $ file if C64name starts with $" hack? I propose just a new option "NODIRFILEENTRY" and perhaps getting rid of the "$C64name" hack which is ugly and counterintuitive (I'd have thought that if I add a "$" this indicates that a $ file entry should be created). BTW what is the encoding of C64name? How to encode commas or CBM special characters? |
|
|
|
Post by spiff on Oct 15, 2008 7:19:10 GMT -5
BTW what is the encoding of C64name? How to encode commas or CBM special characters? I have not given the format a lot of thought. Originally I wanted the format of the index.txt to be compatible with what Hideki's tool used for input. This is where the $ in the C64 filename came from. This also means there is no encoding of the C64 filenames, which means that commas cannot be used. AFAIR (I'm at work now) the only conversion happens when no c64name is given, in which case it converts the filename to a c64name by changing lowercase letters to uppercase and stripping known extensions. This also means that comma cannot be used in file names. One way to work aroun this is to enclose the C64name in quotes if it contains a comma. This could fairly easily be done, and would not break compatibility because the filename may not contain quote anyway. I don't know how many persons originally used Hideki's tool, but I don't hear about it anymore, and I don't think I has been updated since long before I released the first version of dtvmkfs, so I guess the backwards compatibility can be dropped in this respect. But now with your graphical tool, to do much of the work on archive files, we need to be compatible if we add features. And as such I would want to have the archive backward compatible as long as possible, to avoid problems with people using other software or older versions. There are probably a lot of things people have packed up in archives that work with the current version of dtvmkfs. If they download a new version I think they would expect it to work with that too. Of course I may be wrong in this respect  It appears that my decision to use the format from Hideki's tool has come back to haunt me, but I think there would have been a problem getting people using the repository and dtvmkfs if it was in a new format back when it all started. As I see it, we have a number of limitations with the current format. Yet changing the format will just lead to another format, that may solve some problems, but then there is yet another format to deal with. A limitation of the ZIP-files is the needed meta-information to bring it all together. The load and sys addresses and the C64 name are in index.txt for this reason. This also has limitations, and hopefully we can find a solution. Any suggestion for encoding the C64 filenames? I think the current repository has all C64names specified in only uppercase letters, so I guess we could use lowercase for special characters, but really there is no easy way to do this properly. When I created dtvmkfs I wanted an easy way to make a custom filesystem. With this and the repository I figured other people would benefit from the work of me and many others, and allow them to make their own custom images. This also offloaded me in terms of finding all the games that had been patched and converting them to a format usable by dtvmkfs. I have no problem continuing development of features that people would like, but I would not like to split the userbase or community. I think both of those are small already. |
|
|
|
Post by 1570 on Oct 15, 2008 11:45:49 GMT -5
While I'm especially unconvinced with the leading-$ hack, I second the motion of keeping compatibility with existing repository files.
Concerning the index.txt charset encoding, I suggest to escape the C64 name in URLEncoding/PETSCII. All characters with charToIsoValue(char)!=charToUnshiftedPETSCIIValue(char) as well as "," and "%" must be escaped. "GAME,THE" becomes "GAME%2CTHE" therefore. Using quotes is no option as these can occur in filenames (PETSCII #$A0 is filename end marker for the 1541; PETSCII #$00 is filename end marker for the DTV flash).
Concerning putting data to fixed positions in flash, I favor nojoopa's idea of just importing $1f8000... from an existing image as a program option instead of the FLASHADDR=... approach.
The one use case this is needed for (adding the TRSI kernal) needs also a patched $00e000 kernal. Creating a ZIP containing both $00e000 and $1f8000 kernal and flashing the resulting image using DTVTrans+ will not flash $00xxxx and will confuse users. Also, for patching kernals you should use kernalpatcher and not some existing image downloaded somewhere (think about modded NTSC DTV2s and Hummers).
From the DTVFSEdit point of view, while handling FLASHADDR= would be possible, it's a break in the design. Think about adding such a ZIP to an image; no new file will appear (?). Exporting the image as Spiff ZIP will lose that file (?). No good user experience. Instead, just importing images containing the TRSI kernal works already.
|
|
|
|
Post by spiff on Oct 15, 2008 12:26:04 GMT -5
While I'm especially unconvinced with the leading-$ hack, I second the motion of keeping compatibility with existing repository files. OK, in that case, maybe we could agree to add an option (e.g. ,NOLIST) to the "parameters" on the end of the lines in index.txt. (Yes, I have adopted your notion of ,PRG). Concerning the index.txt charset encoding, I suggest to escape the C64 name in URLEncoding/PETSCII. All characters with charToIsoValue(char)!=charToUnshiftedPETSCIIValue(char) as well as "," and "%" must be escaped. "GAME,THE" becomes "GAME%2CTHE" therefore. Using quotes is no option as these can occur in filenames (PETSCII #$A0 is filename end marker for the 1541; PETSCII #$00 is filename end marker for the DTV flash). Well, you are probably right (as usual) concerning the end of file markers. But at least in the $-file it does not make much sense to have quotes. How does BASIC parse the filenames if they contain quotes? That said, I do like your idea better. It will take a little extra code for the string processing, but really this has not been a concern for me (in fact I have been lazy and not even bothered to free the memory used for strings - such are the wonders of C-programming). Now, which characters map directly, and which fail your "test" above. Concerning putting data to fixed positions in flash, I favor nojoopa's idea of just importing $1f8000... from an existing image as a program option instead of the FLASHADDR=... approach. The one use case this is needed for (adding the TRSI kernal) needs also a patched $00e000 kernal. Creating a ZIP containing both $00e000 and $1f8000 kernal and flashing the resulting image using DTVTrans+ will not flash $00xxxx and will confuse users. Also, for patching kernals you should use kernalpatcher and not some existing image downloaded somewhere (think about modded NTSC DTV2s and Hummers). From the DTVFSEdit point of view, while handling FLASHADDR= would be possible, it's a break in the design. Think about adding such a ZIP to an image; no new file will appear (?). Exporting the image as Spiff ZIP will lose that file (?). No good user experience. Instead, just importing images containing the TRSI kernal works already. As long as everyone is happy with it, this is fine with me. In fact this will be easier to implement. I just thought there was also something about a DTVMON on a fixed address. Specifying an option on the command-line for doing something special should be done when this is indeed something special. Because I started out with the idea of importing the image and adding files to it, I made the kernel import that way. dtvmkfs will always start creation of the flash filesystem from $10000, but an option to only zero-fill (or $FF-fill, i guess) up to a certain address, and setting the conceived flash size smaller would be easy to implement, and allow importing from the top part of the image also, while dtvmkfs will still tell you if there is not enough room. |
|
|
|
Post by 1570 on Oct 15, 2008 17:47:59 GMT -5
But at least in the $-file it does not make much sense to have quotes. How does BASIC parse the filenames if they contain quotes? LIST works but LOAD doesn't of course (without adding a wildcard). There's a number of D64 images using quotes in filenames though, and being able to convert a D64 directly to a Spiff ZIP would be nice. Loading files with quotes in the name in ASM works. $20-$5B plus $5D are okay here (minus $25="%" and $2C=","). BTW in DTVFSEdit I used some Unicode<=>PETSCII mapping I found on the net. Using a real CBM charset for display would be nicer but I was too lazy for adding that (and I wouldn't know how to do that in Java). DTVFSEdit imports existing images and builds a bytes used map. Bytes obviously occupied by files (as indicated in the $010000 directory) are marked as are bytes that are not FF (see DtvFlashFs1.java). dtvmkfs could use the same approach. Then you don't need any special options to exclude DTVMON/TRSI Kernal areas when adding files. |
|
|
|
Post by spiff on Oct 16, 2008 3:28:59 GMT -5
DTVFSEdit imports existing images and builds a bytes used map. Bytes obviously occupied by files (as indicated in the $010000 directory) are marked as are bytes that are not FF (see DtvFlashFs1.java). dtvmkfs could use the same approach. Then you don't need any special options to exclude DTVMON/TRSI Kernal areas when adding files. This is basically how I was thinking of making it work when importing files on a fixed address. A map would be needed for that, since files COULD be included in multiple places. Thinking about it, what I will do is importing the whole filesystem and detecting the free space. Then there could be an option for wiping the flash above $10000, which would give the behavior I have today. I think I need to sit down and look at the options (hopefully sometime during the weekend). |
|
|
|
Post by spiff on Dec 6, 2008 19:34:39 GMT -5
New version 0.91 released. - Handles c64names as per the new repository specification (including %XX-escape).
- Added NOLIST-option.
- Simple import function, e.g. for importing TRSI kernel.
- Fixed a bug with incorrectly parsing the load address for some PRGs.
The load address bug was discovered when I was converting Impossible Mission 2 (multi-part with some weird load addresses) to the new repository format. symlink.dk/nostalgia/dtv/dtvmkfs/ |
|
