========================================================================== 2/21/89 ========================================================================== This file contains a review of PKZIP v0.90 and some comments on the implications of this new program for BBS systems. It originated on the Computer Connections PCBoard in Washington, DC. ========================================================================== ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» ÍÍÍ͹ Bulletin 1 -- Comments on New Files ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Computer Connections PCBoard, Washington, D.C. Robert Blacher, Sysop (202) 547-2008 -- Public (free) (202) 547-7621 & (202) 543-8088 -- Limited access (supporters) ==> Note: Anyone may also call (202) 547-3037 to upload a file. That is the only activity permitted on this 3rd number so as to make it easy to get through. ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ Files marked with an L* are limited in some manner and are less useful than the versions sent to those who pay for the program. ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ ÿ ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» ÿÍÍÍ͹ February 21: This BBS will adopt the ZIP file format ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ ÿ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ First, I want to thank Phil Katz for taking the time to reply to the various questions that I asked. I raised, in essence, 3 issues: (1) Did PKWARE claim any property interest in the compression techniques used in the ZIP programs? Phil's answer quite fairly points out that the implementing code is copyrighted, but he also makes clear that other parties can create ZIP-compatible (and ZIP competitive) programs without needing to seek any license from PKWARE. I find his answer fully satisfactory. (2) Because I had read that some of the algorithms used in producing reduced files were licensed from Graeme McRae (author of SCRNCH), it was also important to determine whether any other party could require a ZIP utility author to obtain a license. Phil's answer is clear and unequivocal -- no. I have received similar information from Graeme McRae in separate correspondence. (3) Finally, there is the question of documenting ZIP. Thus far, we have APPNOTE.TXT, part of the v0.90 distribution package. The format of a ZIP file is clearly documented. So, too, is the method of extracting a REDUCED file. The method of extracting a SHRUNK file is somewhat less clearly documented, but adequately so. What is least clearly documented is the process of creating a SHRUNK file and, even more so, the method for building REDUCED files. I am told by people whose judgment I trust that they will eventually be able to figure out how to shrink files. But, those same people indicate strongly that there simply is not enough information in the current application notes to figure out how to REDUCE a file, and that's *very* troubling. Phil indicates a willingness to provide additional information as needed by other authors seeking to write compatible programs. And, in fact, he has been helpful to Richard Byrne, author of PCBoard's Zdoor program, in resolving some issues about how to extract a SHRUNK file. It would certainly be preferable if Phil would, as time goes on and the same questions are raised again and again, release a more detailed set of application notes. But, the current situation, while by no means ideal for a "standard" to be adopted by BBS's, is tolerable (and, it's early -- getting v0.90 out the door was no small task. Hopefully, as the program matures, so too will the application notes). Where does that leave us? With perfectly satisfactory answers to the first two issues I raised and a not entirely satisfactory answer to the third, ZIP must then be looked at in comparison to similar programs. Compared to the current BBS standard, the ARChive standard, ZIP is *much* less encumbered by legal pitfalls and problems. Compared to POLICY.SEA, the official statement by THom Henderson et al. on creating ARChive compatible utilities, ZIP is clearly to be preferred in terms of legal issues. The ZIP file format and extension are in the public domain and neither the author nor any other party is claiming any property interest in the compression algorithms. That's makes ZIP a hands down winner compared to SEA's policy on ARC, which is a confusing thicket of copyright, trademark and other issues that inhibits development of third-party programs. And, even on the issue of documenting the standard, ARC is not necessarily to be preferred. The only real documentation for ARChives was the source code itself which SEA did release. But, in light of SEA's current policies, anyone who uses that source code for anything is opening themselves up to potential legal claims. If looking at the source code is out of bounds, than ARC is not documented at all by SEA (and, presumably, all documentation of the format by others is tainted by having been derived from SEA's source). That brings us to performance and reliability. I have been testing the ZIP programs extensively since their first official release. There is little question that, with the REDUCED algorithms turned on, ZIP can produce substantial savings in file sizes. That saves disk space for Sysops and download time for callers. It does take longer to produce a ZIP file in this manner but that's a one-time cost, more than offset by the amount of time it will save callers when downloading the files. Extraction of ZIP files is very fast, regardless of the compression method used. I have also seen no reliability problems thus far. In short, the program is more efficient than ARC in terms of file sizes, will be cost-effective for BBS use, and is a clear winner here as well. Finally, ZIP, like ARC, is shareware. The licensing terms are similar (free for noncommercial use with a contribution requested) but, in fact, ZIP's license in general is more liberal and the demands for payment less harsh. The difference here is really one of tone. As I've mentioned elsewhere, I hope if you decide to use the PKWARE programs you will make the requested contribution. It's rare that we see this kind of shareware license these days and it ought to be encouraged. Bottom line: This BBS will use the ZIP format. As the various PCBoard utilities and DOORs are updated, eventually all files on this BBS will be converted to ZIP format. All new files posted will be converted by me to ZIP format if they are uploaded as ARChives. So, for a little while, both file types will be on the system, but I expect the ARChive format to disappear entirely from this BBS, and most others, over time. Until somebody builds a better mousetrap, the ZIP format will be the standard on this BBS. I'd like to thank all of you who participated in the discussion of this issue here and kept the tone at a high level. While the world wouldn't stop turning regardless of the outcome of this issue, it is important for BBS's to make this decisions carefully. I believe we have done that and hope the future bears that out. What follows, for the sake of context, is my original set of questions to Phil Katz, his answers, and correspondence with Graeme McRae. Also included is my original review of PKZIP v0.90 written when the program was first released. ==================================== Questions from me to Phil Katz 2/17/89 ==================================== (1) Do you claim that any of the compression techniques used in PKZIP are copyrighted, trade secrets or otherwise proprietary? (2) Assuming the answer to the first question is no, are any of the compression techniques used in PKZIP licensed from other parties? If so, can those parties claim property interests in the compression methods and require licenses from others who wish to use these techniques? (3) Finally, APPNOTES.TXT, included in the PKZIP v0.90 distribution package, may not give sufficient information to allow third parties to extract files from within ZIPs and, in particular, does not appear to provide sufficient information to allow third parties to build ZIP archives using all of the same compression options and levels employed in your own program. Are you committed to providing more detailed information about the compression algorithms, including implementation techniques and suggestions, and, if so, when will that additional information be available? ================================ Phil Katz's reply 2/19/89 ================================ Those are good questions. First, PKWARE does not consider the algorithms used in the software to be propietary in the sense that we would not allow someone else to implement them or would require that someone who did write a program using those algorithms to be licensed from PKWARE. However, the implementations of those algorithms as contained in PKZIP and PKUNZIP and PKSFX (R) are copyrighted. If someone were to lift the machine code out of PKZIP, that would be a problem. In answer to the second question, no requirements or obligations are required to third parties by users of the software or the information contained in the application notes. Lastly, several people have informed my that they have written or are writing programs to extract or compress files using the information in the application notes for the software. Therefore, it appears that for someone who is willing to think on there own that the information provided is sufficient. While I don't want to write people's source code for them, I am willing to offer assistance or information above what is in the docs on a reasonable basis if someone asks. FOr example, I have had several conversations with Mr. Burns [sic -- he means Richard Byrne], author of ZDOOR for PCBoard systems. He has successfully written code to Expand (UnReduce) a file and was having a few minor problems with UnSquishing, and I was happy to answer his questions and offer some suggestions as to what the cause of his problems could be. >Phil> ======================================================= Correspondence with Graeme McRae, author of SCRNCH ======================================================= #: 17060 S6/File Utilities [S] 19-Feb-89 15:45:14 Sb: #16924-New PKZ/2ZIP Fm: Bob Blacher 72677,3305 To: Graeme W. McRae 73307,2453 Graeme -- There was mention in the PKZIP software developer's toolkit that Phil Katz had licensed from you some of the techniques used by PKZIP to produce "reduced" files. If that's true, 2 questions: (1) Would any other author who wanted to produce a ZIP-compatible program (including a program that competed directly with PKZIP by producing ZIP files) be obliged to come to you for a license as well? (2) Are either you or Phil Katz planning to release additional information, beyond what is in APPNOTES.TXT, on how to "reduce" files in a manner compatible with PKZIP (how to extract reduced files seems adequately documented)? Thanks. =============================================================== #: 17346 S6/File Utilities [S] 20-Feb-89 17:32:18 Sb: #17060-New PKZ/2ZIP Fm: Graeme W. McRae 73307,2453 To: Bob Blacher 72677,3305 Bob, My agreement with Phil is that he has my permission to use programming techniques that I developed and used in SCRNCH, and he may disclose any of these techniques necessary to document the ZIP file format. The ZIP file format is in the public domain, so no one may collect "license fees" for its use. I have no connection with the ZIP program, so I won't be releasing any information about it. I don't know what Phil's plans are. Sorry I don't have very much info for you. --Graeme =============================================================== ÿ ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» ÿÍÍÍ͹ February 15: First thoughts on PK ZIP utils v0.90 ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ ÿ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Phil Katz has released the first, official version of his new storage & compression utils (a leaked beta had been widely circulated some time ago but was not posted for download here). As with Phil's prior archive utils, there is much to be impressed about in these programs, particularly in efficiency and speed. By default, PKZIP creates files that are slightly larger than PK's arc utils (more information about each file is stored and that's the primary reason for the slight increase in size). But, you can turn on additional compression, at the cost of some speed, and then PKZIP seems to produce consistently smaller files than PKARC/PKPAK did. The savings can range from trivial to quite significant. There are also some very clever and useful extensions to the command set of the arc utils. PKZIP can recursively store files preserving their subdirectory structure, print files directly from within a *.ZIP file, and more. The hooks are also there for support of some other operating systems. All in all, it's very impressive for a first release (particularly a v0.90 release). The programs are, by my definition, true shareware. Noncommercial use (whatever that means exactly) is free, with a $25 contribution requested. $47 brings a copy of the next version of the program on disk, along with a printed manual. In this day and age, those are very liberal shareware terms, and Phil is to be commended for releasing the product in this manner. If you do choose to use these programs, I hope you'll consider supporting them financially to encourage this type of "old-style" shareware approach. So, is this BBS going to convert all its files to ZIP format? The answer at the moment is no. First, although the ZIP file format and file extension have been "dedicated" to the public domain, the status of the compression algorithms used in PKZIP remains unclear. There is some documentation of these new algorithms (actually, variations on well-known LZW compression) within the distribution package, but their exact legal status is *not* clearly spelled out. That MUST be clarified. Furthermore, it is unclear, to me at least, whether enough information has been provided about these algorithms to allow other authors to create compatible programs that can extract files from within ZIPs and, indeed, build compatible ZIP files. If that information is not all currently there, then it will be a question of how cooperative Phil is with 3rd party developers in providing the necessary information. If someone wants to develop a product that competes with Phil's ZIP package directly (as Phil's ARC programs competed with SEA), what is Phil's position with regard to that? In short, I'm not willing to jump out of the frying pan into the fire. I have very serious concerns about what SEA did with regard to ARChives. I'm perfectly willing to change away from that format. But, until the questions I've raised above are answered, there's no guarantee that ZIP won't ultimately lead us down the same road. Finally, there are some practical concerns. I am not going to convert over 200 meg of files on this BBS to ZIP format until I am *VERY* sure that this program always produces valid files. This is a first, preliminary release and needs thorough testing. Second, this BBS has several DOORs available that process ARChive files, including allowing you to read files in ARCs and extract some members for download. Mail is also arc'd for you if you wish for download. For ProDoor, supporting the new ZIP format is relatively trivial as it already invokes Katz's archive utils for most of its processing. But, Zdoor does *all* of its ARC processing internally. That's a more efficient and safe process and saves a great deal of wear and tear on this BBS. The author of Zdoor has indicated a willingness to support ZIP files if sufficient information is provided about the compression algorithms used. As I said above, that's an open question at the moment. In sum, until the legal questions I've raised above can be resolved, until we see whether Phil has provided, or will provide, sufficient information so that compatible programs can be created, and until the ZIP program itself has been thoroughly tested, any consideration of converting all files on this BBS to ZIP format is premature. If you upload a file in ZIP format, I'll simply convert it to an ARC. I re-ARC all uploads anyway so that's really no bother for me (it's just one more batch file to run during maintenance). No rush, folks. Let's sort this out carefully so that we don't repeat the ARC fiasco. ZIP may be a viable alternative, but it's just too soon to say that with certainty.