The DemOS FAQ

This is the official DemOS FAQ. It was last updated February 2nd 1996.

1. What is The Demo Project?
1.1 General info
1.2 Distribution sites for Palantír
1.3 Submissions
1.4 Related information

2. Background
2.1.1 Why not Dos?
2.1.2 Why not Windows 95?
2.1.3 Why not Linux?
2.2 The thought about a new OS
2.3 The initiative

3. About the OS
3.1 Hardware requirements
3.2 The kernel
3.3 I/O
3.4 Multitasking
3.5 Video support
3.6 Sound support
3.7.1 The file system
3.7.2 The file system specs
3.8 Shells
3.9 Programming aids
3.10 The boot process
3.11 Plug and Play??

4. Inside the project
4.1 Crew
4.2 Project groups

5. Joining the project
5.1 Who do we need?
5.1.1 Graphician needed
5.2 How do I join?
5.3 What privilegiums do I get by joining?
5.4 What should my development system be like when I can't use DemOS right now?
5.5 What's the projects current status?

6. Misc.
6.1 Music
6.2 Graphics

7. Creating demos and programs for DemOS
7.1 Is all my prior programming experience wasted?
7.2 Programming concepts


What is The DemOS Project?

General Info

Palantír is a newsletter which will be released on monthly basis. Its purpose is to keep you up to date about The DemOS Project.

The DemOS will be a new Operating System which has only one main purpose: To run demos at the best performance possible at your home computer. As much of the computer resources as possible are laid at the coders feet. The only limit is your computer and your imagination.

Distribution Sites

Palant¡r is distributed at the following sites, groups and BBS'es:

Bulletin Board Systems:


Norway

Portugal

Switzerland

France

U.S.A.

It will be distributed through ftp.cdrom.com within a few weeks.

If you'd like to leave an article about your view on The DemOS Project, about a feature you'd like to have included or (if you're inside the project) about how thing's are going, what's new or anything at all which might be of interest of the readers of Palant¡r, please send it to beren@infolink.no, and we'll include it in our next issue. Until then!

Related Information

[subscription information will be released at later occasion]

Background

Why not DOS?

Why not DOS? The reasons are many. First of all, DOS uses BIOS calls for whatever it needs to be done. This slows down the overall performance. Secondly, DOS works with 16-bits programs. 16 bits? That's 286 talk!!! We want our demos to be pure 32-bits, don't we? Most scene dudes use a 486 DX33 or better anyway. No need to stay behind! Okay, many demos nowadays tend to use DOS Extenders, but this isn't really as good as it ought to be. Especially when you have memory managers going in the background which already have put the computer in some crazy mode. Thirdly, DOS uses FAT on both floppies and harddrives. Do you realise how much space this wastes? And FAT gives you the 8+3 filename headache! No, give me a decent file system!

Why not Windows '95?

Why not Windows 95? First of all, Windows 95 lays on top of DOS. They had to change DOS somewhat to make it VFAT compatible and let them take more control of your machine, a control which Windows 95 doesn't give you back. Think of Windows 95 as a bad kind of Bilbo and the democoder is Gollum (the character, not the group) 'Thief, thief! We hates it!' ;-) Secondly, the VFAT system is a rather poor system. Ever tried to copy VFAT files with a program that does only know FAT? There you've got a fucked up system! Thirdly, as I said, Windows 95 doesn't give you back the control, WHAT SO EVER! When you want to use a graphical routine, you'll have to go to some DLL and ask it to be so kind and fulfil your wish. Then it goes and checks if it's OK by the driver, the system and every other application. If nothing sais no, it gives in and grants you your wish. But by the time it sais OK, you've already lost three frames if you've got a rather quick PC.

Why not Linux? Hey, I'm sick and tired of all these why nots. Finrod, take this one. I've got some Coke to drink! :-)

Why not Linux?

Well, once when I was about half the height of this gun, my dad took me on his lap and he said, son, one day you're gonna be a coder. And you're gonna make demos. And son, I want you to know that Linux is not an OS to write demos in! So promise me that you'll never, ever write a demo under Linux. And as the obedient son that I was (but that was way back. Now I'm not quite sure what "obedient" means, it just *looks* good if you know what I mean.) So anyway, Dave - sorry, *Jimmy* - Letterman's top ten reasons not to use Linux for demos:

The thought about a new OS

This is not at all a new idea. The Amiga used to boot from floppies. Most games ran without OS. This was of course to use the computer best for the games needs. In comp.sys.ibm.pc.demos, most people agreed that it was a good idea to have an OS to run demos under, but that was it. And I was to impatient to wait for an OS. If every single group was to make their own OS'es, the 'f', 'd', 'i', 's' and 'k' keys on my computer would quickly wear out. What we needed was a standard OS for The Scene, and to make it a standard, we'd have to make people use it. But for everyone to like the OS, we'd have to let everyone work on it... And me, as a musician with poor system programming experience [make that poor programming experience at all ;) -Finrod] [Hey! I know enough to survive, all right? -Beren] [Nah, just kidding -Finrod], would have to wait endlessly for this OS to come out, IF anybody would do anything about it, that is. OR, I could host a co-operation myself. So I did.

The initiative

Some lousy afternoon the phone rings at Finrod's. With much effort he lets go of the computer for long enough to answer the phone. It's that wierd guy Beren again. Yes, he has read comp.sys.ibm.pc.demos lateley, and yes he has read the dreams about an operating system for demos and yes no one is taking any initiative at all. Beren had not long ago asked if anyone was doing anything about it. One guy had actually begun working on such an operating system. But no one had considered the thought of co-operation with other demogroups over the net as anything but ridiculous. 'Someone's got to take an initiative,' Beren complains to Finrod. 'How would you like us to host such a project?'


About the OS

Hardware requirements

DemOS will run on any 386 or better computer, with 1 MB RAM or more, and a few megs of free harddisk space. However the minimum realistic configuration for a DemOS system is a 486/33 (DX or SX does not matter as long as no application you wish to run needs floating point) with 4 MB RAM and enough harddisk space ("enough" simply because we have no idea yet how big the OS will be. However, the OS itself, complete with microkernel, kernel modules and command interpreter, should require less than 5 megs. Such a stripped- down configuration is more than adequate if you plan to just *watch* demos. If you want to use DemOS for "serious" work, however, such as coding, tracking, drawing, or whatever, you'll need a lot more.)

Most of you are probably thinking "Yeah, right, 486/33/4, that's what IBM said about OS/2, and look at the result." However we have very good reasons to believe that these figures are realistic. Amongst other reasons, DemOS (in its initial configuration) has no fancy GUI, nor tons of useless utils (who needs ramdrive, fastopen, nlsfunc, dosshell, msbackup, or qbasic anyway?). Basically, if it can run DOS at a satisfying speed, it can run DemOS even faster.

The Kernel

We are planning to base DemOS on a microkernel architecture with most of the OS functionality lying in external modules. All modules will comply to a standard API, with standard APIs for each subcategory. This simply means that all modules will behave the same for common functions such as driver identification, version number, etc., and that all modules in a given category (for instance, all sound drivers, all file system drivers, etc.) will behave in a similar fashion. Provisions will be made to allow partial or extended functionality - for instance, AdLib drivers will be allowed not to implement recording :) and Matrox Millenium drivers will be allowed to add 3D functionality (or maybe we'll define an API for 3D functionality and allow VGA drivers to ignore it)

I/O

I/O ports will not be privileged in DemOS. To do so would be pointless as it would significantly slow down anything that uses them, i.e. most - if not all - demos, intros and games. However it is possible that we provide a mechanism to lock specific I/O ports - for instance the disk subsystem might lock the SCSI adapter's I/O ports to prevent accidental data loss. However only privileged processes (i.e. OS modules) will be allowed to lock I/O ports. Thus we simply set the IOPL to 0 and give everybody the same I/O permission bitmap, and avoid the hassle of setting one up for each task.

Multitasking

We do not plan to implement multitasking from the beginning. However we mean to design the OS and especially the API in such a way that multi- tasking can later be added with no programmatic differences (i.e. old programs will run just as well with multitasking).

Video support

A standard API for video cards will be defined, and DemOS will ship will drivers for VGA and probably a few accelerators. Forget VESA, it's far too slow. I don't know how the coder (or coders) in charge of the video subsystem will handle this, but quite possibly they will define a list of standard mode numbers. Minimal SVGA support should be easy to achieve for most adapters; the only really critical functions that need to be implemented for each card are identification, mode initialisation and status report.

Needless to say DemOS will make provisions for linear addressing. The idea so far is to map the linear video buffer at a fixed position in each application's address space, and provide an OS call which will return the offset to that location. With some work it is possible, through paging, to simulate linear addressing even on cards that do not support it.

Sound support

DemOS will ship with a set of sound support modules. These drivers are meant for coders who don't want to devote all their time and effort to implementing support for whichever sound board they feel like but rather to the demo (or intro or whatever) itself. Note that the use of these drivers will NOT be compulsory. Coders are free to write their own sound routines rather than use the DemOS sound API - or if they wish, to write and distribute their own drivers.

The file system

DemOS will use a dual file system. The microkernel contains code for a proprietary file system loaded at boot time; this is the only file system allowed on the boot partition because of the kernel's need to easily locate and load dynamic link OS modules. Additional file system drivers may be used by DemOS to mount other partitions, such as FAT12, FAT16, VFAT, EXT2FS or HPFS partitions. Apart from FAT16, and most probably FAT12, it is not yet clear which file systems will be supported at release time.

The additional file systems will be implemented as external modules which can easily be exchanged. Specs for the file system API are in section 3.7.2 to allow programmers to design and/or implement their own file systems.

The file system specs

The design of the file system has been completed, except for a couple of implementation details. The design will provide a fast and efficient way to access files, and data. This is done, by using a combination of addressing schemes. No, FAT, is not one of them! In fact the design more resembles UNIX - FS meets OS/2 - HPFS. The directory structure will consist of BTrees. Since BTrees are fast, efficient and lend themselves very well for what was needed, they were the obvious choice. For those of you that do not know what a BTree is, it is basically much like a binary tree, except that instead of having only two branchs it has multiple branchs. Because of the complexity of BTrees, it is impossible to implement a DOS-like deletion. So, to provide a means to undelete items, some other method has to be used. It has been mentioned to use a trashcan sort of method, where items are not immediately deleted, but instead kept in a trashcan list, and are only raped when needed or after a specific amount of time has passed. I would like to here comments on this, so if you do have a comment, please feel free to write me.

Speaking of comments I have another item that I would like peoples opinion on. This has to do with how you want different partitions to be handled. Do you want drive letters (as per DOS), or would you prefer having more of a UNIX like method of handling multiple partitions. Again, for those that are not familiar with the UNIX method, it basically makes all partitions, and physical drives look like one big directory tree, since drives are mounted into a particular directory. This directory is not special in any way, and does not appear any different to the user than any other directory. Either method can be implemented into the file system, but I would like to hear what you (the potential user) has to say!

Now onto some boring numbers about the address space of the file system. There essentially is no limit as to how big a partition, or a particular file can be. The only limit there is, is because of the use of 32 bit pointers for block addresses. This gives a 2^32 possible block addresses. If the block size is 4096 bytes then you can address a total of:


2^32 * 4096 = 1.759218e+13 bytes or 16,777,216 Megs


Using the indexing scheme of addressing blocks, a file can grow up to


9 + 1024 + 1048576 + 1.073741e+09 = 1.074791e+09 blocks


So, again if we used 4096 bytes blocks, the total bytes that can be addressed is:


1.074791e+09 * 4096 = 4.402345e+12 bytes or 4,198,404 Megs


This is only if a file does not happen to be contiguous. If the file is contiguous, then the entire address space can be addressed by any file, using the other addressing scheme, that is within the file system. There are two different addressing schemes because, one is better suited toward disjointed, non-contiguous files, whereas the other is better suited toward linear or contiguous files. To keep track of all used/free blocks the file system will use bitmaps. The bitmaps will be spread out across the disk. Each bitmap will "service" a set of blocks, and will reside in the middle of the blocks that they service. This will reduce the seek time that is needed if a bitmap is not already in a buffer and needs to be read in.

Since this was to be a general overview, I have purposely left out a lot of specifics. A more detailed description will be available later, when some of the open concerns have been resolved. I will note that the implementation has already begun, and is coming along. Again if there are any question or comments, feel free to contact Mage at: bhanks@sunfse.ese.lmsc.lockheed.com

Shells

Whichever you want - even write your own if you like. The shell is just a program like any other (contrary to DOS). You can even specify your favourite word processor (anyone want to port WordPerfect to DemOS?) as a shell if you like.

Programming aids

When the kernel is ready, we will start working on a linker and probably port GCC/GPP to DemOS. The advantage of a GNU C/C++ compiler is that the code can easily be ported from system to system and grants us access to a huge collection of for instance Linux programs which will help us much while the number of programs running under DemOS is still very limited. For instance, porting an editor and an assembler could greatly accelerate development.

The boot process

DemOS (or whatever we'll call it on release) can be booted from floppy diskette as well as from fixed disk. The guy who takes care of most of the booting stuff is called OSLO (OS Loader) and resides in a 7.5K contiguous area right after the boot block. Our little boot-slave :-) is loaded and invoked by the boot sector code on a floppy disk or by the master boot record on a fixed disk (also via the root partition's boot sector...). If installed on a harddisk, OSLO will behave as a boot manager allowing the user to select the partition he wants to boot from. Otherwise it will simply load the kernel. OSLO does not need a separate partition (as does the OS/2 bootmgr) but uses space on the DemOS root partition (just as LILO). OSLO runs in protected mode (I'll use gcc to write it) so that the bootsector code is switching to protected mode before invoking OSLO. The OS Loader will use a v86 task to communicate with BIOS. You don't need to use OSLO as a boot manager (although it will be compatible with Linux, OS/2 and DOS/Win95). It will then behave as if booted from floppy diskette (i.e. loading the kernel). Ok, now to give you a quick overview what has already been done: Boot sector is done, loads correctly and switches to protected mode. The OSLO virtual 8086 interface is almost completed so that OSLO itself should be done in a week or so. Hmm, well, that was it IMHO.. For further questions just ask *me* :-). Uhm btw, I'm developing under Linux so if you've got any questions about using gcc/gas to code for this project, feel free to ask.

Plug and Play??

Right... let's talk about dreams other places. Of course, it wouldn't be that hard to implement, but then again, if you want to use Windows '95, why don't you just do that? Plug and Play would require more time to start the OS, take some of the performance away from the OS and with it the demos, take us a lot of work which no-one would ever thank us for and make if we wanted to do Bill Gates then... well, you get the point! Plug and Play my ass!


Inside the Project

Crew

Since we launched the project officially, several people have joined us, which is quite fortunate since we need people in this project. What we still need is people who want to make specific programs and utilities. That's i.e. WWW-browsers, IRC-clients and stuff like that, it's offliners, music-players, trackers, painting programs and graphical converters, it's editors, it's shells, it's even games. If you've made ANYTHING and want to port it to DemOS, contact us and we'll provide you with any information you'd need.

Project groups

We've divided the entire project into smaller groups. Each group has its leader who is free to divide the group into subgroups at need. The groups operating at present are:

Group (Leader) Who's in the group?
Kernel (Token) Token, Finrod
Graphic Subsystem (Erise)Erise, Inferno, Jestyr, Clef


Joining the project

Who do we need?

Primarily, we need people with expertise in graphics (low-level VGA programming), sound (sound drivers) and disk drives (file system), but all in all any coder who wants to join and feels he has something to bring to the project is welcome to apply. If you feel you don't have the time or the energy to participate in the coding effort, you can join as an alpha tester. In my opinion it is an advantage for alpha testers to participate, even passively, in the development so they have a good knowledge of the system and know what to do and where to look for bugs.

Finally we need beta testers and anyone willing to write programs for DemOS, both to test the OS and to ease our work.

Graphician needed

For this issue of Palant¡r, I got WhiteWater of Inferiors to make the ASCII-logo and ending. However, a small problem arose when he suddenly went off to The Party and I couldn't get him. Well, anyway, for the next issues of Palant¡r, I would need to work with a graphician. You should know how to make ASCII-, ANSI- and normal bitmap-pictures (320x200x256). Please write to beren@infolink.no. I need you real bad!

[PS: We might also need you to make pictures for the OS itself - Finrod]

How do I join?

To join The DemOS Project, write to me, beren@infolink.no, and tell me what you want to work with and I'll send you everything you'll need of information and tell you who you'll be working with.

What privileges do I get by joining?

What should my development system be like when I can't use DemOS right now?

So you want to begin creating utillities for DemOS right now? Great! :-) For a superior development environment, it's reccommended that you use a Unix-clone of some kind. Linux and FreeBSD are good and practically free Unix Operating Systems. For coding purposes, you should use GNU C/C++ (GCC) (preferably GCC 2.7.2pentium) and/or GNU Assembler (GAS)

What's the projects current status?


Misc.

Music

DemOS' drivers will support module-formats and be players as well. By making own drivers for module-formats, new module formats can be added into your favourite module player or tracker just by adding a driver. And by making module players as drivers, we can provide a music system which is as easy as Midas to use for demopurposes. DemOS will as default support 4- & 8-channel MODs, S3Ms, XMs, ITs, MIDs and EarthTracker files.

Graphics

Just like music formats, we will support drivers for graphic format. We will as default support the JPEG and PiNG format.


Creating demos and programs for DemOS

Is all my prior programming experience wasted?

You'd like that, wouldn't ya? Well... No. It's still the x86 architecture, so you ASM-freaks are saved from that trouble. When we have finished the kernel, we'll have to make a linker and a GNU C/C++ compiler. If we don't get enough people who're interested in making a Pascal compiler, it won't be made until somebody else does it. So, if your favourite programming language is Pascal, you'd be safer studying C or ASM, and if you don't care anything about ANSI or GNU compatibility when you're programming C, you've got a bit to learn. But hey, if the OS becomes a breakthrough, these things might come slowly.

Programming concepts

Programm

Coding under DemOS shouldn't be to hard if you already know how to code for the x86 processors in 32-bits mode. While you have to use a DOS Extender in DOS, you can skip that bit of the work with this system. You don't need to bother about memory limits no more, but I would strongly recommend not to exceed 8 MB because not everybody has 16+ MB RAM.