Storage access: files or memory?

This forum is dedicated to those who wish to suggest new features.
Questo forum è dedicato a chi desidera suggerire nuove feature.
Ce forum est dédié à ceux qui souhaitent suggérer de nouvelles fonctionnalités.
Post Reply
spotlessmind1975
Site Admin
Posts: 171
Joined: Fri Oct 06, 2023 8:25 pm

Storage access: files or memory?

Post by spotlessmind1975 »

[italian and french follow]

:arrow: english

Hi everybody!

Ok, as you well know, I've been asked for some time to add support for storage devices, i.e. mass storage. Apart from the fact that I opened an issue about it, in general it involves giving support to four different things:
  • give the possibility to support deferred loading of (ugBASIC) programs, with the so-called "chaining";
  • allow the dynamic loading of modules within ugBasic programs, with the so-called overlay;
  • allow deferred loading of resources (graphics, audio, game data);
  • allow to save the game state, for later reload.
Image

Each of these points will need dedicated instructions, or an adaptation of those that are already there (backwards compatible adaptation, of course). However, they all come up against a fundamental decision: does it make sense to use the file metaphor, or is it more reasonable to see the medium as an addressable memory space?

The difference is not insignificant.

On the one hand, maintaining the metaphor of files, directories, and so on, we have a simple way to insert data into media using a PC, a way that programs on the home computer itself can access. On the other hand, access becomes very laborious, less performant and, in the end, a very demanding development.

On the other hand, seeing the medium as a storage space without a form, we can give it any form. We can therefore quickly access what interests us, load it and save it as we want. The disadvantage? No one else will be able to access that data if he doesn't know how.

And you, what do you think?

:arrow: italian

Ciao a tutti!

Ok, come ben sapete, da tempo mi viene chiesto di aggiungere il supporto per i dispositivi di archiviazione, cioè le memorie di massa. A parte il fatto che ho aperto un ticket a riguardo, in generale si tratta di dare supporto a quattro funzionalità differenti:
  • dare la possibilità di supportare il caricamento differito dei programmi (ugBASIC), con il cosiddetto "chaining";
  • permettere il caricamento dinamico dei moduli all'interno dei programmi ugBasic, con il cosiddetto overlay;
  • consentire il caricamento differito delle risorse (grafica, audio, dati di gioco);
  • consentire di salvare lo stato del gioco, per ricaricarlo successivamente.
Image

Ognuno di questi punti avrà bisogno di istruzioni dedicate, o di un adattamento di quelle già presenti (adattamento retrocompatibile, ovviamente). Tutti però si scontrano con una decisione fondamentale: ha senso usare la metafora del file, o è più ragionevole vedere il mezzo come uno spazio di memoria indirizzabile?

La differenza non è insignificante.

Da un lato, mantenendo la metafora di file, directory e così via, abbiamo un modo semplice per inserire dati nei media utilizzando un PC, un modo a cui possono accedere i programmi sul computer di casa stesso. D’altro canto, l’accesso diventa molto laborioso, meno performante e, in definitiva, uno sviluppo molto impegnativo.

D'altra parte, considerando il mezzo come uno spazio di archiviazione senza forma, possiamo dargli qualsiasi forma. Possiamo quindi accedere velocemente a ciò che ci interessa, caricarlo e salvarlo come vogliamo. Lo svantaggio? Nessun altro potrà accedere a quei dati se non sa come.

E voi, cosa ne pensate?

:arrow: french

Salut tout le monde!

Ok, comme vous le savez bien, on me demande depuis un certain temps d'ajouter le support des périphériques de stockage, c'est-à-dire le "mass storage". Outre le fait que j'ai ouvert un ticket à ce sujet, il s'agit en général d'apporter son soutien à quatre différentes choses:
  • donnent la possibilité de prendre en charge le chargement différé des programmes (ugBASIC), avec ce que l'on appelle le "chaining" ;
  • autoriser le chargement différé des ressources (graphiques, audio, données de jeu) ;
  • permet de enregistrer l'état du jeu, pour un rechargement ultérieur.
Image

Chacun de ces points nécessitera des instructions dédiées, ou une adaptation de celles qui existent déjà (adaptation rétrocompatible bien entendu). Cependant, ils se heurtent tous à une décision fondamentale : est-il judicieux d’utiliser la métaphore du fichier, ou est-il plus raisonnable de considérer le support comme un espace mémoire adressable ?

La différence n'est pas négligeable.

D’une part, en conservant la métaphore des fichiers, des répertoires, etc., nous disposons d’un moyen simple d’insérer des données dans un support à l’aide d’un PC, un moyen auquel les programmes de l’ordinateur personnel lui-même peuvent accéder. En revanche, l'accès devient très laborieux, moins performant et, au final, un développement très exigeant.

En revanche, en considérant le support comme un espace de stockage sans forme, on peut lui donner n'importe quelle forme. On peut donc accéder rapidement à ce qui nous intéresse, le charger et le sauvegarder comme bon nous semble. Le désavantage? Personne d'autre ne pourra accéder à ces données s'il ne sait pas comment.

Et vous, qu'en pensez-vous ?
User avatar
wysardry
Posts: 26
Joined: Sun Oct 08, 2023 6:45 pm

Re: Storage access: files or memory?

Post by wysardry »

For me, the file metaphor makes more sense as I would want to be able to load and save data via cassette tape and disks on physical machines.

Unless I'm missing something, seeing the medium as an addressable memory space wouldn't work well with the serial nature of tapes.
TRS-Eric
Posts: 4
Joined: Wed Oct 18, 2023 4:43 am

Re: Storage access: files or memory?

Post by TRS-Eric »

I want to be able to load and save off of Floppy disk, files will work. Memory access would work too. Memory access might be best to get the ball rolling because you can always build out a file system on top of that.
ldir_hector
Posts: 13
Joined: Sun Oct 08, 2023 11:47 am

Re: Storage access: files or memory?

Post by ldir_hector »

I think that if you make an abstraction of floppies to manage them like virtual memory, it would be very slow and not very practical.
Most floppies are organised with tracks and sectors so accessing any data inside them individually would be quite inneficient.

I think that even if files are not the most efficient way to manage data on floppy disk, at least they are very easy to deal with and quite universal.
But what I don't know is will you use the default standard format of each machine or will you make your own multi floppies format ?

I think that each file format have their own limitations. On the amstrad CPC OOTB for example,you don't have any named directories (only 15 user directories from number 0 to 15) and you can only have 64 files on a floppy disk face (you have to turn the disk to read the other face !).

Anyhow, I think the file system is the more realistic. Accessing directly to the disk without the file system is really very dependant of what kind of floppy you have and can't be abstracted easilly I think.
spotlessmind1975
Site Admin
Posts: 171
Joined: Fri Oct 06, 2023 8:25 pm

Re: Storage access: files or memory?

Post by spotlessmind1975 »

Hi everybody, and thank you for your interesting answers! :D

I remind everyone that we are looking for an isomorphic approach to the problem. This means that we are not looking for a "lowest common multiple" but rather a way to best express the potential of each target.
wysardry wrote: Wed Nov 22, 2023 11:44 am For me, the file metaphor makes more sense as I would want to be able to load and save data via cassette tape and disks on physical machines.
TRS-Eric wrote: Wed Nov 22, 2023 3:29 pm I want to be able to load and save off of Floppy disk, files will work.
ldir_hector wrote: Wed Nov 22, 2023 9:44 pm I think that even if files are not the most efficient way to manage data on floppy disk, at least they are very easy to deal with and quite universal.
In general, therefore, it seems to me that maintaining the concept of file is perceived as essential because it is universal. I quite agree with you. To answer wysardry, files make all accesses sequential, even those to a non-sequential device like disks. Of course, you can move randomly through the file when the hardware subsystem allows it, as with disks but not with tapes.

So, in summary, we can think of files as an "entry point", behind which to hide the complexity of the hardware: could this be good? In that case, the BASIC primitives would still be the same, but which file to use would make a difference.
wysardry wrote: Wed Nov 22, 2023 11:44 am Unless I'm missing something, seeing the medium as an addressable memory space wouldn't work well with the serial nature of tapes.
TRS-Eric wrote: Wed Nov 22, 2023 3:29 pm Memory access would work too. Memory access might be best to get the ball rolling because you can always build out a file system on top of that.
Once we establish that files are "entry points," we might think of memory as a huge file, perhaps with a particular name. So let's move up and down this file. The metaphor could work..?
ldir_hector wrote: Wed Nov 22, 2023 9:44 pm I think that if you make an abstraction of floppies to manage them like virtual memory, it would be very slow and not very practical.
Most floppies are organised with tracks and sectors so accessing any data inside them individually would be quite inneficient.
ldir_hector wrote: Wed Nov 22, 2023 9:44 pm Anyhow, I think the file system is the more realistic. Accessing directly to the disk without the file system is really very dependant of what kind of floppy you have and can't be abstracted easilly I think.
Ok, this is unfortunately also true with files. In fact, it's even worse, because the filesystem has to take care of occupying the disk surface: BAM, allocation maps, and so on. In reality I was thinking of something simpler: it was ok to move "randomly" along the file, but if reading / writing could only take place at a certain size (e.g. 256 bytes at a time), this size could coincide with the size of the logical sector of the support. So, in practice, you would only have to calculate an offset within the medium, and then write to or read from it.
ldir_hector wrote: Wed Nov 22, 2023 9:44 pm But what I don't know is will you use the default standard format of each machine or will you make your own multi floppies format ?
This is a moot point. On the one hand, I would like it to be as "standard" as possible, so that it is possible to write an ugBASIC program capable of reading data from a medium that did not produce the compiler. On the other hand, this introduces a significant level of complexity, which is not easily addressed.

I'll give you an example: there are targets where you are forced to talk to another "computer", to access the data, with a serial protocol (read: Commodore). Imagine having to make the transition from one system to another "transparently", i.e. with the same primitives, but without introducing abstractions.

It is a really hard task! :shock:

But I think that from our exchange the best solution for everyone will certainly emerge. 8-)
ldir_hector
Posts: 13
Joined: Sun Oct 08, 2023 11:47 am

Re: Storage access: files or memory?

Post by ldir_hector »

Hello !
It's been a while !

I haven't done much with ugBasic till now, but I am glad it's still progressing !

I hope you'll find a way to deal with storage access. I think that dealing with a distant storage access connected through a serial cable is a very rare use case. I know that on a Commodore C64, the floppy drive is accessed like this technically and that their floppy drives were nearly a complete computer themselves ! However when using Commodore Basic, all this was hidden behind very simple command like LOAD.

I think that communicationg through serial or parallel port shall be another technical subject and that storage access shall be only on local hardware. I think that in that case, you should keep the original system and just define a very "basic" way to access storage with all what theses computers dealt with : Text files that can be loaded either line by line or data by data (with fixed length or variable length fields) and binary files that are loaded where we want in memory. Wether you can open several files at the same time will depend on the target system computer, but I bet there wouldn't be many that can open two files at the same time !
spotlessmind1975
Site Admin
Posts: 171
Joined: Fri Oct 06, 2023 8:25 pm

Re: Storage access: files or memory?

Post by spotlessmind1975 »

Hi ldir_hector, nice to see you again!
ldir_hector wrote: Mon Oct 14, 2024 9:23 pm I haven't done much with ugBasic till now, but I am glad it's still progressing !
In the meantime, I'm happy to read you again, and yes, it is and I'm trying to carry it forward in my spare time, and despite everything. :D
ldir_hector wrote: Mon Oct 14, 2024 9:23 pm I hope you'll find a way to deal with storage access
Yes, I found it. As of today ugBASIC is (or rather, would be able to 8-) ) call the system routines of the Amstrad CPC 664 ROM. So, at least in theory, it would be able to read from disk. I just checked the sources, and I found that the DLOAD command has been implemented for cpc target.

Here you can find the manual page about DLOAD.

I think you'll have to do some testing because, honestly, I don't remember how to set up the disk, apart from using the ugBASIC's BEGIN STORAGE mechanism, which I described on the Facebook group. Here you can find an example of how to insert an image into a disk, and have ugBASIC read it. Please note, I have not tested it, so I cannot guarantee that it works (the resources are in the repository):

Code: Select all

    BITMAP ENABLE
    STORAGE "disco" AS "disk1"
        IMAGE "token_red.png" AS "tokenred"
    ENDSTORAGE    
    CLS
    tokenImage := NEW IMAGE(16, 16)
    DLOAD "tokenred" TO VARPTR(tokenImage)
    PUT IMAGE tokenImage AT 32, 32
If you go to the examples page (STORAGE MANAGEMENT), you should find a series of other examples. Again, I have not tested them recently, but they should work. Let me know. ;)

About the serial access, it was actually a "fake" serial. It was, more than anything, an original way of communicating between two computationally equivalent devices, namely the Commodore 64 and the 1541-II. In practice, it is more a matter of parallel programming than serial communication. It was a clever idea for the time, but I must admit that it is really complicated to deal with today. In practice, it is more convenient to call the KERNAL routines, which ugBASIC also does, especially for loading data from the floppy to the REU memory expansion.
Post Reply