|
Softpanorama
(slightly skeptical)
Open Source Software Educational Society |
May the
source be with you,
but remember the KISS principle ;-)
|
FTP Filesystems
The two most common implementations of ftp
filesystem are Netdrive and Gnome. The idea go back to Plan 9, the research
successor to Unix
by the Computing Sciences Research Center at
Bell Labs
between the mid-1980s and 2002
There is no ftp(1) command under Plan 9. Instead
there is an ftpfs fileserver, and each FTP connection looks like a
file system mount. ftpfs automatically translates open, read, and
write commands on files and directories under the mount point into FTP
protocol transactions. Thus, all ordinary file-handling tools such as ls(1),
mv(1) and cp(1) simply work, both underneath the FTP mount point and across
the boundaries with the rest of the user's view of the namespace. The only
difference the user (or his scripts and programs) will notice is retrieval
speed.[Plan 9 The Way the
Future Was]
Independently this idea was implemented in Orthodox
file managers (OFM) and later found its was to Unix Midnight Commander and
CurlFtpFS. There is also
GNOME File
System modeled after midnight Commander VFS. It is used in Nautilus,
but the latter is pretty buggy:
You can use the file manager to access
FTP sites. To access an FTP site, enter the URL for the site in the
field on the location bar, then press Return. The contents of the
site are displayed in the view pane. To copy a file from the FTP
site, drag the file to the new location.
To access an FTP site
that requires a username and password, you can enter the URL in the
following form:
ftp://username:password@hostname.domain
To reload the FTP site in the view pane, choose View -> Reload.
To stop loading the FTP site, choose View -> Stop.
Gnome VFS has wrapper for Perl:
VFSGnome -
Gnome Virtual Filesystem for Perl - search.cpan.org
VFS::Gnome is a TIEHANDLE module that uses the
gnome-vfs library from the Gnome project (http://www.gnome.org).
The gnome-vfs library (Virtual File System) allows uniform access to various
uri types such as http://, https://, file://, ftp:// etc.
SshFSAuthor: Miklos Szeredi / miklos
at szeredi hu
Homepage:
http://fuse.sourceforge.net/sshfs.html
- Description
- This is a filesystem client based on the
SSH File Transfer Protocol. Since most SSH servers already
support this protocol it is very easy to set up: i.e. on the
server side there's nothing to do. On the client side mounting
the filesystem is as easy as logging into the server with ssh.
HTTPFS
... is a FUSE-based file system
. You may think of it, as
a downloader with the speed of light.
...Here's something I threw together. I too am very interested an seeing
the writeups of the gnome-vfs and kio as I only know very high level
things about them.
Attached is some kind of architecture diagram and some text describing
various ideas in it. I guess its an elaboration on what I was saying
yesterday. I've also inserted the text part below. If anything maybe the
design goals part tries to sum up what we've talked about in this thread
so far.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
See attached diagram:
Legend:
Dotted red arrows = "relaization" in the uml sense
Green arrows = D-BUS transactions
White boxes = software implementations
Yellow boxes = D-Bus interfaces ('services')
Overview
--------
I think this design is essentially something along the lines of what
people have been thinking about, with the ability to 'hook' into
various parts of the process by passing references to services that
have well-defined interfaces. It's kind of like a call back, but it
uses D-BUS to let you define much more interesting interactions than
just a simple callback. Backends that use protocols that weren't meant
for async io can still be written in a fairly sync matter, it would
just have to block on certain things (like authentication).
Design goals
------------
* Simplicity: This is i think from the point of view of the
application programmer and the backend developer. There should not
be too many things that they have to worry about. From the user's
perspective, it should just work (tm), but I think this specifically
means that passing in URI's to various applications should result in
regular and predictable consistent behaviour, including the kinds of
authentication mechanisms presented.
* User-interaction decoupling: All the parts that require user
interaction are hidden behind D-BUS services with specific message
interfaces. This lets client apps not have to worry about anything
unless they want to, and it allows desktops to provide integrated
'default' solutions that can implement their own policies with
regards to authentications, password caching, sessions management,
etc. This allows for a situation like, if you run a KDE app in
gnome, it should be able to use the proper gnome dialogs and
policies.
* Client control: If a client wants to be really involved in the
backend, like with browsers, it should be able to. My design so far
allows clients to provide their own authentication services, as well
as monitor backends in a way that may be specific to that backend.
I suppose it would be useful to have a backend also specify any
backend-specific control interfaces so that a client can have lots
of control over a backend if it wanted to. Or maybe a mechanism by
which a client can bypass the central vfs service altogether and use
a backend directly and exculsively.
* Provide High level operations: I like the queryInterfaces() type of
idea, save for that I'm not sure most application programmers want
to deal with adapting to all kinds of different capabilities of
various backends. So there should be someone in the middle that
provides very high level operations and translates them to whatever
capabilities a backend may provide, or tells you that they're not
possible. This is probably hard to do very well, so it should only
be done once.
VFS Service
-----------
The Vfs service is at the core. It is sort of a management center that
provides high level vfs operations. This service would be running
regardless of whether a desktop was, so that it could provide vfs
services to console applications as well.
It also is responsible for activating back ends (or maybe this can be
done by d-bus directly), knowing what backends are available, providing
calls to query what kinds of backends are currently running, and to
provide a standard way to observe the state of the vfs in real time. I
think it would be nice to also bring something like fam into this
component so that we only need one interface to support monitoring
on all kinds of vfs's.
I think this would also be the place to put high level-> low level
operation composition kind of stuff. (i.e. implement recursive
directory delete as a series of change directories and deletes, as
many ftp programs do) I think kio does this, not sure about gnome-vfs.
But these highlevel operations are useful to everyone so they should
share an implementation.
A desktop when it runs should be able to tell the vfs to notify it of
events. I'm not sure exactly what kind of events, but there must be
something that the desktop would like to know in real time.
VFS Backend
-----------
This is where the actual work happens. This design tries to extract
all the GUI and user-interaction requiring stuff out of the module.
Each backend instance would be passed several service names as
parameters, one which implements an authentication service that is
specific to this backend (or i guess the interface could be shared if
i multiple backends had the same kind of authentication requirements)
and an observer interface, that would be capable of capturing events
within a backend that maybe specific to that back end.
For example i could say give me a sftp backend instance passing it
the gnome sftp authentication adapter, and the gnome sftp observer. The
authentication adapter would give me gnome dialogs that adhere
to gnome's policy of 'sessions' and password caching (maybe a dialog
wouldn't even pop up at all) and would also say, pop up gnome dialogs
to show the current status (like say if downloading af file)
It seems like things like the status dialogs that come up when you do
a big copy in konqueror or nautilus should be able to be handled at
either the desktop level OR specifically by the client. You wouldn't
want little dialog windows popping up every time konqueror loads a
page, so an app like konq should be able to pass in its own observer
that could monitor the backend's status and update its own GUI in an
appropriate way. A console ftp app could also pass in its own observer
that just updates its own internal structures and displays information
in some way more appropriate for the console.
In the case that the app doesn't care what kind of authentication or
observer is used, there should of course be a call that just means
'use whatever is there'. The best way to do this is probably use the
'service ownership' idea in D-BUS so that when gnome starts up for
example, it would have something that takes ownership of a well known
auth adapter service name, org.vfs.auth.sftp or something similar. If
when the backend is created, not specific auth adapters or observers
are passed in, then it could just be hardwired to lookup the well
known service name.
The only thing I'm not sure about is, say you're just running in the
console, so no application may have claimed the default well-known
auth service. In that case, if you try to do something that requires a
service, the activation mehcanism will kick in, but you want it to do
something appropriate for the console. Is there a way in D-BUS that
you can say 'give me something that works on the console', or rather
is there a way to implement that kind of thing?
Issues
------
It seems kind of fundamental, but how does the client app actually get
the data it wants? In this design's organization, I suppose the back
end would deliver it to the vfs service, which would then deliver it
to the client application. I guess if D-BUS can do some kind of
shared-memory avoid-copying thing then this could be a reasonable
solution. Scheduling latency might be a problem with so many mutually
out-of-process interactions. In the case of network transfers, since
the network will likely be the bottleneck, maybe this isn't so much of
a problem. But if its dog slow, nobody's gonna use it.. especially for
'file://'
AKBP-II 1999 Gruppe 4
FTP filesystem für Solaris, implementiert mit RPCs und User-Level
Daemon
Ziel war es ein FTP filesystem für Solaris zu bauen. Prinzipiell
gab es zwei Methoden um mit dem Kern zu kommunizieren: Die Implementierung eines
neuen System-Calls oder eine Verbindung über RPC. Wir entschieden uns für RPC.
Ein wenig Spass muss ja auch sein!
tech-kern Re proposed changes to etc ()
Subject: Re: proposed: changes to
"etc" (?)
To: None <tech-kern@NetBSD.ORG>
From: Paul B Dokas <dokas@cs.umn.edu>
List: tech-kern
Date: 12/10/1997 11:23:35
On Tue, 9 Dec 1997, Paul Evans wrote:
> One of the specific draw backs of UNIX I keep running up against is the
> difficulties related to configuration across software packages and
> different UNIXices. Specficly I'm talking about files in any dir that
> ends in "etc"
>
> Having said that, I acknowledgbe there's nothing we can do about other
> peoples misguided beliefs (Hello Solaris Dev team are you there?)
>
> I'm not propsing to instigate something the Red Sysadmin book refers to
> as a "vendor gratuitous change" But, I think an initive is needed here,
> for several reasons:
>
> 1. difficulties in management due to differnces in format
> 2. error prone by nature, i.e. typos
> 3. lack of history log
> 4. Security concerns (This last to are far ranging)
> 5. Program inplementations of parsers are redundant (hence more
> possible failure points)
Well, I'll try not to get involved in this one. Personally, I feel that
the current /etc/* files are fine. Perhaps that's just because I've
never known anything else. But, I'm open to suggestions.
> >From what I can see there are two possible alternitive solutions:
>
> 1. A user mode configuration library that uses a protected
> binary file. (a *gasp* registry)
>
> 2. A VFS based solution involving mounting the above to create
> the illusion of the files in "etc" (as ascii), but with
> an API for program use.
>
> Just wanted to pitch this out as I suspect it could have a lot to do with
> the proposed capabilties an HSM or new VFS layers/drivers
[Let me preface this with the declaimer: I'm know very little about
how the kernel implements file systems and I know even less about file
systems themselves. Take everything that I'm about to say with a
lot of salt...]
Now this one is interesting to me. One thing that I'd love to see is
a well implemented user space filesystem (perhaps just a daemon that
talks to the kernel's VFS layer). For example, imagine an RCS filesystem
built with a user space filesystem and the RCS commands. When a file is
opened, the daemon does an 'co -l' on the file. When the file is closed,
the daemon does a 'ci -u'. It could probably even be extended to append
version numbers to files ala VMS (IMO this is ugly as hell, but it is
potentially useful in some cases).
In fact, it would be more useful to have a generic user space filesystem
daemon that executes shell scripts to implement the VFS operations. That
way, we'd have a generic daemon that would be useful for experimenting
with new filesystems without having to muck around in the kernel. The
FTP filesystem would be pretty trivial to implement with such a daemon.
Yes, this would be slow, insecure and open to all kinds of problems. But
it would be useful in some very limited cases and for experimentation.
Just rambling on,
Paul
--
Paul Dokas dokas@cs.umn.edu
In case of broken links
please try to use Google search. If you find the page please notify
us about new location
Internal
External
GNU's
Bulletin, vol. 1 no. 16 - GNU Project - Free Software Foundation (FSF) mirror
is here
GNU's
Bulletin, vol. 1 no. 16
- Alex FTP Filesystem
- Alex is a filesystem that lets users access files in FTP sites around the
world just like they access local files. Alex pathnames are composed of 3 parts.
First is /alex. Second is a reversed hostname. Last is the path on that host.
For example, /alex/edu/berkeley/pub/virus.patch is a file at berkeley.edu.
- CIFS - the Common Internet
File System
- Coda - an advanced
network filesystem
- Coda is an advanced networked filesystem. It has been developed at CMU during
the last six years by the systems group of M. Satyanarayanan. in the SCS deparment.
-
Ext2fs - a structural analysis
-
-
Microsoft Research Scalable Servers
- The Swarm Scaleable
Storage System
- The goal of the Swarm project is to design and implement a scalable network
storage system based on a cluster of personal computers. Swarm is scalable because
its performance is a function of the size of the cluster -- the more nodes in
the cluster, the higher the system performance. This performance improvement
is not merely an increase in aggregate performance, either; clients whose I/O
performance is limited by the storage system will see an improvement in individual
I/O operations if more storage servers are added. This decoupling of I/O performance
from storage server performance allows the nodes of the cluster to be chosen
to optimize cost-performance, rather than absolute performance -- the desired
performance of the overall system is attained by aggregating enough nodes.
- TCFS - the
Transparent Cryptographic File System
Libor Martinek has submitted code for an FTP-capable filesystem implementation.
See his message: http://www.netbeans.org/www-contrib/msg00006.html It lets you connect
to an FTP server and then mount part of the server in NetBeans like a filesystem;
you can then edit files and work with them normally. Any objections to making him
a project for this? Suggested name: "ftpfs" or maybe "remotefs" (it has some sections
which are more generic, then FTP-specific stuff, then a module--right Libor?). -Jesse
HTTPFS talk at USENIX99
Alex FTP Filesystem
Summary: alex - filesystem for transparent access to FTP sites; spoofs NFS into
mounting cached copies of Internet-based FTP sites Version: 940717, 950406 (alpha)
Description: Alex is an experimental filesystem that tries to give users transparent
read access to anonymous FTP sites. Alex pathnames are really composed of 3 parts.
First is /alex. Second is a reversed hostname. Last is the path on that host. For
example, /alex/edu/berkeley/pub/virus.patch is a file at berkeley.edu. .../usr.man/alex.1
Keywords:
alex,
data communication,
File Transfer Protocol,
ftp(1),
Network File System,
NFS Language(s): C Requirements: NFS client capability See Also: ? Restrictions:
See .../src/README References: ?
internet-tools
Alex FTP Filesystem
Alex FTP Filesystem
Alex FTP Filesystem
Alex
Copyright © 1996-2008 by Dr. Nikolai Bezroukov.
www.softpanorama.org was
created as a service to the UN Sustainable Development Networking Programme (SDNP)
in the author free time.
Submit
comments This document is an industrial compilation designed and created
exclusively for educational use and is placed under the copyright of the
Open Content License(OPL).
Original materials copyright belong to respective owners. Quotes are made
for educational purposes only in compliance with the fair use doctrine.
Standard disclaimer: The statements, views and opinions presented on
this web page are those of the author and are not endorsed by, nor do they necessarily
reflect, the opinions of the author present and former employers, SDNP or any other
organization the author may be associated with. We do not warrant the correctness
of the information provided or its fitness for any purpose.
Last modified:
June 02, 2008
