HDFS and FUSE - a mountable distributed file system

The Hadoop Distributed File System (HDFS) is an open source, distributed file system that's designed to run on commodity hardware and provide a fault tolerant data store optimized capable of storing extremely large files across many machines. It's similar in architecture to the Google File System, except that it's something you can install and play with yourself.

Most people who are using HDFS are using it as a storage system for extremely customized distributed application development and the HDFS data is accessed programatically via a Java API or through a few rudimentary shell commands. But what if you just want to use it as a general purpose file system that will automatically replicate many terabytes of data across a number of spare boxes hanging around the office? That's where FUSE comes in.

These projects (enumerated below) allow HDFS to be mounted (on most flavors of Unix) as a standard file system using the mount command. Once mounted, the user can operate on an instance of hdfs using standard Unix utilities such as 'ls', 'cd', 'cp', 'mkdir', 'find', 'grep', or use standard Posix libraries like open, write, read, close from C, C++, Python, Ruby, Perl, Java, bash, etc.

There are currently a few HDFS FUSE projects, some of which seem to be more maintained than others. One, called fuse-j-hdfs, is written with the FUSE for Java library and seems to be the most active project. Outside of FUSE, there's also a Webdav wrapper for HDFS that should provide mountable access from Windows clients.

Are there any HDFS gurus in the room who'd care to chime in with their own experiences with any of these tools?

Mountable HDFS Overview and Instructions
Hadoop Distributed File System


Recent Entries

Comments

Oldest comments listed first.

Posted by: shane on February 26, 2009 at 10:51 AM

I can install, and "play with myself"? Awesome!


Leave a comment


Subscribe to MAKE!Subscribe to MAKE Magazine!

Subscribe today, save 42% and get web access to MAKE free. MAKE Digital Edition is available only to subscribers.

$34.95 / 1 year
(4 Quarterly Issues)

Subscribe now


Void your warranty, violate a user agreement, fry a circuit, blow a fuse, poke an eye out. Make: The risk-takers, the doers, the makers of things... Welcome to Make: Online!


CRAFT Maker Shed Maker Faire MAKE television
Holiday Gift Guides from MAKE
Gifts for Dads
Science and Chemistry
Gifts Under $20
More guides: Santa Claus Machines, Geek Toys for Grown Up Girls & Boys


Check out all of the episodes of Make: television

Alex Rider Dream Gadget Contest
Make: Science Room

Connect with MAKE

Be a MAKE fan on Facebook MAKE on Facebook
Visit our Facebook page and become a fan of MAKE!
MAKE on Twitter MAKE on Twitter
Follow our MAKE tweets!
MAKE Flickr Pool MAKE on Flickr
Join our MAKE Flickr Pool!
    make_tips on Twitter




    Maker SHED

    Advertise here with FM.

    Why advertise on MAKE?
    Read what folks are saying about us!

    Click here to advertise on MAKE!



    Subscribe to MAKE Magazine!

    Make: Online authors!

    Gareth BranwynGareth Branwyn
    Senior Editor


    Phillip TorronePhillip Torrone
    Senior Editor
    | AIM | Twitter


    Becky SternBecky Stern
    Associate Editor
    | AIM | Twitter


    Marc de VinckMarc de Vinck
    Contributing Writer
    | AIM | Twitter


    John ParkJohn Park
    Contributing Writer
    | Twitter


    Sean RaganSean Ragan
    Contributing Writer
    | Twitter


    Matt MetsMatt Mets
    Contributing Writer
    | AIM | Twitter


    Dale DoughertyDale Dougherty
    Editor & Publisher
    | Twitter


    Shawn ConnallyShawn Connally
    Managing Editor
    | Twitter


    Goli MohammadiGoli Mohammadi
    Associate Managing Editor

    Kip KayKip Kay
    Weekend Projects
    | AIM | Twitter


    Collin CunninghamCollin Cunningham
    Contributing Writer
    | AIM | Twitter

    Adam FlahertyAdam Flaherty
    Contributing Writer
    | AIM | Twitter



    More contributors: Mark Frauenfelder (Editor-in-Chief, MAKE magazine), Kipp Bradford (Technical Consultant/Writer), Chris Connors (Education), Diana Eng (Guest Author), Peter Horvath (Intern), Brian Jepson (O'Reilly Media), Robert Bruce Thompson (Science Room)

    Suggest a Site!

    Current Podcast

    itunesdl.gif Weekend Project: Beetlebot Simple robot from your parts bin that avoids obstacles. Thanks go to Jerome Demers for the original article in MAKE, Volume 12. To download the Beetlebot video, click here or subscribe in iTunes. Check out the complete Beetlebot article... More...

    Get the Make: Online sent via email
    Enter your email to receive Make: Online each day:



    MAKE Fascination video series brought to you by Dow

    Make: Education
    MAKE: en EspaƱol MAKE: Japan
    Important please read


    Subscribe to MAKE Magazine!

    Recent Posts from the Craft: Blog