Mar 31, 2015

HDFSHelloWorld


This page aims at creating a "copy-paste"-like tutorial to familiarize with HDFS commands . It mainly focuses on user commands (uploading and downloading data into HDFS).

Requirements

  • SSH (for Windows, use PuTTY and see how to create a key with PuTTY)
  • An account in the DAPLAB, and send your ssh public key to Benoit.
  • A browser -- well, if you can access this page, you should have met this requirement :)

Resources


While the source of truth for HDFS commands is the code source, the documentation page describing the hdfs dfs commands is really useful:

Basic Manipulations

Listing a folder

Your home folder

$ hdfs dfs -ls
Found 28 items
...
-rw-r--r--   3 bperroud daplab_user    6398990 2015-03-13 11:01 data.csv
...
^^^^^^^^^^   ^ ^^^^^^^^ ^^^^^^^^^^^    ^^^^^^^ ^^^^^^^^^^ ^^^^^ ^^^^^^^^
         1   2        3           4          5          6     7        8
Columns, as numbered below, represent:
  1. Permissions, in unix-style syntax
  2. Replication factor (RF in short), default being 3 for a file. Directories have a RF of 0.
  3. Owner
  4. Group owning the file
  5. Size of the file, in bytes. Note that to compute the physical space used, this number should be multiplied by the RF.
  6. Modification date. As HDFS is mostly a write-once-read-many filesystem, this date often means creation date
  7. Modification time. Same as date.
  8. Filename, within the listed folder

Listing the /tmp folder

$ hdfs dfs -ls /tmp

Uploading a file

In /tmp

$ hdfs dfs -copyFromLocal localfile.txt /tmp/
The first arguments after -copyFromLocal point to local files or folders, while the last argument is a file (if only one file listed as source) or directory in HDFS.
Note: hdfs dfs -put is doing about the same thing, but -copyFromLocal is more explicit when you're uploading a local file and thus preferred.

Downloading a file

From /tmp

$ hdfs dfs -copyToLocal /tmp/remotefile.txt .

The first arguments after -copyToLocal point to files or folder in HDFS, while the last argument is a local file (if only one file listed as source) or directory.
hdfs dfs -get is doing about the same thing, but -copyToLocal is more explicit when you're downloading a file and thus preferred.

Creating a folder

In your home folder

$ hdfs dfs -mkdir dummy-folder

In /tmp

$ hdfs dfs -mkdir /tmp/dummy-folder
Note that relative paths points to your home folder, /user/bperroud for instance.




11 comments :

  1. Nice information about the HDFS my sincere thanks for sharing this post and please continue to share this post
    Hadoop Training in Chennai

    ReplyDelete
    Replies
    1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training India . Nowadays Java has tons of job opportunities on various vertical industry.

      Delete
  2. Thank you for excellent article.Great information for new guy like antimalware service executable

    ReplyDelete
  3. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts.It is my work checkout one time- Error 0X000003EB

    ReplyDelete
  4. Garmin Express Update can also track your fitness records. One can use the Garmin connect to link the Garmin the monitor to their respective device and easily use. We are available 24x7!

    ReplyDelete
  5. Hey There. I discovered your weblog the usage of msn. That is a really well written blog. I will be sure to bookmark it and come back to learn extra of your useful information. Thanks for the post. Know about Garmin.com/Express.

    ReplyDelete
  6. toptan iç giyim tercih etmenizin sebebi kaliteyi ucuza satın alabilmektir. Ürünler yine orjinaldir ve size sorun yaşatmaz. Yine de bilinen tekstil markalarını tercih etmelisiniz.

    Digitürk başvuru güncel adresine hoşgeldiniz. Hemen başvuru yaparsanız anında kurulum yapmaktayız.

    tutku iç giyim Türkiye'nin önde gelen iç giyim markalarından birisi olmasının yanı sıra en çok satan markalardan birisidir. Ürünleri hem çok kalitelidir hem de pamuk kullanımı daha fazladır.

    nbb sütyen hem kaliteli hem de uygun fiyatlı sütyenler üretmektedir. Sütyene ek olarak sütyen takımı ve jartiyer gibi ürünleri de mevcuttur. Özellikle Avrupa ve Orta Doğu'da çokça tercih edilmektedir.

    yeni inci sütyen kaliteyi ucuz olarak sizlere ulaştırmaktadır. Çok çeşitli sütyen varyantları mevcuttur. iç giyime damga vuran markalardan biridir ve genellikle Avrupa'da ismi sıklıkla duyulur.

    iç giyim ürünlerine her zaman dikkat etmemiz gerekmektedir. Üretimde kullanılan malzemelerin kullanım oranları, kumaşın esnekliği, çekmezlik testi gibi birçok unsuru aynı anda değerlendirerek seçim yapmalıyız.

    iç giyim bayanların erkeklere göre daha dikkatli oldukları bir alandır. Erkeklere göre daha özenli ve daha seçici davranırlar. Biliyorlar ki iç giyimde kullandıkları şeyler kafalarındaki ve ruhlarındaki özellikleri dışa vururlar.

    ReplyDelete