使用压缩来看看
?
sqoop:000> update job --jid 1
Compression format:
? 0 : NONE
? 1 : DEFAULT
? 2 : DEFLATE
? 3 : GZIP
? 4 : BZIP2
? 5 : LZO
? 6 : LZ4
? 7 : SNAPPY
Choose: 3
Output directory: /home/dimDateGZip
Job was successfully updated with status FINE
?
使用Gzip
同样的job 跑出来的不一样
?
[root@localhost ~]# hadoop fs -ls /home/dimDateGZip 14/03/20 09:39:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 11 items -rw-r--r-- 1 root supergroup 0 2014-03-20 09:35 /home/dimDateGZip/_SUCCESS -rw-r--r-- 1 root supergroup 2266 2014-03-20 09:34 /home/dimDateGZip/part-m-00000.gz -rw-r--r-- 1 root supergroup 2461 2014-03-20 09:34 /home/dimDateGZip/part-m-00001.gz -rw-r--r-- 1 root supergroup 1905 2014-03-20 09:34 /home/dimDateGZip/part-m-00002.gz -rw-r--r-- 1 root supergroup 2814 2014-03-20 09:34 /home/dimDateGZip/part-m-00003.gz -rw-r--r-- 1 root supergroup 1546 2014-03-20 09:35 /home/dimDateGZip/part-m-00004.gz -rw-r--r-- 1 root supergroup 2804 2014-03-20 09:34 /home/dimDateGZip/part-m-00005.gz -rw-r--r-- 1 root supergroup 20 2014-03-20 09:34 /home/dimDateGZip/part-m-00006.gz -rw-r--r-- 1 root supergroup 20 2014-03-20 09:35 /home/dimDateGZip/part-m-00007.gz -rw-r--r-- 1 root supergroup 20 2014-03-20 09:35 /home/dimDateGZip/part-m-00008.gz -rw-r--r-- 1 root supergroup 535 2014-03-20 09:35 /home/dimDateGZip/part-m-00009.gz [root@localhost ~]# hdfs dfs -ls /home/dimDate 14/03/20 09:42:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 11 items -rw-r--r-- 1 root supergroup 0 2014-03-20 09:29 /home/dimDate/_SUCCESS -rw-r--r-- 1 root supergroup 20748 2014-03-20 09:28 /home/dimDate/part-m-00000 -rw-r--r-- 1 root supergroup 22248 2014-03-20 09:28 /home/dimDate/part-m-00001 -rw-r--r-- 1 root supergroup 17461 2014-03-20 09:28 /home/dimDate/part-m-00002 -rw-r--r-- 1 root supergroup 25573 2014-03-20 09:29 /home/dimDate/part-m-00003 -rw-r--r-- 1 root supergroup 14132 2014-03-20 09:29 /home/dimDate/part-m-00004 -rw-r--r-- 1 root supergroup 25693 2014-03-20 09:29 /home/dimDate/part-m-00005 -rw-r--r-- 1 root supergroup 0 2014-03-20 09:29 /home/dimDate/part-m-00006 -rw-r--r-- 1 root supergroup 0 2014-03-20 09:29 /home/dimDate/part-m-00007 -rw-r--r-- 1 root supergroup 0 2014-03-20 09:29 /home/dimDate/part-m-00008 -rw-r--r-- 1 root supergroup 3477 2014-03-20 09:29 /home/dimDate/part-m-00009
?压和没压差10倍.
?
下一步就是把table 搞进hive 打算用RCFile
?
