HPL/SQL is included to Apache Hive since version 2.0
COPY FROM FTP statement allows to copy files from a FTP server to local or any Hadoop compatible file system. Using this statement you can easily copy FTP subdirectories into HDFS i.e.
The NEW option helps you build a ETL process and download only new files from FTP.
COPY FROM FTP host [USER user [PWD password]] [DIR directory] [FILES files_wildcard] [TO [LOCAL] target_directory] [options] options: OVERWRITE | NEW SUBDIR SESSIONS num
Copy new files including files in subdirectories from a FTP server to HDFS location using 3 concurrent connections:
copy from ftp 'ftp.myserver.com' user 'paul' pwd '***' dir data/sales/in subdir files '.*' to /data/sales/raw sessions 3 new
Compatibility: HPL/SQL Extension
Version: HPL/SQL 0.3.17