* 1. Command Line download: download one file or one directory from the command line
* 2. Agent Based download: run from the scheduler
* 3. Command Line locally to import the file(directory): Import one file or one directory from the command line, used by upload from file and upload from server
*
*
* +-----------------------+
* | Command Line download |
* +-----------------------+
*
* To download one file or one directory from the command line:
* example:
* ./wget_agent http://www.aaa.com/bbb
*
* +----------------------+
* | Agent Based |
* +----------------------+
* To download one file or one directory (one URL )from the scheduler:
* example:
* part 1 parameters from the scheduler: 19 - http://g.org -l 1 -R index.html*
* 19 is uploadpk, 'http://g.org' is downloadfile url,
* '-l 1 -R index.html*' is several parameters used by wget_agent
* part 2 parameters from wget_agent.conf: -d /var/local/lib/fossology/agents
* '/var/local/lib/fossology/agent' is directory for downloaded file(directory)
* storage temporarily, after all file(directory) is dowloaded, move them into repo