Download
svn co https://uicrawler.svn.sourceforge.net/svnroot/uicrawler/trunk/
rsync -av uicrawler.svn.sourceforge.net::svn/uicrawler/trunk/* .
Install uicrawler
- Issue this command to install the program
python setup.py install
Using uicrawler
- To see if your template got installed run this command:
paster create --list-templates
- You should see uicrawler as one of them
uicrawler: Template for creating basic datamining, crawling,converting package.
paster create -t 'uicrawler'
- After you answer all questions you should see file structure like this:
- I called my project somedata.
somedata
|-- __init__.py
|-- somedata
| |-- Convert
| | `-- mydata.txt
| |-- HDF5
| |-- RoughData
| | |-- download.sh
| | `-- download_list.txt
| |-- __init__.py
| `-- model
| `-- model.template
|-- somedata.egg-info
| |-- PKG-INFO
| |-- SOURCES.txt
| |-- dependency_links.txt
| |-- entry_points.txt
| |-- not-zip-safe
| |-- paster_plugins.txt
| `-- top_level.txt
|-- setup.cfg
`-- setup.py
- In Convert folder you will find a sample file where you can write your convert script
- In model you can write your file deffinition using sqlalchemy.
- in data you download the actual data.