Datagrepper installation: exhausting but interesting!

In last blog i have introduced you all about the OPW and the project i am interested on. Moving ahead today i will describe my installation process.

 

Firstly, I visited to the github profile of fedora-infrastructure and cloned the project in my local repository. Thereafter I followed the installation steps given on this link. But at various stages i found errors yelling out on my screen so i thought let describe each steps that may not be listed on official installation procedure but might be helpful.

Installation

1. Open up a terminal window.
2. Install virtualenv and  virtualenwrapper with this command. This will provide you to create and work on a separate environment.

$ sudo pip install virtualenv
$ sudo pip install virtualenwrapper

 Configure the PATH for virtualenvwrapper. To do that, append following lines to your .bashrc file

export WORKON_HOME=/home/charul/.virtualenvs;
source /usr/bin/virtualenvwrapper.sh;

Reload your profile

$ source ~/.bashrc

3. Issue this command to create and then enter the virtual environment: 

$ mkvirtualenv datagrepper 
$ workon datagrepper        

btw, if you wants to get out of the virtualenv, then

$ deactivate

4. Install dependencies in datagrepper virtualenv

$ pip install -r requirements.txt
$ pip install psycopg2 

If you get error like this

 error: command 'gcc' failed with exit status 1.

Then,

$ sudo yum install python-devel

If it yells error for this dependency #include<libpq-fe.h> requirement, then install that using command:

$ yum install /usr/include/libpq-fe.h

6. Download the rpm file postgresql-9.2.4-7.fc19.src.rpm and install it.

7. Then, issue this command:

 $ pip install .

8. Refer link  — https://github.com/fedora-infra/datanommer/tree/develop/datanommer.models/alembic. 

 $ git clone https://github.com/fedora-infra/datanommer/
 $ cd datanommer.models 
 $ alembic current                                           

Now you have the cloned the datanommer project; copy the datanommer.models folder into the folder where you datagrepper project is there.

 $ cd /datagrepper/datanommer.models
 $ python setup.py install  

Go to alembic.ini and change sqlalchemy.url to sqlalchemy.url
=postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/data     
                       
Then run these commands: 

 $ alembic stamp 2affa1daa804 
 $ alembic upgrade head 

9. Install postgres:

 $ rm -rf /var/lib/pgsql
 $ sudo yum install -y postgresql-server fedmsg
 $ sudo postgresql-setup initdb

10. Edit the /var/lib/pgsql/data/pg_hba.conf. You might find a line like this:
host       all              all          127.0.0.1/32               ident Change it to:
host       all              all          127.0.0.1/32               md5

11. Then, start up postgres

 $ sudo systemctl restart postgresql.service   

12. Become the postgres user (again) and run the psql command.            

 $ sudo su - postgres
 $ psql
 # create database datanommer;
 # create user datanommer with password 'bunbunbun';
 # grant all privileges on database datanommer to datanommer;
 # \q

13. Back in the bash shell (but still as the postgres user), grab a DB dump and restore it:  

 $ wget http://ianweller.fedorapeople.org/datanommer-2013-04-10-0.5.0.dump.xz
 $ xzcat datanommer-2013-04-10-0.5.0.dump.xz | psql datanommer

14. Edit fedmsg.d/example-datagrepper.py and give it these contents

  config = {
    'datanommer.enabled': False,
    'datanommer.sqlalchemy.url': 'postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/datanommer',
    'fedmsg.consumers.datagrepper-runner.enabled': True,
  }

15. As your normal old user self, run the development server:

  $ workon datagrepper
  $ python runserver.py

16.  In a browser, visit http://localhost:5000 to see the output.

17. First execute this command: 

$ sudo yum install -y httpie.
$ http get https://apps.fedoraproject.org/datagrepper/raw/delta==172800.

If its gives you internal server error due to proxy; for that open /etc/bashrc and add this command

export NO_PROXY=localhost,127.0.0.0/8,127.0.1.1
$ source /etc/bashrc

If you want to kill the running server the just press these keys

CTRL-Z 

Also, to kill all programs listening to port(port:5000), so that if you start it next time it will not yell out error

kill -9 `fuser -n tcp 5000` 

At last it took two days to install. In this process i learnt many things. I have gone through various python packages (virtualenv, flask, jinja2, SQLAlchemy etc) and their documentation to understand more about them. Virtualenv is really a handy tool for software developemnt and flask: best tool for portable web-development.

 

I also want to thank my mentor Ralph Bean and Pierre-Yves. They have helped me a lot during the installation process.

 

Thanks!