Cookiecutter – Modify context in post_gen_project.py

Cookiecutter is a template where you can setup skeleton of a project, and based on parameters from cookiecutter.json it will prefill all files with the supplied values.

Advance Cookiecutter Question:
How can I add new context based on what was submitted from cookiecutter.json, then come up with my own variations, and pass them back to context/extra_context to be rendered.

  • Example1: if {{ cookiecutter.project_name}} == myapp and {{cookiecutter.github_username }} ==lszyba1 then add a new context variable called: mygreatuser=’ProSupport’. Then in template files I would use that variable to fill in some values.
  • Example2: if {{cookiecutter.framework_to_deploy}}==’pyramid’:
    deployment_prod_or_dev_file = ask_more_questions(….) (This allows me to write my own function to ask more question, if user said pyramid then ask X, if he said django then ask Y.)
  • Example3: import os ; workfolder=os.getcwd() ; context[‘workfolder’]=workfolder (this will insert a new variable I can render in template.

Solution:

Cookiecutter has hooks folder, but this does not allow context to be modified out of the box, so we need to add 10 lines of code.
Since pre_gen_project.py and post_gen_project.py get rendered with only the values from cookiecutter.json, I will use post_get_project.py to do my programming, add more values to context then re-render the files using mako.
This allows me to have all files rendered with jinja2 syntax (cookiecutter default), and all my template variables will be left alone, and will be rendered by me using mako.
This also allows me to mix and match code in post_get_project.py where I use cookiecutter original context in my python if statements.

Lets get started:
Create folder hooks and add post_gen_project.py file

myapp/
....
-- {{cookiecutter.folder_name}}
   |-- {{cookiecutter.package_name}}.conf
|-- {{cookiecutter.package_name}}.txt
|-- __init__.py
|-- README.txt
|-- cookiecutter.json
-- DESCRIPTION.rst
-- docs
-- hooks
|-- post_gen_project.py

....

cookiecutter.json contains:
{
"folder_name": "apache2",
"package_name": "myapp",
"domain_name": "example.com",
"framework_to_deploy": ["pyramid", "django"]
}

My .conf file will be called myapp.conf if you just hit enter through prompts.


Now lets go into post_gen_project.py

#First get cookiecutter context dictionary:
context={{cookiecutter}}
#I can add values to it like this.:
context['better_package_name']={{cookiecutter.package_name}}+'2'
or
context['better_package_name']=context['package_name']+'2'
#Left is new context I will use in mako, the right is context and info supplied from original cookiecutter.json file

#Add few more:<code>
import os
workfolder=os.getcwd()
context['workfolder']=os.getcwd()
context['user_name']=os.getlogin()

#Now lets re-render my template(s) with my additional 3 variables (aka workfolder,better_package_name,user_name..)
from mako.template import Template
from mako.lookup import TemplateLookup

mylookup = TemplateLookup(directories=[workfolder],strict_undefined=True)

def serve_template(templatename, **kwargs):
    mytemplate2 = mylookup.get_template(templatename)
    print('Rendering: ' + templatename)
    return mytemplate2.render(**kwargs)

#This loops through a files in a workfolder. I need more testing to confirm the work folder is where I think it is, so for now I will name the files explicatively.
#Render each template explicitly
def save_template(workfolder=None,file_name=None,context=None):
    newtemplate=serve_template(file_name,**context)
    print('Saving: '+workfolder+'/'+file_name)
    f=open(os.path.join(workfolder,file_name),'w')
    f.write(newtemplate)
    f.close()

#Now the lets Re-Render my template with my 3 new variables.
file_name= context['package_name']+'.conf'
#or below is also works.
file_name= {{cookiecutter.package_name}}+'.conf'
save_template(workfolder,file_name,context)

#Done, now the template contains all my new variables.


Here is a template sample.

##########Start of {{cookiecutter.package_name}}.conf ########

#I like your new project {{cookiecutter.package_name}}. I think it will be awsome, but you should consider giving it a better name ${better_package_name}. 

#Beginning the configuration per ${user_name} instructions

#some code,conf,etc...
Alias ${better_package_name}/{{cookiecutter.package_name}}.txt ${workfolder}/${package_name}
##########End of {{cookiecutter.package_name}}.conf ########


Bonus:

#Ask questions based on cookiecutter parameters
def ask_more_questions(question=None):
    try:
        output = input(question)
    except NameError:
        output = raw_input(question)
    return output


if context['framework_to_deploy']=='pyramid':
    deployment_prod_or_dev_file=ask_more_questions('What file you want to deploy: [development.ini] or production.ini :')
    context['deployment_prod_or_dev_file']=(deployment_prod_or_dev_file or 'development.ini')

#Now For every input question from cookiecutter I can do if statements, if a, then b...

Hope you enjoyed it. Have fun creating your awesome new template. Many thanks to cookiecutter team for making simple yet powerful project templating software!
LucasManual.com team

Use the tools available: This post is about being able to programmatically add values based on initial cookiecutter.json context. This is not about which python templating language is better.

Quick Intro to Cassandra vs MongoDB with python

Cassandra Nosql

    Cassandra Conclusion:

  • “One way that Cassandra deviates from Mongo is that it offers much more control on how it’s data is laid out. Consider a scenario where we are interested in laying out large quantities of data that are related, like a friend’s list. Storing this in MongoDB can be a bit tricky – it’s not great at storing lists that are continuously growing. If you don’t store the friends in a single document, you end up risking pulling data from several different locations on disk (or on different servers) which can slow down your entire application. Under heavy load this will impact other queries being performed concurrently.”[1]
  • If you have a project that is mature, it requires a lot of consecutive data that you will want to read later without jumping around to different disks. Cassandra looks like a strong candidate for:
    1. Show last 50 items for “TheMostIntrestingPersonInTheWorld”: item1,item2,..item3000..
    2. Show me last comments on “TheLucasMovie”: comment1,comment2,comment3,
    3. Show water level in Louisiana RiverIoT: level at 8am,level at 8:01am,level at 8:02am, x 100-1000 locations
  • Great if you have data structure already setup, and it fits above model. [2][3]

MongoDB

    MongoDB Conclusion:

  • No structure. import mongodb, mydb = db.myawsomedatabase, mydb.insert(start adding data). Done.
  • You have a project and you are not sure how NoSQL will handle it but you want to try it. [4]
  • You have a working process but its grown to a point where traditional RDMS can’t handle the IO load. [5]
  • You don’t have time to create table structures just now, you just want to get going, and see what happens.
  • You want to find documentation with python fast, and benefit from large community examples.

Cassandra Python
Cassandra Code in Python; Details:
Installation:

#Add cassandra repo to /etc/apt/sources.list
deb http://www.apache.org/dist/cassandra/debian 37x main
sudo apt-get update
update-alternatives --config java  #pick openjdk 8
sudo apt-get install cassandra
#status
nodetool status
nodetool info
nodetool tpstats
#python
virtualenv -p python3 env_py3
source env_py3/bin/activate
pip install cassandra-driver

Python:

from cassandra.cluster import Cluster
cluster=Cluster()
session = cluster.connect()

#nodetool status
#nodetool info
#nodetool tpstats


#https://github.com/dkoepke/cassandra-python-driver/blob/master/example.py
session.execute("CREATE KEYSPACE vindata WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '1' }")
session.execute("use vindata")
#http://www.slideshare.net/ebenhewitt/cassandra-datamodel-4985524 slide 23
session.execute("""
CREATE TABLE emissions (
vin text,
make text,
year text,
zip_code_of_station text,
co2 text,
year_month_key int,
PRIMARY KEY (vin)
)
""")

#https://www.youtube.com/watch?v=97VBdgIgcCU
#Load mydata

import glob
print(glob.glob("./data/*.dat"))
session.execute("use vindata")


for datafile in glob.glob("./data/*.dat"):
    f=open(datafile, 'r')
    data={}
    for row in f.readlines():
        data={}
        data['vin']=row[:20].strip()
        data['make']=row[20:24].strip()
        data['year']=row[24:28].strip()
        data['zip_code_of_station']=row[42:47].strip()
        data['co2']=row[47:48].strip()
        ymk='20'+datafile[-12:-8]
        data['year_month_key']=ymk
        #print(data)
        session.execute(
        """
        INSERT INTO emissions (vin, make, year,zip_code_of_station,co2,year_month_key)
        VALUES (%s,%s,%s,%s,%s,%s)
        """,
        (data['vin'],data['make'],data['year'],data['zip_code_of_station'],data['co2'],data['year_month_key'])
    )
    f.close()

future=session.execute_async("SELECT * FROM emissions where vin='1B4GP33R9TB205257'")
rows = future.result()
for row in rows:
    print(row)

MongoDB and Python
MongoDB Code in Python; Details:

Installation

sudo aptitude install mongodb
/etc/init.d/mongodb start
#python
virtualenv -p python3 env_py3
source env_py3/bin/activate
pip install pymongo

Python

#http://api.mongodb.com/python/current/tutorial.html
from pymongo import MongoClient
client = MongoClient('mongodb://localhost:27017/')
#create database
db = client.vindata
#create collection/table
emissions = db.emissions

#Load data from mydata
import glob
print(glob.glob("./data/*.dat"))
for datafile in glob.glob("./data/*.dat"):
    f=open(datafile, 'r')
    data={}
    for row in f.readlines():
        data={}
        data['vin']=row[:20].strip()
        data['make']=row[20:24].strip()
        data['year']=row[24:28].strip()
        data['zip_code_of_station']=row[42:47].strip()
        data['co2']=row[47:48].strip()
        #data['year_month_key']=201608
        ymk='20'+datafile[-12:-8]
        data['year_month_key']=ymk
        #print(data)
        emissions.insert(data)
    f.close()

emissions.count()
emissions.find_one()
emissions.find_one({"vin":"1B4GP33R9TB205257"})
#http://altons.github.io/python/2013/01/21/gentle-introduction-to-mongodb-using-pymongo/
#https://www.youtube.com/watch?v=f7l8PTjQ160&index=4&list=PLGOsbT2r-igmFK9IKEGAnBaklqtuW7l8W
#https://www.youtube.com/watch?v=FVyIxdxsyok

#-------BONUS--------------
import pandas
cursor=emissions.find({"year_month_key":"201608"})
result=pandas.DataFrame(list(cursor))
result.describe()
result.columns
#http://lucasmanual.com/mywiki/Pandas
#later http://alexgaudio.com/2012/07/07/monarymongopandas.html

Sources:
1. https://academy.datastax.com/mongodb-to-cassandra-migration
2. http://www.slideshare.net/nkorla1share/cass-summit-3?qid=f85a27f7-a560-48bb-9d64-6eaa91c39f24&v=&b=&from_search=8
3. https://www.youtube.com/watch?v=tg6eIht-00M
4. https://www.mongodb.com/customers/city-of-chicago
5. https://www.youtube.com/watch?v=FVyIxdxsyok