Cookiecutter – Modify context in post_gen_project.py

Disclaimer: This post is about being able to programmatically add values based on initial cookiecutter.json context. This is not about which python templating language is better. [Stop arguing about which is better, and instead just code your project and use the tools available]

Cookiecutter is a template where you can setup skeleton of a project, and based on parameters from cookiecutter.json it will prefill all files with the supplied values.

Advance Cookiecutter Question:
How can I add new context based on what was submitted from cookiecutter.json, then come up with my own variations, and pass them back to context/extra_context to be rendered.

  • Example1: if {{ cookiecutter.project_name}} == myapp and {{cookiecutter.github_username }} ==lszyba1 then add a new context variable called: mygreatuser=’ProSupport’. Then in template files I would use that variable to fill in some values.
  • Example2: if {{cookiecutter.framework_to_deploy}}==’pyramid':
    deployment_prod_or_dev_file = ask_more_questions(….) (This allows me to write my own function to ask more question, if user said pyramid then ask X, if he said django then ask Y.)
  • Example3: import os ; workfolder=os.getcwd() ; context[‘workfolder’]=workfolder (this will insert a new variable I can render in template.

Solution:

Cookiecutter has hooks folder, but this does not allow context to be modified out of the box, so we need to add 10 lines of code.
Since pre_gen_project.py and post_gen_project.py get rendered with only the values from cookiecutter.json, I will use post_get_project.py to do my programming, add more values to context then re-render the files using mako.
This allows me to have all files rendered with jinja2 syntax (cookiecutter default), and all my template variables will be left alone, and will be rendered by me using mako.
This also allows me to mix and match code in post_get_project.py where I use cookiecutter original context in my python if statements.

Lets get started:
Create folder hooks and add post_gen_project.py file

myapp/
....
-- {{cookiecutter.folder_name}}
   |-- {{cookiecutter.package_name}}.conf
|-- {{cookiecutter.package_name}}.txt
|-- __init__.py
|-- README.txt
|-- cookiecutter.json
-- DESCRIPTION.rst
-- docs
-- hooks
|-- post_gen_project.py

....

cookiecutter.json contains:
{
"folder_name": "apache2",
"package_name": "myapp",
"domain_name": "example.com",
"framework_to_deploy": ["pyramid", "django"]
}

My .conf file will be called myapp.conf if you just hit enter through prompts.


Now lets go into post_gen_project.py

#First get cookiecutter context dictionary:
context={{cookiecutter}}
#I can add values to it like this.:
context['better_package_name']={{cookiecutter.package_name}}+'2'
or
context['better_package_name']=context['package_name']+'2'
#Left is new context I will use in mako, the right is context and info supplied from original cookiecutter.json file

#Add few more:<code>
import os
workfolder=os.getcwd()
context['workfolder']=os.getcwd()
context['user_name']=os.getlogin()

#Now lets re-render my template(s) with my additional 3 variables (aka workfolder,better_package_name,user_name..)
from mako.template import Template
from mako.lookup import TemplateLookup

mylookup = TemplateLookup(directories=[workfolder],strict_undefined=True)

def serve_template(templatename, **kwargs):
    mytemplate2 = mylookup.get_template(templatename)
    print('Rendering: ' + templatename)
    return mytemplate2.render(**kwargs)

#This loops through a files in a workfolder. I need more testing to confirm the work folder is where I think it is, so for now I will name the files explicatively.
#Render each template explicitly
def save_template(workfolder=None,file_name=None,context=None):
    newtemplate=serve_template(file_name,**context)
    print('Saving: '+workfolder+'/'+file_name)
    f=open(os.path.join(workfolder,file_name),'w')
    f.write(newtemplate)
    f.close()

#Now the lets Re-Render my template with my 3 new variables.
file_name= context['package_name']+'.conf'
#or below is also works.
file_name= {{cookiecutter.package_name}}+'.conf'
save_template(workfolder,file_name,context)

#Done, now the template contains all my new variables.


Here is a template sample.

##########Start of {{cookiecutter.package_name}}.conf ########

#I like your new project {{cookiecutter.package_name}}. I think it will be awsome, but you should consider giving it a better name ${better_package_name}. 

#Beginning the configuration per ${user_name} instructions

#some code,conf,etc...
Alias ${better_package_name}/{{cookiecutter.package_name}}.txt ${workfolder}/${package_name}
##########End of {{cookiecutter.package_name}}.conf ########


Bonus:

#Ask questions based on cookiecutter parameters
def ask_more_questions(question=None):
    try:
        output = input(question)
    except NameError:
        output = raw_input(question)
    return output


if context['framework_to_deploy']=='pyramid':
    deployment_prod_or_dev_file=ask_more_questions('What file you want to deploy: [development.ini] or production.ini :')
    context['deployment_prod_or_dev_file']=(deployment_prod_or_dev_file or 'development.ini')

#Now For every input question from cookiecutter I can do if statements, if a, then b...

Hope you enjoyed it. Have fun creating your awesome new template. Many thanks to cookiecutter team for making simple yet powerful project templating software!
LucasManual.com team

Quick Intro to Cassandra vs MongoDB with python

Cassandra Nosql

    Cassandra Conclusion:

  • “One way that Cassandra deviates from Mongo is that it offers much more control on how it’s data is laid out. Consider a scenario where we are interested in laying out large quantities of data that are related, like a friend’s list. Storing this in MongoDB can be a bit tricky – it’s not great at storing lists that are continuously growing. If you don’t store the friends in a single document, you end up risking pulling data from several different locations on disk (or on different servers) which can slow down your entire application. Under heavy load this will impact other queries being performed concurrently.”[1]
  • If you have a project that is mature, it requires a lot of consecutive data that you will want to read later without jumping around to different disks. Cassandra looks like a strong candidate for:
    1. Show last 50 items for “TheMostIntrestingPersonInTheWorld”: item1,item2,..item3000..
    2. Show me last comments on “TheLucasMovie”: comment1,comment2,comment3,
    3. Show water level in Louisiana RiverIoT: level at 8am,level at 8:01am,level at 8:02am, x 100-1000 locations
  • Great if you have data structure already setup, and it fits above model. [2][3]

MongoDB

    MongoDB Conclusion:

  • No structure. import mongodb, mydb = db.myawsomedatabase, mydb.insert(start adding data). Done.
  • You have a project and you are not sure how NoSQL will handle it but you want to try it. [4]
  • You have a working process but its grown to a point where traditional RDMS can’t handle the IO load. [5]
  • You don’t have time to create table structures just now, you just want to get going, and see what happens.
  • You want to find documentation with python fast, and benefit from large community examples.

Cassandra Python
Cassandra Code in Python; Details:
Installation:

#Add cassandra repo to /etc/apt/sources.list
deb http://www.apache.org/dist/cassandra/debian 37x main
sudo apt-get update
update-alternatives --config java  #pick openjdk 8
sudo apt-get install cassandra
#status
nodetool status
nodetool info
nodetool tpstats
#python
virtualenv -p python3 env_py3
source env_py3/bin/activate
pip install cassandra-driver

Python:

from cassandra.cluster import Cluster
cluster=Cluster()
session = cluster.connect()

#nodetool status
#nodetool info
#nodetool tpstats


#https://github.com/dkoepke/cassandra-python-driver/blob/master/example.py
session.execute("CREATE KEYSPACE vindata WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '1' }")
session.execute("use vindata")
#http://www.slideshare.net/ebenhewitt/cassandra-datamodel-4985524 slide 23
session.execute("""
CREATE TABLE emissions (
vin text,
make text,
year text,
zip_code_of_station text,
co2 text,
year_month_key int,
PRIMARY KEY (vin)
)
""")

#https://www.youtube.com/watch?v=97VBdgIgcCU
#Load mydata

import glob
print(glob.glob("./data/*.dat"))
session.execute("use vindata")


for datafile in glob.glob("./data/*.dat"):
    f=open(datafile, 'r')
    data={}
    for row in f.readlines():
        data={}
        data['vin']=row[:20].strip()
        data['make']=row[20:24].strip()
        data['year']=row[24:28].strip()
        data['zip_code_of_station']=row[42:47].strip()
        data['co2']=row[47:48].strip()
        ymk='20'+datafile[-12:-8]
        data['year_month_key']=ymk
        #print(data)
        session.execute(
        """
        INSERT INTO emissions (vin, make, year,zip_code_of_station,co2,year_month_key)
        VALUES (%s,%s,%s,%s,%s,%s)
        """,
        (data['vin'],data['make'],data['year'],data['zip_code_of_station'],data['co2'],data['year_month_key'])
    )
    f.close()

future=session.execute_async("SELECT * FROM emissions where vin='1B4GP33R9TB205257'")
rows = future.result()
for row in rows:
    print(row)

MongoDB and Python
MongoDB Code in Python; Details:

Installation

sudo aptitude install mongodb
/etc/init.d/mongodb start
#python
virtualenv -p python3 env_py3
source env_py3/bin/activate
pip install pymongo

Python

#http://api.mongodb.com/python/current/tutorial.html
from pymongo import MongoClient
client = MongoClient('mongodb://localhost:27017/')
#create database
db = client.vindata
#create collection/table
emissions = db.emissions

#Load data from mydata
import glob
print(glob.glob("./data/*.dat"))
for datafile in glob.glob("./data/*.dat"):
    f=open(datafile, 'r')
    data={}
    for row in f.readlines():
        data={}
        data['vin']=row[:20].strip()
        data['make']=row[20:24].strip()
        data['year']=row[24:28].strip()
        data['zip_code_of_station']=row[42:47].strip()
        data['co2']=row[47:48].strip()
        #data['year_month_key']=201608
        ymk='20'+datafile[-12:-8]
        data['year_month_key']=ymk
        #print(data)
        emissions.insert(data)
    f.close()

emissions.count()
emissions.find_one()
emissions.find_one({"vin":"1B4GP33R9TB205257"})
#http://altons.github.io/python/2013/01/21/gentle-introduction-to-mongodb-using-pymongo/
#https://www.youtube.com/watch?v=f7l8PTjQ160&index=4&list=PLGOsbT2r-igmFK9IKEGAnBaklqtuW7l8W
#https://www.youtube.com/watch?v=FVyIxdxsyok

#-------BONUS--------------
import pandas
cursor=emissions.find({"year_month_key":"201608"})
result=pandas.DataFrame(list(cursor))
result.describe()
result.columns
#http://lucasmanual.com/mywiki/Pandas
#later http://alexgaudio.com/2012/07/07/monarymongopandas.html

Sources:
1. https://academy.datastax.com/mongodb-to-cassandra-migration
2. http://www.slideshare.net/nkorla1share/cass-summit-3?qid=f85a27f7-a560-48bb-9d64-6eaa91c39f24&v=&b=&from_search=8
3. https://www.youtube.com/watch?v=tg6eIht-00M
4. https://www.mongodb.com/customers/city-of-chicago
5. https://www.youtube.com/watch?v=FVyIxdxsyok

Alt + Speak (Alt+S) = How to enable Text-To-Speech

*Install festival. This will speak the text you select.
apt-get install festival

*Install xsel. This program allows you to pass what you selected into a talk.sh from your graphic interface. In my case I use gnome shell, so when you are in a email or website or reading long marketing notes and you select your text you just press alt+s and computer will speak(say) it to you.
*Lets install xsel.
apt-get install xsel

*Now create a little bash script that passes what you selected to festival
cd /home/lucas
vi talk.sh

*And paste these lines
#!/bin/bash
xsel | festival --tts --pipe

*In order to being able to execute it we need to give it execute permission.
chmod +x talk.sh

*Now you are ready. We just need to add our talk.sh to our keyboard shortcuts. In gnome shell : Search for
System Settings
then press keyboard
then press Shortcuts
then press +

Add keyboard shortcut.
Name: TTS (Text To Speech)
Command: /home/lucas/talk.sh

*And set it for
alt + s

aka Alt + SPEAK!

*You are ready to go. Select some text and press alt+s. (I have noticed that sometimes you need to press and hold s for about 1sec for it to work when there is more text selected.

*Watch a quick video that was very helpful:

*Some notes on the voice. The default festival voice is the most versatile when it comes to reading. I have tried other voices but they are incomplete when it comes to some words. I was surprised that this can read almost every word. You can try other languages here. While I read that Japanese American might be a good replacement. The default festival voice was very easy to get used to. In 1 day I catch up on all the marketing notes. So have the computer read to you and get back to business.

While you are still here help me request this talk.sh being included in all Debian installations by default and keyboard shortcut set to alt+s (alt+speak)

*Enjoy! text-to-speech on Debian Linux

Linux System Recovery part 1

In December 2012, there was a power outage mostly due to blowing fuse at the fuse box. For those that don’t deal with that much, older fuses will trip because the connectors are worn out. Also on a 20amp fuse you should not go over 60% output, so 14amp would be safe. You can replace with new fuse, to minimize this problem, and of course buying a uninterrupted power supply (ups) would mitigate the system turning off and on while we figure out why the fuse is being tripped.

System Crash
System won’t boot. It says that root partition is not available. Unable to load or mount any of the file systems. Looks like all the mount points are gone, or unable to read.

System Rescue

In Debian rescue live CD

#Scan for mdadm device
mdadm --examine --scan

#Update mdadm.conf with what what found
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

#Assemble the raid. Examine scan should show you the name it wants you to use if different them md127
mdadm --assemble --scan /dev/md127

#See what got assembled
cat /proc/mdstat

#Now If you have LVM2 then need to mount LVM

#See if there is LVM group on your newly mounted mdadm device /dev/md127 for example.
pvscan

#Display more about the LVM2 group
vgdisplay

#See what logical disks are there(The /dev/md127 should match to what you found in prior commands)
lvmdiskscan

#Display the drives in md device
lvdisplay |more

#Display the list of volumes
lvscan

#Now we should have our raid device assembled and we should know a list of lvm drives(partitions) on the
#lvm group. We should see from lvscan what are their mount points

#if the lvm group shows inactive you can activate it by

#lvscan shows…Example
#inactive /dev/my_lvmgroup/my_root
#inactive /dev/my_lvmgroup/my_home
#inactive /dev/my_lvmgroup/my_swap

#Then do this. This will change the inactive status to active.
lvchange -a y /dev/my_lvmgroup

#You can deactivate the group but make sure all drives were unmounted by you running command umount on it.

Now that we have mounted the drives its time to recover. First we need to find out what file system are on our /dev/my_lvmgroup/my_root and /dev/my_lvmgroup/my_home. Then we need to do a file system check, and calculate our damages.

mdadm Raid5 – How to replace failed drive GPT partition

Install gdisk. On debian squeeze you need to add main backports
vi /etc/apt/source.list
#Add below
deb http://backports.debian.org/debian-backports squeeze-backports main contrib non-free

Install gdisk
aptitude install gdisk

Drive sdb1 failed? Show details of partition md0
mdadm --detail /dev/md0

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdd1[2]
3907028864 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]


Drive sdb1 is already removed. but if you need to remove it manually you can:

mdadm /dev/mmdadm /dev/md0 -r /dev/sdb1

SHUT DOWN IF YOU NEED TO REPALCE DRIVE. MAKE SURE NEW DRIVE IS STILL sdb.
Look how disk is structured and what partition type it has
sgdisk -p /dev/sdb
Disk /dev/sdb: 3907029168 sectors, 1.8 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 0ED13F81-6EEA-4E12-9F27-DD806CF1F09C
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 8-sector boundaries
Total free space is 0 sectors (0 bytes)

Number Start (sector) End (sector) Size Code Name
1 34 3907029134 1.8 TiB FD00

Now copy partition A structure into partition B

#sgdisk -R=/dev/TO_THIS_DISK /dev/FROM_THIS_DISK
sgdisk -R=/dev/sdb /dev/sda
#Give new GUID since above options clones the disk including GUID
sgdisk -G /dev/sdb

Now readd the drive to md0

mdadm /dev/md0 -a /dev/sdb1

Check the status

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sda1[1] sdc1[2]
3907028864 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]
[>....................] recovery = 0.0% (124204/1953514432) finish=786.3min speed=41401K/sec

Done. Check back in few hours to see if it finished.
Keywords: fdisk,sdisk, sgdisk, gdisk,parted,gpt, mbr,raid5,mdadm,linux,debian,

Upgrade Grub to Grub-pc

During upgrade of Lenny to Squeeze I run into an an issue with upgrading grub to grub2. My 4x2Tb drivers were created using GDP partition. As a result of the upgrade the grub-pc had an issue with my gpt partition. If you receive any of these errors you might be in a similar situation. The fix is pretty easy, so hopefully this will guide you through it.

grub-installer: grub-setup: warn: This GPT partition label has no BIOS
Boot Partition; embedding won’t be possible!

or

grub-installer: grub-setup: error: Embedding is not possible, but this

is required when the root device is on a RAID array or LVM volume.

or

grub loading...
no module name found

or

After restart I can only see “GRUB>”.

(If you have above please read the whole post, and especially the “debug” portion.)

I’ve started from Debian cd (rescue) mode. Then I’ve assembled my raid partition (sdb1,sdc1,sdd1) and
executed into shell of my lvm root group mapper_xyz_root. From there I run “upgrade-from-grub-legecy”

**Long story short.  It seems as the because I am using 4 x 2TB drivers and all 4 were
created using GPT partition table, and my computer/mother board was
purchased before 2010 it does not support GPT EFI, so grub needed more
space then MBR allowed, and was telling me no module found because it
didn’t have enough space to be installed.

Steps:
1. First I tried running the “grub-install /dev/sda” I would get:

grub-installer: grub-setup: warn: This GPT partition label has no BIOS
Boot Partition; embedding won’t be possible!

and
grub-installer: grub-setup: error: Embedding is not possible, but this
is required when the root device is on a RAID array or LVM volume.

I found this post:
<http://www.shuvoovuhs.com/linux/grub-installation-issue-with-2-tb-hdd-gpt-requires-bios-boot-partition/>
which said ( *have to* create a BIOS Boot Partition).

2. Reading up on it I concluded that my motherboard does not support
GPT EFI and therefore I do indeed need that partition:
http://en.wikipedia.org/wiki/BIOS_Boot_partition

3. Now since I only have one boot partition on /dev/sda1 (100mb) I had
199.9GB free. I decided to create a partition right after my 100mb. In
rescue mode cd I installed parted (in case its not installed)
aptitude install parted
then proceeded to install my new partition.
parted /dev/sda print
parted /dev/sda unit MB print
##this showed partion 1 as (31.4kb to 100MB)
parted /dev/sda
mkpart biosboot 100MB 101MB
##above created partition 2
set 2 bios_grub on
##Above enabled the bios grub partition

Then I did grub install and it successfully installed. (I rebooted and
it worked.

I hope you have enough room to create a new partition. This should help you out. Most time took to research why I  “HAVE TO CREATE BIOS BOOT PARTITION”, and how to use “parted” to create a new partition, since none of the fdisk tools work with GPT partition table just yet. Enjoy. See the debug portion for how to get additional information out of your system.

Debug: More details :

In the process of upgrading from debian lenny amd64 to debian squeezy amd64 I was able to successfully upgrade to kernel 32 and new udev as Debian Release notes suggest. Then after reboot I followed with apt-get dist-upgrade.

Everything went fine, but towards the end I was asked to upgrade to
grub-pc. During this choice I was asked to specify mbr to install new
boot loader. I’ve selected my a wrong driver. Instead of selecting “/dev/sda” I’ve selected a different drive. (Always select the first hard drive in the list if you are not sure) (I thought it was “flash” drive that I have used before to hold my “boot” partition, but on this machine my boot parition was on  /dev/sda1″

After restart I can only see “GRUB>”.
At this point I should have tried starting the system by issueing few grub commands. More on it shortly.


While Recovering from grub-pc install failure. I’ve started from cd
(rescue) mode. I then assembled my raid partition (sdb1,sdc1,sdd1) and
executed into shell of my lvm root group mapper_xyz_root. From there I
run “upgrade-from-grub-legecy” and this time I’ve selected my usb and
sda to install grub.

Still no lock.

Then I tried “update-grub”

Now I get “grub loading…
no module name found”

What should I do now? I’ve logged in with rescue cd again and now my
/boot partition no longer holds other files except for “/boot/grub/..”
What happened to my kernel files 26 and 32 that were on the /boot? (This was my fault as it seem /boot partition was not mounted properly.  I have my boot partition somewhere else, yet somehow I was looking at  /boot on my root partition.)

I’ve posted to debian mailing list:

What are my choices on installing grub-pc? Do I need “boot” partition?
What should be on it? Why did /boot kernel files got removed? Should I be
installing grub on my lvm root group? or sda? or /boot flashdrive?

> During this choice I was asked to specify mbr to install new

> boot loader. I’ve selected my “flash” drive that I have used before to
> hold my “boot” partition I believe.

I think the most normal installation is to select your first raw
drive.  That is, if you have /boot on /dev/sda1 and / on /dev/sda5 or
some such then you would install grub on /dev/sda without adding any
partition numbers.

**That is correct, I don’t know why I thought I had a usb, but in this
computer I did not have a usb driver, so I should have selected
/dev/sda.  In rescue mode I did that many times and it still failed.

> After restart I can only see “GRUB>”.

Grub appears to be installed then.  But the problem would seem to be
that grub’s configuration file didn’t point to the root filesystem.

** Correct, at that point I’m not sure if that was still grub 1 or
grub 2(grub-pc). (more on it below) The information below was very
useful after I gut the grub> menu.

If you have “grub> ” you can:

At that point you can issue instructions to grub.  You should be able
to get some good information.  It is a little confusing to describe
but the most important thing to know is that TAB will expand and list
your possible options.  Use this to explore your system at that point
and to see what is where.  You can type in “help” to get a list of
commands available but that will produce a lot of output and will
overwhelm you.

At the grub prompt type in “root (” and then hit TAB to have it
complete.  It will look like this:

grub> root (

Press TAB at that point and it will fill out to the available
options.

grub> root (hd0,

Press TAB again to have it list them out.

grub> root (hd0,
Possible partitions are:
Partition hd0,sda1
Partition hd0,sda2

Then select one of them and repeat to list the contents of that
filesystem.

grub> root (hd0,0)/
Possible files are:
lost+found/ System.map-2.6.32-5-686 vmlinuz-2.6.32-5-686 grub/
config-2.6.32-5-686 initrd.img-2.6.32-5-686

That verifies that on my system hd0,0 (/dev/sda1) is my /boot
partition.
Repeat again with the other partition numbers.

grub> root (hd0,1)/
Possible files are:
bin/ boot/ dev/ home/ lib/ lost+found/ media/ mnt/ opt/ …

That verifies that on the system I tried that hd0,1 (/dev/sda2) is the
root partition.

So to manually tell grub what it needs to boot I can type in the
following:

grub> root (hd0,0)
grub> kernel /vmlinuz-2.6.32-3-amd64 root=/dev/sda2 ro
grub> initrd /initrd.img-2.6.32-3-amd64
grub> boot

Use TAB to complete the filenames to ensure that you have the right
location and to save you from typing in all of the details of the
version numbers and architecture type.

If that works then your problem is not your grub install to the boot
partition but rather your configuration for grub in /boot/grub/* that
is the problem.

> While Recovering from grub-pc install failure. I’ve started from cd
> (rescue) mode.

A debian-installer disk in rescue mode should work okay.

> I then assembled my raid partition (sdb1,sdc1,sdd1) and

Why did you need to assemble the raid?

**I think in the “rescue CD mode it asks you to assemble raid or not”
You can select automatic assemble if you want to or you could pick a
different driver as your root. There was no issue with initrd as you
are suggesting below, but it made me pay attention to what is mounted
and what is not!!

That points to a different problem.  The raid should be automatically assembled by the initial
ram disk (initrd) and if it isn’t then you are past grub and onto the
initrd phase of boot.

Did you by any chance add a disk to the raid but not rebuild the
initrd image?  The initrd has the UUIDs of every disk in the raid as a
copy of the /etc/mdadm/mdadm.conf file in the initrd.  If you have
added a disk to the raid and it is required for the lvm to start then
this also needs to be added to the initrd copy of the mdadm.conf
file.  Otherwise it will fail to start the raid at boot time.

> executed into shell of my lvm root group mapper_xyz_root. From there I
> run “upgrade-from-grub-legecy” and this time I’ve selected my usb and
> sda to install grub.

I think you are mixing issues.  I think you mixing up grub with raid
with lvm but really those are all separate.  This is very easy to
become confused about but just the same I think that is what is
happening.

> Then I tried “update-grub”
>
> Now I get “grub loading…
> no module name found”

That I don’t know.

** It seems as the because I am using 4 x 2TB drivers and all 4 were
created using GPT partition table, and my computer/mother board was
purchased before 2010 it does not support GPT EFI, so grub needed more
space then MBR allowed, and was telling me no module found because it
didn’t have enough space to be installed.

Steps:
1. First I tried running the “grub-install /dev/sda” I would get:

grub-installer: grub-setup: warn: This GPT partition label has no BIOS
Boot Partition; embedding won’t be possible!

and
grub-installer: grub-setup: error: Embedding is not possible, but this
is required when the root device is on a RAID array or LVM volume.

I found this post:
<http://www.shuvoovuhs.com/linux/grub-installation-issue-with-2-tb-hdd-gpt-requires-bios-boot-partition/>
which said ( *have to* create a BIOS Boot Partition).

2. Reading up on it I concluded that my motherboard does not support
GPT EFI and therefore I do indeed need that partition:
http://en.wikipedia.org/wiki/BIOS_Boot_partition

3. Now since I only have one boot partition on /dev/sda1 (100mb) I had
199.9GB free. I decided to create a partition right after my 100mb. In
rescue mode cd I installed parted (in case its not installed)
aptitude install parted
then proceeded to install my new partition.
parted /dev/sda print
parted /dev/sda unit MB print
##this showed partion 1 as (31.4kb to 100MB)
parted /dev/sda
mkpart biosboot 100MB 101MB
##above created partition 2
set 2 bios_grub on
##Above enabled the bios grub partition

Then I did grub install and it successfully installed. (I rebooted and
it almost worked. I got the grub> ) but there was nothing there. I’ve
used what you email me to find my problem here.

When I did

grub-install /dev/sda

my /boot folder was in my “server1_lvmgroup-server1-root but that is
not what I had. My boot was a seperate disc /dev/sda1. So the proper
way I should have do it was:

ls /boot
#above shows only /boot/grub folder
umount /boot
#not mounted
mount -a
#this mounted my boot driver based on the /etc/fstab
ls /boot
#no I see
initrd.img-2.6.26-2-686
initrd.img-2.6.32-5-686
vmlinuz-2.6.26-2-686
vmlinuz-2.6.32-5-686

##Now I do install grub

grub-install /dev/sda
update-grub

##When you do update-grub it will find all kernels…it will say

found kernel 2.6.26…
found kernel 2.6.32….

reboot, and it worked.

I really appreciate your help.
This should really be in the Upgrades from Debian 5.0 (lenny), as if your motherboard does not support GPT EFI then
upgrade-from-grub-legacy will fail, and there is no instructions on how to go back.

I can’t really say at what point during the upgrade I had grub>? but
it all came down to creating a GPT Bios boot partition for grub-pc
(grub2).

Thank you.

***your email was most helpful. When I connected the dots with boot
partition then it all went smooth.

> What should I do now? I’ve logged in with rescue cd again and now my
> /boot partition no longer holds other files except for “/boot/grub/..”
> What happened to my kernel files 26 and 32 that were on the /boot?

Mounted the wrong partition? They should still be there.  Take a deep
breath.  Remain calm.  Try it again.  They should be there.

If you have somehow wiped them out then you will need to either
recover them or reinstall them from the chroot.

I have upgraded many machines from Lenny to Squeeze and although I
think this upgrade has the more problems of any of the previous
upgrades I have never had any of the problems you have mentioned.

> What are my choices on installing grub-pc? Do I need “boot” partition?

Yes.

> What should be on it?

Files such as:

System.map-2.6.26-2-686
System.map-2.6.32-5-686
config-2.6.26-2-686
config-2.6.32-5-686
grub/
initrd.img-2.6.26-2-686
initrd.img-2.6.32-5-686
vmlinuz-2.6.26-2-686
vmlinuz-2.6.32-5-686

> Why did files got removed?

Only you are in a position to know what you did.

> Should I be installing grub on my lvm root group? or sda? or /boot
> flashdrive?

Since you are getting the grub prompt then you have successfully
installed grub and your problem is with a later phase of the boot.

Sources:

http://www.rodsbooks.com/gdisk/booting.html

https://bbs.archlinux.org/viewtopic.php?id=114420

https://help.ubuntu.com/community/Grub2

http://lists.debian.org/debian-user/2011/08/msg00079.html

http://www.shuvoovuhs.com/linux/grub-installation-issue-with-2-tb-hdd-gpt-requires-bios-boot-partition/

http://ubuntuforums.org/showthread.php?t=1736409

http://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#newkernel

http://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#update-grub

Mozilla Lightning on Debian Squeeze

Hello,

To install Mozilla Lightning on Debian squeeze do the following:

aptitude install iceowl-extension

Fixes the following issues:
If you are trying to install lightning plugin version 1.0b1 January 12, 2010 2.5 MB

Works with:

  • Thunderbird 3.0b4 – 3.0.*
  • SeaMonkey 2.0b2 – 2.0.*
and you are using amd64 then you might be getting these errors if you download lightining from mozilla.org

“Could not be installed because it is not compatible with  your Icedove build type (Linux_x86_64-gcc3). Please contact the author of this item about the problem.”

Above installation fixes this issue. Enjoy!

Computer Basics (Domain, Record your desktop, commandline)

Some basic videos that everybody in technology world should know and learn:

Setup Domain Controller and know how to access your shared drives, setup user accounts, join domain, and use tools available to you.

(1)http://www.youtube.com/watch?v=QGfwfWcLFMw

(2)http://www.youtube.com/watch?v=PelA-_gT2qE

(3)http://www.youtube.com/watch?v=8hrhwzN15A4

(4)http://www.youtube.com/watch?v=2-7g_gA9NCg

(5) http://www.youtube.com/watch?v=T1AZgXqi2yQ

Know how to record your desktop so you can record and do training manuals for users.

http://www.youtube.com/watch?v=n9nAQg0B_Gk

Know your command line:

http://www.youtube.com/watch?v=TPxNAp7VRcM